Construction method of lightweight target detection network adaptive to dark light scene

Abstract

A construction method of a lightweight target detection network adapted to a dark light scene comprises the steps of construction of a training set, a verification set and a test set and training and verification of a model, and the model training process comprises the steps that S1, an original image F passes through two convolution layers in sequence to eliminate redundant information, and a feature map F1 and a feature map F2 are obtained; s2, the original image F, the feature map F1 and the feature map F2 are input into a feature enhancement network in parallel for semantic information coding, and output feature maps O, O1 and O2 on three paths are obtained; s3, the output feature map O2 and the output feature map O1 are subjected to channel dimension splicing through a layer of up-sampling network, the channel dimension is adjusted through one-time up-sampling, and a spliced convolution feature map is obtained; carrying out element addition on the output feature map O and the spliced convolution feature map, and decoding to obtain an image-enhanced intermediate image; and S4, performing feature extraction on the intermediate image by adopting MobileViT, fusing the extracted feature information of different scales, inputting the fused feature information into a detection head, and predicting a detection result by a network.

Description

Technical Field

[0001] The present invention belongs to the field of target detection, and specifically relates to a method for constructing a lightweight target detection network that is adapted to dark light scenes, and also relates to a method for performing target detection using the constructed network, and corresponding electronic products. Background Art

[0002] In recent years, with the development of deep learning, object detection tasks have achieved good results on most benchmark datasets. However, most existing detection networks are studied under normal conditions. In real environments, there are often many harsh environmental conditions, such as nighttime and exposure. Therefore, object detection under low-light conditions is a long-standing and continuously concerned challenge in the field of computer vision. Due to insufficient light, objects in images are often difficult to display clearly, resulting in blurred object edges, loss of color and texture information, and other problems. The accuracy of object detection will be seriously affected, thereby affecting the performance of the entire visual system.

[0003] Studies have shown that if the image is properly enhanced and more potential information of the original blurred target is restored according to the external environmental conditions, the target detection model can adapt to different low-light conditions, which is of great significance for improving the performance and stability of the computer vision system. In addition, during the deployment phase, the lightweight model will reduce the pressure on the terminal device and enable the model to better adapt to resource-limited environments such as embedded devices and mobile devices. Therefore, the present invention provides a method for target detection in dark light scenes using a feature enhancement network and a lightweight feature extraction network, so that the specific location information of the target object can be accurately detected even when the target object is in low-light conditions. Summary of the invention

[0004] In view of the problems and shortcomings of the prior art, the present invention provides a method for constructing a lightweight target detection network in dark light scenes.

[0005] Based on the above purpose, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a method for constructing a lightweight target detection network in a dark light scene, comprising the following steps:

[0007] (1) Collect single-view videos in dark scenes, divide the image sequence, annotate the images and divide them into training set, validation set and test set;

[0008] (2) Using the training set to train the model, the overall network loss function is continuously optimized and the weight of the trained model is adjusted. After the training is completed, a model that can detect targets in dark light scenes is obtained;

[0009] (3) The model obtained in step (2) is evaluated using a validation set and a test set, wherein the validation set is used to evaluate the parameter indicators of the model, and the test set is used to verify the detection performance of the model.

[0010] The specific process of model training in step (2) is as follows:

[0011] S1. Make the original image F pass through two convolutional layers in sequence to eliminate redundant information, and obtain feature maps F1 and F2; the original image is represented by the data size F∈R H×W×3 , the data size of the feature map is 1 / 4 and 1 / 8 of the original image, respectively, expressed as and

[0012] S2. Input the original image F, feature map F1, and feature map F2 into the feature enhancement network module in parallel to obtain the output feature maps O, O1, and O2 on the three paths, denoted as O∈R H×W×32 ,

[0013] S3. The output feature map O2 is concatenated with the output feature map O1 in terms of channel dimension through a layer of upsampling network, and then the channel dimension is adjusted through upsampling again to obtain a concatenated convolutional feature map; the output feature map O is element-wise added to the concatenated convolutional feature map, and the image is decoded to obtain an intermediate image after image enhancement;

[0014] S4. A lightweight feature extraction backbone network MobileViT is used to extract features from the intermediate image. The extracted feature information of different scales is fused at different levels to obtain a multi-layer fused feature map. The multi-layer fused feature map is input into the detection head, and finally the network predicts the detection result.

[0015] Preferably, the method for dividing the training set, validation set and test set in step (1) is:

[0016] a. Collecting original image sequences: Videos collected in low-light environments are divided into 10 image sequences;

[0017] b. Create image labels: Use the labelImg tool to manually label the object categories in the image and export the image label file;

[0018] c. Divide the image into training set, validation set and test set: Divide the collected image sequence into training set, validation set and test set in a ratio of 7:1:2.

[0019] Furthermore, the scene data collected in step a is collected by a ZED new generation binocular camera, and the corrected left camera video is obtained through a camera program package.

[0020] Preferably, in S1, redundant information is eliminated through two convolutional layers, and the convolution kernel size of the convolutional layer is 3, which can be expressed as:

[0021] F1 = Conv(F1), S = 4,

[0022] F2 = Conv(F1), S = 2,

[0023] In the formula, Conv represents the convolution operation and S represents the stride of the convolution.

[0024] Preferably, the feature enhancement network described in S2 is composed of six layers of CBR modules with residual connections, wherein the first layer and the sixth layer are provided with residual connections, the second layer and the fifth layer are provided with residual connections, and the third layer and the fourth layer are provided with residual connections; each CBR module performs Cov, BN and Relu operations in sequence, the convolution kernel size of the Cov convolution operation is 3, the step size is 1, the size of the feature map is not changed during the encoding process, the number of channels is increased from 3 to 32, and the extracted information is integrated into the channel dimension.

[0025] Preferably, the process in S3 is expressed by a mathematical formula:

[0026]

[0027] In the formula, the upsample operation represents upsampling, the concat operation represents concatenation along the channel dimension, and the add operation represents element-wise addition of feature maps. Represents the feature map of the final output.

[0028] Preferably, the specific method of feature extraction in S4 is: encode each area of ​​the intermediate image into three vectors, namely the query vector Q, the key-value vector K and the value vector V, calculate the similarity between the Q of each area and the K of the remaining areas, perform weighted summation according to the weights, and extract effective information.

[0029] Preferably, the overall network loss function in step (2) is defined as the reconstruction loss L1 and the detection loss L d , the reconstruction loss is mainly used to constrain the information loss of the input image after passing through the feature enhancement network; the detection loss is mainly used to constrain the detection loss of the target detection network, including the target loss L obj , category loss L cls and the regression loss L loc , expressed as:

[0030] L=λ1L1+λ d L d =λ1L1+λ obj L obj +λcls L cls +λ loc L loc

[0031] Among them, λ1, λ obj , cls and λ loc Represent the weight coefficients of each part of the loss.

[0032] In a second aspect, a target detection method is provided, comprising the steps of inputting an image to be detected into the target detection network constructed in the first aspect for target detection.

[0033] In a third aspect, an electronic product is provided, comprising a processor and a memory communicatively connected to the processor; the memory is used to store instructions; and the processor is used to call the instructions stored in the memory to execute the target detection method described in the second aspect.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. When introducing the feature enhancement module, the present invention adopts a three-path parallel input method to fuse feature information of different scales, so as to obtain features with stronger robustness and richer semantic information. The feature map obtained after the feature enhancement network module is easier to fuse; and in the information extraction network, a residual connection is provided to ensure that important information is not lost while expanding the spatial information of the feature map, and enriching the color and texture information of the image during the encoding and decoding process.

[0036] 2. The present invention obtains an intermediate image after preliminary decoding in the decoding stage. During the training process, the effect of preliminary enhancement can be monitored more intuitively, which solves the problem of achieving accurate and stable detection when the semantic information of the image is not obvious.

[0037] 3. The present invention adopts a lightweight feature extraction backbone network MobileViT, which not only has good feature extraction capabilities, but also makes the model better adaptable to resource-limited environments such as embedded devices and mobile devices.

[0038] 4. The detection method of the present invention is mainly aimed at performing stable and accurate detection of targets when the external environment is not good and the light is insufficient. It utilizes a feature enhancement network to improve the overall training speed of the model. It utilizes a lightweight feature extraction backbone network MobileViT, which not only has good feature extraction capabilities, but also can make the model better adaptable to resource-limited environments such as embedded devices and mobile devices, so that the network can be applied to autonomous robots equipped with a visual system, enabling them to better cope with dark light scenes, ensure the integrity of image information, and improve the accuracy and robustness of the robot's detection capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is an overall framework diagram of a lightweight target detection network adapted to dark light scenes in a specific embodiment of the present invention;

[0040] Figure 2 is a schematic diagram of a feature enhancement network module in a specific embodiment of the present invention;

[0041] Figure 3 It is a schematic diagram of a decoding module in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention is described in detail below in conjunction with specific implementation modes.

[0043] A lightweight target detection network adapted to dark light scenes. The overall framework is as follows Figure 1 As shown, the construction method includes the following steps:

[0044] (1) Use a binocular camera to collect single-view videos in dark scenes. Take the video collected by the left camera as the benchmark, build a self-built image sequence dataset, and divide it into a training set, a validation set, and a test set. Specifically:

[0045] a. Collect the original image sequence: Equip the new generation ZED binocular camera on the visual platform, initialize the device, ensure that the collected scene data is in a low-light environment, obtain the corrected left camera video through the camera package, and divide the collected video into 10 image sequences.

[0046] b. Create image labels: Use the labelImg tool to manually label the object categories in the image and export the image label file to ensure the quality of the data set provided to the network training.

[0047] c. Divide the dataset into training set, validation set and test set: Divide the acquired image sequence into training set, validation set and test set in a ratio of 7:1:2.

[0048] (2) The training set is used for model training. By continuously optimizing the overall network loss function and adjusting the weight of the trained model, a model that can detect targets in dark light scenes is obtained after the training is completed. Specifically:

[0049] S1. The original image is represented by the data size F∈R H×W×3 , so that the original image passes through two convolutional layers in sequence to eliminate redundant information: the convolution kernel size of the first convolutional layer is 3, the step size is 4, and the feature map F1 is obtained. The size of the feature map F1 becomes 1 / 4 of the original one, and the channel dimension remains unchanged, that is, The feature map F1 passes through the second convolutional layer. The convolution kernel size of the second convolutional layer is 3 and the stride is 2. The feature map F2 is obtained. The size of the feature map F2 becomes 1 / 8 of the original one, and the channel dimension remains unchanged, which is The above process is expressed as follows:

[0050] F1=Conv(F), S=4,

[0051] F2 = Conv(F1), S = 2,

[0052] In the formula, Conv represents the convolution operation and S represents the stride of the convolution.

[0053] S2. Input the original image F, feature map F1, and feature map F2 into the feature enhancement network module in parallel to obtain the output feature maps O, O1, and O2 on the three paths, denoted as O∈R H×W×32 ,

[0054] Feature enhancement network modules such as Figure 2 As shown in the figure, it consists of six layers of CBR modules with residual connections to encode semantic information and extract important information. Each CBR module performs Cov, BN and Relu operations in sequence. The convolution kernel size of the Cov convolution operation is 3 and the step size is 1. The size of the feature map is not changed during the encoding process. The number of channels is increased from 3 to 32, and the extracted information is integrated into the channel dimension. In the feature enhancement network module, the first layer and the sixth layer are connected with residual connections, the second layer and the fifth layer are connected with residual connections, and the third layer and the fourth layer are connected with residual connections to ensure that important information is not lost.

[0055] S3. The output feature map O2 is concatenated with the output feature map O1 in channel dimension through a layer of upsampling network, and then the channel dimension is adjusted through upsampling to obtain a concatenated convolution feature map; the output feature map O is element-wise added to the concatenated convolution feature map, and the intermediate image after image enhancement is obtained through a decoding module. The decoding module is as follows: Figure 3 As shown, the process is expressed mathematically as:

[0056]

[0057] In the formula, the upsample operation represents upsampling, the concat operation represents concatenation along the channel dimension, and the add operation represents element-wise addition of feature maps. Represents the feature map of the final output.

[0058] S4. Use the lightweight feature extraction backbone network MobileViT to extract features from the intermediate image, encode each area of ​​the intermediate image into three vectors, namely the query vector Q, the key vector K and the value vector V, calculate the similarity between the Q of each feature block and the K of the remaining feature blocks, perform weighted summation and extract effective information according to the weight. The extracted feature information of different scales is fused at different levels (channel dimension splicing) to obtain a multi-layer fused feature map; the multi-layer fused feature map is input into the detection head, and the network finally predicts the detection result (position and category information).

[0059] According to the prediction results, the overall network loss function is optimized and the training model weights are adjusted. The overall network loss function is defined as the reconstruction loss L1 and the detection loss L d , the reconstruction loss is mainly used to constrain the information loss of the input image after passing through the feature enhancement network; the detection loss is mainly used to constrain the detection loss of the target detection network, including the target loss L obj , category loss L cls and the regression loss L loc , expressed as:

[0060] L=λ1L1+λ d L d =λ1L1+λ obj L obj +λ cls L cls +λ loc L loc

[0061] Among them, λ1, λ obj , cls and λ loc Represent the weight coefficients of each part of the loss.

[0062] (3) The model obtained in step 2 is evaluated using a validation set and a test set, where the validation set is used to evaluate the parameter indicators of the model, and the test set is used to verify the detection performance of the model.

[0063] The above is a description of a preferred embodiment of the present invention, but it should be understood that the invention is not limited to the contents disclosed herein. As long as non-substantial improvements are made using the method concept and technical solution of the present invention, or the method concept and technical solution of the present invention are applied to other occasions, they are all within the scope of protection of the present invention.

Claims

1. A method for constructing a lightweight target detection network adapted to dark light scenes, characterized in that: The following steps are involved: (1) Collect single-view videos in dark scenes, divide the image sequence, annotate the images and divide them into training set, validation set and test set; (2) Using the training set to train the model, the overall network loss function is continuously optimized and the weight of the trained model is adjusted. After the training is completed, a model that can detect targets in dark light scenes is obtained; (3) using a validation set and a test set to evaluate the model obtained in step (2), wherein the validation set is used to evaluate the parameter indicators of the model, and the test set is used to verify the detection performance of the model; The specific process of model training in step (2) is as follows: S1. Make the original image F pass through two convolutional layers in sequence to eliminate redundant information, and obtain feature maps F1 and F2; the original image is represented by the data size F∈R H×W×3 , the data size of the feature map is 1 / 4 and 1 / 8 of the original image, respectively, expressed as and S2. Input the original image F, feature map F1, and feature map F2 into the feature enhancement network module in parallel to obtain the output feature maps O, O1, and O2 on the three paths, denoted as O∈R H×W×32 , S3. The output feature map O2 is concatenated with the output feature map O1 in terms of channel dimension through a layer of upsampling network, and then the channel dimension is adjusted through upsampling again to obtain a concatenated convolutional feature map; the output feature map O is element-wise added to the concatenated convolutional feature map, and the image is decoded to obtain an intermediate image after image enhancement; S4. A lightweight feature extraction backbone network MobileViT is used to extract features from the intermediate image. The extracted feature information of different scales is fused at different levels to obtain a multi-layer fused feature map. The multi-layer fused feature map is input into the detection head, and finally the network predicts the detection result.

2. The method for constructing a lightweight target detection network adapted to dark light scenes as claimed in claim 1, characterized in that: The method for dividing the training set, validation set and test set in step (1) is: a. Collecting original image sequences: Videos collected in low-light environments are divided into 10 image sequences; b. Create image labels: Use the labelImg tool to manually label the object categories in the image and export the image label file; c. Divide the image into training set, validation set and test set: Divide the collected image sequence into training set, validation set and test set in a ratio of 7:1:

2.

3. The method for constructing a lightweight target detection network adapted to dark light scenes as claimed in claim 1, characterized in that: S1 is described as eliminating redundant information through two convolutional layers, the convolution kernel size of the convolutional layer is 3, which can be expressed as: F1=Conv(F),S=4, F2=Conv(F1),S=2, In the formula, Conv represents the convolution operation and S represents the stride of the convolution.

4. The method for constructing a lightweight target detection network adapted to dark light scenes as claimed in claim 3, characterized in that: S2 The feature enhancement network is composed of six layers of CBR modules with residual connections, wherein the first layer and the sixth layer are provided with residual connections, the second layer and the fifth layer are provided with residual connections, and the third layer and the fourth layer are provided with residual connections; each CBR module performs Cov, BN and Relu operations in sequence, the convolution kernel size of the Cov convolution operation is 3, the step size is 1, the size of the feature map is not changed during the encoding process, the number of channels is increased from 3 to 32, and the extracted information is integrated into the channel dimension.

5. The method for constructing a lightweight target detection network adapted to dark light scenes as claimed in claim 4, characterized in that: The process of adding the output feature map O and the concatenated convolution feature map element by element in S3 is expressed as follows: In the formula, the upsample operation represents upsampling, the concat operation represents concatenation along the channel dimension, and the add operation represents element-wise addition of feature maps. Represents the feature map of the final output.

6. The method for constructing a lightweight target detection network adapted to dark light scenes as claimed in claim 5, characterized in that: The specific method of feature extraction described in S4 is: encode each area of ​​the intermediate image into three vectors, namely the query vector Q, the key value vector K and the value vector V, calculate the similarity between the Q of each area and the K of the remaining areas, perform weighted summation according to the weight, and extract effective information.

7. The method for constructing a lightweight target detection network adapted to dark light scenes as claimed in claim 1, characterized in that: The overall network loss function in step (2) is defined as the reconstruction loss L1 and the detection loss L d , the reconstruction loss is mainly used to constrain the information loss of the input image after passing through the feature enhancement network; the detection loss is mainly used to constrain the detection loss of the target detection network, including the target loss L obj , category loss L cls and the regression loss L loc , expressed as: L=λ1L1+λ d L d =λ1L1+λ obj L obj +λ cls L cls +λ loc L loc Among them, λ1, λ obj , cls and λ loc Represent the weight coefficients of each part of the loss.

8. A target detection method, comprising the steps of inputting an image to be detected into the target detection network constructed according to claims 1-7 to perform target detection.

9. An electronic product, comprising a processor and a memory communicatively connected to the processor; the memory is used to store instructions; and the processor is used to call the instructions stored in the memory to execute the target detection method as claimed in claim 8.

Images