Low-light image detection algorithm based on multi-scale feature extraction

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a low-light image detection algorithm based on multi-scale feature extraction, which comprises a multi-scale feature extraction module, a loss function and a space-to-depth sampling module, and is characterized in that the multi-scale feature extraction module is added into a feature extraction network of an enhancement model; the loss function comprises an exposure control loss function, a color constancy loss function, an illumination smooth loss function and a mixed loss function, and the space-to-depth sampling module is composed of a space-to-depth layer and a non-convolution step length layer; multi-branch step-by-step pixel feature extraction is carried out on a multi-level feature extraction network, space and channel reconstruction convolution is used for replacing standard convolution operation in the last-level feature extraction stage, redundant features are reduced, calculation complexity is reduced, and meanwhile the correlation degree between pixels can be well improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular to a low-light image detection algorithm based on multi-scale feature extraction. Background Art

[0002] As an important application branch in the field of artificial intelligence, object detection has played an important role in many fields such as autonomous driving, criminal investigation, and industrial maintenance with the continuous development of artificial intelligence in recent years. In these applications, high-quality images and videos are crucial for target recognition and detection. However, under unfavorable or even harsh conditions, such as low-light environments, traditional object detection models trained based on high-quality images often fail to achieve ideal detection results.

[0003] The current mainstream low-light detection method is to first perform illumination enhancement processing on the low-light image through the existing image enhancement framework, and then send the enhanced image to the detector for detection. However, almost all of the currently well-performing enhancement frameworks are based on human vision and serve humans to see images with better visual quality. Experimental exploration shows that although the enhancement framework changes the visual quality of the image, it inevitably destroys the inherent properties of some pixels, resulting in a decrease in the target detection performance of the enhanced image. In order to solve the above problems, based on the Zero-DCE algorithm, an image enhancement algorithm that is more suitable for image detection is proposed, and by combining it with the most mainstream YOLOv8 detector, the performance of target detection in low-light conditions is improved.

[0004] Therefore, to address the above problems, a low-light image detection algorithm based on multi-scale feature extraction is proposed. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the shortcomings of the prior art, the present invention provides a low-light image detection algorithm based on multi-scale feature extraction, which solves the problem that the inherent properties of some pixels are inevitably destroyed during the image enhancement processing in the prior art, resulting in a decrease in the target detection performance of the enhanced image.

[0007] (II) Technical solution

[0008] To achieve the above object, the present invention provides the following technical solutions: a low-light image detection algorithm based on multi-scale feature extraction, comprising a multi-scale feature extraction module, a loss function and a space-to-depth sampling module;

[0009] Add the multi-scale feature extraction module to the feature extraction network of the enhanced model;

[0010] The loss functions include exposure control loss, color constancy loss, lighting smoothness loss and hybrid loss function;

[0011] The spatial-to-depth sampling module consists of a spatial-to-depth layer and a convolution-free stride layer.

[0012] Preferably, a multi-scale feature extraction module is added to the feature extraction network of the enhancement model, the input of the multi-scale feature extraction module is the feature extracted by the previous convolution kernel, pixel features are extracted level by level through multi-branch on the multi-level feature extraction network, and spatial and channel reconstruction convolution is used instead of standard convolution operation in the last level feature extraction stage;

[0013] The input features are divided into 4 groups, and the 3×3 filters of the previous n channels are replaced with smaller filter groups. After 1×1 convolution, the feature map is evenly divided into s feature map subsets, denoted as x i , where i∈{1, 2, …, s}, each feature subset x i It has the same spatial size as the input feature map, the number of channels is 1 / s, and except for x1, each x i There are corresponding convolution operations K i (), use y i K i The output of (), feature map subset x i With K i-1 () outputs are added and then sent to K as output i (), y i It can be expressed as:

[0014]

[0015] The feature maps of all groups are concatenated and fed into another 1×1 filter.

[0016] Preferably, the spatial and channel reconstruction convolution includes a spatial reconstruction unit and a channel reconstruction unit, the two units are arranged in sequence, and the input X first passes through the spatial reconstruction unit to obtain the spatial refinement feature X ω , and then through the channel reconstruction unit, the channel refined feature Y is obtained as the output, and the feature map with large information content is separated from the feature map with small information content through the separation operation. Through the reconstruction operation, the two weighted different feature information are merged to obtain X ω1 and X ω2 , after connection, we get the spatial refinement feature map X ω , the formula is:

[0017]

[0018] Preferably, the channel reconstruction unit uses a segmentation, conversion, and fusion method to convert the X obtained after SRU processingω Divide into two parts with different number of channels, compress these two parts using 1×1 convolution kernel to obtain feature X up and X low , the network will X up As the input of rich feature extraction, after GWC and PWC operations respectively, the sum is obtained to get the output Y1, and X low As a supplement to the previous extraction, the result is combined with the original input to obtain the output Y2. Using the idea of ​​neural network, the outputs Y1 and Y2 of the conversion stage are adaptively merged. Global average pooling is used to collect global spatial information. The channel statistics S m The calculation formula is:

[0019]

[0020] S1 and S2 are superimposed together, and feature weight vectors β1 and β2 are obtained through Softmax. The feature weight vector is combined with the stage output to obtain the channel refinement feature Y. The calculation formula of Y is:

[0021] Y=β1Y1+β2Y2.

[0022] Preferably, exposure control loss: The exposure level is controlled by exposure control loss, which measures the difference between the average intensity of the local area and the well-exposed area. The formula is:

[0023]

[0024] Where E is the grayscale level in the RGB space, M represents the number of non-overlapping local regions of size 16×16, and Y is the average intensity value of the local region in the enhanced image.

[0025] Preferably, color constant loss: the average value of the RGB channels is close to the global average value of the entire image, and the color constant loss is used to correct the potential color deviation and establish the connection between the three channels. The formula is:

[0026]

[0027] Preferably, lighting smoothness loss: an lighting smoothness loss is added to each generated curve parameter map A, and its formula is:

[0028]

[0029] Where N is the number of iterations, and Represents horizontal and vertical gradient operations respectively.

[0030] Preferably, the hybrid loss function integrates the MS-SSIM loss and L1 loss that perform well in computer vision tasks, and its formula is:

[0031]

[0032] Among them, W col and W tvA is the weight corresponding to the loss function.

[0033] Preferably, the SPD component divides the input feature X into several sub-feature maps according to a scale factor, and concatenates the sampled sub-feature maps along the channel dimension to obtain an intermediate feature map X', X' reduces a scale factor in the spatial dimension and increases a scale factor in the channel dimension, and performs a convolution operation with stride=1 on the intermediate feature map to obtain a feature map with intact features, and places the space-to-depth sampling module before each c2f module.

[0034] (III) Beneficial effects

[0035] Compared with the prior art, the present invention provides a low-light image detection algorithm based on multi-scale feature extraction, which has the following beneficial effects:

[0036] 1. This low-light image detection algorithm based on multi-scale feature extraction extracts pixel features step by step through multi-branch extraction on a multi-level feature extraction network, and uses spatial and channel reconstruction convolution instead of standard convolution operations in the last level of feature extraction. It reduces redundant features and computational complexity while also being able to improve the correlation between pixels.

[0037] 2. This low-light image detection algorithm based on multi-scale feature extraction reduces the spatial and channel redundancy between features in the convolutional neural network by introducing spatial and channel reconstruction convolution, improves the correlation between pixels and improves the overall performance of the network.

[0038] 3. This is a low-light image detection algorithm based on multi-scale feature extraction. By placing the space-to-depth sampling module before each c2f module, the information with more features is handed over to c2f to better perform feature fusion and improve the performance of detecting low-resolution and small target objects.

[0039] 3. This low-light image detection algorithm based on multi-scale feature extraction guides the generation of low-light enhanced images that are more conducive to target detection by selecting exposure control loss, color constancy loss, lighting smoothing loss and mixed loss. While ensuring the high correlation of the global features of the generated image, it can also well preserve the color and brightness of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0041] Figure 1 It is a structural diagram of the multi-scale feature extraction module of the present invention;

[0042] Figure 2 It is the overall structure diagram of the spatial and channel reconstruction convolution of the present invention;

[0043] Figure 3 It is a structural diagram of the spatial and channel reconstruction convolution module of the present invention;

[0044] Figure 4 It is the model structure and application diagram of the space-to-depth sampling module of the present invention;

[0045] Figure 5 It is a comparison chart of ablation experiment results of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Specific examples are given below.

[0048] See also Figure 1-Figure 5 ,The present invention provides a low-light image detection algorithm based on multi-scale feature extraction, including a multi-scale feature extraction module, a loss function and a space-to-depth sampling module;

[0049] Add the multi-scale feature extraction module to the feature extraction network of the enhanced model;

[0050] The loss functions include exposure control loss, color constancy loss, lighting smoothness loss and hybrid loss function;

[0051] The spatial-to-depth sampling module consists of a spatial-to-depth layer and a convolution-free stride layer;

[0052] A multi-scale feature extraction module is added to the feature extraction network of the enhanced model. The input of the multi-scale feature extraction module is the feature extracted by the previous convolution kernel. The pixel features are extracted level by level through multi-branch on the multi-level feature extraction network, and the spatial and channel reconstruction convolution is used instead of the standard convolution operation in the last level feature extraction stage.

[0053] The input features are divided into 4 groups, and the 3×3 filters of the previous n channels are replaced with smaller filter groups. After 1×1 convolution, the feature map is evenly divided into s feature map subsets, denoted as x i , where i∈{1, 2, …, s}, each feature subset x i It has the same spatial size as the input feature map, the number of channels is 1 / s, and except for x1, each x i There are corresponding convolution operations K i (), use y i K i The output of (), feature map subset x i With K i-1 () outputs are added and then sent to K as output i (), y i It can be expressed as:

[0054]

[0055] Concatenate the feature maps of all groups and send them to another 1×1 filter;

[0056] The information can be completely fused to obtain feature maps with higher pixel correlation and extract more diverse information.

[0057] The spatial and channel reconstruction convolution includes a spatial reconstruction unit and a channel reconstruction unit. The two units are arranged in sequence. The input X first passes through the spatial reconstruction unit to obtain the spatial refinement feature X. ω , and then through the channel reconstruction unit, the channel refined feature Y is obtained as the output, and the feature map with large information content is separated from the feature map with small information content through the separation operation. Through the reconstruction operation, the two weighted different feature information are merged to obtain X ω1 and X ω2 , after connection, we get the spatial refinement feature map X ω , the formula is:

[0058]

[0059] The channel reconstruction unit uses segmentation, conversion, and fusion methods to convert the X obtained after SRU processing ω Divide into two parts with different number of channels, compress these two parts using 1×1 convolution kernel to obtain feature X up and X low , the network will X up As the input of rich feature extraction, after GWC and PWC operations respectively, the sum is obtained to get the output Y1, and X lowAs a supplement to the previous extraction, the result is combined with the original input to obtain the output Y2. Using the idea of ​​neural network, the outputs Y1 and Y2 of the conversion stage are adaptively merged. Global average pooling is used to collect global spatial information. The channel statistics S m The calculation formula is:

[0060]

[0061] S1 and S2 are superimposed together, and feature weight vectors β1 and β2 are obtained through Softmax. The feature weight vector is combined with the stage output to obtain the channel refinement feature Y. The calculation formula of Y is:

[0062] Y=β1Y1+β2Y2;

[0063] It can reduce the spatial and channel redundancy between features in convolutional neural networks, increase the correlation between pixels and improve the overall performance of the network.

[0064] Exposure control loss: The exposure level is controlled by exposure control loss, which measures the difference between the average intensity of the local area and the well-exposed area. The formula is:

[0065]

[0066] Where E is the grayscale level in RGB space, M represents the number of non-overlapping local regions of size 16×16, and Y is the average intensity value of the local region in the enhanced image;

[0067] Color constancy loss: The average value of the RGB channels is close to the global average value of the entire image. Color constancy loss is used to correct potential color deviations and establish the connection between the three channels. The formula is:

[0068]

[0069] Lighting smoothness loss: Add an illumination smoothness loss to each generated curve parameter map A, and its formula is:

[0070]

[0071] Where N is the number of iterations, and Represents horizontal and vertical gradient operations respectively;

[0072] Hybrid loss function: It integrates MS-SSIM loss and L1 loss which have excellent performance in computer vision tasks. Its formula is:

[0073]

[0074] Among them, Wcol and W tvA is the weight corresponding to the loss function;

[0075] While ensuring the high correlation of the global features of the generated image, the color and brightness of the image can also be well preserved.

[0076] The SPD component divides the input feature X into several sub-feature maps according to a scale factor, and concatenates the sampled sub-feature maps along the channel dimension to obtain an intermediate feature map X'. X' reduces a scale factor in the spatial dimension and increases a scale factor in the channel dimension. The intermediate feature map is convolved with stride = 1 to obtain a feature map with intact features. The space-to-depth sampling module is placed before each c2f module;

[0077] All discriminative feature information is retained to facilitate low-resolution and small target detection, and information with more features is handed over to c2f for better feature fusion, thereby improving the performance of detecting low-resolution and small target objects.

[0078] The proposed method is compared with the existing Zero-DCE, MBLLEN, EnlightenGAN, RUAS, and URetinexNet low-light image enhancement frameworks combined with the YOLOv8 detector. URetinexNet proposes a deep unfolding network based on the Retinex theory, and carefully designs three learnable modules to achieve noise suppression and detail retention of the decomposition results, and finally generates a satisfactory low-light enhanced image. RUAS is a low-light image enhancement module based on unsupervised learning. It obtains the illumination map and generates the required image with good visual performance through a cooperative reference-free two-layer learning strategy. The detector used in this paper is unified as the YOLOv8 version. Table 1 shows the comparison results of different enhancement models combined with YOLOv8 on the ExDarkPlus dataset;

[0079]

[0080] Table 1 Detection results of different enhancement frameworks on the ExDarkPlus dataset

[0081] Through experimental data, we verified the previous statement that most existing low-light enhancement frameworks only focus on enhancing the visual effects of images, resulting in ignoring or destroying the perception of pixel semantic level, which in turn affects the subsequent object detection work.

[0082] In order to further verify the effectiveness of the improved algorithm proposed in this paper, this section designs an ablation experiment on the multi-scale feature extraction module and the space-to-depth sampling module. By selectively adding modules, the role of the modules in the entire work can be verified. The experimental results are shown in Table 2. Figure 5 The comparison of ablation experiment results is shown.

[0083]

[0084] Table 2 Ablation experiment results

[0085] The basic principles, main features and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A low-light image detection algorithm based on multi-scale feature extraction, characterized by: Includes multi-scale feature extraction module, loss function and space-to-depth sampling module; Add the multi-scale feature extraction module to the feature extraction network of the enhanced model; The loss functions include exposure control loss, color constancy loss, lighting smoothness loss and hybrid loss function; The spatial-to-depth sampling module consists of a spatial-to-depth layer and a convolution-free stride layer.

2. The low-light image detection algorithm based on multi-scale feature extraction according to claim 1, characterized in that: A multi-scale feature extraction module is added to the feature extraction network of the enhanced model. The input of the multi-scale feature extraction module is the feature extracted by the previous convolution kernel. The pixel features are extracted level by level through multi-branch on the multi-level feature extraction network, and the spatial and channel reconstruction convolution is used instead of the standard convolution operation in the last level feature extraction stage. The input features are divided into 4 groups, and the 3×3 filters of the previous n channels are replaced by smaller filter groups. After 1×1 convolution, the feature map is evenly divided into s feature map subsets, denoted as xi, where i∈{1, 2, …, s}. Each feature subset xi has the same spatial size as the input feature map, and the number of channels is 1 / s. Except for x1, each x i There are corresponding convolution operations K i (), use y i K i The output of (), feature map subset x i With K i-1 () outputs are added and then sent to K as output i (), y i It can be expressed as: The feature maps of all groups are concatenated and fed into another 1×1 filter.

3. The low-light image detection algorithm based on multi-scale feature extraction according to claim 1, characterized in that: The spatial and channel reconstruction convolution includes a spatial reconstruction unit and a channel reconstruction unit. The two units are arranged in sequence. The input X first passes through the spatial reconstruction unit to obtain the spatial refinement feature X. ω , and then through the channel reconstruction unit, the channel refined feature Y is obtained as the output, and the feature map with large information content is separated from the feature map with small information content through the separation operation. Through the reconstruction operation, the two weighted different feature information are merged to obtain X ω1 and X ω2 , after connection, we get the spatial refinement feature map X ω , the formula is:

4. The low-light image detection algorithm based on multi-scale feature extraction according to claim 1, characterized in that: The channel reconstruction unit uses segmentation, conversion, and fusion methods to convert the X obtained after SRU processing ω Divide into two parts with different number of channels, compress these two parts using 1×1 convolution kernel to obtain feature X up and X low , the network will X up As the input of rich feature extraction, after GWC and PWC operations respectively, the sum is obtained to get the output Y1, and X low As a supplement to the previous extraction, the result is combined with the original input to obtain the output Y2. Using the idea of ​​neural network, the outputs Y1 and Y2 of the conversion stage are adaptively merged. Global average pooling is used to collect global spatial information. The channel statistics S m The calculation formula is: S1 and S2 are superimposed together, and feature weight vectors β1 and β2 are obtained through Softmax. The feature weight vector is combined with the stage output to obtain the channel refinement feature Y. The calculation formula of Y is: Y=β1Y1+β2Y2.

5. The low-light image detection algorithm based on multi-scale feature extraction according to claim 1, characterized in that: Exposure control loss: The exposure level is controlled by exposure control loss, which measures the difference between the average intensity of the local area and the well-exposed area. The formula is: Where E is the grayscale level in the RGB space, M represents the number of non-overlapping local regions of size 16×16, and Y is the average intensity value of the local region in the enhanced image.

6. The low-light image detection algorithm based on multi-scale feature extraction according to claim 1, characterized in that: Color constancy loss: The average value of the RGB channels is close to the global average value of the entire image. Color constancy loss is used to correct potential color deviations and establish the connection between the three channels. The formula is:

7. The low-light image detection algorithm based on multi-scale feature extraction according to claim 1, characterized in that: Lighting smoothness loss: Add an illumination smoothness loss to each generated curve parameter map A, and its formula is: Where N is the number of iterations, ▽ x and y Represents horizontal and vertical gradient operations respectively.

8. The low-light image detection algorithm based on multi-scale feature extraction according to claim 1, characterized in that: Hybrid loss function: It integrates MS-SSIM loss and L1 loss which have excellent performance in computer vision tasks. Its formula is: Among them, W col and W tvA is the weight corresponding to the loss function.

9. The low-light image detection algorithm based on multi-scale feature extraction according to claim 1, characterized in that: The SPD component divides the input feature X into several sub-feature maps according to a scaling factor, concatenates the sampled sub-feature maps along the channel dimension, and obtains an intermediate feature map X'. X' reduces a scale factor in the spatial dimension and increases a scale factor in the channel dimension. The intermediate feature map is convolved with stride = 1 to obtain a feature map with intact features. The space-to-depth sampling module is placed before each c2f module.

Images