Infrared image enhancement methods and computer program products based on multi-scale information exchange

Abstract

This invention pertains to an infrared image enhancement method. To address the technical challenge of current convolutional neural network-based image enhancement methods still needing improvement in detail representation and overall visual effect, this invention provides an infrared image enhancement method and computer program product based on multi-scale information exchange. During image enhancement, the original image at its original size undergoes convolutional learning to obtain a coarse feature map. This coarse feature map is then upsampled and downsampled for further feature extraction, better complementing the original-scale feature learning process. Simultaneously, during multi-scale information mining, a random selection and fusion module exchanges feature information from different scales, enabling cross-scale feature learning. By combining feature information from multiple scales, the spatial details of the original image are preserved, thus enhancing the image's detail representation.

Description

Technical Field

[0001] This invention relates to an infrared image enhancement method, specifically an infrared image enhancement method and computer program product based on multi-scale information exchange. Background Technology

[0002] Infrared imaging plays a crucial role in medical imaging, security monitoring, environmental pollution detection, and military target detection. However, infrared images typically suffer from drawbacks such as low contrast and blurred details, which limit the observation of infrared targets and the further development of infrared imaging applications. Therefore, it is necessary to enhance infrared images to obtain high-quality results.

[0003] Currently, most methods for infrared image enhancement are generalized using spatial and frequency domains, including histogram equalization, contrast adjustment and transformation, and empirical mode decomposition. Histogram equalization methods have been introduced to equalize the histogram distribution of infrared images, improving the overall dynamic range of grayscale. However, these methods often suffer from over-enhancement or under-enhancement. In contrast, methods based on convolutional neural networks (CNNs) have achieved excellent performance in various visual tasks, such as object detection, image recognition, and low-level vision.

[0004] Choi et al. first proposed a CNN-based method for enhancing infrared images, utilizing a relatively shallow CNN. Subsequently, Retinex-inspired networks were developed, such as Shen, L., Yue, Z., Feng, F., Chen, Q., Liu, S., Ma, J.'s 2017 paper "Msr-net: Low-light image enhancement using deep convolutional network" and Zhang, Y., Hang, J., Guo X.'s 2019 paper "Kindling the darkness: A practical low-light image enhancer". Encoder-decoder networks, such as the 2019 papers "Low-light image enhancement via a deep hybrid network" by Ren, W., Liu, S., Ma, L., Xu, Q., Xu, X., Cao, X., Du, J., Yang, MH, and ... However, currently, image enhancement methods based on convolutional neural networks still need further improvement in terms of image detail representation and overall visual effect. Summary of the Invention

[0005] This invention addresses the technical problem that current image enhancement methods based on convolutional neural networks still need improvement in terms of image detail representation and overall visual effect. It provides an infrared image enhancement method and computer program product based on multi-scale information exchange.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] An infrared image enhancement method based on multi-scale information exchange is characterized by the following steps:

[0008] S1, by performing convolutional learning on the original image, a coarse feature map F0 of the original image is obtained;

[0009] S2, perform upsampling and downsampling on the coarse extracted feature map F0 respectively to obtain the upsampled coarse extracted feature map F1 and the downsampled coarse extracted feature map F2;

[0010] S3, sequentially through the bottleneck attention module and the random selection and fusion module, performs one attention feature extraction and multi-scale feature exchange on F1, F0 and F2, then through the bottleneck attention module, performs a second attention feature extraction, and then through the feature fusion module, performs multi-scale feature fusion to obtain a multi-scale feature fusion map;

[0011] S4. Repeat steps S2 and S3 multiple times, and each time use the multi-scale feature fusion map obtained in the previous step S3 as the coarse extracted feature map F0 in step S2 to obtain a multi-level feature extraction map.

[0012] S5, input the multi-level feature extraction map into the convolutional layer to obtain the preliminary reconstructed image;

[0013] S6, the original image is downsampled and upsampled sequentially, and then connected with the preliminarily reconstructed image to obtain the final enhanced image.

[0014] Further, in step S3, the first attention feature extraction specifically involves:

[0015] Sa.1, calculate the channel feature attention map and spatial feature attention map of F1, F0 and F2 respectively, and obtain the spatial-channel feature attention maps M(F1), M(F0) and M(F2) of F1, F0 and F2 respectively;

[0016] Sa.2, M(F1), M(F0) and M(F2) are multiplied pixel by pixel with the corresponding F1, F0 and F2 respectively, and the multiplication results are superimposed with F1, F0 and F2 to obtain the corresponding first-order output feature maps L1, L0 and L2.

[0017] Further, in step S3, the multi-scale feature exchange specifically involves:

[0018] Sb.1, perform downsampling and upsampling on L1 and L2 respectively to restore L1 and L2 to the same scale as L0. Then add L0, L1 and L2 restored to the same scale as L0 to obtain the feature map L.

[0019] Sb.2, global average pooling is performed on feature map L to encode the information of the same channel into a real number;

[0020] Sb.3 uses a convolutional layer as a channel-reduced convolutional layer to perform convolutional learning and generate a channel feature;

[0021] Sb.4, For the channel features obtained in step Sb.3, use three parallel convolutional layers of the same dimension as channel expansion convolutions to generate three channel feature maps V1, V0 and V2 respectively;

[0022] Sb.5, multiply the three channel feature maps V1, V0 and V2 with L0, and L1 and L2 restored to the same scale as L0 in the feature dimension, respectively, to obtain feature maps L'1, L'0 and L'2 after multi-scale feature information exchange;

[0023] Sb.6, randomly selects channels from V1, V0 and V2 respectively, and obtains constants s1, s2 and s3 after softmax activation;

[0024] Sb.7, combining feature maps L'1, L'0 and L'2, and constants s1, s2 and s3, yields the multi-scale feature exchange result U0;

[0025] Sb.8, then repeat steps Sb.6 and Sb.7 twice to obtain two different multi-scale feature exchange results U1 and U2.

[0026] Furthermore, in step S3, the secondary attention feature extraction is performed using the same method as the primary attention feature extraction, resulting in secondary output feature maps L21, L20, and L22.

[0027] Furthermore, in step S3, the multi-scale feature fusion specifically involves:

[0028] Sc.1, upsample or downsample L21 and L22 respectively to make L21, L20 and L22 have the same scale;

[0029] Sc.2, add L20, L21 and L22 after upsampling or downsampling, to obtain the resulting feature map M;

[0030] In Sc.3, the resulting feature map M is added to the coarse extracted feature map F0 using residual connections to obtain a multi-scale feature fusion map.

[0031] Furthermore, in step S3, the loss functions in the bottleneck attention module, the random selection and fusion module, and the feature fusion module are:

[0032]

[0033] in, Represents the loss function. Indicates the output image, I *Let represent the input image, and ε represent a constant.

[0034] Furthermore, in step Sa.2, the channel feature attention map is obtained in the following way:

[0035] Global average pooling is used to encode the global information in each channel into a real number. Then, a hidden layer consisting of two fully connected layers is used to reduce and restore the feature channels. Finally, a batch normalization layer is used to normalize the data, resulting in a channel feature attention map.

[0036] Furthermore, in step Sa.2, the spatial feature attention map is obtained in the following way:

[0037] First, a 1x1 convolution is used to map the coarsely extracted feature map into a low-dimensional space R. C / r×H×W After two more 3x3 dilated convolutions, a 1x1 convolution is used to map the result to space R. 1×H×W Finally, a normalization layer is used to normalize the data to obtain a spatial feature attention map;

[0038] Where C represents the number of channels in the coarsely extracted feature map, r represents, H represents, and W represents.

[0039] Furthermore, in step S4, the repeated execution of steps S2 and S3 specifically means repeating the execution three times.

[0040] The present invention also provides a computer program product, including a computer program, which is characterized in that, when the program is executed by a processor, it implements the steps of the above-described infrared image enhancement method based on multi-scale information exchange.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. This invention proposes an infrared image enhancement method based on multi-scale information exchange. During image enhancement, the original image at its original size undergoes convolutional learning to obtain a coarse feature map. This coarse feature map is then upsampled and downsampled for feature extraction, better complementing the original-scale feature learning process. Simultaneously, during multi-scale information mining, a random selection and fusion module is used to exchange feature information from different scales, enabling cross-scale feature learning. By combining feature information from multiple scales, the spatial details of the original image are preserved, improving the image's detail representation capabilities.

[0043] 2. In this invention, attention feature extraction is performed through a bottleneck attention module. After calculating the spatial-channel feature attention map, it is multiplied with the coarse extracted feature map. The result of the multiplication is then superimposed on the coarse extracted feature map to complete the attention feature extraction, so that the target in the image can be emphasized in both spatial and channel dimensions.

[0044] 3. In this invention, when reconstructing an image, the original image is downsampled and upsampled sequentially, and then connected with the initially reconstructed image. This can effectively improve the quality of the reconstructed image and avoid the disturbance of the result by noise in the original image.

[0045] 4. The present invention also provides a computer program product capable of executing the above method steps, which can extend and apply the method of the present invention and achieve integration on corresponding hardware devices. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating an embodiment of the infrared image enhancement method based on multi-scale information exchange of the present invention.

[0047] Figure 2 This is a schematic diagram illustrating the principle of MSIEB in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the principle of BAB in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the RSFB principle in an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram illustrating the principle of FFB in an embodiment of the present invention; Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0052] This invention proposes an infrared image enhancement method based on multi-scale information exchange, which can improve the detail representation and overall visual effect of images. For example... Figure 1 The diagram shown illustrates the overall flow of an embodiment of the infrared image enhancement method of the present invention. The original image undergoes convolutional learning using a convolutional layer (ConV), then is processed sequentially through three multi-scale enhancement modules (MSIEB), before being input into the convolutional layer for preliminary reconstruction. Finally, it is concatenated with the original image, which has undergone downsampling and upsampling, to complete the enhancement and obtain the final enhanced image. The specific implementation steps are as follows:

[0053] Step 1: Extract features from the input original image.

[0054] The original input image is subjected to convolutional learning using convolutional layers to obtain a coarse feature map F0 of the original image.

[0055] Step 2, Deep Information Mining

[0056] Step 2 is mainly completed by three structurally identical multi-scale enhancement modules (MSIEB) to perform deep feature mining on the coarse feature map F0 extracted in Step 1. For example... Figure 2 Each MSIEB mainly contains a Bottleneck Attention Module (BAB), a Random Selection and Fusion Module (RSFB), and a Feature Fusion Module (FFB). The specific steps executed in each MSIEB are as follows:

[0057] Step 2.1, First Attention Feature Extraction

[0058] The main structure used to implement step 2.1 is the Bottleneck Attention Module (BAB). The BAB calculates the channel feature attention map and spatial feature attention map of the coarsely extracted feature map, then merges them to form the spatial-channel feature attention map of the coarsely extracted feature map. The spatial-channel feature attention map is then fused with the input coarsely extracted feature map to form the output feature map improved by attention feature extraction. The extraction method of the output feature map can be expressed by the following formula:

[0059]

[0060] Where F′ represents the output feature map after attention feature extraction improvement, F represents the coarse extracted feature map, and M(F) represents the spatial-channel feature attention map. This represents pixel-wise multiplication. M(F) uses the spatial feature attention map M. s (F) and Feature Attention Map M c (F) After extended fusion and activation, it can be expressed as follows:

[0061] M(F)=σ(M C (F)+M S (F))

[0062] like Figure 3 M s (F) and M c (F) is calculated using both spatial and channel attention branches. The spatial attention branch first passes through a 1×1 convolution, mapping the features of the coarsely extracted feature map into a low-dimensional space R. C / r×H×W Then, it undergoes two layers of 3×3 dilated convolutions to increase the network's receptive field, followed by a 1×1 convolution mapping to R. 1×H×W Finally, a normalization layer is used to normalize the data, resulting in the spatial feature attention map M. s(F), where H and W represent the height and width of the coarsely extracted feature map, respectively, C / r ​​represents the number of channels in the low-dimensional space, and r represents the channel mapping ratio from the coarsely extracted features to the low-dimensional space. The channel attention branch first encodes the global information in each channel into a real number through global average pooling, thus obtaining the vector F. C ∈R C×1×1 Then, a hidden layer (consisting of two fully connected layers) is used to reduce and restore the feature channels, thereby reducing the additional overhead. Finally, a batch normalization (BN) layer is used to normalize the data, resulting in a channel feature attention map.

[0063] After calculating the spatial-channel feature attention map, it is multiplied pixel by pixel with the coarse extracted feature map F. The resulting feature map is then superimposed with the coarse extracted feature map F to obtain the output feature map L. At this point, the target in the image is emphasized in both spatial and channel dimensions.

[0064] The coarse feature map extracted in step 1 is downsampled and upsampled respectively to form a three-layer multi-scale feature map with the coarse feature map corresponding to the original image. The coarse feature map F0, the upsampled coarse feature map F1, and the downsampled coarse feature map F2 are input into BAB respectively. BAB outputs output feature maps L1, L0 and L2 weighted by channel feature attention and spatial feature attention, which correspond to F1, F0 and F2 respectively.

[0065] Step 2.2, multi-scale feature exchange.

[0066] The main structure for executing step 2.2 is RSFB, and the RSFB structure is as follows: Figure 4 As shown. The input consists of output feature maps L1, L0, and L2 at different scales. L1 and L2 are upsampled and downsampled respectively to recover the same scale as L0, and the three are added together to obtain feature map L. Global average pooling is applied to feature map L to encode the information of the same channel into a real number, resulting in R. 1×1×C Where C represents the number of channels, and a single convolutional layer is used as a channel reduction convolutional layer to generate a compact channel feature, denoted as R. 1 ×1×r1Where r1 = C / n, r1 represents the number of channels in the compact channel feature, and n is an integer representing the scaling factor from the number of channels C to r. Finally, three parallel convolutional layers are used as channel expansion convolutions to generate three channel feature maps, denoted as V1, V0, and V2, respectively. The dimensions of the three parallel convolutional layers are all 1×1×C. V1, V0, and V2 are multiplied with the output feature maps L1, L0, and L2 of the corresponding layers in the feature dimension, respectively, to obtain feature maps L'1, L'0, and L'2 after multi-scale feature information exchange, and then the output U of RSFB is obtained, U = s1L'1 + s2L'2 + s3L'3, where s1, s2, and s3 are obtained by randomly selecting channels from V1, V0, and V2 and performing softmax activation.

[0067] like Figure 2 The output of BAB is processed by three RSFBs. The s1, s2 and s3 of the three RSFBs are different in the processing, which results in multi-scale feature exchange results U1, U0 and U2.

[0068] Step 2.3, Secondary Attention Feature Extraction

[0069] The multi-scale feature exchange results U1, U0, and U2 are input into BAB, and attention feature extraction is performed again. Specifically, the method of secondary attention feature extraction is exactly the same as that of primary attention feature extraction. U1, U0, and U2 are used as coarse extracted feature maps in primary attention feature extraction, and the corresponding secondary output feature maps L21, L20, and L22 are obtained after secondary attention feature extraction.

[0070] Step 2.4, multi-scale feature fusion.

[0071] like Figure 5 The three-layer multi-scale features are obtained through steps 2.1-2.3. Figure 2 The output feature maps L21, L20, and L22 are then upsampled or downsampled to the same scale as L20 using FFB and summed to obtain the result feature map M. Simultaneously, residual connections are used to sum M with the input feature map of the MSIEB, i.e., the coarse extracted feature map, to form the result feature map of the MSIEB.

[0072] After completing steps S2.1 to S2.4 above, the feature extraction process of one MSIEB is completed. In this embodiment, a total of three MSIEBs with the same structure are used to perform multi-level feature extraction on the coarse feature map to obtain a multi-level feature extraction map.

[0073] Step 3, Image Reconstruction

[0074] Step 2 yields feature maps that have undergone multi-scale feature exchange and depth extraction. Step 3 then reconstructs the image. The output features of the multi-level feature extraction map obtained in Step 2 are input into a convolutional layer to obtain a preliminary reconstructed image. To improve the quality of the reconstructed image and to avoid noise in the original image from affecting the result, the original image is sequentially downsampled and upsampled before being concatenated with the preliminary reconstructed image to form the final enhanced image.

[0075] The infrared image enhancement method of the present invention described above is mainly processed by three MSIEBs. The loss functions of the bottleneck attention module, the random selection and fusion module, and the feature fusion module in the MSIEB can all adopt the following loss function:

[0076]

[0077] in, Represents the loss function. Indicates the output image, I * Let represent the input image, and ε represent a constant.

[0078] In other embodiments of the present invention, the number of MSIEBs can also be adjusted according to actual needs. Practical verification has shown that the method of the present invention can improve the detail representation of the image while preserving the spatial details of the original image.

[0079] Furthermore, the infrared image enhancement method of the present invention can also be applied as a computer program product to a terminal device. The terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the infrared image enhancement method of the present invention. The terminal device here can be a computer, laptop, handheld computer, or various cloud servers, etc., and the processor can be a general-purpose processor, digital signal processor, application-specific integrated circuit, or other programmable logic device.

[0080] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An infrared image enhancement method based on multi-scale information exchange, characterized in that, Includes the following steps: S1, by performing convolutional learning on the original image, a coarse feature map F0 of the original image is obtained; S2, perform upsampling and downsampling on the coarse extracted feature map F0 respectively to obtain the upsampled coarse extracted feature map F1 and the downsampled coarse extracted feature map F2; S3, sequentially through the bottleneck attention module and the random selection and fusion module, performs one attention feature extraction and multi-scale feature exchange on F1, F0 and F2, then through the bottleneck attention module, performs a second attention feature extraction, and then through the feature fusion module, performs multi-scale feature fusion to obtain a multi-scale feature fusion map; The specific steps of the first attention feature extraction are as follows: Sa.1, calculate the channel feature attention map and spatial feature attention map of F1, F0 and F2 respectively, and obtain the spatial-channel feature attention maps M(F1), M(F0) and M(F2) of F1, F0 and F2 respectively; Sa.2, M(F1), M(F0) and M(F2) are multiplied pixel by pixel with the corresponding F1, F0 and F2 respectively, and the multiplication results are superimposed with F1, F0 and F2 to obtain the corresponding first-order output feature maps L1, L0 and L2; The multi-scale feature exchange specifically refers to: Sb.1, perform downsampling and upsampling on L1 and L2 respectively to restore L1 and L2 to the same scale as L0. Then add L0, L1 and L2 restored to the same scale as L0 to obtain the feature map L. Sb.2, global average pooling is performed on feature map L to encode the information of the same channel into a real number; Sb.3 uses a convolutional layer as a channel-reduced convolutional layer to perform convolutional learning and generate a channel feature; Sb.4, For the channel features obtained in step Sb.3, use three parallel convolutional layers of the same dimension as channel expansion convolutions to generate three channel feature maps V1, V0 and V2 respectively; Sb.5, multiply the three channel feature maps V1, V0 and V2 with L0, and L1 and L2 restored to the same scale as L0 in the feature dimension, respectively, to obtain feature maps L'1, L'0 and L'2 after multi-scale feature information exchange; Sb.6, randomly selects channels from V1, V0 and V2 respectively, and obtains constants s1, s2 and s3 after softmax activation; Sb.7, combining feature maps L'1, L'0 and L'2, and constants s1, s2 and s3, yields the multi-scale feature exchange result U0; Sb.8, then repeat steps Sb.6 and Sb.7 twice more to obtain two different multi-scale feature exchange results U1 and U2; S4. Repeat steps S2 and S3 multiple times, and each time use the multi-scale feature fusion map obtained in the previous step S3 as the coarse extracted feature map F0 in step S2 to obtain a multi-level feature extraction map. S5, input the multi-level feature extraction map into the convolutional layer to obtain the preliminary reconstructed image; S6, the original image is downsampled and upsampled sequentially, and then connected with the preliminarily reconstructed image to obtain the final enhanced image.

2. The infrared image enhancement method based on multi-scale information exchange according to claim 1, characterized in that, In step S3, the secondary attention feature extraction is performed using the same method as the primary attention feature extraction, resulting in secondary output feature maps L21, L20, and L22.

3. The infrared image enhancement method based on multi-scale information exchange according to claim 2, characterized in that, In step S3, the multi-scale feature fusion specifically involves: Sc.1, upsample or downsample L21 and L22 respectively to make L21, L20 and L22 have the same scale; Sc.2, add L20, L21 and L22 after upsampling or downsampling, to obtain the resulting feature map M; In Sc.3, the resulting feature map M is added to the coarse extracted feature map F0 using residual connections to obtain a multi-scale feature fusion map.

4. The infrared image enhancement method based on multi-scale information exchange according to any one of claims 1 to 3, characterized in that: In step S3, the loss functions in the bottleneck attention module, the random selection and fusion module, and the feature fusion module are: in, Represents the loss function. Indicates the output image, I * Let represent the input image, and ε represent a constant.

5. The infrared image enhancement method based on multi-scale information exchange according to claim 4, characterized in that, In step Sa.2, the channel feature attention map is obtained in the following way: Global average pooling is used to encode the global information in each channel into a real number. Then, a hidden layer consisting of two fully connected layers is used to reduce and restore the feature channels. Finally, a batch normalization layer is used to normalize the data, resulting in a channel feature attention map.

6. The infrared image enhancement method based on multi-scale information exchange according to claim 5, characterized in that, In step Sa.2, the spatial feature attention map is obtained in the following way: First, a 1x1 convolution is used to map the coarsely extracted feature map into a low-dimensional space R. C / r×H×W After two more 3x3 dilated convolutions, a 1x1 convolution is used to map the result to space R. 1×H×W Finally, a normalization layer is used to normalize the data to obtain a spatial feature attention map; Where C represents the number of channels in the coarse feature map, r represents the channel mapping ratio from the coarse feature map to the lower-order space, H represents the height of the coarse feature map, and W represents the width of the coarse feature map.

7. The infrared image enhancement method based on multi-scale information exchange according to claim 6, characterized in that: In step S4, the repeated execution of steps S2 and S3 specifically means repeating the execution three times.

8. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps of the infrared image enhancement method based on multi-scale information exchange as described in any one of claims 1 to 7.

Images