Infrared dual-band image fusion method based on semantic loss function driving

Abstract

The invention discloses an infrared dual-band image fusion method based on semantic loss function driving, which belongs to the technical field of image fusion in digital image processing, and comprises the following steps of: guiding a fusion network by utilizing a designed loss function on the basis of deeply researching the existing image fusion method based on deep learning; weak and small target information in the fused image is effectively enhanced, background clutter interference is suppressed, and finally the fusion purpose is achieved. Through the fusion network trained by the joint loss function, background clutters are suppressed while target information is highlighted, and the promotion effect of image fusion on downstream tasks is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image fusion in digital image processing, and in particular relates to an infrared dual-band image fusion method driven by a semantic loss function. Background Art

[0002] With the advancement of infrared stealth and jamming technology, the limitations of single-band infrared thermal imaging systems have gradually emerged. In many application scenarios, there are defects such as poor target detection and recognition capabilities and high false alarm rates. To solve this type of problem, by studying multimodal image fusion detection technology, camouflaged and stealth targets can be better identified. For infrared dual-band image fusion tasks, the grayscale value and signal-to-clutter ratio of the target in the medium-wave infrared image are relatively high, but there are complex interferences such as bright clouds and large areas of sea bright bands, resulting in a large number of suspected targets. The background clutter interference of long-wave infrared images is weak, and the impact on tasks such as target detection and recognition is weak, but the grayscale value and signal-to-clutter ratio of the target are low, and it is not easy to distinguish from the background. Therefore, the dual-band image fusion technology can be used to effectively integrate the inherent differences and complementary information of the target and the background in infrared images of different bands, thereby effectively improving the performance of downstream tasks (such as target detection, target recognition and anti-interference, etc.).

[0003] Traditional image fusion methods fuse images through multi-scale transformation, sparse representation, etc., but they do not fully consider the differences between different modal features because they need to use artificially designed features and fusion strategies. In recent years, image fusion methods based on deep learning have been widely used and developed in the field of image fusion due to their powerful feature extraction capabilities. For different downstream tasks, image fusion methods based on deep learning can guide the network to learn the best feature fusion strategy through loss function design to achieve adaptive feature fusion. However, the current deep learning fusion methods still have problems that need to be improved in the infrared dual-band image fusion task: the current mainstream deep learning fusion methods only focus on the improvement of the fused image in the evaluation index, ignoring the role of image fusion in promoting downstream tasks. The commonly used loss functions in the model, such as intensity loss and structural similarity loss, often focus on the integration of global image information, and do not fully consider the connection between the target and background characteristics in infrared images. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and to provide an infrared dual-band image fusion method driven by a semantic loss function, so as to solve the problem that the deep fusion method in the prior art does not consider the promoting effect of image fusion on downstream tasks and does not fully consider the connection between the target and background characteristics in the infrared image.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: An infrared dual-band image fusion method driven by a semantic loss function includes the following steps: S1, obtaining an infrared dual-band image to be fused, wherein the infrared dual-band image to be fused includes a medium-wave image to be fused and a long-wave image to be fused; S2, fusing the infrared dual-band images to be fused through a fusion network, wherein the fusion network is obtained through training of a data set, and the loss function of the training process is a joint loss function, which includes pixel intensity loss, wavelet smoothing mask loss, and background residual loss; The acquisition process of the wavelet smoothing mask loss is as follows: obtaining a long-wave background image through a long-wave image in a data set, and obtaining a medium-wave target image through a medium-wave image in a data set; performing wavelet transform on the long-wave background image to obtain a long-wave background image after background suppression, adding the long-wave background image after background suppression and the medium-wave target image pixel by pixel to obtain a desired image, and obtaining a wavelet smoothing mask loss based on the desired image and the fused image; S3, output fused image.

[0006] A further improvement of the present invention is: Preferably, in S2, in the process of acquiring the wavelet smoothing mask loss, the medium wave image and the long wave image in the data set are first subjected to binarization processing to obtain a binarized target mask image; the medium wave image and the long wave image in the data set are subjected to debinarization processing to obtain a debinarized background mask image; The long-wave background image is obtained through the long-wave image and the background mask image, and the medium-wave target image is obtained through the medium-wave image and the target mask image.

[0007] Preferably, in S2, the process of obtaining the long-wave background image through the long-wave image and the background mask image is to multiply the long-wave image and the background mask image pixel by pixel to obtain an image containing only the background as the long-wave background image.

[0008] Preferably, in S2, the process of obtaining the medium wave target image through the medium wave image and the target mask image is to multiply the medium wave image and the target mask image pixel by pixel to obtain an image containing only the target as the medium wave target image.

[0009] Preferably, the process of performing wavelet transform on the long-wave background image to obtain the long-wave background image after background suppression is: performing Haar wavelet transform on the long-wave background image to obtain each sub-band of the long-wave background image. , , , , and then through threshold processing, retain the low-frequency components and remove the high-frequency components. Finally, the long-wave background image after background suppression is obtained through inverse Haar wavelet transform.

[0010] Preferably, the wavelet smoothing mask loss in S2 is the mean absolute error calculated between the desired image and the fused image.

[0011] Preferably, the calculation process of the background residual loss is: obtaining a binary segmentation result map of the fused image through a detection algorithm, calculating the background residual in the binary segmentation result map, and taking the mean absolute error of the background residual as the background residual loss.

[0012] Preferably, the background residual degree is the ratio of the number of residual non-zero pixels in the binary segmentation result image to the number of pixels in the entire image.

[0013] Preferably, the detection algorithm is ADMD.

[0014] Preferably, the calculation formula of the joint loss function is: (8) in, , , is the weight coefficient, obtained through experiments, where is the pixel intensity loss, is the background residual loss, is the wavelet smoothing mask loss.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The invention discloses an infrared dual-band image fusion method driven by a semantic loss function. The core of the method is to design a semantic-driven joint loss function, which aims to guide the image fusion network to retain the high signal-to-clutter ratio target information in the medium-wave image and effectively integrate the weak background clutter information in the long-wave image during the fusion process. The specific implementation method is as follows: First, a classic pixel intensity loss function is introduced as part of the image content loss. This function focuses on maintaining the gray value information of the target in the medium-wave image to ensure the accurate transmission of the target information during the fusion process. Secondly, in order to more effectively suppress the background clutter in the long-wave image, a wavelet smoothing mask loss is proposed. The loss function uses wavelet transform technology to finely process the long-wave image, and then accurately superimposes it with the medium-wave target image pixel by pixel. In this way, a desired image that contains both a high signal-to-clutter ratio target and weak background clutter is obtained. Subsequently, the mean absolute difference loss between the fused image and the desired image is calculated and used as another important image content loss function. In addition, in order to further improve the quality of the fused image, a background residual loss function is also introduced. This function uses the advanced ADMD detection method to detect targets in the fused image, and counts the ratio of the number of residual non-zero pixels in the detection result to the number of pixels in the entire image, that is, the background residual degree. By calculating the mean absolute error of the background residual degree of the fused image, a loss function that reflects the semantic information of the image is constructed, thereby effectively reducing the background clutter interference in the fused image. Finally, the above three loss functions are weighted and combined to form a comprehensive joint loss function. This function not only considers the retention and fusion effect of the image content, but also fully considers the accuracy of the image semantic information and the suppression effect of background clutter. Based on the in-depth study of the existing image fusion methods based on deep learning, this method uses the designed loss function to guide the fusion network, effectively enhances the weak target information in the fused image, and suppresses the background clutter interference, and finally achieves the fusion purpose. Through the fusion network trained by the joint loss function, while highlighting the target information, suppressing the background clutter, and enhancing the promotion of image fusion to downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The calculation principle diagram of the expected image in wavelet smoothing mask loss; Figure 2 This is an example diagram of a small target detection method based on ADMD; Figure 3 It is a semantic-driven infrared dual-band image fusion network framework; Figure 4 This is the infrared dual-band image fusion result. DETAILED DESCRIPTION

[0017] In the following, the terms "first", "second", "third", and "fourth" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first", "second", "third", and "fourth" may explicitly or implicitly include one or more of the features.

[0018] The co-shooting method provided in the embodiment of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiment of the present application does not impose any restrictions on the specific type of the terminal device.

[0019] It should be noted that the terms "first", "second", etc. in the specification and drawings of the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] The present invention discloses an infrared dual-band image fusion method driven by a semantic loss function, and the fusion method comprises the following steps: S1, acquiring an infrared dual-band image to be fused, wherein the infrared dual-band image to be fused includes a medium-wave image to be fused and a long-wave image to be fused; S2, fusing the infrared dual-band images to be fused through a fusion network to obtain a fused image; S3, output fused image.

[0021] The fusion networks are common fusion networks, such as a fusion network based on wavelet transform, a fusion network based on non-subsampled contourlet transform (NSCT), and a fusion network based on deep learning.

[0022] The fusion network is trained with the dataset before use, and a joint loss function is used in the training process. The joint loss function includes pixel intensity loss, wavelet smoothing mask loss, and background residual loss; Pixel intensity loss is a pixel-level loss function that calculates the difference between the intensity value of each pixel in the generated image or processed image and the intensity value of the corresponding pixel in the original image or reference image. This difference is usually calculated by some distance metric, such as mean square error (MSE) or mean absolute error. In the present invention, pixel intensity loss can be obtained by common methods.

[0023] The process of obtaining the wavelet smoothing mask loss is as follows: obtain the long-wave background image through the long-wave image in the data set, and obtain the medium-wave target image through the medium-wave image in the data set; perform wavelet transform on the long-wave background image to obtain the long-wave background image after background suppression, add the long-wave background image after background suppression and the medium-wave target image pixel by pixel to obtain the expected image, and obtain the wavelet smoothing mask loss based on the expected image and the fused image.

[0024] The process of obtaining the background residual loss is as follows: the process of calculating the background residual loss is as follows: obtaining the binary segmentation result map of the fused image through the detection algorithm, calculating the background residual in the binary segmentation result map, and taking the average absolute error of the background residual as the background residual loss.

[0025] The method for obtaining the joint loss function includes the following steps: Step 1: Obtain infrared dual-band image data and construct a fused image dataset ,in represents a medium wave image, Represents a long-wave image.

[0026] Step 2, binarize the medium wave image or the long wave image to obtain a binarized target mask image for locating the target, and obtain a background mask image after debinarization for locating the background; select the target mask image obtained by the medium wave image or the long wave image, and the background mask image obtained by the medium wave image or the long wave image to form the following data; ,in represents the target mask image, Represents the background mask image.

[0027] Step 3, Medium Wave Image With the target mask image Perform pixel-by-pixel multiplication to obtain an image containing only medium-wave targets , is the medium wave target image, long wave image With background mask image Perform pixel-by-pixel multiplication to obtain an image containing only the long-wave background , is the long-wave background image. Its mathematical expression is as follows: (1) (2) Step 4: Perform wavelet smoothing on the long-wave background image to obtain the long-wave background image after background suppression. The process is as follows: Perform Haar wavelet transform on the long-wave background image to obtain the sub-bands of the long-wave background image. , , , , respectively containing the low-frequency information of the image and the detailed information of the high-frequency part of the original image in the horizontal, vertical and diagonal directions. The Haar wavelet transform formula is as follows: (3) Then, through threshold processing, the low-frequency components are retained and the high-frequency components are removed to reduce the background clutter in the image. Finally, the long-wave background image after background suppression is obtained through inverse Haar wavelet transform.

[0028] Step 5, perform pixel-by-pixel addition operation on the medium-wave target image and the long-wave background image after background suppression to obtain the expected image. The calculation principle of the expected image is as follows: Figure 1 The mean absolute error (MAE), i.e., L1 loss, is calculated between the expected image and the fused image and is defined as the wavelet smoothing mask loss: (4) (5) in, To fuse the images, For the desired image, represents the wavelet smoothing operation, represents L1 loss.

[0029] The fused image is a medium wave image. and long wave images The fused image obtained by any method.

[0030] Step 6: Use the traditional infrared small target detection algorithm (Absolute Directional MeanDifference, ADMD) to perform preliminary detection on the fused image and obtain the binary segmentation result image. Figure 2 As shown, it contains the target and background clutter with similar characteristics to the target.

[0031] For ease of understanding, the present invention marks the real target and the false target, using a green frame to represent the real target and a red frame to represent the false target.

[0032] Step 7: The ratio of the number of residual non-zero pixels (0 for background, 1 for target) in the binary segmentation result image to the number of pixels in the entire image is defined as the background residual degree, which is mathematically expressed as follows: (6) In the formula, are the height and width of the source image, respectively. Indicates that the ADMD algorithm is used to detect the input image. Indicates counting the non-zero pixels in the detection results.

[0033] The mean absolute error, i.e., L1 loss, is calculated for the background residual of the fused image, which is defined as the background residual loss and is mathematically expressed as follows: (7) In the formula, To fuse the images, Refers to calculating the background residual of the image.

[0034] Step 8: Weight the classic pixel intensity loss, wavelet smoothing mask loss, and background residual loss to obtain a joint loss function: (8) in, , , is the weight coefficient obtained through experiments.

[0035] Step 9: Train the fused image dataset through the image fusion network. In this process, the joint loss function is calculated based on the target, background mask image data and fused image to guide the network to highlight the target and suppress the background. The overall architecture of the semantic-driven infrared dual-band fusion network is as follows: Figure 3 As shown in Figure 3, the content loss function includes pixel intensity loss and wavelet smoothing mask loss, and the semantic loss function is background residual loss.

[0036] The present invention mainly designs a semantically driven joint loss function to guide the image fusion network to simultaneously retain the high signal-to-clutter ratio target information in the medium wave image and the weak background clutter information in the long wave image. The classic pixel intensity loss is used as one of the image content loss functions to retain the gray value information of the target in the medium wave image; a wavelet smoothing mask loss is proposed, and the wavelet transform is used to suppress the background clutter in the long wave image. The pixel-by-pixel addition operation is performed with the medium wave target image to obtain the expected image containing the high signal-to-clutter ratio target and the weak background clutter. The mean absolute difference loss is calculated for the fused image and the expected image as one of the image content loss functions; a background residual loss is proposed, and the ADMD detection method is used to obtain the target detection result of the fused image. The ratio of the number of residual non-zero pixels in the result image to the number of pixels in the entire image is defined as the background residual degree. The mean absolute error of the background residual degree of the fused image is calculated as the image semantic loss function to reduce the background clutter in the fused image; the above three loss functions are weighted to obtain a joint loss function. This method effectively enhances the weak target information in the fused image, while suppressing the interference of background clutter, and realizes image fusion.

[0037] The specific implementation method of the present invention is further described in conjunction with specific embodiments: Example In this embodiment, the collected medium-wave image and long-wave image are placed in a fusion network, and the fusion network fuses the medium-wave image and the long-wave image to obtain a fusion network.

[0038] The experimental results are as follows Figure 4 As shown, there are three groups of infrared dual-band images under complex backgrounds. The first group is a bright cloud background, the second group is a sea and sky background, and the third group is a sea, mountain and sky background. The red boxed areas in the figure are the locations of weak targets. The fusion results represent the image fusion results based on the above image fusion mode, which can effectively highlight the weak target information.

[0039] This paper proposes an innovative infrared dual-band image fusion technology. The core lies in constructing a semantically driven comprehensive loss function to optimize the image fusion process, ensuring that the high signal-to-noise ratio target information in the medium-wave image can be accurately retained and the weak background information in the long-wave image can be effectively fused. First, the traditional pixel intensity loss function is considered as the basic component of image content loss, focusing on maintaining the grayscale value accuracy of the target in the medium-wave image to ensure the accurate transmission of the target information during the fusion process. Secondly, in order to more effectively filter out the background noise in the long-wave image, a wavelet smoothing mask loss function is calculated. This function uses wavelet transform technology to carefully process the long-wave image and accurately fuses it with the medium-wave target image at the pixel level, thereby generating a desired image that contains both a clear target and a weak background. Then, the mean absolute difference loss function is introduced to further optimize the image content by comparing the difference between the fused image and the desired image. In addition, in order to further improve the quality of the fused image, a background residual loss function is also introduced. This function uses advanced ADMD detection technology to detect targets on the fused image and calculates the proportion of residual non-zero pixels in the detection result, that is, the background residual. By minimizing the mean absolute error of the background residual of the fused image, a loss function that can reflect the semantic information of the image is constructed, which effectively reduces the interference of background noise. The above three loss functions are weighted and integrated to form a joint loss function. This function not only focuses on the retention and fusion effect of the image content, but also fully considers the accuracy of the image semantic information and the suppression effect of background noise. By deeply studying and improving the image fusion method based on deep learning, the designed joint loss function is used to guide the fusion network, which significantly enhances the weak target information in the fused image, effectively suppresses the background noise, and achieves high-quality image fusion. The fusion network trained by the joint loss function effectively suppresses the background noise while highlighting the target information, which significantly improves the role of image fusion in promoting subsequent tasks.

[0040] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for infrared dual-band image fusion driven by semantic loss function, characterized in that: The following steps are involved: S1, obtaining an infrared dual-band image to be fused, wherein the infrared dual-band image to be fused includes a medium-wave image to be fused and a long-wave image to be fused; S2, fusing the infrared dual-band images to be fused through a fusion network, wherein the fusion network is obtained through training of a data set, and the loss function of the training process is a joint loss function, which includes pixel intensity loss, wavelet smoothing mask loss, and background residual loss; The acquisition process of the wavelet smoothing mask loss is as follows: obtaining a long-wave background image through a long-wave image in a data set, and obtaining a medium-wave target image through a medium-wave image in a data set; performing wavelet transform on the long-wave background image to obtain a long-wave background image after background suppression, adding the long-wave background image after background suppression and the medium-wave target image pixel by pixel to obtain a desired image, and obtaining a wavelet smoothing mask loss based on the desired image and the fused image; S3, output fused image.

2. According to claim 1, the infrared dual-band image fusion method driven by semantic loss function is characterized in that: In S2, in the process of obtaining the wavelet smoothing mask loss, the medium wave image and the long wave image in the data set are firstly binarized to obtain a binarized target mask image; The medium wave image and the long wave image in the data set are subjected to debinarization processing to obtain a debinarized background mask image; The long-wave background image is obtained through the long-wave image and the background mask image, and the medium-wave target image is obtained through the medium-wave image and the target mask image.

3. The infrared dual-band image fusion method based on semantic loss function driving according to claim 2 is characterized in that: In S2, the process of obtaining the long-wave background image through the long-wave image and the background mask image is to multiply the long-wave image and the background mask image pixel by pixel to obtain an image containing only the background as the long-wave background image.

4. The infrared dual-band image fusion method based on semantic loss function driving according to claim 2 is characterized in that: In S2, the process of obtaining the medium wave target image through the medium wave image and the target mask image is to multiply the medium wave image and the target mask image pixel by pixel to obtain an image containing only the target as the medium wave target image.

5. The infrared dual-band image fusion method based on semantic loss function drive according to claim 1 is characterized in that: The process of performing wavelet transform on the long-wave background image to obtain the long-wave background image after background suppression is as follows: performing Haar wavelet transform on the long-wave background image to obtain each sub-band of the long-wave background image. , , , , and then through threshold processing, retain the low-frequency components and remove the high-frequency components. Finally, the long-wave background image after background suppression is obtained through inverse Haar wavelet transform.

6. The infrared dual-band image fusion method based on semantic loss function drive according to claim 1 is characterized in that: The wavelet smoothing mask loss described in S2 is the mean absolute error calculated between the desired image and the fused image.

7. The infrared dual-band image fusion method based on semantic loss function drive according to claim 1, characterized in that: The calculation process of the background residual loss is: obtaining a binary segmentation result map of the fused image through a detection algorithm, calculating the background residual in the binary segmentation result map, and taking the mean absolute error of the background residual as the background residual loss.

8. The infrared dual-band image fusion method based on semantic loss function drive according to claim 7 is characterized in that: The background residual degree is the ratio of the number of residual non-zero pixels in the binary segmentation result image to the number of pixels in the entire image.

9. The infrared dual-band image fusion method based on semantic loss function drive according to claim 7, characterized in that: The detection algorithm is ADMD.

10. The infrared dual-band image fusion method based on semantic loss function drive according to any one of claims 1 to 9, characterized in that: The calculation formula of the joint loss function is: （8） in, , , is the weight coefficient, obtained through experiments, where is the pixel intensity loss, is the background residual loss, is the wavelet smoothing mask loss.

Images