A semantic loss function-driven infrared dual-band image fusion method

Abstract

This paper discloses a semantic loss function-driven infrared dual-band image fusion method, belonging to the field of image fusion technology in digital image processing. Based on in-depth research on existing deep learning-based image fusion methods, this method uses a designed loss function to guide the fusion network, effectively enhancing the weak target information in the fused image while suppressing background clutter interference, ultimately achieving the fusion purpose. By combining the fusion network trained with the loss function, the target information is highlighted while background clutter is suppressed, thus enhancing the promotion effect of image fusion on downstream tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of image fusion in digital image processing, and in particular relates to an infrared dual-band image fusion method driven by a semantic loss function. Background Art

[0002] With advances in infrared stealth and jamming technologies, the limitations of single-band infrared thermal imaging systems have become increasingly apparent. These systems suffer from poor target detection and recognition capabilities and high false alarm rates in many application scenarios. To address these issues, research into multimodal image fusion detection technology is underway to better identify camouflaged and stealthed targets. For dual-band infrared image fusion tasks, targets in medium-wave infrared images have high grayscale values ​​and signal-to-clutter ratios, but complex interference such as bright clouds and large sea-bright bands can lead to a large number of suspected targets. Long-wave infrared images have less background clutter, thus minimizing the impact on target detection and recognition tasks. However, targets have low grayscale values ​​and signal-to-clutter ratios, making them difficult to distinguish from the background. Therefore, dual-band image fusion can effectively integrate the inherent differences and complementary information between the target and background in infrared images of different bands, thereby significantly improving the performance of downstream tasks such as target detection, recognition, and anti-jamming.

[0003] Traditional image fusion methods fuse images through multi-scale transformations and sparse representations. However, due to the need to use manually designed features and fusion strategies, they do not fully consider the differences between features of different modalities. In recent years, deep learning-based image fusion methods have been widely used and developed in the field of image fusion due to their powerful feature extraction capabilities. For different downstream tasks, deep learning-based image fusion methods can guide the network to learn the optimal feature fusion strategy through loss function design, achieving adaptive feature fusion. However, current deep learning fusion methods still have problems that need to be improved in the task of infrared dual-band image fusion: the current mainstream deep learning fusion methods only focus on improving the evaluation indicators of the fused image, ignoring the role of image fusion in promoting downstream tasks. Commonly used loss functions in the model, such as intensity loss and structural similarity loss, often focus on integrating global image information and do not fully consider the connection between target and background characteristics in infrared images. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide an infrared dual-band image fusion method driven by a semantic loss function to solve the problems that the deep fusion method in the prior art does not consider the promoting effect of image fusion on downstream tasks and does not fully consider the relationship between the target and background characteristics in the infrared image.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] An infrared dual-band image fusion method driven by a semantic loss function includes the following steps:

[0007] S1, obtaining an infrared dual-band image to be fused, wherein the infrared dual-band image to be fused includes a medium-wave image to be fused and a long-wave image to be fused;

[0008] S2, fusing the infrared dual-band images to be fused through a fusion network, wherein the fusion network is obtained through training of the data set, and the loss function of the training process is a joint loss function, which includes pixel intensity loss, wavelet smoothing mask loss, and background residual loss;

[0009] The wavelet smoothing mask loss is obtained by: obtaining a long-wave background image through a long-wave image in a data set, and obtaining a medium-wave target image through a medium-wave image in a data set; performing a wavelet transform on the long-wave background image to obtain a long-wave background image after background suppression; adding the long-wave background image after background suppression and the medium-wave target image pixel by pixel to obtain a desired image; and obtaining a wavelet smoothing mask loss based on the desired image and the fused image;

[0010] S3, output fused image.

[0011] A further improvement of the present invention is:

[0012] Preferably, in S2, during the acquisition of the wavelet smoothing mask loss, the medium wave image and the long wave image in the data set are first subjected to binarization processing to obtain a binarized target mask image; and the medium wave image and the long wave image in the data set are debinarized to obtain a debinarized background mask image;

[0013] The long-wave background image is obtained by using the long-wave image and the background mask image, and the medium-wave target image is obtained by using the medium-wave image and the target mask image.

[0014] Preferably, in S2, the process of obtaining the long-wave background image through the long-wave image and the background mask image is to multiply the long-wave image and the background mask image pixel by pixel to obtain an image containing only the background as the long-wave background image.

[0015] Preferably, in S2, the process of obtaining the MW target image through the MW image and the target mask image is to multiply the MW image and the target mask image pixel by pixel to obtain an image containing only the target as the MW target image.

[0016] Preferably, the process of performing wavelet transform on the long-wave background image to obtain the long-wave background image after background suppression is as follows: performing Haar wavelet transform on the long-wave background image to obtain the sub-bands of the long-wave background image. 、 、 、 , then through threshold processing, retain the low-frequency components and remove the high-frequency components, and finally obtain the long-wave background image after background suppression through inverse Haar wavelet transform.

[0017] Preferably, the wavelet smoothing mask loss in S2 is the mean absolute error calculated between the desired image and the fused image.

[0018] Preferably, the calculation process of the background residual loss is: obtaining a binary segmentation result map of the fused image through a detection algorithm, calculating the background residual in the binary segmentation result map, and taking the mean absolute error of the background residual as the background residual loss.

[0019] Preferably, the background residual degree is the ratio of the number of residual non-zero pixels in the binary segmentation result image to the number of pixels in the entire image.

[0020] Preferably, the detection algorithm is ADMD.

[0021] Preferably, the calculation formula of the joint loss function is:

[0022] (8)

[0023] in, 、 、 is the weight coefficient, obtained through experiments, where is the pixel intensity loss, is the background residual loss, is the wavelet smoothing mask loss.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] This paper discloses a semantically driven dual-band infrared image fusion method. The core of this method lies in the design of a semantically driven joint loss function, which aims to guide the image fusion network in preserving high-SNR target information in the medium-wavelength image while effectively integrating weak background clutter information in the long-wavelength image. The specific implementation is as follows: First, a classic pixel intensity loss function is introduced as part of the image content loss. This function focuses on preserving the grayscale value information of the target in the medium-wavelength image, ensuring accurate transmission of target information during the fusion process. Second, to more effectively suppress background clutter in the long-wavelength image, a wavelet smoothing mask loss is proposed. This loss function utilizes wavelet transform technology to finely process the long-wavelength image and then accurately superimposes it with the medium-wavelength target image on a pixel-by-pixel basis. This method produces a desired image that contains both the high-SNR target and weak background clutter. Subsequently, the mean absolute difference loss between the fused image and the desired image is calculated and used as another important image content loss function. Furthermore, a background residual loss function is introduced to further improve the quality of the fused image. This function uses the advanced ADMD detection method to detect objects in the fused image and calculates the ratio of the number of residual non-zero pixels in the detection result to the total number of pixels in the entire image, i.e., the background residual degree. By calculating the mean absolute error of the background residual degree of the fused image, a loss function that reflects the image semantic information is constructed, effectively reducing background clutter in the fused image. Finally, a weighted combination of the three loss functions is formed to form a comprehensive joint loss function. This function not only considers the preservation of image content and the fusion effect, but also fully considers the accuracy of image semantic information and the suppression of background clutter. Based on in-depth research on existing deep learning-based image fusion methods, this method uses the designed loss function to guide the fusion network, effectively enhancing the weak target information in the fused image while suppressing background clutter, ultimately achieving the fusion goal. The fusion network trained with the joint loss function highlights the target information while suppressing background clutter, enhancing the contribution of image fusion to downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The calculation principle diagram of the expected image in wavelet smoothing mask loss;

[0027] Figure 2 This is an example diagram of the small target detection method based on ADMD;

[0028] Figure 3 It is a semantic-driven infrared dual-band image fusion network framework;

[0029] Figure 4 This is the infrared dual-band image fusion result. DETAILED DESCRIPTION

[0030] Hereinafter, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features indicated. Thus, a feature identified as "first," "second," "third," or "fourth" may explicitly or implicitly include one or more of such features.

[0031] The co-shooting method provided in the embodiments of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0032] It should be noted that the terms "first," "second," and the like in the description and drawings of the present invention are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0033] The present invention discloses an infrared dual-band image fusion method driven by a semantic loss function, which comprises the following steps:

[0034] S1, acquiring an infrared dual-band image to be fused, wherein the infrared dual-band image to be fused includes a medium-wave image to be fused and a long-wave image to be fused;

[0035] S2, fusing the infrared dual-band images to be fused through the fusion network to obtain a fused image;

[0036] S3, output fused image.

[0037] The fusion networks are common fusion networks, such as the fusion network based on wavelet transform, the fusion network based on non-subsampled contourlet transform (NSCT), and the fusion network based on deep learning.

[0038] The fusion network is trained on the dataset before use, and a joint loss function is used during the training process. The joint loss function includes pixel intensity loss, wavelet smoothing mask loss, and background residual loss.

[0039] Pixel intensity loss is a pixel-level loss function that calculates the difference between the intensity value of each pixel in the generated or processed image and the intensity value of the corresponding pixel in the original or reference image. This difference is typically calculated using a distance metric such as mean squared error (MSE) or mean absolute error. In the present invention, pixel intensity loss can be obtained using common methods.

[0040] The acquisition process of wavelet smoothing mask loss is as follows: a long-wave background image is obtained through the long-wave image in the data set, and a medium-wave target image is obtained through the medium-wave image in the data set; the long-wave background image is subjected to wavelet transform to obtain a long-wave background image after background suppression; the long-wave background image after background suppression and the medium-wave target image are added pixel by pixel to obtain the expected image; and the wavelet smoothing mask loss is obtained based on the expected image and the fused image.

[0041] The process of obtaining the background residual loss is as follows: the process of calculating the background residual loss is as follows: obtaining the binary segmentation result map of the fused image through the detection algorithm, calculating the background residual in the binary segmentation result map, and taking the average absolute error of the background residual as the background residual loss.

[0042] The method for obtaining the joint loss function includes the following steps:

[0043] Step 1: Obtain infrared dual-band image data and construct a fused image dataset ,in represents a medium wave image, Represents a long-wave image.

[0044] Step 2: Binarize the medium-wave image or the long-wave image to obtain a target mask image after binarization for locating the target, and obtain a background mask image after debinarization for locating the background; the target mask image obtained from the medium-wave image or the long-wave image and the background mask image obtained from the medium-wave image or the long-wave image are selected to form the following data; ,in represents the target mask image, Represents the background mask image.

[0045] Step 3, Medium Wave Image and target mask image Perform pixel-by-pixel multiplication to obtain an image containing only medium-wave targets , is the medium wave target image, long wave image With background mask image Perform pixel-by-pixel multiplication to obtain an image containing only the long-wave background , is the long-wave background image. Its mathematical representation is as follows:

[0046] (1)

[0047] (2)

[0048] Step 4: Perform wavelet smoothing on the long-wave background image to obtain the long-wave background image after background suppression. The process is as follows: Perform Haar wavelet transform on the long-wave background image to obtain the sub-bands of the long-wave background image. 、 、 、 , respectively containing the low-frequency information of the image and the detailed information of the high-frequency parts of the original image in the horizontal, vertical and diagonal directions. The Haar wavelet transform formula is as follows:

[0049] (3)

[0050] Then, through threshold processing, the low-frequency components are retained and the high-frequency components are removed to reduce the background clutter in the image. Finally, the long-wave background image after background suppression is obtained through inverse Haar wavelet transform.

[0051] Step 5: Perform pixel-by-pixel addition of the medium-wave target image and the long-wave background image after background suppression to obtain the expected image. The calculation principle of the expected image is as follows: Figure 1 The mean absolute error (MAE) between the expected image and the fused image is calculated, i.e., the L1 loss, which is defined as the wavelet smoothing mask loss:

[0052] (4)

[0053] (5)

[0054] in, To fuse the images, For the desired image, represents the wavelet smoothing operation, represents L1 loss.

[0055] The fused image is the medium wave image and long-wave images The fused image obtained by any method.

[0056] Step 6: Use the traditional infrared small target detection algorithm (Absolute Directional MeanDifference, ADMD) to perform preliminary detection on the fused image and obtain the binary segmentation result image. Figure 2 As shown, it contains the target and background clutter with similar characteristics to the target.

[0057] For ease of understanding, the present invention marks the real target and the false target, using a green frame to represent the real target and a red frame to represent the false target.

[0058] Step 7: The ratio of the number of residual non-zero pixels (0 for background, 1 for target) in the binary segmentation result image to the number of pixels in the entire image is defined as the background residual degree, which is mathematically expressed as follows:

[0059] (6)

[0060] Where, are the height and width of the source image, Indicates that the ADMD algorithm is used to detect the input image. Indicates counting the non-zero pixels in the detection results.

[0061] The mean absolute error, i.e., L1 loss, is calculated for the background residual of the fused image, which is defined as the background residual loss, and its mathematical expression is as follows:

[0062] (7)

[0063] Where, To fuse the images, Refers to calculating the background residual of an image.

[0064] Step 8: Weight the classic pixel intensity loss, wavelet smoothing mask loss, and background residual loss to obtain a joint loss function:

[0065] (8)

[0066] in, 、 、 is the weight coefficient, obtained through experiments.

[0067] Step 9: The image fusion network is used to train the fused image dataset. During this process, a joint loss function is calculated based on the target, background mask image data, and fused image to guide the network to highlight the target and suppress the background. The overall architecture of the semantic-driven infrared dual-band fusion network is as follows: Figure 3 As shown in Figure 3, the content loss function includes pixel intensity loss and wavelet smoothing mask loss, and the semantic loss function is background residual loss.

[0068] This paper primarily designs a semantically driven joint loss function to guide the image fusion network to simultaneously preserve high signal-to-clutter ratio target information in medium-wavelength images and weak background clutter information in long-wavelength images. A classic pixel intensity loss is used as one of the image content loss functions to preserve the grayscale value information of the target in the medium-wavelength image. A wavelet smoothing mask loss is proposed, which uses wavelet transforms to suppress background clutter in the long-wavelength image. This image is then added pixel by pixel to the medium-wavelength target image to obtain a desired image containing a high signal-to-clutter ratio target and weak background clutter. The mean absolute difference loss is calculated between the fused and desired images as one of the image content loss functions. A background residual loss is proposed, which uses the ADMD detection method to obtain the target detection result of the fused image. The ratio of the number of residual nonzero pixels in the result image to the total number of pixels in the entire image is defined as the background residual. The mean absolute error of the background residual in the fused image is calculated as the image semantic loss function to reduce background clutter in the fused image. The three loss functions are weighted to obtain a joint loss function. This method effectively enhances the weak target information in the fused image while suppressing background clutter interference, achieving image fusion.

[0069] The specific implementation method of the present invention is further described with reference to specific embodiments:

[0070] Example

[0071] In this embodiment, the collected medium-wave image and long-wave image are placed in a fusion network, and the fusion network fuses the medium-wave image and the long-wave image to obtain a fusion network.

[0072] The experimental results are as follows Figure 4 As shown, there are three groups of infrared dual-band images under complex backgrounds. The first group is a bright cloud background, the second group is a sea and sky background, and the third group is a sea, mountain and sky background. The red boxed area in the figure is the location of the weak target. The fusion result represents the image fusion result based on the above image fusion mode, which can effectively highlight the weak target information.

[0073] This paper proposes an innovative infrared dual-band image fusion technique. Its core lies in constructing a semantically driven comprehensive loss function to optimize the image fusion process, ensuring both accurate preservation of the high signal-to-noise ratio target information in the medium-wavelength image and effective integration of the faint background information in the long-wavelength image. First, a traditional pixel intensity loss function is considered as a fundamental component of image content loss. This function focuses on maintaining the accuracy of the grayscale values ​​of the target in the medium-wavelength image, ensuring the precise transmission of target information during the fusion process. Second, a wavelet smoothing mask loss function is designed to more effectively filter out background noise in the long-wavelength image. This function uses wavelet transform technology to meticulously process the long-wavelength image and accurately fuses it with the medium-wavelength target image at the pixel level, generating a desired image that contains both a clear target and a faint background. Next, a mean absolute difference loss function is introduced to further optimize image content by comparing the difference between the fused image and the desired image. Furthermore, a background residual loss function is introduced to further enhance the quality of the fused image. This function uses advanced ADMD detection technology to detect targets in the fused image and calculates the proportion of residual non-zero pixels in the detection result, i.e., the background residual. By minimizing the mean absolute error of the background residuals in the fused images, a loss function that reflects the semantic information of the images was constructed, effectively reducing the interference of background noise. The three loss functions mentioned above were weighted and integrated to form a joint loss function. This function not only focuses on the preservation of image content and the fusion effect, but also fully considers the accuracy of image semantic information and the suppression of background noise. By in-depth research and improvement of deep learning-based image fusion methods, the designed joint loss function is used to guide the fusion network, significantly enhancing the weak target information in the fused image while effectively suppressing background noise, achieving high-quality image fusion. The fusion network trained with the joint loss function effectively suppresses background noise while highlighting the target information, significantly improving the role of image fusion in promoting subsequent tasks.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A semantic loss function driven infrared dual-band image fusion method, characterized in that: The following steps are involved: S1, obtaining an infrared dual-band image to be fused, wherein the infrared dual-band image to be fused includes a medium-wave image to be fused and a long-wave image to be fused; S2, fusing the infrared dual-band images to be fused through a fusion network, wherein the fusion network is obtained through training of the data set, and the loss function of the training process is a joint loss function, which includes pixel intensity loss, wavelet smoothing mask loss, and background residual loss; The wavelet smoothing mask loss is obtained by: obtaining a long-wave background image through a long-wave image in a data set, and obtaining a medium-wave target image through a medium-wave image in a data set; performing a wavelet transform on the long-wave background image to obtain a long-wave background image after background suppression; adding the long-wave background image after background suppression and the medium-wave target image pixel by pixel to obtain a desired image; and obtaining a wavelet smoothing mask loss based on the desired image and the fused image; In the process of obtaining the wavelet smoothing mask loss, the medium wave image and the long wave image in the data set are first subjected to binarization processing to obtain a binarized target mask image; The medium-wave image and the long-wave image in the data set are subjected to debinarization processing to obtain a debinarized background mask image; A long-wave background image is obtained through a long-wave image and a background mask image, and a medium-wave target image is obtained through a medium-wave image and a target mask image; The process of obtaining the long-wave background image by using the long-wave image and the background mask image is as follows: multiplying the long-wave image and the background mask image pixel by pixel to obtain an image containing only the background as the long-wave background image; The process of obtaining the medium wave target image through the medium wave image and the target mask image is to multiply the medium wave image and the target mask image pixel by pixel to obtain an image containing only the target as the medium wave target image; The calculation process of the background residual loss is as follows: obtaining a binary segmentation result map of the fused image through a detection algorithm, calculating the background residual in the binary segmentation result map, and taking the mean absolute error of the background residual as the background residual loss; The background residual degree is the ratio of the number of residual non-zero pixels in the binary segmentation result image to the number of pixels in the entire image; S3, output fused image.

2. The infrared dual-band image fusion method based on semantic loss function drive according to claim 1 is characterized in that: The process of performing wavelet transform on the long-wave background image to obtain the long-wave background image after background suppression is as follows: performing Haar wavelet transform on the long-wave background image to obtain the sub-bands of the long-wave background image. 、 、 、 , then through threshold processing, retain the low-frequency components and remove the high-frequency components, and finally obtain the long-wave background image after background suppression through inverse Haar wavelet transform.

3. The infrared dual-band image fusion method based on semantic loss function driving according to claim 1 is characterized in that: The wavelet smoothing mask loss in S2 is the mean absolute error calculated between the desired image and the fused image.

4. The infrared dual-band image fusion method based on semantic loss function driving according to claim 1 is characterized in that: The detection algorithm is ADMD.

5. The infrared dual-band image fusion method based on semantic loss function drive according to any one of claims 1 to 4, characterized in that: The calculation formula of the joint loss function is: （8） in, 、 、 is the weight coefficient, obtained through experiments, where is the pixel intensity loss, is the background residual loss, is the wavelet smoothing mask loss.

Images