A pixel-level real-time multispectral image fusion method based on adaptive weights and target perception

Abstract

This invention discloses a pixel-level real-time multispectral image fusion method based on adaptive weights and target perception. The method includes the following steps: pixel-level real-time multispectral image fusion: stitching an infrared image and a visible light image on channels to obtain a stitched image; generating adaptive pixel-level weights on the stitched image to obtain a fusion weight map of the infrared image and a fusion weight map of the visible light image; performing element-wise multiplication of the fusion weight map of the infrared image and the infrared image to obtain a first image; performing element-wise multiplication of the fusion weight map of the visible light image and the visible light image to obtain a second image; performing element-wise addition of the first image and the second image to obtain a third image; and normalizing the third image to obtain a fused image. This invention can significantly improve the processing speed of image fusion tasks, enabling real-time deployment on resource-constrained mobile platforms.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a pixel-level real-time multispectral image fusion method based on adaptive weights and target perception. Background Technology

[0002] Multispectral image fusion is a fundamental and crucial technology that has been widely applied in fields such as autonomous driving, remote sensing, and optoelectronic pods. Effective fusion methods aim to fully utilize the spatiotemporal correlation and complementary information of two or more sensors to generate information-rich fused images, thereby benefiting human visual perception or subsequent computer vision tasks. In recent years, image fusion network models based on deep learning technology have been continuously innovated, significantly improving the processing accuracy of their algorithms. However, current high-precision image fusion networks typically require substantial memory and computational resources, making it difficult to deploy existing methods in real-time on resource-constrained mobile platforms.

[0003] Image fusion can generally be divided into three types: pixel-based fusion, feature-based fusion, and decision-based fusion. Traditional infrared and visible light image fusion methods mostly employ pixel-based fusion, with key processing stages including image transformation, transform coefficient fusion, and inverse transform. To achieve higher image fusion quality, these methods design complex models and fusion rules, leading to a significant decrease in computational efficiency. Due to the powerful feature extraction and reconstruction capabilities of deep neural networks, learning-based infrared and visible light image fusion networks tend to use feature-based fusion rather than pixel-based fusion. Compared to traditional methods, learning-based methods significantly improve image fusion quality. However, state-of-the-art networks typically focus on designing sophisticated network models to continuously improve image fusion quality, requiring substantial memory and computational resources, making them difficult to deploy on resource-constrained platforms. Although some lightweight methods attempt to improve efficiency by reducing network size, they still employ feature-based fusion, and their image fusion quality and processing speed remain unsatisfactory. Summary of the Invention

[0004] To address the problems existing in the prior art, the purpose of this invention is to propose a pixel-level real-time multispectral image fusion method based on adaptive weights and target perception. This invention can achieve lightweight design while ensuring fusion quality.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] A pixel-level real-time multispectral image fusion method based on adaptive weights and target perception includes the following process:

[0007] Pixel-level real-time multispectral image fusion: Infrared and visible light images are stitched together on a channel to obtain a stitched image; adaptive pixel-level weighting is performed on the stitched image to obtain a fusion weight map of the infrared image and a fusion weight map of the visible light image; the fusion weight map of the infrared image and the infrared image are multiplied element-wise to obtain a first image; the fusion weight map of the visible light image and the visible light image are multiplied element-wise to obtain a second image; the first and second images are added element-wise to obtain a third image; the third image is normalized to obtain a fused image.

[0008] Preferably, the visible light image is a grayscale image or a color image;

[0009] When the visible light image is a grayscale image, the infrared image and the visible light image are directly stitched together on the channel;

[0010] When the visible light image is a color image, the visible light image is converted to the YCbCr color space, and then the Y channel is used to stitch it with the infrared image;

[0011] After obtaining the fused image, the Y channel of the visible light image is replaced with the fused image, and then the color space is converted back from YCbCr to RGB to obtain a colored fused image.

[0012] Preferably, when performing adaptive pixel-level weight generation on the stitched image to obtain the fusion weight map of the infrared image and the fusion weight map of the visible light image, four 3×3 convolutional layers are used to learn the fusion weights, and a sigmoid layer is used to fix the range of the fusion weights between 0 and 1.

[0013] The output channels of the four convolutional layers are 32, 64, 32, and 2, respectively, and no downsampling operation is performed; all are performed at the original resolution.

[0014] Preferably, the third image is normalized using the max-min normalization method to obtain the fused image.

[0015] The present invention also provides a system for implementing the above-described pixel-level real-time multispectral image fusion method based on adaptive weights and target perception, comprising:

[0016] A pixel-level real-time multispectral image fusion network is used to stitch infrared and visible light images together on a channel to obtain a stitched image; adaptive pixel-level weight generation is performed on the stitched image to obtain a fusion weight map of the infrared image and a fusion weight map of the visible light image; element-wise multiplication is performed between the fusion weight map of the infrared image and the infrared image to obtain a first image; element-wise multiplication is performed between the fusion weight map of the visible light image and the visible light image to obtain a second image; element-wise addition is performed between the first image and the second image to obtain a third image; and the third image is normalized to obtain a fused image.

[0017] Preferably, the visible light image is a grayscale image or a color image;

[0018] When the visible light image is a grayscale image, the pixel-level real-time multispectral image fusion network directly stitches the infrared image and the visible light image on the channel;

[0019] When the visible light image is a color image, the pixel-level real-time multispectral image fusion network converts the visible light image into the YCbCr color space, then uses the Y channel to stitch with the infrared image to obtain a fused image, and then replaces the Y channel of the visible light image with the fused image; then the pixel-level real-time multispectral image fusion network converts from YCbCr back to the RGB color space to obtain a color fused image.

[0020] The present invention also provides a target detection method, comprising the following steps:

[0021] The infrared image and the visible light image are processed by the pixel-level real-time multispectral image fusion method based on adaptive weight and target perception as described above in this invention to obtain a fused image;

[0022] The target in the fused image is detected to obtain the detection result.

[0023] The present invention also provides a target detection system, including

[0024] Pixel-level real-time multispectral image fusion network: used to process infrared images and visible light images using the pixel-level real-time multispectral image fusion method based on adaptive weights and target perception as described above, to obtain a fused image;

[0025] Target detection subnetwork: used to detect targets in the fused image and obtain detection results.

[0026] Preferably, the loss function L of the target detection system is as follows:

[0027] L = 0.15 × L fusion +L detection

[0028] L fusion =0.01×L intensity +L texture

[0029]

[0030]

[0031] Among them, L fusion L is the loss function for pixel-level real-time multispectral image fusion networks. detection Let L be the loss function of the target detection subnetwork. intensity Let L be the intensity loss function. texture Let H be the texture loss function, and W be the height and width of the image, respectively. f To fuse images, I ir For infrared images, I vi Visible light image, Let |·| denote the Laplace operator, and |·| denotes taking the absolute value.

[0032] Preferably, the object detection sub-network adopts the YOLOv5s object detection model or the mask-RCNN object detection model.

[0033] Compared with the prior art, the present invention has at least the following beneficial technical effects:

[0034] This invention proposes a pixel-level real-time multispectral image fusion method based on adaptive weights and target perception. The proposed pixel-level fusion method learns pixel-by-pixel fusion weights for infrared and visible light images, obtaining a fused image through pixel-by-pixel weighting. This enables real-time deployment of the image fusion task on embedded platforms, thus achieving both lightweight design and high fusion quality. Specifically, in terms of fusion quality, the designed infrared and visible light image fusion network achieves performance comparable to advanced networks on common fusion metrics, and the fused image achieves accuracy comparable to advanced methods in downstream target detection tasks. Regarding fusion speed, the designed infrared and visible light image fusion network achieves an inference speed of 420 FPS on a high-performance GPU and 68 FPS on an embedded platform (resolution 512×512). Attached Figure Description

[0035] Figure 1 This is the overall framework of the target detection network of the present invention;

[0036] Figure 2 This is a schematic diagram of the pixel-level real-time multispectral image fusion network of the present invention. Detailed Implementation

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0038] This invention designs an efficient fusion network to adaptively obtain pixel-by-pixel fusion weights, and obtains the fused image through pixel-by-pixel weighting, thereby enabling real-time deployment of image fusion tasks on embedded platforms. Furthermore, this invention constructs a joint optimization framework incorporating the proposed fusion network and existing detection models to collaboratively optimize these two tasks. The end-to-end joint training framework employed in this invention allows the training effects of image fusion and object detection tasks to mutually promote and enhance each other, resulting in better optimization results. The specific details of this invention are described below.

[0039] like Figures 1-2 As shown, the target detection network provided by this invention comprises two parts: a pixel-level real-time multispectral image fusion network and a target detection sub-network. The pixel-level real-time multispectral image fusion sub-network enables real-time fusion of infrared and visible light images, while the target detection sub-network can directly employ existing detection models (such as YOLOv5, mask-RCNN, etc.) to detect salient targets in the fused image. This pixel-level real-time multispectral image fusion network can seamlessly integrate with existing target detection models.

[0040] 1) Figure 1 This is a schematic diagram of the overall framework of the target detection network of this invention. The input of this invention is a registered infrared and visible light image pair I. ir and I vi The output is the detection result D of the fused image. f The target detection network mainly consists of two sub-networks: a pixel-level real-time multispectral image fusion network and a target detection sub-network. Specifically, the registered infrared and visible light images are compared with the target detection sub-network. ir and I vi The images are stitched together on the channel and input into a pixel-level real-time multispectral image fusion network to generate weight maps for infrared and visible light images. The weight maps for infrared and visible light images are then used to obtain a fused image. Next, the fused image is input into a target detection sub-network to obtain the detection result of the fused image.

[0041] The input visible light image can be either a grayscale image or a color image. A grayscale image is directly stitched with an infrared image, while a color image requires color space conversion before stitching. Specifically, if the visible light image is a color RGB image, it is first converted to the YCbCr color space, and then the Y channel is fused with the infrared image. After obtaining a grayscale fused image through a pixel-level real-time multispectral image fusion network, the Y channel of the visible light image is replaced with the grayscale fusion result. Finally, the image is converted back from the YCbCr color space to the RGB color space to obtain a color fused image, which serves as the input to the subsequent object detection sub-network.

[0042] 2) Figure 2 This is a schematic diagram of the pixel-level real-time multispectral image fusion network of the present invention. Given a pair of registered infrared and visible light image pairs I ir and I vi First, the infrared and visible light images are stitched together on the channel and then fed into multiple convolutional layers. Next, after a sigmoid operation, two weight maps containing pixel-wise weights are generated (i.e., the fused weight map of the infrared image and the fused weight map of the visible light image). Then, the weight maps and their corresponding images are multiplied element-wise and added together (i.e., the fused weight map of the infrared image and the infrared image are multiplied element-wise to obtain the first image, the fused weight map of the visible light image and the visible light image are multiplied element-wise to obtain the second image, and the first and second images are added element-wise to obtain the third image). Finally, the third image is normalized to obtain the fused image. Specifically, in this invention, four 3×3 convolutional layers are used to learn the fusion weights, and a sigmoid layer fixes the range of the fusion weights between 0 and 1. The output channels of the four convolutional layers are 32, 64, 32, and 2, respectively, and no downsampling operation is performed; all are performed at the original resolution. The fusion weights W of the infrared image are obtained. ir The fusion weight map W with the visible light image vi Then, each fused weight map is multiplied pixel-by-pixel with its corresponding original image, and then summed:

[0043] I f =I ir ·W ir +I vi ~W vi

[0044] Finally, the fused image is obtained using max-min normalization:

[0045]

[0046] To make image fusion and object detection more effective, this invention constructs the following loss functions for these two parts respectively:

[0047] For image fusion, intensity loss and texture loss are used to ensure that the fused image contains comprehensive intensity information and rich texture information. The purpose of intensity loss is to guide the fused image to retain significant pixel intensity information from both the infrared and visible light images; its specific formula is as follows:

[0048]

[0049] Where H and W are the height and width of the image.

[0050] The purpose of texture loss is to guide the fused image to contain more texture details, and its specific formula is:

[0051]

[0052] in, Let |·| denote the Laplace operator, and |·| denotes taking the absolute value.

[0053] For target detection, this invention directly adopts the loss function L of the detection model used. detection .

[0054] After calculating the above loss function, the overall loss function for target detection in this invention is:

[0055] L = 0.15 × L fusion +L detection

[0056] Among them, the loss function L of the fusion network fusion =0.01×L intensity +L texture .

[0057] To improve learning convergence speed and prevent getting stuck in local optima, this invention selects the Adam optimizer to update model parameters. The model is trained on the MSRS dataset following the above process, with random initialization of fusion network parameters and initialization of detection network parameters using publicly available pre-trained parameters from the corresponding detection model. At this point, model optimization is complete, and the model can be used for online inference tasks.

[0058] Through the joint optimization and design of the two sub-networks described above, this invention enables image fusion and object detection tasks to mutually promote and enhance each other. In practical applications, even on resource-constrained platforms, this method can achieve real-time infrared and visible light image fusion.

Claims

1. A pixel-level real-time multispectral image fusion method based on adaptive weights and target perception, characterized in that, The process includes the following: Pixel-level real-time multispectral image fusion: Infrared and visible light images are stitched together on a channel to obtain a stitched image; adaptive pixel-level weight generation is performed on the stitched image to obtain a fusion weight map of the infrared image and a fusion weight map of the visible light image; the fusion weight map of the infrared image and the infrared image are multiplied element-wise to obtain a first image; the fusion weight map of the visible light image and the visible light image are multiplied element-wise to obtain a second image; the first and second images are added element-wise to obtain a third image; the third image is normalized to obtain a fused image. The visible light image is a grayscale image or a color image; When the visible light image is a grayscale image, the infrared image and the visible light image are directly stitched together on the channel; When the visible light image is a color image, it is converted to the YCbCr color space, and then the Y channel is stitched with the infrared image. After obtaining the fused image, the Y channel of the visible light image is replaced with the fused image, and then it is converted back from YCbCr to RGB color space to obtain the color fused image. When performing adaptive pixel-level weight generation on the stitched image to obtain the fusion weight map of the infrared image and the fusion weight map of the visible light image, four 3×3 convolutional layers are used to learn the fusion weights, and a sigmoid layer is used to fix the range of the fusion weights between 0 and 1. The output channels of the four convolutional layers are 32, 64, 32, and 2, respectively, and no downsampling operation is performed; all are performed at the original resolution.

2. The pixel-level real-time multispectral image fusion method based on adaptive weights and target perception according to claim 1, characterized in that, The third image is normalized using the max-min normalization method to obtain the fused image.

3. A pixel-level real-time multispectral image fusion system based on adaptive weights and target perception, used to implement the pixel-level real-time multispectral image fusion method based on adaptive weights and target perception as described in claim 1 or 2, characterized in that, include: Pixel-level real-time multispectral image fusion network: used to stitch infrared images and visible light images on channels to obtain a stitched image; Adaptive pixel-level weight generation is performed on the stitched image to obtain a fusion weight map of the infrared image and a fusion weight map of the visible light image; the fusion weight map of the infrared image and the infrared image are multiplied element-wise to obtain a first image; the fusion weight map of the visible light image and the visible light image are multiplied element-wise to obtain a second image; the first image and the second image are added element-wise to obtain a third image. The third image is normalized to obtain the fused image.

4. The pixel-level real-time multispectral image fusion system based on adaptive weights and target perception according to claim 3, characterized in that: The visible light image is a grayscale image or a color image; When the visible light image is a grayscale image, the pixel-level real-time multispectral image fusion network directly stitches the infrared image and the visible light image on the channel; When the visible light image is a color image, the pixel-level real-time multispectral image fusion network converts the visible light image into the YCbCr color space, then uses the Y channel to stitch with the infrared image to obtain a fused image, and then replaces the Y channel of the visible light image with the fused image; then the pixel-level real-time multispectral image fusion network converts from YCbCr back to the RGB color space to obtain a color fused image.

5. A target detection method, characterized in that, The process includes the following: The infrared image and the visible light image are processed by the pixel-level real-time multispectral image fusion method based on adaptive weight and target perception as described in any one of claims 1-2 to obtain a fused image; The target in the fused image is detected to obtain the detection result.

6. A target detection system, characterized in that, include Pixel-level real-time multispectral image fusion network: used to process infrared images and visible light images using the pixel-level real-time multispectral image fusion method based on adaptive weights and target perception as described in any one of claims 1-2, to obtain a fused image; Target detection subnetwork: used to detect targets in the fused image and obtain detection results.

7. The target detection system according to claim 6, characterized in that, The loss function of the target detection system as follows: in, Let be the loss function of a pixel-level real-time multispectral image fusion network. Let the loss function be that of the target detection subnetwork. Let the intensity loss function be... For texture loss function, These are the height and width of the image, respectively. To merge images, Infrared image, For the visible light image, ▽ represents the Laplacian operator. This indicates taking the absolute value.

8. The target detection system according to claim 6, characterized in that, The object detection sub-network uses either the YOLOv5s object detection model or the mask-RCNN object detection model.

Images