Method and system for infrared and visible light image fusion and target detection

Abstract

This invention provides a method and system for infrared and visible light image fusion and target detection. The method includes: acquiring infrared and visible light images; inputting them into a GCNet fusion network built based on a lightweight GhostConv module to generate a high-quality fused image; and then feeding the fused image into a target detection network to obtain detection results. The core of this invention lies in employing a joint optimization training strategy, using a total loss consisting of image fusion loss (including gradient, intensity, and structural similarity components) and target detection loss to perform end-to-end joint training of the fusion and detection networks. The model of this invention is lightweight, efficient, and has a fast inference speed. Through task co-optimization, it significantly improves the quality of the fused image and the accuracy of target detection, enabling real-time processing, and is particularly suitable for edge computing platforms with limited computing power, such as assisted driving and drones.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, and more specifically, to a method and system for infrared and visible light image fusion and target detection. Background Technology

[0002] Traditional target detection networks are mostly trained on single-modal image streams (such as visible light or infrared images), which cannot simultaneously utilize the complementary information of two modules for target detection to improve the performance. For example, in visible light detection scenarios, due to the limitations of the sensor itself, sufficient information cannot be obtained for target detection in situations such as smoke, dust, glare, and low light, which is not conducive to effective detection in all weather conditions. In infrared detection scenarios, due to the low resolution of infrared images and the lack of color and detail information, it is difficult to accurately distinguish different targets.

[0003] Traditional image fusion networks and object detection networks are often treated as two separate tasks, ignoring the complementary information in the features of the two networks. This increases network complexity and results in poor detection performance, making it impossible to effectively deploy on edge application platforms such as surveillance and drones where computing power is limited.

[0004] Therefore, how to design a lightweight end-to-end network that can effectively fuse infrared and visible light images and improve target detection performance, while meeting the requirements of real-time and edge deployment, is a technical problem that urgently needs to be solved in the field of target detection. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a method and system for infrared and visible light image fusion and target detection, to address the limitations of existing target detection methods based on a single modality (such as visible light only or infrared only). Visible light images are difficult to obtain effective information in adverse weather conditions (smoke, dust) or poor lighting conditions (glare, low light); while infrared images, although able to highlight thermal targets, lack color and fine texture details, and their resolution is usually low, making target differentiation difficult and failing to meet the requirements of all-day, all-weather detection. Although there are methods that perform image fusion before detection, fusion and detection are often treated as two independent and separate tasks, ignoring their mutually reinforcing relationship. Furthermore, the network structure is often complex and computationally intensive, making real-time deployment difficult on edge computing platforms such as monitoring equipment and drones with limited computing power.

[0006] Therefore, the present invention aims to provide a lightweight and efficient infrared and visible light image fusion and target detection method and system to solve the performance bottleneck problems caused by the bloated models, poor real-time performance, and decoupling of fusion and detection tasks in the prior art.

[0007] To achieve the above and other related objectives, the present invention provides a method for infrared and visible light image fusion and target detection, comprising:

[0008] Step 1: Acquire pre-registered infrared and visible light images;

[0009] Step 2: Perform image fusion on the infrared image and the visible light image to obtain a fused image. The image fusion includes the following sub-steps:

[0010] a) Convert the visible light image from the RGB color space to the YUV color space and extract its Y channel image, wherein the Y channel image is the luminance channel image;

[0011] b) Stitch the brightness channel image and the infrared image along the channel dimension;

[0012] c) Input the stitched image from sub-step b) into the feature extraction module in the fusion network for feature extraction. The feature extraction module contains multiple cascaded feature extraction units.

[0013] d) Based on the features extracted by the feature extraction module, the fusion weights of the infrared image and the brightness channel image are generated by the fusion weight generation module in the fusion network;

[0014] e) The infrared image and the brightness channel image are weighted and fused based on the fusion weights to obtain a fused brightness channel image;

[0015] f) Merge the fused luminance channel image with the UV channel image (chrominance channel image) of the visible light image to generate the fused image;

[0016] Step 3: Input the fused image into the target detection network for target detection to obtain the target detection result;

[0017] Step 4: Employ a joint optimization training strategy to jointly optimize and train the fusion network and the target detection network based on image fusion loss and target detection loss.

[0018] Preferably, in step two (c), the feature extraction module includes multiple cascaded lightweight convolutional blocks as the feature extraction unit, specifically, the lightweight convolutional block is a GhostConv convolutional block.

[0019] Preferably, in step two (c), the feature extraction module includes five cascaded GhostConv convolutional blocks.

[0020] Preferably, in step two (c), each GhostConv convolutional block in the feature extraction module is configured to maintain the same size of its output feature map as the size of its input feature map during feature extraction.

[0021] Preferably, in step two (c), the first GhostConv convolutional block in the five-level cascaded GhostConv convolutional block uses a 3x3 convolutional kernel for convolution operation.

[0022] Preferably, in step two (c), the fusion network further includes a skip connection, which connects the stitched image from sub-step (b) between the output of the fourth-level GhostConv convolutional block and the input of the fifth-level GhostConv convolutional block in the five-level cascaded GhostConv convolutional blocks.

[0023] Preferably, in step 2d), the fusion weight generation module applies a sigmoid activation function to process the extracted features to generate the fusion weights, and the fusion weights take values ​​in the range of (0,1).

[0024] Preferably, in step four, the image fusion loss includes gradient loss, intensity loss, and structural similarity loss.

[0025] Preferably, the gradient loss (L_grad) is calculated based on the difference between the gradient of the fused luminance channel image, the gradient of the luminance channel image, and the gradient of the infrared image, in order to encourage the fused luminance channel image to retain detailed texture.

[0026] Preferably, the gradient loss (L_grad) is calculated by: using the Sobel operator to calculate the gradients of the fused luminance channel image, the luminance channel image, and the infrared image; taking the pixel-wise maximum value of the gradients of the luminance channel image and the infrared image; calculating the L2 norm difference between the gradient of the fused luminance channel image and the pixel-wise maximum value, and then normalizing it.

[0027] Preferably, the intensity loss (L_intensity) is calculated based on the difference between the pixel intensity of the fused luminance channel image, the pixel intensity of the luminance channel image, and the pixel intensity of the infrared image, and is used to encourage the fused luminance channel image to retain significant target information and increase brightness.

[0028] Preferably, the intensity loss (L_intensity) is calculated by: taking the pixel-wise maximum value of the intensity of the luminance channel image and the infrared image; calculating the L2 norm difference between the intensity of the fused luminance channel image and the pixel-wise maximum value, and then normalizing it.

[0029] Preferably, the structural similarity loss (L_ssim) is calculated based on the structural similarity between the fused luminance channel image and a reference image, which is generated from the luminance channel image and the infrared image.

[0030] Preferably, the structural similarity loss (L_ssim) is calculated as follows: first, a reference image is generated by taking the pixel-wise maximum value of the luminance channel image and the infrared image; then, the structural similarity index between the fused luminance channel image and the reference image is calculated; finally, the structural similarity index is subtracted from 1 to obtain the structural similarity loss.

[0031] Preferably, in step four, the image fusion loss (L_fusion) is a weighted sum of the gradient loss, the intensity loss, and the structural similarity loss.

[0032] Preferably, in step four, the image fusion loss (L_fusion) is calculated as follows: L_fusion = L_grad + α * L_intensity + β * L_ssim, where L_grad is the gradient loss, L_intensity is the intensity loss, L_ssim is the structural similarity loss, and α and β are preset first and second hyperparameters.

[0033] Preferably, in step four, the joint optimization training strategy is optimized using a total loss (L_joint), which is a weighted sum of the image fusion loss and the target detection loss generated by the target detection network.

[0034] Preferably, the total loss (L_joint) is calculated as follows: L_joint = L_fusion + γ * L_detect, where L_fusion is the image fusion loss, L_detect is the target detection loss, and γ is a preset third hyperparameter.

[0035] Preferably, in step three, the target detection network is a YOLO11n network.

[0036] Accordingly, this aspect also provides an infrared and visible light image fusion and target detection system, including:

[0037] The image acquisition unit is used to acquire pre-registered infrared and visible light images;

[0038] An image fusion network, connected to the image acquisition unit, is used to receive the infrared image and the visible light image, and perform an image fusion process to generate a fused image. The image fusion network includes:

[0039] The color space conversion unit is used to convert the visible light image from the RGB color space to the YUV color space and extract its Y channel image (luminance channel image).

[0040] A channel stitching unit is used to stitch the brightness channel image and the infrared image in the channel dimension;

[0041] The feature extraction module includes multiple cascaded feature extraction units, which are used to extract features from the stitched image output by the channel stitching unit;

[0042] The fusion weight generation module is used to generate fusion weights for the infrared image and the brightness channel image based on the features extracted by the feature extraction module.

[0043] A weighted fusion unit is used to perform weighted fusion on the infrared image and the brightness channel image based on the fusion weight to obtain a fused brightness channel image;

[0044] An image merging unit is used to merge the fused luminance channel image with the UV channel image (chroma channel image) of the visible light image to generate the fused image;

[0045] An object detection network, connected to the image fusion network, is used to receive the fused image and perform object detection to generate object detection results;

[0046] The network parameters of the image fusion network and the target detection network are obtained through a joint optimization training strategy, which jointly optimizes the image fusion network and the target detection network based on the image fusion loss and the target detection loss.

[0047] Preferably, the feature extraction unit in the feature extraction module is a GhostConv convolutional block.

[0048] Preferably, the feature extraction module comprises five cascaded GhostConv convolutional blocks.

[0049] Preferably, each GhostConv convolutional block in the feature extraction module is configured to maintain the same size as the input feature map when performing feature extraction.

[0050] Preferably, the image fusion network further includes a skip connection structure configured to connect the output of the channel stitching unit between the output of the fourth-level GhostConv convolutional block and the input of the fifth-level GhostConv convolutional block in the five-level cascaded GhostConv convolutional blocks.

[0051] Preferably, the fusion weight generation module includes a sigmoid activation function to process the extracted features to generate the fusion weights with values ​​ranging from (0,1).

[0052] Preferably, the target detection network is a YOLO11n network.

[0053] As described above, the infrared and visible light image fusion and target detection method and system of the present invention have the following beneficial effects:

[0054] The fused image generated by this method clearly highlights the target in the infrared image while preserving the rich background details and textures in the visible light image, resulting in superior overall visual quality. Based on this high-quality image, downstream target detection tasks become more accurate, especially in identifying small targets, distant targets, or partially occluded targets, where the performance is significantly improved.

[0055] This invention employs a joint optimization training strategy, combining image fusion loss and object detection loss into a unified optimization objective. This enables the training process of the fusion network to be guided by improving downstream detection performance, adaptively preserving and enhancing the features most beneficial to the detection task.

[0056] This invention successfully resolves the contradiction between high performance and high efficiency. Through lightweight network design and task collaboration optimization, it achieves superior fusion and detection performance while significantly reducing computational resource consumption. This enables the proposed method and system to be deployed cost-effectively and efficiently on various computing-constrained platforms, demonstrating high practical value and broad application prospects in fields such as autonomous driving, security monitoring, and unmanned systems. Attached Figure Description

[0057] Figure 1 The diagram shows the overall process flow of the infrared and visible light image fusion and target detection method proposed in this invention.

[0058] Figure 2 The diagram shows the overall technical architecture of the method of this invention, illustrating the main modules from input to output.

[0059] Figure 3 The diagram shows a detailed structural schematic of the image fusion network (GCNet) in this invention;

[0060] Figure 4 The diagram shows the structure of the lightweight GhostConv convolutional block used in this invention.

[0061] Figure 5 The figure shows a quantitative comparison of the target detection performance of the method of the present invention with other existing methods on the M3FD dataset.

[0062] Figure 6 The image shown is a quantitative evaluation result of the image fusion quality of the method of the present invention and other existing technologies on the M3FD dataset.

[0063] Figure 7 The image shown is a comparison of the fused image generated by the method of this invention with that generated by other methods.

[0064] Figure 8 The image shows a comparison of the target detection results of the method of this invention with those of other methods.

[0065] Figure 9 The graph shows a comparison of the computational efficiency of the GCNet fusion network in this invention with other methods.

[0066] Figure 10 The diagram shows the deployment architecture of the system of the present invention on an edge computing platform;

[0067] Figure 11 The image shown demonstrates the effect of real-time inference using the method of this invention on an edge computing platform. Detailed Implementation

[0068] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention.

[0069] Please see Figure 1 This invention provides a method for infrared and visible light image fusion and target detection, the overall architecture of which can be referred to Figure 2 As shown.

[0070] The method includes at least:

[0071] Step 1: Acquire pre-registered infrared (IR) and visible light (RGB) images. For ease of description, the infrared image will be denoted as I_ir in the following text. Pre-registration ensures that the two modalities are spatially aligned, which is the basis for subsequent effective fusion and accurate detection. Preprocessing can be achieved using existing or specially designed image registration algorithms.

[0072] Step 2: Perform image fusion on the infrared image and the visible light image to obtain a fused image. This fusion process combines the advantages of the two images to generate an image that contains both infrared target salience and visible light detail texture, thereby improving the performance of subsequent target detection.

[0073] Please see Figure 3 Image fusion is performed within a GCNet fusion network, which includes the following sub-steps:

[0074] a) Convert the visible light image from the RGB color space to the YUV color space and extract its Y channel image. The Y channel image is the luminance channel image, which will be referred to as I_vi in ​​the following text. By converting to the YUV space, the luminance information (Y channel) and chrominance information (UV channel) can be separated, which facilitates subsequent fusion based only on the luminance information, while preserving the color of the original visible light image.

[0075] b) The luminance channel image (I_vi) and the infrared image (I_ir) are stitched together along the channel dimension. By stacking two important single-channel grayscale information sources together, a multi-channel input is formed, preparing the subsequent network to process these two types of information simultaneously.

[0076] c) The stitched image from sub-step b) is input into the feature extraction module of the fusion network (GCNet) for feature extraction. The feature extraction module contains multiple cascaded feature extraction units. The deep convolutional neural network structure automatically learns and extracts multi-level features from the image, including low-level edges and textures and high-level semantic information, laying the foundation for subsequent accurate weight generation and fusion.

[0077] In some embodiments, in step two (c), the feature extraction module includes multiple cascaded lightweight convolutional blocks as feature extraction units. Specifically, the lightweight convolutional blocks are GhostConv convolutional blocks. See also... Figure 4 For convolutions with a target output channel count of CO, the GhostConv module first generates CO / 2 feature maps through a regular convolutional block, and then generates another CO / 2 feature maps through inexpensive grouped convolutions (such as depthwise separable convolutions). Using GhostConv as the basic feature extraction unit significantly reduces computation and parameter count while maintaining feature extraction capabilities, making the entire fusion network more lightweight and improving operational efficiency and deployment on edge devices.

[0078] In some embodiments, in step two (c), the feature extraction module includes a five-stage cascaded GhostConv convolutional block, such as... Figure 3 The GhostConv1 to GhostConv5 diagrams illustrate how stacking multiple levels of convolutional blocks can progressively expand the receptive field, extract feature representations from shallow to deep layers, and capture image information at different scales.

[0079] In some embodiments, in step c), each GhostConv convolutional block in the feature extraction module is configured to maintain the same size as the input feature map when performing feature extraction. This design helps reduce spatial information loss during feature transfer, especially in pixel-level image fusion tasks, and can better preserve the details and structure of the image.

[0080] In some embodiments, in step two (c), the first GhostConv convolutional block in the five-level cascaded GhostConv convolutional blocks uses a 3x3 convolutional kernel for convolution operations. Using a slightly larger convolutional kernel (such as 3x3 instead of 1x1) in the early stages of the network helps to obtain a larger receptive field at the beginning and capture richer contextual information.

[0081] In some embodiments, in step two c), the converged network further includes hop connections, such as... Figure 3 As shown, the skip connection connects the initial two-channel image stitched together in sub-step b) between the output of the fourth-level GhostConv convolutional block (GhostConv4) and the input of the fifth-level GhostConv convolutional block (GhostConv5) in a five-level cascaded GhostConv convolutional block. Introducing skip connections, specifically connecting shallow features (input stitched images containing rich details of the original images) to deeper layers, helps to directly pass low-level detail features to the back end, enhancing the preservation of details in the original image by the fused image. Furthermore, since the connected features have the same number of channels and feature map size, this connection does not introduce excessive additional computational overhead.

[0082] d) Based on the features extracted by the feature extraction module, the fusion weights W_IR and W_RGB (corresponding to the luminance channel Y) of the infrared image (I_ir) and the luminance channel image (I_vi) are generated by the fusion weight generation module in the fusion network (e.g., after the final convolutional layer processing). According to the learned feature information, fusion weights are adaptively assigned to each pixel of the infrared image and the visible light luminance image to determine the contribution of each information to the final fusion result.

[0083] In some embodiments, in step 2d), the fusion weight generation module applies a sigmoid activation function to process the extracted features to generate fusion weights, such as... Figure 3 As shown in the Sigmoid layer, the fusion weights range from (0,1). The Sigmoid function can be used to map the network output to the (0,1) interval.

[0084] e) The infrared image (I_ir) and the luminance channel image (I_vi) are weighted and fused based on the fusion weights to obtain the fused luminance channel image I_fu. The infrared image I_ir and the visible light luminance image I_vi are multiplied by their corresponding fusion weights W_IR and W_Y (i.e., W_RGB), respectively. Then, element-wise multiplication and addition are performed to obtain the Y channel image of the fused image. This Y channel image is then combined with the UV channels of the visible light YUV image to preserve visible light colors.

[0085] f) Merge the fused luminance channel image I_fu with the UV channel image (chrominance channel image) of the visible light image to generate a fused image. Combine the optimized luminance information obtained from the fusion with the original visible light chrominance information (from the UV channel converted in step a) to finally generate a fused image that retains the saliency of the target and some details while possessing natural colors. If necessary, the YUV image can be converted back to the RGB color space for display or input to subsequent modules that only accept RGB images.

[0086] The entire image fusion process in step two aims to generate a high-quality fused image. For example... Figure 7 As shown, a comparison with existing image fusion methods such as TarDAL, DetFusion, and SeAFusion reveals that the fused image generated by the method of this invention (Ours) not only clearly highlights the target (such as people and vehicles), but also retains rich image details and complementary structural information in the visible light and infrared images, with clear and sharp edges and the best overall visual effect.

[0087] Step 3: Input the fused image into an object detection network (such as YOLOv11n) for object detection to obtain the object detection results.

[0088] In some embodiments, in step three, the object detection network is a YOLO11n network. Using a lightweight YOLO11n network as the object detection backend helps maintain the real-time performance of the entire system, especially when cascaded with a lightweight fusion network GCNet, enabling end-to-end real-time image fusion and object detection.

[0089] Step 4: Employ a joint optimization training strategy, jointly optimizing and training the fusion network (GCNet) and the object detection network (YOLO11n) based on the image fusion loss (L_fusion) and object detection loss (L_detect). Unlike treating fusion and detection as two independent tasks, joint optimization training ensures that the fusion network's learning objective is not only to generate high-quality fused images but also to benefit the downstream object detection task. Simultaneously, feedback from the detection task can guide the fusion network to retain more features beneficial to detection, achieving mutual promotion between the two tasks and improving overall performance.

[0090] In some embodiments, step four includes gradient loss, intensity loss, and structural similarity loss. By designing combinations of various loss functions, the learning process of the fusion network can be constrained from different dimensions, ensuring that the fused image achieves good results in terms of detail, saliency, structural information, and other aspects.

[0091] In some embodiments, in step four, the gradient loss (L_grad) is calculated based on the difference between the gradient of the fused luminance channel image and the gradients of the luminance channel image (I_vi) and the infrared image (I_ir), to encourage the fused luminance channel image to retain detailed texture. Gradient information reflects the edge and texture details of the image, and by minimizing the difference between the gradient of the fused image and the most significant gradient in the source image, the fused image can be encouraged to inherit richer details from the source image.

[0092] In some embodiments, step four, the gradient loss (L_grad) is calculated as follows: using the Sobel operator to calculate the gradients of the fused luminance channel image, luminance channel image, and infrared image; taking the pixel-wise maximum value of the gradients of the luminance channel image (I_vi) and the infrared image (I_ir); calculating the L2 norm difference between the gradient of the fused luminance channel image and the pixel-wise maximum value, and normalizing it. Note: The corresponding formula is: L_grad=(1 / (WH))*|||∇I_fu|-max(|∇I_vi|,|∇I_ir|)||_2, where I_fu is the fused luminance channel image, ∇I_vi, ∇I_ir, and ∇I_fu represent the gradients of the visible light image, infrared image, and fused image, respectively, max selects the pixel with the largest gradient, ∇ is the Sobel operator, W and H are the width and height of the source image, respectively, and ||||_2 represents taking the L2 norm of the pixel difference between the fused image and the source image.

[0093] In some embodiments, in step four, the intensity loss (L_intensity) is calculated based on the differences between the pixel intensities of the fused luminance channel image, the luminance channel image, and the infrared image. This is used to encourage the fused luminance channel image to retain significant target information and enhance brightness. Intensity information is directly related to the visibility of the target and the overall brightness of the image. By minimizing the difference between the intensity of the fused image and the intensity of brighter pixels in the source image, it helps to highlight the target and maintain image contrast.

[0094] In some embodiments, step four, the calculation of intensity loss (L_intensity) includes: taking the pixel-wise maximum value of the intensity of the luminance channel image (I_vi) and the infrared image (I_ir); calculating the L2 norm difference between the fused luminance channel image intensity and the pixel-wise maximum value, and normalizing it. Note: The corresponding formula is: L_intensity=(1 / (WH))*||I_fu-max(I_vi,I_ir)||_2, where I_fu is the fused luminance channel image, I_vi is the visible light image, I_ir is the infrared image, W and H are the image width and height, and ||·||_2 is the L2 norm.

[0095] In some embodiments, in step four, the structural similarity loss (L_ssim) is calculated based on the structural similarity between the fused brightness channel image and a reference image generated from the source image. This calculation method aims to maximize the preservation of the most salient features in the source image from the structural information of the fused image. Specific calculation methods include:

[0096] First, a reference image max(I_vi, I_ir) is generated by taking the pixel-wise maximum value of the luminance channel image (I_vi) and the infrared image (I_ir). This reference image combines the stronger signals from the two source images at each pixel.

[0097] Next, the structural similarity index (SSIM) between the fused luminance channel image (I_fu) and the aforementioned generated reference image is calculated.

[0098] Finally, subtract 1 from the structural similarity index to obtain the final structural similarity loss. Note: The corresponding formula is: L_ssim = 1 - SSIM(I_fu, max(I_vi, I_ir)), where I_fu is the fused luminance channel image, I_vi is the luminance channel image, I_ir is the infrared image, and SSIM is the structural similarity index function.

[0099] In some embodiments, in step four, the image fusion loss (L_fusion) is a weighted sum of gradient loss, intensity loss, and structural similarity loss. By weighting and combining different loss components, different emphases on detail, saliency, and structure can be balanced to obtain a fusion result with better overall performance.

[0100] In some embodiments, in step four, the image fusion loss (L_fusion) is calculated as follows: L_fusion = L_grad + α * L_intensity + β * L_ssim, where L_grad is the gradient loss, L_intensity is the intensity loss, L_ssim is the structural similarity loss, and α and β are preset first and second hyperparameters. The hyperparameters α and β can be adjusted according to specific application scenarios and datasets to obtain the optimal fusion effect.

[0101] In some embodiments, in step four, the joint optimization training strategy uses a total loss (L_joint) for optimization. The total loss is a weighted sum of the image fusion loss and the object detection loss generated by the object detection network. Combining the fusion loss and the detection loss as the final optimization objective allows the network to simultaneously strive to improve both image fusion quality and object detection accuracy during training.

[0102] In some embodiments, in step four, the total loss (L_joint) is calculated as: L_joint = L_fusion + γ * L_detect, where L_fusion is the image fusion loss, L_detect is the object detection loss, and γ is a preset third hyperparameter. The hyperparameter γ is used to balance the importance of the fusion task and the detection task. Adjusting it can control the degree of emphasis on which task the optimization process focuses on, thereby achieving synergistic improvement of the two tasks.

[0103] Through the aforementioned joint optimization training strategy, the method of this invention can significantly improve the quality of fused images and the performance metrics (such as mAP50, mAP50-95) of object detection tasks, surpassing traditional methods that involve fusing images before detection or using only single-modal images for detection. For example, the quantitative evaluation results of object detection on the M3FD dataset (see reference) Figure 5 ) and quantitative evaluation results of image fusion on the M3FD dataset (reference) Figure 6 The method of this invention (Ours) has achieved best or second-best performance in various indicators, especially in mAP50 and mAP50-95. (Reference) Figure 8 The comparative experimental results of target detection also clearly show that the method of the present invention has better detection effect and scene adaptability than other methods in different scenarios.

[0104] Please see Figure 9 The image fusion network (GCNet) described in this embodiment improves computational efficiency. To further quantify this performance advantage, this invention conducted a performance comparison experiment on an NVIDIA RTX 3090 GPU platform, targeting images with a resolution of 1024x768, comparing GCNet with several advanced fusion networks in existing technologies.

[0105] The comparative results clearly demonstrate the significant advantages of the GCNet invention:

[0106] The GCNet of this invention has a computational complexity (FLOPs) of only 4.11 G, a network parameter count of only 0.005 M (i.e., about 5000 parameters), and an average inference time of only 11.10 milliseconds (ms) per image.

[0107] Other comparative methods, such as TarDAL, DetFusion, SeAFusion, EMMA, and MRFS, have computational complexities ranging from a minimum of 106.3G to a maximum of 356.0G; the number of parameters varies from 0.168M to 135.0M; and the corresponding inference time is generally above 70ms, with the slowest reaching 445.2ms.

[0108] The experimental data above show that the GCNet of this invention has a computational complexity and a number of network parameters that are one to several orders of magnitude lower than existing methods. This order-of-magnitude reduction fundamentally improves its inference speed. The inference time of only 11.10 ms fully demonstrates the superior real-time performance of the fusion network of this invention. This allows it to be successfully deployed on edge computing platforms with limited computing power, such as monitoring equipment, drones, and assisted driving systems, to achieve real-time image fusion. When the fusion network is cascaded with a lightweight object detection network, real-time end-to-end image fusion and object detection capabilities can be achieved.

[0109] This invention also provides an infrared and visible light image fusion and target detection system. This system can realize the fusion and detection process described in the above method, and its overall architecture can be referred to Figure 10 As shown, it can be deployed on servers or edge computing platforms (such as Nvidia Jetson Orin Nano) and includes at least:

[0110] An image acquisition unit is used to acquire pre-registered infrared and visible light images. For example, it can be a hardware interface and driver program connecting an infrared camera and a visible light camera, or a module for reading image data from a storage device.

[0111] Image fusion networks (such as) Figure 2 The GCNet shown is connected to the image acquisition unit and is used to receive infrared and visible light images, and perform an image fusion process to generate a fused image. The image fusion network includes:

[0112] The color space conversion unit is used to convert visible light images from the RGB color space to the YUV color space and extract their Y channel image (luminance channel image). It can be implemented by standard image processing library functions or dedicated hardware processing units.

[0113] The channel stitching unit is used to stitch the luminance channel image and the infrared image along the channel dimension.

[0114] In some embodiments, the feature extraction module's feature extraction unit is a GhostConv convolutional block (its structure is as follows). Figure 3 As shown, this lightweight design significantly reduces the computational complexity and number of parameters of the network. It includes multiple cascaded feature extraction units for extracting features from the stitched image output by the channel stitching unit. In some embodiments, the feature extraction module includes five cascaded GhostConv convolutional blocks, which can effectively extract multi-scale features. In some embodiments, each GhostConv convolutional block in the feature extraction module is configured to maintain the same size of its output feature map as the input feature map during feature extraction. This helps to reduce information loss during feature extraction and preserve more spatial details.

[0115] In some embodiments, the fusion weight generation module includes a sigmoid activation function (e.g., ...). Figure 3 As shown in the figure, it is used to generate fusion weights for infrared images and brightness channel images based on the features extracted by the feature extraction module, and generates fusion weights with values ​​ranging from (0,1).

[0116] The weighted fusion unit is used to perform weighted fusion of the infrared image and the brightness channel image based on the fusion weights to obtain a fused brightness channel image. It performs a weighted addition operation.

[0117] The image merging unit is used to merge the fused luminance channel image with the UV channel image (chrominance channel image) of the visible light image to generate a fused image.

[0118] In some embodiments, the image fusion network also includes skip connection structures (such as...) Figure 3 As shown, this structure is configured to connect the output of the channel stitching unit between the output of the fourth-level GhostConv convolutional block and the input of the fifth-level GhostConv convolutional block in a five-level cascaded GhostConv convolutional block. This allows low-level features of the original image to be introduced into deep features, enhancing detail preservation while maintaining low computational overhead.

[0119] An object detection network, connected to an image fusion network, receives the fused image and performs object detection to generate object detection results; in some embodiments, this network is a YOLO11n network. Choosing a lightweight object detection network ensures end-to-end system efficiency.

[0120] The network parameters of the image fusion network and the object detection network are obtained through a joint optimization training strategy. This strategy jointly optimizes the image fusion network and the object detection network based on the image fusion loss and the object detection loss. The system obtained through joint optimization training not only achieves high-quality fused images but also significantly improves object detection accuracy, outperforming traditional discrete processing methods.

[0121] The system proposed in this invention is lightweight and highly efficient, enabling real-time infrared and visible light image fusion and target detection on platforms with limited computing power (such as monitoring equipment, UAV-borne systems, and assisted driving systems). [Reference] Figure 10 This demonstrates a system architecture for deploying a real-time inference network for image fusion and object detection on an Nvidia Jetson Orin Nano, with inference images of size 640*480. (Reference) Figure 11 The image shows the results of real-time inference on this edge platform. It is evident that the real-time image fusion and object detection inference network exhibits high inference accuracy and good target recognition performance for small targets and occluded objects. When there are many objects to be identified in the image, the FPS ranges from a minimum of 27.20 to a maximum of 42.21, meeting real-time requirements. Timing was started from the beginning of the main thread's runtime and ended when 4200 infrared and visible light image pairs were inferred. The average inference time for each image fusion and object detection operation was measured to be 27.95ms, resulting in an average FPS of 35.78. This demonstrates excellent real-time performance, meeting the requirements of applications with high real-time demands, such as assisted driving.

[0122] It should be noted that the illustrations provided in this embodiment are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0123] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for infrared and visible light image fusion and target detection, characterized in that, At least including: Step 1: Acquire pre-registered infrared and visible light images; Step 2: Perform image fusion on the infrared image and the visible light image to obtain a fused image. The image fusion includes the following sub-steps: a) Convert the visible light image from the RGB color space to the YUV color space and extract its Y channel image, wherein the Y channel image is the luminance channel image; b) Stitch the brightness channel image and the infrared image along the channel dimension; c) Input the stitched image from sub-step b) into the feature extraction module in the fusion network for feature extraction. The feature extraction module contains multiple cascaded feature extraction units, each composed of cascaded lightweight convolutional blocks. The fusion network also includes a skip connection structure, which connects the stitched image from sub-step b) to the input of the subsequent lightweight convolutional block in the cascaded lightweight convolutional blocks. d) Based on the features extracted by the feature extraction module, the fusion weights of the infrared image and the brightness channel image are generated by the fusion weight generation module in the fusion network; e) The infrared image and the brightness channel image are weighted and fused based on the fusion weights to obtain a fused brightness channel image; f) Merge the fused luminance channel image with the UV channel image (i.e., chroma channel image) of the visible light image to generate the fused image; Step 3: Input the fused image into the target detection network for target detection to obtain the target detection result; Step 4: Employ a joint optimization training strategy to jointly optimize and train the fusion network and the object detection network based on image fusion loss and object detection loss. The image fusion loss includes: a gradient loss to ensure that the fused brightness channel image retains detailed textures, an intensity loss to retain significant target information and increase brightness, and a structural similarity loss calculated based on the structural similarity between the fused brightness channel image and a reference image generated from the source image. The joint optimization training strategy uses a total loss for optimization, which is a weighted sum of the image fusion loss and the object detection loss generated by the object detection network.

2. The infrared and visible light image fusion and target detection method according to claim 1, characterized in that: In step 2c), the lightweight convolutional block is a GhostConv convolutional block.

3. The infrared and visible light image fusion and target detection method according to claim 2, characterized in that: In step 2c), the feature extraction module includes five cascaded GhostConv convolutional blocks.

4. The infrared and visible light image fusion and target detection method according to claim 2 or 3, characterized in that: In step 2c), each GhostConv convolutional block in the feature extraction module is configured to maintain the same size of its output feature map as the size of its input feature map during feature extraction.

5. The infrared and visible light image fusion and target detection method according to claim 3, characterized in that: In step 2c), the first GhostConv convolutional block in the five-level cascaded GhostConv convolutional block uses a 3x3 convolutional kernel for convolution operation.

6. The infrared and visible light image fusion and target detection method according to claim 3, characterized in that: In step 2c), the fusion network further includes a skip connection that connects the stitched image from sub-step b) between the output of the fourth-level GhostConv convolutional block and the input of the fifth-level GhostConv convolutional block in the five-level cascaded GhostConv convolutional blocks.

7. The infrared and visible light image fusion and target detection method according to claim 1, characterized in that: In step 2d), the fusion weight generation module applies a sigmoid activation function to process the extracted features to generate the fusion weights, and the fusion weights take values ​​in the range of (0,1).

8. The infrared and visible light image fusion and target detection method according to claim 7, characterized in that: The gradient loss is calculated based on the difference between the gradient of the fused luminance channel image, the gradient of the luminance channel image, and the gradient of the infrared image, and is used to encourage the fused luminance channel image to retain detailed texture.

9. The infrared and visible light image fusion and target detection method according to claim 8, characterized in that: The gradient loss is calculated as follows: using the Sobel operator to calculate the gradients of the fused luminance channel image, the luminance channel image, and the infrared image; taking the pixel-wise maximum value of the gradients of the luminance channel image and the infrared image; calculating the L2 norm difference between the gradient of the fused luminance channel image and the pixel-wise maximum value, and then normalizing it.

10. The infrared and visible light image fusion and target detection method according to claim 1, characterized in that: The intensity loss is calculated based on the difference between the pixel intensity of the fused luminance channel image, the pixel intensity of the luminance channel image, and the pixel intensity of the infrared image, and is used to encourage the fused luminance channel image to retain significant target information and increase brightness.

11. The infrared and visible light image fusion and target detection method according to claim 10, characterized in that: The intensity loss is calculated by: taking the pixel-wise maximum value of the intensity of the luminance channel image and the infrared image; calculating the L2 norm difference between the intensity of the fused luminance channel image and the pixel-wise maximum value, and then normalizing it.

12. The infrared and visible light image fusion and target detection method according to claim 11, characterized in that: The calculation method for the structural similarity loss includes: first, generating a reference image by taking the pixel-wise maximum value of the luminance channel image and the infrared image; then, calculating the structural similarity index between the fused luminance channel image and the reference image; and finally, subtracting the structural similarity index from 1 to obtain the structural similarity loss.

13. The infrared and visible light image fusion and target detection method according to claim 1, characterized in that: In step four, the image fusion loss is calculated as follows: L_fusion=L_grad+α*L_intensity+β*L_ssim, where L_grad is the gradient loss, L_intensity is the intensity loss, L_ssim is the structural similarity loss, and α and β are preset first and second hyperparameters.

14. The infrared and visible light image fusion and target detection method according to claim 1, characterized in that: The formula for calculating the total loss is: L_joint = L_fusion + γ * L_detect, where L_fusion is the image fusion loss, L_detect is the target detection loss, and γ is a preset third hyperparameter.

15. The infrared and visible light image fusion and target detection method according to claim 1, characterized in that: In step three, the target detection network is a YOLO11n network.

16. A system for infrared and visible light image fusion and target detection, characterized in that, At least including: The image acquisition unit is used to acquire pre-registered infrared and visible light images; An image fusion network, connected to the image acquisition unit, is used to receive the infrared image and the visible light image, and perform an image fusion process to generate a fused image. The image fusion network includes: The color space conversion unit is used to convert the visible light image from the RGB color space to the YUV color space and extract its Y channel image, i.e., the luminance channel image. A channel stitching unit is used to stitch the brightness channel image and the infrared image in the channel dimension; The feature extraction module includes multiple cascaded feature extraction units for extracting features from the stitched image output by the channel stitching unit. Each feature extraction unit is composed of cascaded lightweight convolutional blocks. The fusion network also includes a skip connection structure, which connects the stitched image to the input of the subsequent lightweight convolutional block in the cascaded lightweight convolutional blocks. The fusion weight generation module is used to generate fusion weights for the infrared image and the brightness channel image based on the features extracted by the feature extraction module. A weighted fusion unit is used to perform weighted fusion on the infrared image and the brightness channel image based on the fusion weight to obtain a fused brightness channel image; An image merging unit is used to merge the fused luminance channel image with the UV channel image (i.e., chroma channel image) of the visible light image to generate the fused image; An object detection network, connected to the image fusion network, is used to receive the fused image and perform object detection to generate object detection results; The network parameters of the image fusion network and the object detection network are obtained through a joint optimization training strategy. The joint optimization training strategy jointly optimizes the image fusion network and the object detection network based on image fusion loss and object detection loss. The image fusion loss includes: gradient loss to retain detailed texture of the fused brightness channel image, intensity loss to retain significant target information and increase brightness, and structural similarity loss calculated based on the structural similarity between the fused brightness channel image and the reference image generated from the source image. The joint optimization training strategy uses a total loss for optimization, which is a weighted sum of the image fusion loss and the object detection loss generated by the object detection network.

17. The infrared and visible light image fusion and target detection system according to claim 16, characterized in that: The feature extraction unit in the feature extraction module is a GhostConv convolutional block.

18. The infrared and visible light image fusion and target detection system according to claim 17, characterized in that: The feature extraction module contains five cascaded GhostConv convolutional blocks.

19. The infrared and visible light image fusion and target detection system according to claim 17 or 18, characterized in that: Each GhostConv convolutional block in the feature extraction module is configured to maintain the same size as the input feature map when performing feature extraction.

20. The infrared and visible light image fusion and target detection system according to claim 18, characterized in that: The image fusion network further includes a skip connection structure configured to connect the output of the channel stitching unit between the output of the fourth-level GhostConv convolutional block and the input of the fifth-level GhostConv convolutional block in the five-level cascaded GhostConv convolutional blocks.

21. The infrared and visible light image fusion and target detection system according to claim 16, characterized in that: The fusion weight generation module includes an S-shaped activation function, which processes the extracted features to generate the fusion weights with values ​​ranging from (0,1).

22. The infrared and visible light image fusion and target detection system according to claim 16, characterized in that: The target detection network is a YOLO11n network.

Images