A method for infrared and visible light image fusion for joint target detection
Through the infrared and visible light image fusion method under the convolutional neural network framework, the problems of low detection accuracy and insufficient real-time performance in the existing technology are solved, and efficient target detection is achieved under poor lighting conditions, which is suitable for high-level visual tasks in more scenarios.
Patent Information
- Application Number
- CN202310147643.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-22
- Publication Date
- 2025-09-16
- Estimated Expiration
- 2043-02-22
AI Technical Summary
Existing infrared and visible light image fusion methods have low detection accuracy under poor lighting conditions, lack real-time performance, and cannot effectively cooperate with high-level visual tasks. In addition, traditional methods have poor generalization and robustness, and deep learning-based methods have complex structures and are difficult to balance in training.
Using convolutional neural networks as the main framework, a method for fusion of infrared and visible light images for joint target detection is designed. By pre-training the detection network and fusion network, using feature extraction network modules and image reconstruction modules, and combining fusion loss and detection loss functions for training, dense networks and channel attention mechanisms are introduced to achieve fast and efficient image fusion.
The generated fused image can significantly improve the accuracy and real-time performance of target detection, adapt to the high-level visual task requirements of more scenarios, improve the average accuracy and meet the real-time detection requirements.
Smart Images
Figure CN116188342B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of infrared and visible light image fusion, and in particular relates to an infrared and visible light image fusion method for joint target detection. Background Art
[0002] Existing object detection methods have limited application scenarios. Object detection methods based on visible light images can detect various targets in well-lit scenes and are widely used in areas such as autonomous driving and real-time monitoring. However, recognition rates drop significantly in low-visibility conditions such as darkness, rain, snow, and fog. Therefore, object detection based solely on visible light images is limited in its application scenarios.
[0003] Directly applying the fused images generated by existing infrared and visible light image fusion methods to target detection results in low accuracy. Because scholars in the field are limited to subjective analysis and objective image quality evaluation metrics for fusion methods, the generated fused images, while achieving high image quality evaluation metrics, do not work well with high-level vision tasks. Experiments have shown that directly applying existing infrared and visible light image fusion methods to high-level vision tasks does not achieve high accuracy, making infrared and visible light image fusion methods incompatible with high-level vision tasks and unable to be applied in more scenarios.
[0004] Existing infrared and visible light image fusion methods for joint target detection suffer from poor real-time performance. Existing infrared and visible light image fusion methods, including some for joint target detection, utilize complex model structures based on generative adversarial networks (GANs). These structures make balanced network training difficult, and the generation of fused images slow. Since most target detection applications require real-time detection, GAN-based image fusion methods lack real-time performance and are unable to meet the requirements of subsequent high-level vision tasks.
[0005] Many existing infrared and visible light image fusion methods have been developed to improve visual quality, which are mainly divided into traditional image fusion methods and deep learning-based image fusion methods.
[0006] Traditional image fusion methods rely primarily on manually designed feature extractors for feature extraction, requiring specialized knowledge and complex parameter tuning. Furthermore, each method is application-specific. Furthermore, traditional methods typically use the same transformation or representation to extract features from source images, failing to account for the inherent differences between different source images. Furthermore, manually designed fusion rules are inadequate for complex fusion scenarios. Furthermore, in pursuit of improved fusion performance, the design of feature extraction methods and fusion rules has become increasingly complex, making the process of generating the fused image less real-time. Traditional image fusion methods suffer from generally weak generalization and poor robustness.
[0007] In recent years, deep learning technology has rapidly developed in the fields of computer vision and image processing. An increasing number of researchers are applying deep learning to image fusion, resulting in the emergence of a large number of deep learning-based image fusion algorithms. Based on their common network architectures, these algorithms can be categorized into three main categories: convolutional neural networks (CNNs), generative adversarial networks (GANs), and auto-encoders (AEs). Most deep learning-based infrared and visible image fusion methods, including FusionDN (A unified densely connected network for image fusion) and IFCNN (Image Fusion framework based on the Convolutional Neural Network), achieve good fusion results. However, these methods fail to integrate the fusion network with higher-level visual tasks, resulting in lower accuracy when the fused image is used in conjunction with higher-level visual tasks. Consequently, these infrared and visible image fusion methods are not suitable for high-level visual tasks and are therefore not applicable to a wider range of scenarios. The TDAL (Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection) method combines infrared and visible light image fusion with high-level vision tasks. It designs a target-aware dual adversarial learning network with relatively few parameters for detection-oriented fusion. Because the model structure is based on a generative adversarial network (GAN), it is relatively complex and difficult to balance network training, resulting in slow generation of fused images. Since most object detection applications require real-time detection, image fusion methods using GANs are not real-time enough to meet the needs of subsequent high-level vision tasks. Summary of the Invention
[0008] In order to overcome the above-mentioned shortcomings of the prior art, the object of the present invention is to provide a method for fusion of infrared and visible light images for joint target detection;
[0009] Objective 1 of the present invention: Under poor lighting conditions, the saliency information provided by infrared images can significantly improve the accuracy of target detection. The present invention combines target detection with the design of an infrared and visible light image fusion method, which can achieve the purpose of image fusion as a low-level visual task to assist high-level visual tasks.
[0010] The second purpose of the present invention: The infrared and visible light image fusion method of the present invention is combined with the target detection design, and the average accuracy in the target detection task is more than two percentage points higher than that of other methods, achieving the purpose of high accuracy in the target detection task.
[0011] The third purpose of the present invention: The present invention adopts a convolutional neural network as the main framework of the image fusion method, which is easy to train and has a fast fusion speed, and can achieve the purpose of generating a fused image in real time, meeting the needs of the target detection task.
[0012] In order to achieve the above object, the technical solution adopted by the present invention is:
[0013] A method for fusion of infrared and visible light images for joint target detection comprises the following steps:
[0014] 1) Using an existing dataset, we loaded visible light images and image label files into the YOLO v5 model. We then performed data augmentation and convolutional layer modules on the network while simultaneously calculating the loss function. Based on the backpropagation loss, we adjusted the network weights during training to minimize the loss. After multiple iterations, when the loss function converged, we used the parameters at that point to form a pre-trained detection network.
[0015] 2) The visible light and infrared light images are input to the input of the fusion network. The images pass through the feature extraction network module and the image reconstruction module to train the fusion network. At this time, the pre-trained detection network is used to generate the detection loss. The detection loss is returned and the fusion loss is used to train the fusion network together.
[0016] 3) The fusion network outputs the generated fusion image, which is then input into the detection network for formal training of the detection network to obtain the training weights of the detection network; after completing all steps, the fusion network and detection network with trained network parameters are obtained.
[0017] In step 1), pre-training is performed early to assist with formal training in subsequent steps. By pre-training the detection network, the parameters of the detection network form a detection loss, which assists in training the fusion network in subsequent steps. The detection network is a single-stage object detection network, YOLO v5 (You Only Look Once), whose architecture includes Mosaic data augmentation, feature pyramids, and a path aggregation network, offering advantages such as lightweight and real-time performance.
[0018] The step 2) is specifically as follows:
[0019] The infrared image and the visible light image are loaded into the fusion network. Both images are passed through the dense network module and the channel attention module of the feature extraction network respectively.
[0020] The image features obtained by the two feature extraction modules are passed through the image reconstruction module to generate a fused image. The fused image, the input image, and the mask information are combined to calculate the fusion loss according to the fusion loss function formula. At the same time, the output fused image is loaded into the detection network. The detection network, which has been pre-trained to obtain parameters, returns the detection loss to the fusion network. The fusion loss function and the detection loss function jointly guide the training of the fusion network. The process is as follows:
[0021] According to the back-propagation fusion loss and detection loss, the fusion network updates the network weights in the direction of reducing the two types of losses during training. After multiple iterations, the two types of loss functions tend to converge. The parameters at this time are used as the fusion network parameters to obtain a trained infrared and visible light image fusion network.
[0022] The formula of the infrared image and visible light image fusion network is expressed as:
[0023] {F ir , F vi}={E F (I ir ), E F (I vi )}
[0024] I f =R I (C(F ir , F vi ))
[0025] I ir , I vi is the input infrared and visible light images, E F is the feature extraction network, F ir , F vi is the extracted infrared and visible light image features, R I is the image reconstruction module, C is the feature splicing, I f is the output fused image.
[0026] The training part of the fusion network uses a fusion loss function and a detection loss function;
[0027] The fusion loss function consists of two parts: pixel loss and gradient loss. The pixel loss limits the intensity of the pixel fusion image to be consistent with the source image, while the gradient loss forces the fusion image to contain more detailed information.
[0028] The fusion loss function extracts mask information from the infrared image. Under the constraints of the loss function, the information of the fused image can be defined as follows:
[0029] I result =I mask I ir +(1-I mask )·I vi
[0030] Among them I result is the fused image, I mask is the mask information extracted from the infrared light source image, I ir is the infrared light source image, I vi For visible light source images, during the training phase, the detection loss of the target detection network flows back to the fusion network, and the fusion network is trained together with the fusion loss, so that the fusion network is combined with the detection network to obtain an infrared and visible light fusion network that is more suitable for target detection.
[0031] The fusion network includes a feature extraction network module and an image reconstruction module. The feature extraction of the fusion network is a parallel dense network module and a channel attention module. The dense network (Densenet) module is connected in parallel with the channel attention module to increase important information in infrared images and visible light images.
[0032] The mainstream dense network module implements feature extraction; the tributary attention mechanism highlights important features; the parallel structure reuses features through the main dense stream and tributaries to enhance the fusion network's ability to describe fine-grained spatial details. The dense network module includes multi-scale convolution, activation functions, and residual connections to establish dense connections between all previous layers and subsequent layers. Through dense connections, the fusion network reuses features, thereby achieving better feature extraction performance.
[0033] The attention mechanism is a dynamic selection process that adaptively weights features based on the importance of the input, and can retain more important features.
[0034] The image reconstruction module includes multi-scale convolution and activation functions, and generates a fused image by passing the image features obtained by the two feature extraction branches through the image reconstruction module.
[0035] The formal training of step 3) is:
[0036] The fused image and image label files are loaded into the main program of the YOLO v5 model. The loss function is calculated simultaneously through the network's data enhancement module and convolutional layer module. According to the loss of back propagation, the network weights are updated in the direction of loss reduction during the training process. After multiple iterations, the loss function tends to converge, and the obtained network is the result of formal training. Beneficial effects of the present invention:
[0037] This paper designs a method for combining infrared and visible light image fusion for target detection. The resulting fused image is both rich in information and improves the average accuracy of target detection tasks. This fusion of infrared and visible light images better complements target detection tasks and serves target detection applications. Compared to independently designed fusion methods, this method achieves better results for high-level visual tasks, thus offering broader practical applications.
[0038] The image fusion method of the present invention is highly efficient and accurate. The network for the infrared and visible light image fusion method of the present invention is easy to train and can quickly output fusion results for target detection, making it more efficient than other methods. When the TDAL method and the present invention were trained on the same dataset, the present invention achieved higher image quality evaluation and target detection metrics, resulting in higher accuracy.
[0039] The present invention introduces the attention mechanism into the fusion network, effectively improving the saliency of the fused image target.
[0040] The present invention provides network weights trained on a standard dataset. Based on these weights, the fusion network and the detection network can generate a fused image and detect targets within it. Thus, the present invention takes infrared and visible light images as input and outputs a fused image and a corresponding target detection frame.
[0041] The present invention uses a convolutional neural network, which offers high precision and efficiency. Since most target detection applications require real-time detection, the infrared and visible light image fusion method must be highly efficient. Therefore, the present invention uses a convolutional neural network, which is easy to train, to achieve both high precision and high real-time performance and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 This is a schematic diagram of the overall network structure of the present invention.
[0043] Figure 2 This is a schematic diagram of the infrared and visible light image fusion network structure of the present invention.
[0044] Figure 3 It is a flow chart of the present invention.
[0045] Figure 4 Schematic diagram of visible light image.
[0046] Figure 5 Schematic diagram of infrared light image.
[0047] Figure 6 Schematic diagram of fused image.
[0048] Figure 7 This is the detection result of the fused image.
[0049] Figure 8 This is the output result folder of the target detection network. DETAILED DESCRIPTION
[0050] The present invention will be described in further detail below with reference to the accompanying drawings.
[0051] The structure of the present invention is described as follows: Figure 1 As shown:
[0052] This paper designs a real-time model for joint target detection using convolutional neural networks to fuse infrared and visible light images. This model rapidly generates fused images and enables subsequent target detection. The key to this approach lies in the design of the infrared and visible light image fusion method. The target detection model utilizes YOLO v5 (You Only Look Once), a popular single-stage target detection algorithm, which offers advantages such as lightweight and real-time performance.
[0053] In the fusion network, this paper builds a convolutional neural network framework, combining the fusion of infrared and visible light images with the object detection task. Within the overall framework, the image fusion module and the object detection module are cascaded, allowing the detection losses of the object detection module to flow back to the image fusion module, effectively improving the performance of object detection on the fused image.
[0054] The method steps of the present invention are as follows:
[0055] 1) Using an existing dataset, we loaded visible light images and image label files into the YOLO v5 model. We then performed data augmentation and convolutional layer modules on the network while simultaneously calculating the loss function. Based on the backpropagation loss, we adjusted the network weights during training to minimize the loss. After multiple iterations, when the loss function converged, we used the parameters at that point to form a pre-trained detection network.
[0056] 2) The visible light and infrared light images are input to the input of the fusion network. The images pass through the feature extraction network module and the image reconstruction module to train the fusion network. At this time, the pre-trained detection network is used to generate the detection loss. The detection loss is returned and the fusion loss is used to train the fusion network together.
[0057] 3) The fusion network outputs the generated fusion image, which is then input into the detection network for formal training of the detection network to obtain the training weights of the detection network; after completing all steps, the fusion network and detection network with trained network parameters are obtained.
[0058] Principles of the method steps:
[0059] 1. Loss function principle
[0060] In the field of deep learning, loss functions are primarily used during the model training phase. After each batch of training data is fed into the model, a predicted value is output through forward propagation. The loss function then calculates the difference between the predicted value and the true value, or the loss value. After determining the loss value, the network updates various parameters through backpropagation to reduce the loss between the true value and the predicted value, bringing the model's predicted value closer to the true value, thereby achieving the learning goal.
[0061] The fusion network training part of the present invention uses a fusion loss function and a detection loss function. The fusion loss function includes two parts: pixel loss and gradient loss. The pixel loss limits the intensity of the pixel fusion image to be consistent with the source image, while the gradient loss forces the fusion image to contain more detailed information. In order to make the fusion image contain more effective information, the present invention uses the mask information extracted from the source image in the calculation part of the fusion loss function. Mask (mask, mask) is a common operation in deep learning. Simply put, it is equivalent to covering the original tensor with a mask, thereby shielding or selecting some specific elements, so it is equivalent to constructing a filter for the tensor. Since infrared images usually contain more significant target information and are easier to extract effective specific elements, the mask extracted from the infrared image is used in the calculation of the loss function. Under the restriction of the loss function, the information of the fused image can be defined as follows:
[0062] I result =I mask I ir +(1-I mask )·I vi
[0063] Among them I result is the fused image, I mask is the mask information extracted from the infrared light source image, I ir is the infrared light source image, I vi During the training phase, the detection loss of the target detection network is fed back to the fusion network and used together with the fusion loss to train the fusion network. This allows the fusion network to be combined with the detection network to obtain an infrared and visible light fusion model that is more suitable for target detection.
[0064] 2. Image fusion structure principle
[0065] The present invention uses the structure of a convolutional neural network in the fusion network structure. The fusion framework based on the convolutional neural network can realize implicit feature extraction, fusion and image reconstruction under the guidance of a complex loss function. During the training process, the fused image continuously preserves the thermal radiation information from the infrared image and the gradient information from the visible light image.
[0066] like Figure 2 The figure shows the convolutional neural network architecture used by the fusion network to extract information from each modality. Within this architecture, a dense network (DenseNet) module is designed and connected in parallel with a channel attention module to enhance important information from the source image. This parallel architecture enables feature reuse between the main dense stream and its tributaries, enhancing the fusion network's ability to describe fine-grained spatial details. The dense network primarily uses the main stream for feature extraction, deriving its name from its ability to establish dense connections between all previous layers and subsequent ones. Through this dense connection, the fusion network achieves feature reuse, resulting in superior feature extraction performance.
[0067] The attention mechanism essentially aims to apply human perception and attentional behavior to machines, allowing them to learn to perceive important and unimportant parts of the data. In visual systems, the attention mechanism can be viewed as a dynamic selection process. The core idea is to identify correlations within the original data and then highlight certain key features. Methods can be categorized as channel attention, pixel attention, and multi-level attention, achieving this by adaptively weighting features based on their importance. This architecture utilizes a channel attention mechanism in the tributary, adding attention to the channel dimension. It uses automatic learning, using a new neural network, to determine the importance of each channel in the feature map. This weight is then used to assign a value to each feature, allowing the neural network to prioritize certain feature channels. This promotes channels in the feature map that are useful for the task at hand and suppresses channels that are less useful. This allows the main neural network to focus on channels of high importance, allowing feature extraction to retain more important features.
[0068] The fusion network's feature extraction architecture, through a cascade of dense network modules and channel attention modules, effectively extracts important information from the source images. By passing the features extracted from both images through an image reconstruction module, the fusion network can output a fused image that retains the important features of the original images.
[0069] Another technical solution of the present invention:
[0070] 1) Part of the fusion network structure of the present invention can be replaced. To achieve the invention's objective of "infrared and visible light image fusion for joint target detection," the fusion network can be replaced with other structures, such as generative adversarial networks, autoencoders, or other convolutional neural network structures. Replacing the fusion portion and mimicking the training method of the present invention may achieve the objectives of the present invention.
[0071] 2) The detection network of the present invention can be replaced. While the present invention focuses on designing a fusion network, the detection network uses the existing lightweight one-stage model YOLO v5. There are many mature detection models in the field of object detection, and new detection models are being rapidly developed. Combining the fusion network with other detection models may achieve the objectives of the present invention.
[0072] 3) The training steps of the present invention can be replaced. The present invention adopts the three steps of pre-training the detection network and training the two networks in stages. In the multi-task training of neural networks, other training steps can also be adopted, such as training multiple networks simultaneously.
[0073] The present invention provides network weights that have been trained on a standard data set. Figure 3 As shown in the flowchart, infrared and visible light images are fed into the network based on the weights of the fusion network. The network, composed of a dense network and an attention mechanism, generates a fused image. Based on the weights of the object detection network, the fused image passes through the detection network to generate detection results for the fused image. The detection results include a bounding box on the image, its corresponding confidence score, and a label file containing the coordinates of the bounding box on the image. This approach implements the process of inputting infrared and visible light images and outputting a fused image and the object detection results for it.
[0074] Figure 4 and Figure 5 is the visible light image and infrared light image at the input end of the present invention, corresponding to Figure 3 The input image in . Figure 6 and Figure 7 They are the fusion image and its target detection result map, corresponding to Figure 3 The output image of the fusion network and the output image of the object detection network.
[0075] Figure 8 This is the output folder of the object detection network of this invention. For each pair of input images, the output folder contains one image and one label file. The image is attached with a detection box and its corresponding confidence level, and the label file contains the coordinate information of the detection box on the image.
Claims
1. A method for fusion of infrared and visible light images for joint target detection, characterized in that: The following steps are included: 1) Using an existing dataset, load visible light images and image label files into the YOLO v5 model, perform data augmentation and convolutional layer modules on the network, and simultaneously calculate the loss function. Based on the backpropagation loss, adjust the network weights in the direction of reducing the loss during training. After multiple iterations, the loss function tends to converge, and the parameters at this time are used in the detection network to obtain a pre-trained detection network; 2) The visible light and infrared light images are input to the input of the fusion network. The images pass through the feature extraction network module and the image reconstruction module to train the fusion network. At this time, the pre-trained detection network is used to generate the detection loss. The detection loss is returned and the fusion loss is used to train the fusion network together. 3) The fusion network outputs the generated fusion image, which is then input into the detection network for formal training of the detection network to obtain the training weights of the detection network. After completing all steps, the fusion network and detection network with trained network parameters are obtained; The step 2) is specifically as follows: The infrared image and the visible light image are loaded into the fusion network. Both images are passed through the dense network module and the channel attention module of the feature extraction network respectively. The image features obtained by the two feature extraction modules are passed through the image reconstruction module to generate a fused image. The fused image, the input image, and the mask information are combined to calculate the fusion loss according to the fusion loss function formula. At the same time, the output fused image is loaded into the detection network. The detection network, which has been pre-trained to obtain parameters, returns the detection loss to the fusion network. The fusion loss function and the detection loss function jointly guide the training of the fusion network. The process is as follows: According to the back-propagation fusion loss and detection loss, the fusion network updates the network weights in the direction of decreasing the two types of losses during training. After multiple iterations, the two types of loss functions tend to converge. The parameters at this time are used as the fusion network parameters to obtain the trained infrared and visible light image fusion network; The fusion network includes a feature extraction network module and an image reconstruction module. The feature extraction of the fusion network is a parallel dense network module and a channel attention module. The dense network (Densenet) module is connected in parallel with the channel attention module to increase important information in infrared images and visible light images. The mainstream dense network module implements feature extraction; the tributary attention mechanism highlights important features; the parallel structure reuses features through the main dense stream and tributaries to enhance the fusion network's ability to describe fine-grained spatial details. The dense network module includes multi-scale convolution, activation functions, and residual connections to establish dense connections between all previous layers and subsequent layers. Through dense connections, the fusion network reuses features, thereby achieving better feature extraction performance. The attention mechanism is a dynamic selection process that adaptively weights features based on the importance of the input, and can retain more important features. The image reconstruction module includes multi-scale convolution and activation functions, and generates a fused image by passing the image features obtained by the two feature extraction branches through the image reconstruction module.
2. The infrared and visible light image fusion method for joint target detection according to claim 1, characterized in that: In step 1), pre-training is performed in the early stage to assist the formal training in the subsequent steps; by pre-training the detection network, the parameters of the detection network form the detection loss, which assists the training of the fusion network in the subsequent steps.
3. The infrared and visible light image fusion method for joint target detection according to claim 1, characterized in that: The formula of the infrared image and visible light image fusion network is expressed as: {F ir ,F vi }={E F (I ir ),HAVE BEEN F (I vi )} I f =R I (C(F ir ,F vi )) I ir ,I vi is the input infrared and visible light images, E F is the feature extraction network, F ir ,F vi is the extracted infrared and visible light image features, R I is the image reconstruction module, C is the feature splicing, I f is the output fused image.
4. The infrared and visible light image fusion method for joint target detection according to claim 1, characterized in that: The training part of the fusion network uses a fusion loss function and a detection loss function; The fusion loss function consists of two parts: pixel loss and gradient loss. The pixel loss limits the intensity of the pixel fusion image to be consistent with the source image, while the gradient loss forces the fusion image to contain more detailed information. The fusion loss function extracts mask information from the infrared image. Under the constraints of the loss function, the information of the fused image can be defined as follows: I result =I mask ·I ir +(1-I mask )·I vi Among them I result is the fused image, I mask is the mask information extracted from the infrared light source image, I ir is the infrared light source image, I vi For visible light source images, during the training phase, the detection loss of the target detection network flows back to the fusion network, and the fusion network is trained together with the fusion loss, so that the fusion network is combined with the detection network to obtain an infrared and visible light fusion network that is more suitable for target detection.
5. The infrared and visible light image fusion method for joint target detection according to claim 1, characterized in that: The formal training of step 3) is: The fused image and image label files are loaded into the main program of the YOLO v5 model. The loss function is calculated simultaneously through the network's data augmentation module and convolutional layer module. According to the backpropagation loss, the network weights are updated in the direction of decreasing loss during the training process. After multiple iterations, the loss function tends to converge, and the resulting network is the result of formal training.