Infrared and visible light image fusion detection method based on cross attention mechanism

Abstract

The invention discloses an infrared and visible light image fusion detection method based on a cross attention mechanism, and the method comprises the steps: obtaining a multi-source heterogeneous known scene image, carrying out the classification and marking, and building a standard data set; constructing an image detection fusion network, and performing iterative training on the image detection fusion network based on the standard data set to obtain an ideal fusion model for target detection in the multi-source heterogeneous scene; and obtaining a to-be-detected scene image, inputting the ideal fusion model for target detection, and determining whether the target object is included according to the output of the ideal fusion model. The method focuses on solving the influence of neglecting infrared and visible light complementary features on original feature extraction in the fusion process, constructs an end-to-end network structure at the same time, achieves the detection of the fusion image, and provides an objective basis for the judgment of the fusion quality.

Description

Technical Field

[0001] This invention belongs to the field of image information processing technology, specifically relating to a method for infrared and visible light fusion detection based on a cross-attention mechanism. Background Technology

[0002] Due to the spectral sensing characteristics of imaging sensors, a single sensor type cannot capture all spectral information of the entire scene, making target detection susceptible to interference from environmental factors. For example, infrared sensors capture the thermal radiation information of objects and are sensitive to objects with large temperature differences, but cannot obtain the texture details of objects and scenes. Conversely, visible light sensors capture the visible light reflected by objects and can obtain rich texture and structural information, but are easily affected by environmental factors. Therefore, fusing infrared and visible light images can produce an image that contains both target thermal radiation information and rich texture information, which is helpful for target detection in complex scenes.

[0003] Currently, image fusion algorithms are mainly divided into traditional fusion methods and deep learning-based fusion methods. Although traditional fusion methods have achieved certain results, the fused images still suffer from problems such as blurriness, unstable fusion, and poor visual perception. With the continuous development of deep learning, its application in the field of image fusion is becoming increasingly widespread. Currently, deep learning-based fusion methods mainly include those based on convolutional neural networks (CNNs), autoencoders (AEs), generative adversarial networks (GANs), and visual attention mechanisms (Vision Transformers). Among deep learning methods, fusion methods based on CNNs and autoencoders mainly rely on convolutional blocks for feature extraction, thus often ignoring the different texture and detail information between infrared and visible light images, leading to insufficient fusion. Fusion methods based on generative adversarial networks, due to their adversarial training method, are prone to instability and cannot achieve ideal fusion results. Therefore, in recent years, the fusion network has mainly adopted Transformers.

[0004] Although Transformer has achieved satisfactory results in the field of image fusion, there are still some shortcomings: (1) Existing attention-based fusion networks do not fully consider the influence of complementary features on the extraction of original features. The feature extraction of infrared images and visible light images is carried out independently and lacks correlation, such as YDTR, Densefuse and other methods; (2) Most fusion algorithms do not carry out further work after fusion, and only rely on various evaluation indicators to judge the fusion quality, and cannot determine the actual help of the fused image to subsequent work. Summary of the Invention

[0005] To address the aforementioned shortcomings, the present invention aims to focus on resolving the impact of neglecting complementary infrared and visible light features on the extraction of original features during the fusion process. Simultaneously, it constructs an end-to-end network structure to detect the fused image, providing an objective basis for judging the fusion quality.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: an infrared and visible light image fusion detection method based on a cross-attention mechanism, which performs the following steps to achieve image detection and recognition:

[0007] Step S1: Acquire multi-source heterogeneous known scene images and classify and label them to establish a standard dataset;

[0008] Step S2: Construct an image detection fusion network and iteratively train the image detection fusion network based on a standard dataset to obtain an ideal fusion model for target detection in multi-source heterogeneous scenes;

[0009] Step S3: Obtain the scene image to be detected, input the ideal fusion model for object detection, and determine whether the target object is contained based on the output of the ideal fusion model.

[0010] In step S1, multi-source heterogeneous scene images containing various detection objects are captured by an image acquisition sensor set in a real scene at different angles and under different lighting conditions; and the objects are classified into N types according to known object classification labels, and the target objects in the images are labeled with bounding boxes to obtain a standard dataset.

[0011] The multi-source heterogeneous images are visible light color images and infrared thermal images.

[0012] In step S2, the image detection fusion network is constructed, including a preprocessing module, a primary feature extractor, a secondary feature extractor, a fusion feature extraction module, and a classifier. The preprocessor is used to initially extract features from multi-source heterogeneous known scene images. The primary feature extractor is used to learn the interrelationships between heterogeneous features. The secondary feature extractor is used to learn source feature details by combining image space and channels. The fusion feature extraction module is used to fuse the output features of the primary and secondary feature extractors and extract deep channel features for output to the classifier. The classifier is used to learn deep features, fit the interrelationships between heterogeneous features, the correlation between source feature details and classification labels, identify targets in the current scene image, and output the target category and location.

[0013] The preprocessor employs multiple convolutional layers in series to process multi-source heterogeneous images I. vis and I ir The single-source features are extracted through calculation to obtain the visible light features f. vis and infrared features fir .

[0014] The main feature extractor includes infrared and visible light channels. Each channel is implemented using three visual attention modules (Transform), and the channels are cross-connected to fully consider their corresponding complementary features when extracting features from infrared or visible light images. The main feature extractor learns the heterogeneous features f input. vis and f ir The interrelationships between them yield the main feature f. p .

[0015] The secondary feature extractor is implemented through a fusion spatial-channel attention mechanism, including:

[0016] a. Through channel attention mechanism, heterogeneous visible light features of the input are analyzed. vis and infrared features f ir By fusing the data, channel features are obtained.

[0017] b Combining heterogeneous features f vis and f ir Further processing was performed to obtain enhanced visible light characteristics. and infrared right and Further image addition operations are used to obtain features. Features Input spatial attention mechanism module to extract spatial fusion features

[0018]

[0019] c Spatial fusion characteristics Continue in features and The spatial features are then refined and added together to obtain the final output secondary feature f. s .

[0020] The fusion feature extraction module consists of N parallel convolutional and pooling layers, used to extract the main feature f. p Secondary features f s The fusion is performed to obtain the fusion feature f. fusion It learns and generates N layers of features at different depths, which are then output to the classifier; the loss function during training includes structural similarity (SSIM) and fusion quality (Q). abf SSIM calculates the loss of texture detail in image brightness, contrast, and structure. abf Used for loss calculation of the main objective.

[0021] The classifier adopts the detector head and classifier head structure in the Faster-RCNN network model to realize the recognition function of the Region Candidate Network (RPN) and the Region of Interest Pooling (RoIPooling) parts, which are used to locate and identify target objects and generate classification results and candidate boxes.

[0022] The present invention has the following beneficial effects and advantages:

[0023] 1. The main feature generator used in this invention employs a cross-attention mechanism to extract fused features from infrared and visible light images. This helps to fully consider the influence of complementary features when extracting features from a single channel, thereby enhancing the accuracy of the main features.

[0024] 2. The secondary feature extractor used in this invention employs a fusion spatial and channel attention mechanism, which no longer performs feature extraction on a single infrared or visible light channel, reducing the number of parameters in the overall model and facilitating its practical application.

[0025] 3. This invention designs a complete fusion detection system rather than a separate fusion, which can be directly applied to actual target detection and can effectively improve the accuracy and recall rate of target detection in harsh environments. Attached Figure Description

[0026] Figure 1 This is a diagram of the infrared and visible light image fusion detection network structure.

[0027] Figure 2 This is a schematic diagram of the attention module (MHA) structure;

[0028] Figure 3(a) shows the visible light detection and recognition image;

[0029] Figure 3(b) shows the infrared light detection and recognition image;

[0030] Figure 3(c) shows the fusion detection and recognition image; Detailed Implementation

[0031] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0033] To achieve the above objectives, this invention provides an infrared and visible light image fusion detection method based on an end-to-end cross-attention mechanism to improve the accuracy of target detection in different scenarios. The method mainly includes the following steps: preprocessing the infrared and visible light images to extract preliminary features; constructing an end-to-end cross-attention network, which includes a primary feature extractor and a secondary feature extractor; a fusion feature extraction module, which can extract high-level features such as edges, textures, and shapes. These features are crucial for target detection, helping to identify the contours and key parts of objects, and simultaneously outputting five-channel features for use in the subsequent target detection module (classifier); and constructing a target detection module to detect and classify the fused image.

[0034] In the image preprocessing section, a two-part convolution and ReLU activation function are used to initially extract features from visible light and infrared images, obtaining f. vis and f ir Next, f will be placed separately. vis and f ir The features f are obtained by feeding them into the primary and secondary feature extractors. p and f s The main feature extractor includes infrared and visible light channels, which are cross-connected to ensure that complementary features are fully considered when extracting features from infrared or visible light images. Each channel relies on three visual attention modules (Transform). The infrared and visible light dual-branch channels are independent of each other, receiving the same input but different outputs during training, ensuring the extraction of key features from different images. The main components include a normalization layer, a multi-head attention mechanism, and a fully connected layer. The specific implementation is as follows:

[0035] 1. Receive the input visible light characteristics f vis ∈R H×W×C and infrared features f ir ∈R H×W×C Features are obtained by passing through the first attention module respectively. and

[0036] 2. The input received by the second attention module for both the infrared and visible light channels is:

[0037]

[0038] Thus, the features are obtained. and

[0039] 3. The input data received by the third attention module is:

[0040]

[0041] Obtain features and This leads to the final output of the main feature extractor:

[0042]

[0043] The secondary feature extractor is mainly implemented through a fusion spatial-channel attention mechanism, and the specific steps are as follows:

[0044] 1. Receive the same input as the main feature extractor, i.e., visible light feature f vis ∈R H×W×C and infrared features f ir ∈R H×W×C First, the features are fused across channels, and then the fused channel features are extracted using a channel attention mechanism. The formula is as follows:

[0045]

[0046]

[0047] Where σ represents the ReLU activation function, MLP is a shared perceptron, AvgPool represents global average pooling, and MaxPool represents global max pooling.

[0048] 2. The obtained channel fusion features are further processed to obtain the features. and Since the spatial attention mechanism is not sensitive to channels, the enhanced features are obtained by image addition. Then extract spatial fusion features

[0049]

[0050]

[0051]

[0052]

[0053] Where σ represents the sigmoid activation function, conv7×7 This indicates that a 7x7 convolution operation is performed on the result. AvgPool represents global average pooling, and MaxPool represents global max pooling.

[0054] 3. After obtaining the spatial fusion features, continue to refine the features... and The spatial features are further refined to obtain the secondary features f of the output. s The specific formula is as follows:

[0055]

[0056]

[0057]

[0058] The primary features (such as people, vehicles, scenery, and objects) and secondary features (such as texture, color, and details) are obtained after fusion, and then the final fusion is performed. The fusion formula is as follows:

[0059]

[0060] The fusion feature extraction module consists of five parallel convolutional and pooling layers, used to extract the fusion feature f. fusion Deep extraction is performed, generating five layers of features at different depths, which are then output to the target detection module.

[0061] The object detection module primarily employs the detection and classification heads from the Faster R-CNN algorithm, implementing two main functions: the Region Proposal Network (RPN) and Region of Interest Pooling (RoIPooling), to generate candidate bounding boxes. This task comprises two aspects: classification to determine the presence of an object within an anchor, and bounding box regression to refine anchors and obtain more accurate proposals. RoIPooling collects the proposals (coordinates of each box) generated by the RPN and extracts corresponding feature maps from the fused infrared and visible light features, generating proposal feature maps which are then fed into subsequent fully connected layers for further classification and regression.

[0062] For overall network training, this invention employs a two-stage training approach. First, the fusion feature extraction module is trained, using loss functions including structural similarity (SSIM) and fusion quality (Q). abf SSIM primarily calculates the loss of texture detail in image brightness, contrast, and structure. abf It is mainly used to calculate the loss of the main target. The specific loss function is as follows:

[0063]

[0064] Where α represents the visible light weight, which was set to 0.8 in experiments; β represents the weight of the infrared image, which was designed to be 0.2 in experiments; and SSIM represents image structural similarity. Q represents any three-channel feature in the fused features. abf Visible light image features f vis Infrared image features f ir .

[0065] In the training of the object detection module, the weights of the fusion module are frozen, and the original loss function of Faster-RCNN is used for detection and classification. This method ensures that the fused features of infrared and visible light images can be effectively utilized in both extraction and detection processes, thereby improving the accuracy and reliability of object detection.

[0066] This invention requires manually labeling the target location and category of the acquired raw visible light and infrared images for subsequent training.

[0067] like Figure 1 The diagram shows the structure for infrared and visible light image fusion detection. The solid lines represent the feature extraction and fusion process, while the dashed lines represent target detection based on the fused features. This invention extracts preliminary features using convolutional blocks, which are then fed into a cross-attention module and channel and spatial attention modules to extract and fuse the primary features (such as people, vehicles, and objects) and secondary features (texture details, color) of the infrared and visible light images, respectively. Because this invention fully considers complementary features during the fusion process, the final fused image retains the best features for that point in both images, preserving the most useful information for subsequent detection modules.

[0068] like Figure 2 As shown, the network structure of a single attention module in the cross-attention mechanism includes embedding patches, a normalization layer, a multi-head attention mechanism, and a fully connected layer (MLP). The embedding layer is primarily responsible for adding positional information to the input image, enabling correlation between different image patches. The normalization layer calculates the mean and variance of all feature maps for the input sample and then normalizes the input sample. The multi-head attention mechanism focuses on complementary image features in the infrared and visible light images, thereby achieving subsequent fusion. The fully connected layer is responsible for expanding the feature dimension to ensure that the input and output dimensions are the same.

[0069] As shown in Figures 3(a), 3(b), and 3(c), the overall fusion detection network output results show that the network retains the texture details and color features of the visible light image, while also fusing thermal information (such as the human body) in the infrared image to a certain extent. This enables the final fusion result to detect people in the fog with an accuracy rate of 99%. The individual image fusion results are also better than the existing algorithms, as shown in Table 1.

[0070] Table 1

[0071]

[0072] Where Qabf represents fusion quality, EN represents fusion information entropy, VIF represents visual fidelity, MI represents mutual information, and SF represents the spatial frequency of the fused image. All of these values ​​are better the larger they are.

[0073] This invention proposes an end-to-end infrared and visible light image fusion detection network that can be directly applied to real-world scenarios. The key technical solutions are: employing a cross-attention mechanism to extract the main target features from the two images, which facilitates the correlation between features and enhances the sufficiency of the fusion; and employing a spatial-channel attention mechanism to extract secondary features, which better focuses on texture details and color features in the visible light image, contributing to the final confirmation of the detection scene.

[0074] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should be considered within the scope of protection of the present invention.

Claims

1. An infrared and visible image fusion detection method based on a cross-attention mechanism, characterized in that, The image detection recognition is realized by executing the following method steps, and the method comprises the following steps: Step S1: acquiring multi-source heterogeneous known scene images and classifying and labeling, and establishing a standard data set; Step S2: constructing an image detection fusion network, and iteratively training the image detection fusion network based on the standard data set to obtain an ideal fusion model for target detection in a multi-source heterogeneous scene; Step S3: acquiring a scene image to be detected, inputting the ideal fusion model for target detection, and determining whether the ideal fusion model contains a target object according to the output of the ideal fusion model.

2. The method of claim 1, wherein, In step S1, an image acquisition sensor arranged in a real scene is used to capture multi-source heterogeneous scene images containing various detection objects under different angles and light conditions; and a standard data set is obtained by labeling the target objects in the images with a boundary box according to the known object classification labels for N object types.

3. The method of claim 1, wherein, The multi-source heterogeneous images are visible light color images and infrared thermal images.

4. The method of claim 1, wherein, In step S2, the image detection fusion network comprises a preprocessing module, a main feature extractor, a secondary feature extractor, a fusion feature extraction module, and a classification recognizer; the preprocessor is used to preliminarily extract features of the multi-source heterogeneous known scene images, the main feature extractor is used to learn the mutual correlation between heterogeneous features, and the secondary feature extractor is used to learn source feature details in combination with image space and channels; The fusion feature extraction module is used to fuse and deeply extract deep channel features of the output features of the main feature extractor and the secondary feature extractor and output them to the classification recognizer; the classification recognizer is used to learn deep features, fit the correlation between heterogeneous features and the correlation degree between source feature details and classification labels, recognize targets in the current scene image, and output target categories and positions.

5. The method of claim 4, wherein, The preprocessor adopts multiple convolution layers in series, respectively calculates and extracts single-source features from the multi-source heterogeneous images I vis and I ir , to obtain visible light features f vis and infrared features f ir .

6. The method of claim 4, wherein, The main feature extractor includes an infrared channel and a visible light channel, each channel is implemented by three visual attention modules Transform, and the channels are cross-connected, so as to fully consider the corresponding complementary features when extracting the features of the infrared image or the visible light image; the main feature extractor learns the correlation between the input heterogeneous features f vis and f ir , and obtains the main features f p .

7. The method of claim 4, wherein, The secondary feature extractor is realized by fusing a space-channel attention mechanism, which comprises: a fusion of the input heterogeneous visible light features f vis and infrared features f ir through a channel attention mechanism to obtain channel features b pair combined heterogeneous features f vis and f ir deeper processing to obtain enhanced visible light features and infrared pair and further using image addition operation to obtain features features f input into a spatial attention mechanism module to extract spatial fusion features c spatially fusing features continuing to perform deepening, adding operations on the spatial features and further obtaining final output secondary features f s .

8. The method of claim 4, wherein, The fusion feature extraction module includes N layers of parallel convolutional layers and pooling layers, which are used to fuse the main feature f p , the secondary feature f s to obtain the fusion feature f fusion , and learn to generate N layers of different depth feature outputs for the classification identifier; in the training, the loss function includes structural similarity SSIM and fusion quality Q abf , the SSIM is used for loss calculation of image brightness, contrast, structure and texture details, and the Q abf is used for loss calculation of the main target.

9. The method of claim 4, wherein, The classification recognizer adopts the detection head and classification head structure in the Faster-RCNN network model to realize the recognition functions of the region proposal network RPN and the region of interest pooling RoI Pooling, and is used to locate and recognize target objects and generate classification results and candidate boxes.

Images