An attention mechanism-based dual-branch infrared and visible light image fusion method and system

Abstract

The application discloses a kind of dual-branch infrared light and visible light image fusion method and system based on attention mechanism.The purpose is to solve the feature extraction cannot establish long-distance dependence relationship, the noise is too high in the process of infrared image feature extraction, the problem of blurred edge information after image fusion in the existing infrared light and visible light image fusion technology.It includes: design a dual-branch network.Use dual attention mechanism (channel attention mechanism and spatial attention mechanism) module to carry out multi-scale feature extraction to the input infrared image and visible light image.Transform module is designed to capture global context information from source image, which enables the model to capture long-distance dependence relationship.Edge feature enhancement module is used to enhance the edge features of the image, and the output image of the Transform module is added.Finally, through multi-scale fusion block, the two images are fused to obtain the output fusion image.The application can effectively improve the quality of the fusion image and achieve good fusion effect.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, specifically to a method and system for fusing infrared and visible light images. Background Technology

[0002] With the continuous development of the information age, we are faced with a large amount of data from different sources, and people have increasingly diverse technical means to obtain relevant data information. With the rapid development of deep learning-related technologies, multimodal image fusion has always been an important research direction in the field of computer vision.

[0003] Visible light images are among the most common types of images we encounter in our daily lives. Visible light images reflect the reflective properties of an object's surface, have high spatial resolution, and contain clear details and textures. They are rich in detail and are more easily understood by the human visual system. However, their imaging process is highly dependent on the external environment. While they are beneficial for human vision to perceive scenes, their imaging quality is highly dependent on ambient lighting conditions, and the imaging effect will decrease under low light conditions.

[0004] Infrared imaging is highly dependent on the thermal radiation of the target object and is not sensitive to external light. It can acquire image information of the target object in low-light and weak-light environments. The higher the temperature of the target object, the more favorable it is for infrared imaging. It is particularly effective in expressing information of prominent targets such as people, buildings, and vehicles. This type of image has extremely strong penetrating power and is not affected by environmental interference, which reduces its imaging ability. It is especially suitable for nighttime reconnaissance and work in inclement weather and has all-weather operation capability.

[0005] How can we effectively integrate the valuable information from infrared and visible light images using appropriate fusion rules to obtain a high-quality image containing more complete information? Fusion of infrared and visible light images can provide more comprehensive and accurate target information, improving military reconnaissance, surveillance, and target identification capabilities. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings and deficiencies of the existing technology by providing an infrared and visible light image fusion method. This method aims to solve a series of problems in existing infrared and visible light image fusion technologies, such as the inability to establish long-range dependencies in feature extraction, excessive noise during infrared image feature extraction, and blurred edge information after image fusion.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a method for fusing dual-branch infrared and visible light images based on an attention mechanism, comprising the following steps:

[0008] Two source images, an infrared image and a visible light image, are input into the dual attention mechanism feature extraction block of a dual-branch network to obtain multi-scale features of the two source images;

[0009] The multi-scale features of the infrared and visible light images are input into the Transformer module to capture the global context information of the image and establish long-range dependencies.

[0010] Edge feature enhancement is performed on the feature map that establishes long-distance dependencies, and the image before enhancement is added to the image after enhancement;

[0011] The infrared image and the visible light image are fused using multi-scale features to obtain a high-quality fused image.

[0012] The multi-scale features of the two source images are obtained as follows:

[0013] The dual-attention mechanism feature extraction block combines channel attention and spatial attention mechanisms to form a feature extraction and storage module;

[0014] The feature extraction and storage module includes a residual block, a channel attention mechanism block, and a spatial attention mechanism block.

[0015] The residual block (REB), channel attention block (CAMB), and spatial attention block (SAMB) include:

[0016] REB consists of three convolutional layers with a kernel size of 5*5, two batch normalization layers, and one ReLU activation layer to extract shallow features from the image.

[0017] CAMB consists of three convolutional layers with a kernel size of 3*3, one global average pooling layer, one ReLU activation layer, one Sigmoid activation layer, and two fully connected layers.

[0018] SAMB consists of three convolutional layers with a kernel size of 3*3, one ReLU activation layer, and one Sigmoid activation layer.

[0019] The feature extraction expression includes:

[0020] F1 = REB(I)

[0021] Where I is the input image of the dual attention mechanism feature extraction module, where I is converted from the input RGB source image into a grayscale image with only the brightness channel, and the grayscale image is cropped to a size of 128*128. F1 is the result of the preprocessed source image after passing through the residual block (REB). Since this structure is a dual-branch structure, I represents the preprocessed visible light image and infrared image in the two branches, respectively.

[0022] F2 = CAMB(F1)

[0023] F2 is the output of F1 after passing through the Channel Attention Block (CAMB);

[0024] F3 = SAMB(F1)

[0025] F3 is the output of F2 after passing through the Spatial Attention Block (SAMB); in the above process, F1, F2, and F3 are all four-dimensional tensors with 128 channels;

[0026] The output of the dual attention mechanism feature extraction module is as follows:

[0027] F = F1 + (F1 * F2) + (F1 * F3).

[0028] Furthermore, the Transformer module of the present invention includes:

[0029] The Transformer module consists of three layers of normalization, a multi-head self-attention mechanism, and a multilayer perceptron, and performs multiple residual connections on it.

[0030] Furthermore, the Transformer module of the present invention includes:

[0031] The input feature image is divided into non-overlapping windows of size M*M, generating a... feature.

[0032] in This represents the total number of windows. All of these windows undergo standard self-attention computation. In the self-attention mechanism, features within each window are correlated with features in other windows to capture dependencies between global features.

[0033] Furthermore, the self-attention mechanism described in this invention includes:

[0034] This self-attention calculation is typically achieved by multiplying by the weight matrix of the query, key, and value. The resulting attention score is then processed by the Softmax function to obtain normalized weights. Finally, the values ​​are multiplied by the weights and summed to obtain the final self-attention output.

[0035] Furthermore, the correspondence between the query (Q), key (K), and value (V) and the window feature (X) of the self-attention mechanism described in this invention is as follows:

[0036] Q = X·W Q ,

[0037] K = X·W K ,

[0038] V = X·W V ,

[0039] Where X represents the input window feature, W Q W K and W V Projection matrices for Q, K, and V, respectively, are used to map the input features to higher-level feature representations. These matrices map X to a space of query (Q), key (K), and value (V), with dimensions of [missing information]. Where d is the feature dimension, Q, K,

[0040] Attention value is calculated as follows:

[0041]

[0042] d is the scaling factor. S(·) represents the Softmax function, d represents the dimension, and B represents the learnable relative position encoding.

[0043] Furthermore, the edge feature enhancement described in this invention includes:

[0044] The edge feature enhancement process employs compressed attention and residual attention mechanisms to focus on important information and ignore irrelevant information, and then performs addition and subtraction operations.

[0045] One of the modules is an edge feature extraction module, which aggregates and integrates features from local areas to extract more meaningful features and reduce unnecessary interference and noise. This ultimately achieves edge feature enhancement.

[0046] Furthermore, the edge feature extraction module of the present invention comprises the following components:

[0047] It contains two 1x1 convolutional layers to adjust the channels, five 3x3 convolutional layers, two LReLU activation layers, and residual connections to them.

[0048] Furthermore, the feature fusion process of the present invention includes:

[0049] Feature fusion involves three convolutional layers of the same size to reduce the number of image channels, and incorporates the ELU activation function, which helps reduce the vanishing gradient problem and makes the model more stable. Finally, element-wise addition is performed to obtain the fused output image.

[0050] The present invention also provides a dual-branch infrared and visible light image fusion system based on an attention mechanism, characterized in that the system comprises the following:

[0051] Feature extraction module: The feature extraction module uses channel attention mechanism and spatial attention mechanism to extract features from the input source image at multiple scales;

[0052] Transformer module: Captures image context information and establishes remote dependencies;

[0053] Edge feature enhancement module: Employs compressed attention mechanism and residual attention mechanism to focus on information in specific regions of the image and ignore irrelevant information to reduce interference;

[0054] Edge feature extraction module: used to aggregate features of local regions, thereby extracting richer and more representative features;

[0055] Feature fusion module: Gradually reduces the number of channels in two images that have undergone a series of processing steps, and performs element-wise addition operations. Further, the system of the present invention includes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 8.

[0056] F1=REB(I)F2=CAMB(F1)F3=SAMB(F1)F=F1+(F1*F2)+(F1*F3)Q=X·W Q K = X·W K V = X·W V

[0057] Beneficial effects:

[0058] 1. This invention effectively solves the problems of existing infrared and visible light image fusion techniques, such as the inability to establish long-range dependencies in feature extraction, excessive noise during infrared image feature extraction, and blurred edge information after image fusion. A dual-branch network is designed. A dual-attention mechanism (channel attention and spatial attention) module is used to extract multi-scale features from the input infrared and visible light images. The Transformer module is designed to capture global contextual information from the source images, enabling the model to capture long-range dependencies. An edge feature enhancement module enhances the edge features of the image and adds them to the output image of the Transformer module. Finally, a multi-scale fusion block fuses the two images to obtain the output fused image.

[0059] 2. This invention can effectively improve the quality of fused images and achieve good fusion results. Attached Figure Description

[0060] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0061] To facilitate the implementation of this invention, specific examples are provided below for further illustration. These embodiments are only used to more clearly illustrate the technical solution of this invention, but the scope of protection of this invention is not limited to the following embodiments.

[0062] Example 1

[0063] like Figure 1 As shown, this invention designs a dual-branch infrared and visible light image fusion method based on an attention mechanism, comprising the following steps:

[0064] Two source images, an infrared image and a visible light image, are input into the dual attention mechanism feature extraction block of a dual-branch network to obtain multi-scale features of the two source images;

[0065] The multi-scale features of the infrared and visible light images are input into the Transformer block to capture the global context information of the image and establish long-range dependencies.

[0066] Edge features are extracted from the feature maps that establish long-distance dependencies;

[0067] Enhance the extracted edge features;

[0068] Add the image before enhancement to the image after enhancement;

[0069] The infrared image and the visible light image are fused using multi-scale feature fusion to obtain a high-quality fused image. Specifically, the steps include:

[0070] Input the two source images into the feature extraction module to obtain the multi-scale features of the two source images;

[0071] The feature extraction module includes a channel attention block, a spatial attention block, a residual block, and skip connections.

[0072] The source image, after passing through the feature extraction module, outputs the following expression:

[0073] F1 = REB(I)

[0074] F2 = CAMB(F1)

[0075] F3 = SAMB(F1)

[0076] F = F1 + (F1 * F2) + (F1 * F3)

[0077] I represents the input image to the dual-attention mechanism feature extraction module. Here, I is converted from the input RGB source image into a grayscale image with only a luminance channel, and the grayscale image is cropped to a size of 128*128. F1 is the output result of the preprocessed source image after passing through the residual block (REB). F2 is the output result of F1 after passing through the channel attention block. F3 is the output result of F1 after passing through the spatial attention block. In the above process, F1, F2, and F3 are all four-dimensional tensors with 128 channels. F is the output result of the input source image after passing through the feature extraction module to obtain multi-scale features.

[0078] The multi-scale features of the source image are obtained and input into the Transformer block;

[0079] The Transformer block is composed of the following:

[0080] This module consists of three layers of normalization, a multi-head self-attention mechanism, and a multilayer perceptron, and is subjected to multiple residual connections.

[0081] The implementation details of the Transformer block are as follows:

[0082] The input feature image is divided into non-overlapping windows of size M*M, generating a... feature;

[0083] in This is the total number of windows, all of which undergo standard self-attention calculations.

[0084] In the self-attention mechanism, features within each window are correlated with features within other windows to capture the dependencies between global features.

[0085] Furthermore, the specific principle of the multi-head self-attention mechanism is as follows:

[0086] This self-attention calculation method typically processes the input in a weighted manner using a weight matrix of queries, keys, and values;

[0087] First, the query, key, and value are multiplied by their respective weight matrices. Then, the attention scores are normalized using the Softmax function to obtain the final weights.

[0088] Finally, the values ​​are multiplied by these normalized weights and summed to obtain the final self-attention output. The correspondence between the query (Q), key (K), and value (V) and the window feature (X) is as follows:

[0089] Q = X·W Q ,

[0090] K = X·W K ,

[0091] V = X·W V ,

[0092] Where X represents the input window feature, W Q W K and W V Projection matrices for Q, K, and V, respectively, are used to map the input features to higher-level feature representations. These matrices map X to a space of query (Q), key (K), and value (V), with dimensions of [missing information]. Where d is the feature dimension, Q, K,

[0093] Attention value is calculated as follows:

[0094]

[0095] d is the scaling factor. S(·) represents the Softmax function, d represents the dimension, and B represents the learnable relative position encoding.

[0096] To enhance image edge features, an edge feature enhancement module was designed, and its specific components are as follows:

[0097] The edge feature enhancement process utilizes compressed attention and residual attention mechanisms to focus on important information and ignore irrelevant information;

[0098] First, by employing a compressed attention mechanism, key edge features in the image are highlighted while irrelevant information is suppressed. Then, a residual attention mechanism is used to ensure an effective combination of the compressed key features and the original image features, thereby further enhancing the importance of edge features.

[0099] Finally, the processed edge features are optimized through addition and subtraction operations to ensure that the clarity and contrast of the image edge information are enhanced.

[0100] On the other hand, there is also an edge feature extraction module, which is used to aggregate and integrate the features of local areas to extract more meaningful features and reduce unnecessary interference and noise;

[0101] Ultimately, this achieves the effect of enhancing edge features.

[0102] Furthermore, the edge feature extraction module consists of the following components:

[0103] It contains two 1x1 convolutional layers to adjust the channels, five 3x3 convolutional layers, two LReLU activation layers, and residual connections to them.

[0104] Finally, the enhanced images are fused to obtain the output image.

[0105] Example 2

[0106] This invention designs a dual-branch infrared and visible light image fusion method based on an attention mechanism. This embodiment is a specific application based on Embodiment 1. The experimental datasets used are MSRS and TNO. The specific implementation scheme is as follows:

[0107] 1. Preprocessing operations

[0108] The two datasets were randomly divided into training, validation, and test sets, with 80%, 10%, and 10% of the datasets being used respectively.

[0109] The source images in the dataset are preprocessed to divide these grayscale images into multiple image blocks of the same size (128×128).

[0110] 2. Extract source image features

[0111] Two source images, an infrared image and a visible light image, are input into the dual-attention mechanism feature extraction block of the dual-branch network. In this step, a feature extraction and storage module is formed by combining channel attention and spatial attention mechanisms. This module consists of a residual block (REB), a channel attention (CAMB) block, and a spatial attention (SAMB) block.

[0112] The residual block (REB) consists of three convolutional layers with a kernel size of 5x5, two batch normalization layers, and one ReLU activation layer, used to extract shallow features of the image.

[0113] Channel Attention Block (CAMB): Contains three 3x3 convolutional layers, one global average pooling layer, one ReLU activation layer, one Sigmoid activation layer, and two fully connected layers. The channel attention mechanism is used to extract channel correlation features of an image.

[0114] Spatial Attention Block (SAMB): Contains three 3x3 convolutional layers, one ReLU activation layer, and one Sigmoid activation layer. The spatial attention mechanism is used to extract spatial correlation features from an image. The expression after feature extraction is as follows:

[0115] F1 = REB(I)

[0116] F2 = CAMB(F1)

[0117] F3 = SAMB(F1)

[0118] F = F1 + (F1 * F2) + (F1 * F3)

[0119] Where I represents the preprocessed source image input to the dual-attention mechanism feature extraction module. Since this structure is a dual-branch structure, I represents the preprocessed visible light image and infrared image in the two branches, respectively. F1 is the result of the source image processed by the residual block, F2 is the result of F1 processed by the channel attention block, and F3 is the result of F1 processed by the spatial attention block. In the above process, F1, F2, and F3 are all four-dimensional tensors with 128 channels.

[0120] 3. Establish image remote dependencies

[0121] In this invention, establishing long-range dependencies aims to capture global contextual information of the image and ensure that the correlation between different regions is fully considered during image fusion. First, the multi-scale feature maps extracted by the dual-branch network are fed into the Transformer block. This step aims to further process the feature maps to capture global contextual information.

[0122] The feature map is divided into M*M non-overlapping windows. These windows cover the entire feature map, with each window representing a local region in the image. Self-attention is computed on the features within each window. This means that within each window, features are correlated with features in other windows. The attention weights between each window and other windows are calculated using a weight matrix of query (Q), key (K), and value (V).

[0123] Finally, the normalized weights are summed with their corresponding values ​​to obtain the final self-attention output for each window.

[0124] Through the above steps, the long-range dependencies between various local regions in the image were successfully established, providing more comprehensive contextual information for subsequent image fusion, thereby improving the quality and effect of image fusion.

[0125] 4. Edge Feature Enhancement

[0126] The edge feature enhancement process utilizes compressed attention mechanism and residual attention mechanism, as well as edge feature extraction module, to process the image, focusing on important information and ignoring irrelevant information, and ultimately achieving the goal of enhancing edge features.

[0127] By using a compressed attention mechanism, image features are filtered and compressed, focusing attention more on edge information. This step helps reduce irrelevant information in the image and increases the importance of edge features.

[0128] By utilizing a residual attention mechanism, the compressed key edge features are weighted and superimposed with the original image features to preserve the information of the original image and highlight the importance of edge features. This operation can effectively enhance the sharpness and contrast of image edges.

[0129] In addition, an edge feature extraction module is introduced to aggregate and integrate features of local regions, thereby extracting more meaningful features and reducing unnecessary interference and noise, ultimately achieving the effect of edge feature enhancement.

[0130] The edge feature extraction module includes the following components:

[0131] Two 1×1 convolutional layers are used to adjust the channels in preparation for passing the input feature map to subsequent processing layers;

[0132] Five 5×5 convolutional layers are used to perform deep feature extraction on local regions to capture higher-level feature information;

[0133] Two LReLU activation layers are used to introduce nonlinearity and enhance the expressive power of the model;

[0134] Residual connections are used to maintain the dimensionality and information integrity of the feature maps by performing residual connections between the outputs and inputs of all convolutional layers, ensuring that no important information is lost.

[0135] 5. Feature Fusion

[0136] Dimensionality reduction convolutional layers and feature fusion first employ three convolutional layers of the same size to reduce the number of image channels. These convolutional layers transform the features through convolution operations to reduce the number of channels, helping to reduce the number of parameters and computational load while preserving the useful information of the features.

[0137] After the dimensionality reduction convolutional layer, the ELU activation function is used. The ELU activation function helps to solve the gradient vanishing problem, making the training process more stable and improving the model's non-linear representation capability.

[0138] Finally, the features from different branches are fused using element-wise addition. This step adds the features from each branch to integrate their information and produce the final fused image.

[0139] Through the above feature fusion processing, the features extracted from different branches can be effectively integrated, thereby obtaining image output with richer information and higher quality.

[0140] This embodiment, based on an attention mechanism, can effectively extract multi-scale features from infrared and visible light images and enhance their edge features, revealing rich texture details. This results in a fused image with stronger contrast and detail, making it clearer.

[0141] Example 3

[0142] This embodiment provides a dual-branch infrared and visible light image fusion device based on an attention mechanism, which effectively solves a series of problems such as the inability to establish long-range dependencies during feature extraction, excessive noise during infrared image feature extraction, and blurred edge information after image fusion, making the fused image clearer and achieving better fusion results. The device includes:

[0143] Dual attention mechanism feature extraction block: used to obtain multi-scale features from two source images;

[0144] Transformer block: Captures global context information of the image and establishes remote dependencies;

[0145] Edge Feature Enhancement Block: This block includes edge feature extraction and enhancement. Edge feature extraction aggregates features from local regions, resulting in richer and more representative features. The edge feature enhancement process employs compressed attention and residual attention mechanisms to focus on important information and enhance edge features.

[0146] Feature fusion block: The fusion features at various scales obtained after the source image has undergone a series of operations described above are input into the fusion network to obtain a fused image of the infrared image and the visible light image.

[0147] Example 4

[0148] This embodiment provides an infrared and visible light image fusion device based on an attention mechanism, including a processor and a storage medium;

[0149] The storage medium is used to store instructions;

[0150] The processor is configured to operate according to the instructions to execute the steps of the method described in Embodiment 2.

[0151] Example 5

[0152] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in Embodiment 2:

[0153] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0155] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for fusing dual-branch infrared and visible light images based on an attention mechanism, characterized in that, Includes the following steps: Two source images, an infrared image and a visible light image, are input into the dual attention mechanism feature extraction block of a dual-branch network to obtain multi-scale features of the two source images; The multi-scale features of the infrared and visible light images are input into the Transformer module to capture the global context information of the image and establish long-range dependencies. Edge feature enhancement is performed on the feature map that establishes long-distance dependencies, and the image before enhancement is added to the image after enhancement; The infrared image and the visible light image are fused using multi-scale features to obtain a high-quality fused image. The multi-scale features of the two source images are as follows: The dual-attention mechanism feature extraction block combines channel attention and spatial attention mechanisms to form a feature extraction and storage module; The feature extraction and storage module includes a residual block (REB), a channel attention mechanism block (CAMB), and a spatial attention mechanism block (SAMB); the residual block (REB), channel attention mechanism block (CAMB), and spatial attention mechanism block (SAMB) include: REB consists of three convolutional layers with a kernel size of 5*5, two batch normalization layers, and one ReLU activation layer to extract shallow features from the image. CAMB consists of three convolutional layers with a kernel size of 3*3, one global average pooling layer, one ReLU activation layer, one Sigmoid activation layer, and two fully connected layers. SAMB consists of three convolutional layers with a kernel size of 3*3, one ReLU activation layer, and one Sigmoid activation layer. The feature extraction expression includes: F1 = REB(I) I represents the input image to the dual attention mechanism feature extraction module. I is converted from the input RGB source image into a grayscale image with only the luminance channel, and the grayscale image is cropped to a size of 128*128. F1 is the result of the preprocessed source image after passing through the residual block (REB). Since this structure is a dual-branch structure, I represents the preprocessed visible light image and infrared image in the two branches, respectively. F2 = CAMB(F1) F2 is the output of F1 after passing through the Channel Attention Block (CAMB); F3 = SAMB(F1) F3 is the output of F2 after passing through the Spatial Attention Block (SAMB); in the above process, F1, F2, and F3 are all four-dimensional tensors with 128 channels; The output of the dual attention mechanism feature extraction module is as follows: F = F1 + (F1 * F2) + (F1 * F3).

2. The method for fusing dual-branch infrared and visible light images based on an attention mechanism according to claim 1, characterized in that, The Transformer module includes: The Transformer module consists of three layers of normalization, a multi-head self-attention mechanism, and a multilayer perceptron, and performs multiple residual connections on it.

3. The method for fusing dual-branch infrared and visible light images based on an attention mechanism according to claim 2, characterized in that, The Transformer module includes: The input feature image is divided into non-overlapping windows of size M*M, generating a... feature; in It represents the total number of windows. All of these windows will undergo standard self-attention computation. In the self-attention mechanism, features within each window will be correlated with features within other windows to capture the dependencies between global features.

4. The method for fusing dual-branch infrared and visible light images based on an attention mechanism according to claim 2, characterized in that, The self-attention mechanism mentioned includes: This self-attention calculation is typically achieved by multiplying by the weight matrix of the query, key, and value. The resulting attention score is then processed by the Softmax function to obtain normalized weights. Finally, the values ​​are multiplied by the weights and summed to obtain the final self-attention output.

5. The method for fusing dual-branch infrared and visible light images based on an attention mechanism according to claim 4, characterized in that, The correspondence between the query (Q), key (K), and value (V) of the self-attention mechanism and the window feature (X) is as follows: Q=X·W Q , K=X·W K , V=X·W V , Where X represents the input window feature, W Q W K and W V The projection matrices for Q, K, and V are used to map the input features to higher-level feature representations. These matrices map X to a space of query (Q), key (K), and value (V), with dimensions of 1. Where d is the feature dimension. Attention value is calculated as follows: Where d is the scaling factor, S(·) represents the Softmax function, d represents the dimension, and B represents the learnable relative position encoding.

6. The method for fusing dual-branch infrared and visible light images based on an attention mechanism according to claim 1, characterized in that, The edge feature enhancement includes: The edge feature enhancement process employs compressed attention and residual attention mechanisms to focus on important information and ignore irrelevant information, and then performs addition and subtraction operations. One of them is an edge feature extraction module, which is used to aggregate and integrate features of local areas to extract more meaningful features and reduce unnecessary interference and noise, ultimately achieving the effect of edge feature enhancement.

7. The method for fusing dual-branch infrared and visible light images based on an attention mechanism according to claim 6, characterized in that, The edge feature extraction module consists of the following components: It contains two 1x1 convolutional layers to adjust the channels, five 3x3 convolutional layers, two LReLU activation layers, and residual connections to them.

8. The method for fusing dual-branch infrared and visible light images based on an attention mechanism according to claim 1, characterized in that, Feature fusion processing includes: Feature fusion involves three convolutional layers of the same size to reduce the number of image channels, and combines them with the ELU activation function, which helps to reduce the gradient vanishing problem and makes the model more stable. Finally, element-wise addition is performed to obtain the fused output image.

9. A dual-branch infrared and visible light image fusion system based on an attention mechanism, characterized in that, The system includes: Feature extraction module: The feature extraction module uses channel attention mechanism and spatial attention mechanism to extract features from the input source image at multiple scales; Transformer module: Captures image context information and establishes remote dependencies; Edge feature enhancement module: Employs compressed attention mechanism and residual attention mechanism to focus on information in specific regions of the image and ignore irrelevant information to reduce interference; Edge feature extraction module: used to aggregate features of local regions, thereby extracting richer and more representative features; Feature fusion module: Gradually reduce the number of channels in two images that have undergone a series of processing steps, and then perform element-wise addition operations.

10. A dual-branch infrared and visible light image fusion system based on an attention mechanism according to claim 9, characterized in that, The system includes a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 8.

Images