A cross-modal image fusion method based on multi-scale dilated attention

Abstract

This invention discloses a cross-modal image fusion method based on multi-scale dilated attention, comprising: acquiring multiple sets of visible light-infrared image pairs; constructing a two-stream network; constructing a differential modal multi-scale dilated attention module (DFMDA); inserting three DMDAs into the constructed two-stream network; using the two-stream network to extract hierarchical features from the visible light-infrared images; using the three DMDAs to enhance the extracted hierarchical features with multi-modal complementary information to obtain multi-scale semantic information; and adding the multi-scale cross-modal image features output by the DMDAs for image fusion. This invention solves the problems of computational complexity, limited utilization of information between cross-modal features, and poor robustness in processing images from complex environments such as nighttime in existing image fusion techniques.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a cross-modal image fusion method based on multi-scale hole attention. Background Technology

[0002] Due to the complementary nature of image information from different modalities, the fusion of infrared and visible light has been widely applied in target detection, target tracking, person re-identification, and semantic segmentation. Traditional image fusion techniques mainly employ Gaussian filtering, bilateral filtering, and data-driven methods. These methods transform the source images into a transform domain through mathematical transformations and design corresponding fusion rules to achieve image fusion. However, the problem is that these methods have high computational complexity, making it difficult to achieve higher fusion performance and failing to meet the requirements of real-time computing applications.

[0003] With the rapid development of deep learning, image fusion research is moving towards a data-driven approach, mainly falling into three categories: methods based on autoencoders (AE), methods based on convolutional neural networks (CNN) (such as PIAFusion and MBNet), and methods based on generative adversarial networks (GAN) (such as FusionGan). However, these methods are all implemented based on convolutional blocks, resulting in a small receptive field and an inability to model long-range dependencies between arbitrary image patches.

[0004] In existing technologies, CFT utilizes the self-attention mechanism of Transformer blocks to better fuse image information and has a large receptive field. However, because CFT uses a global receptive field, it increases the cost of secondary computation, and the modeling dependencies between all image blocks may be redundant in image fusion tasks. Furthermore, CFT lacks the ability to extract multi-scale features, cannot effectively address the problem of target scale variations, and rarely utilizes information between cross-modal features, resulting in poor robustness when processing images in complex environments such as nighttime. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a cross-modal image fusion method based on multi-scale dilated attention that improves the network's ability to extract multi-scale semantic information and reduces the redundancy and computational cost of self-attention mechanisms.

[0006] Technical Solution: To achieve the above objectives, the present invention provides a cross-modal image fusion method based on multi-scale dilated attention, comprising the following steps:

[0007] Step S1: Acquire multiple sets of visible light-infrared image pairs;

[0008] Step S2: Construct a two-stream network using the backbone network structure of YOLOv5;

[0009] Step S3: Construct the Differential Modal Multiscale Hollow Attention Module (DFMDA);

[0010] Step S4: Insert three constructed differential modal multiscale holed attention modules (DFMDA) into the constructed two-stream network;

[0011] Step S5: Use the dual-stream network from step S4 to extract the layered features of the visible light-infrared image;

[0012] Step S6: Use the three differential modal multi-scale dilated attention modules (DFMDA) from step S4 to enhance the extracted hierarchical features with multi-modal complementary information to obtain multi-scale semantic information.

[0013] Step S7: Add the outputs of each Differential Modal Multiscale Hollow Attention Module (DFMDA) after feature extraction of visible light and infrared modal features to obtain cross-modal fusion image features at different scales.

[0014] Step S1 involves acquiring multiple sets of visible light-infrared image pairs, wherein the visible light image and the infrared image in each set of image pairs have the same number of channels.

[0015] The construction of a dual-stream network using the YOLOv5 backbone in step S2 refers to the construction of a visible light mode branch and an infrared mode branch. Each branch uses the same YOLOv5 backbone network structure. Specifically, the five feature extraction layers Fr1, Fr2, Fr3, Fr4, and Fr5 serve as the visible light mode branch to extract the visible light mode feature Fr; and the five feature extraction layers Ft1, Ft2, Ft3, Ft4, and Ft5 serve as the infrared mode branch to extract the infrared mode feature Ft.

[0016] The Differential Modal Multiscale Hollow Attention Module (DFMDA) described in step S3 includes a feature difference structure and two multiscale hollow attention modules.

[0017] The step S5, which involves using a dual-stream network to extract layered features from visible light and infrared images, refers to inputting the visible light image and the infrared image into the visible light modal branch and the infrared modal branch, respectively. The Fr1 and Fr2 feature extraction layers of the visible light modal branch and the Ft1 and Ft2 feature extraction layers of the infrared modal branch are selected to capture low-level information in the shallow stage, while the Fr3, Fr4, and Fr5 feature extraction layers of the visible light modal branch and the Ft3, Ft4, and Ft5 feature extraction layers of the infrared modal branch are selected to extract high-level semantic information.

[0018] The enhancement of multimodal complementary features of the extracted hierarchical features by using the three differential modal multiscale holed attention modules (DFMDA) in step S4 in step S6 refers to using three DFMDAs to process the modal features extracted from the feature extraction layers of the visible light modal branches Fr2, Fr3, Fr4 and the infrared modal branches Ft2, Ft3, Ft4 respectively, and then adding the output complementary feature maps to the source modality. The output layers are Fr3, Fr4, Fr5 and Ft3, Ft4, Ft5 respectively. Specifically, this includes first using the differential structure in the DFMDA to obtain the modal difference features of the same layer, and then using the two multi-head holed attention modules (MHDA) in the DFMDA to process the modal difference features.

[0019] The aforementioned method of obtaining modal difference features of the same layer using a differential structure refers to inputting the visible light modal features Fr and infrared modal features Ft of the deep feature layer into the DFMDA, and obtaining the visible light modal difference features by subtracting the visible light modal features Fr from the infrared modal features Ft. The infrared modal difference feature is obtained by subtracting the infrared modal feature Ft from the visible light modal feature Fr.

[0020] The aforementioned use of two multi-headed hollow attention modules (MHDA) to process modal difference features refers to using two MHDA modules to process the difference features of visible light modes separately. Difference characteristics between infrared modes The two input matrices Q of the MHDA are respectively the outputs of the difference structure. and The input matrices K and V are the visible light mode feature Fr and the infrared mode feature Ft, respectively. The complementary feature maps of the two MHDA outputs are added to another mode to achieve multimodal information complementarity. The specific process is as follows:

[0021]

[0022] The output visible light mode feature F r 'and infrared modal characteristics F t 'As input to the next deep feature layer; specifically including the following steps:'

[0023] Step S301: The Multi-Head Hollow Attention Module (MHDA) is composed of multiple hollow sliding window attention modules (DSWA), given the visible light modal difference features as input. Infrared modal differential features Visible light modal characteristics Fr and infrared modal characteristics Ft.

[0024] Step S302: ... Fr and Ft are linearly mapped and divided according to the set number of multi-heads n. The number of multi-heads n for the three DFMDA modules is 4, 8, and 16 respectively. In different attention heads, dilated sliding window attention DSWA is performed using different dilation coefficients r to obtain the output feature h of each head. i n h i The features are concatenated together and then aggregated through linear layer mapping to obtain feature map Y.

[0025] Step S303: Perform Layer Normalization (LayerNorm) and Multilayer Perceptron (MLP) operations on the feature map Y, and then perform residual connection with the previous input to obtain the output Z. The output Z is the final result after processing by the Multi-Head Diffuse Attention (MHDA) module. The MLP consists of two linear layers and a GELU activation function. The specific process is as follows:

[0026]

[0027] Where, r i Q is the expansion rate of the i-th head. j K j and V j This represents a slice of the feature map input to the j-th head, and this slice of the feature map is expressed in matrix form; Q 1 ,K 1 V 1 , Y 1 Z 1 This represents the feature map output from the calculation of the visible light modal components, Q. 2 ,K 2 V 2 , Y 2 Z 2 This represents the feature map output from the calculation of the infrared modal component;

[0028] Step S302 describes performing a holed sliding window attention DSWA using different hole coefficients r in different attention heads. The specific process is as follows:

[0029] The Diffuse Sliding Window Attention (DSWA) technique employs a sliding window to perform self-attention on all query matrices Q. Given a hole coefficient r, within a w×w sliding window centered at position (a,b), the query vector Q sparsely selects positions q from the surrounding keyword matrix K using the hole coefficient r. ab To calculate self-attention, the DSWA formula is expressed as:

[0030]

[0031] Where the inputs Q,K,V∈R C×H×W Q, K, and V represent the query matrix, keyword matrix, and value matrix, respectively. Each row of these three matrices represents a single query vector, keyword vector, and value vector. C, H, and W are the height and width of the input matrices Q, K, and V. r and V r This represents the keys and values ​​selected from the key matrix K and the value matrix V by the void coefficient r;

[0032] Among them, K r and V r The formula for calculating the coordinates (a', b') is as follows:

[0033]

[0034] Step S7 describes adding the outputs of each Differential Modal Multiscale Hollow Attention Module (DFMDA) after feature extraction of visible and infrared modal features to obtain cross-modal fusion image features at different scales. Specifically, the visible and infrared modal features output from the three DFMDA modules are fed into deep feature extraction layers Fr3, Fr4, Fr5 and Ft3, Ft4, Ft5, respectively. The outputs of feature extraction layers Fr3 and Ft3 are added to obtain P3, the outputs of feature extraction layers Fr4 and Ft4 are added to obtain P4, and the outputs of feature extraction layers Fr5 and Ft5 are added to obtain P5. P3, P4, and P5 are cross-modal image fusion features at different scales.

[0035] Beneficial effects: The present invention has the following advantages: 1. The differential structure used in this method can explicitly extract the difference features between different modalities, thereby improving the network's sensitivity to image features from another modality, enhancing multimodal complementary features, improving the network's ability to extract multi-scale semantic information, and also improving the feature extraction ability for complex environment images such as night.

[0036] 2. The Multi-Head Hollow Attention Module (MHDA) of this invention fuses different modal information in the extracted hierarchical features, which can effectively aggregate local features of long and short distances and effectively reduce the redundancy and computational load of the self-attention mechanism. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the invention method flow;

[0038] Figure 2 This is a schematic diagram of the DFMDA network structure;

[0039] Figure 3 This is a schematic diagram of the process for processing visible light and infrared modal features using the Three Differential Mode Multiscale Hollow Attention Module (DFMDA). Detailed Implementation

[0040] The technical solution of the present invention will be described in detail below with reference to the embodiments and accompanying drawings.

[0041] like Figure 1 As shown, the cross-modal image fusion method based on multi-scale dilated attention described in this invention includes the following steps:

[0042] Step S1: Acquire multiple sets of visible light-infrared image pairs;

[0043] Step S2: Construct a two-stream network using the backbone network structure of YOLOv5;

[0044] Step S3: Construct the Differential Modal Multiscale Hollow Attention Module (DFMDA);

[0045] Step S4: Insert three constructed differential modal multiscale holed attention modules (DFMDA) into the constructed two-stream network;

[0046] Step S5: Use the dual-stream network from step S4 to extract the layered features of the visible light-infrared image;

[0047] Step S6: Use the three differential modal multi-scale dilated attention modules (DFMDA) from step S4 to enhance the extracted hierarchical features with multi-modal complementary information to obtain multi-scale semantic information.

[0048] Step S7: Add the outputs of each Differential Modal Multiscale Hollow Attention Module (DFMDA) after feature extraction of visible light and infrared modal features to obtain cross-modal fusion image features at different scales.

[0049] Step S1 involves acquiring multiple sets of visible light-infrared image pairs, wherein the visible light image and the infrared image in each set of image pairs have the same number of channels.

[0050] The construction of a dual-stream network using the YOLOv5 backbone in step S2 refers to the construction of a visible light mode branch and an infrared mode branch. Each branch uses the same YOLOv5 backbone network structure. Specifically, the five feature extraction layers Fr1, Fr2, Fr3, Fr4, and Fr5 serve as the visible light mode branch to extract the visible light mode feature Fr; and the five feature extraction layers Ft1, Ft2, Ft3, Ft4, and Ft5 serve as the infrared mode branch to extract the infrared mode feature Ft.

[0051] The Differential Modal Multiscale Hollow Attention Module (DFMDA) described in step S3 includes a feature difference structure and two multiscale hollow attention modules.

[0052] The step S5, which involves using a dual-stream network to extract layered features from visible light and infrared images, refers to inputting the visible light image and the infrared image into the visible light modal branch and the infrared modal branch, respectively. The Fr1 and Fr2 feature extraction layers of the visible light modal branch and the Ft1 and Ft2 feature extraction layers of the infrared modal branch are selected to capture low-level information in the shallow stage, while the Fr3, Fr4, and Fr5 feature extraction layers of the visible light modal branch and the Ft3, Ft4, and Ft5 feature extraction layers of the infrared modal branch are selected to extract high-level semantic information.

[0053] The enhancement of multimodal complementary features of the extracted hierarchical features by using the three differential modal multiscale holed attention modules (DFMDA) in step S4 in step S6 refers to using three DFMDAs to process the modal features extracted from the feature extraction layers of the visible light modal branches Fr2, Fr3, Fr4 and the infrared modal branches Ft2, Ft3, Ft4 respectively, and then adding the output complementary feature maps to the source modality. The output layers are Fr3, Fr4, Fr5 and Ft3, Ft4, Ft5 respectively. Specifically, this includes first using the differential structure in the DFMDA to obtain the modal difference features of the same layer, and then using the two multi-head holed attention modules (MHDA) in the DFMDA to process the modal difference features.

[0054] The aforementioned method of obtaining modal difference features of the same layer using a differential structure refers to inputting the visible light modal features Fr and infrared modal features Ft of the deep feature layer into the DFMDA, and obtaining the visible light modal difference features by subtracting the visible light modal features Fr from the infrared modal features Ft. The infrared modal difference feature is obtained by subtracting the infrared modal feature Ft from the visible light modal feature Fr.

[0055] The aforementioned use of two multi-headed hollow attention modules (MHDA) to process modal difference features refers to using two MHDA modules to process the difference features of visible light modes separately. Difference characteristics between infrared modes The two input matrices Q of the MHDA are respectively the outputs of the difference structure. and Input matrices K and V represent the visible light mode feature Fr and the infrared mode feature Ft, respectively. The outputs of the two MHDAs are added to another mode to achieve multimodal information complementarity, such as... Figure 2 The diagram shows the structure of the Multi-Head Hollow Attention Module (MHDA), and its operation process is as follows:

[0056]

[0057]

[0058] The output visible light mode feature F r 'and infrared modal characteristics F t As input to the next deep feature layer, the specific steps include:

[0059] Step S301: The Multi-Head Hollow Attention Module (MHDA) is composed of multiple hollow sliding window attention modules (DSWA), given the visible light modal difference features as input. Infrared modal differential features Visible light modal characteristics Fr and infrared modal characteristics Ft.

[0060] Step S302: ... Fr and Ft are linearly mapped and divided according to the set number of multi-heads n. The number of multi-heads n for the three DFMDA modules is 4, 8, and 16 respectively. In different attention heads, dilated sliding window attention DSWA is performed using different dilation coefficients r to obtain the output feature h of each head. i n h i The features are concatenated together and then aggregated through linear layer mapping to obtain feature map Y.

[0061] Step S303: Perform Layer Normalization (LayerNorm) and Multilayer Perceptron (MLP) operations on the feature map Y, and then perform residual connection with the previous input to obtain the output Z. The output Z is the final result after processing by the Multi-Head Diffuse Attention (MHDA) module. The MLP consists of two linear layers and a GELU activation function. The specific process is as follows:

[0062]

[0063] Where, r i Q is the expansion rate of the i-th head. j K j and V j This represents a slice of the feature map input to the j-th head, and this slice of the feature map is expressed in matrix form; Q 1 ,K 1 V 1 , Y 1 Z 1 This represents the feature map output from the calculation of the visible light modal components, Q. 2 ,K 2 V 2 , Y 2 Z 2 This represents the feature map output from the calculation of the infrared modal component;

[0064] Step S302 describes performing a holed sliding window attention DSWA using different hole coefficients r in different attention heads. The specific process is as follows:

[0065] The Diffuse Sliding Window Attention (DSWA) technique employs a sliding window to perform self-attention on all query matrices Q. Given a hole coefficient r, within a w×w sliding window centered at position (a,b), the query vector Q sparsely selects positions q from the surrounding keyword matrix K using the hole coefficient r. ab To calculate self-attention, the DSWA formula is expressed as:

[0066]

[0067] Where the inputs Q,K,V∈R C×H×W Q, K, and V represent the query matrix, keyword matrix, and value matrix, respectively. Each row of these three matrices represents a single query vector, keyword vector, and value vector. C, H, and W are the height and width of the input matrices Q, K, and V. r and V r This represents the keys and values ​​selected from the key matrix K and the value matrix V by the void coefficient r;

[0068] Among them, K r and V r The formula for calculating the coordinates (a', b') is as follows:

[0069]

[0070] The step S7, which involves adding the outputs of each Differential Modal Multiscale Hollow Attention Module (DFMDA) after feature extraction of visible light and infrared modal features to obtain cross-modal fusion image features at different scales, refers to feeding the outputs of the three Differential Modal Multiscale Hollow Attention Modules (DFMDA) into deep feature extraction layers Fr3, Fr4, Fr5 and Ft3, Ft4, Ft5, respectively. The outputs of feature extraction layers Fr3 and Ft3 are added together to obtain P3, the outputs of feature extraction layers Fr4 and Ft4 are added together to obtain P4, and the outputs of feature extraction layers Fr5 and Ft5 are added together to obtain P5. P3, P4, and P5 are cross-modal image fusion features at different scales.

[0071] like Figure 3The diagram illustrates the feature processing flow of the Three Differential Modal Multiscale Hollow Attention Module (DFMDA) for visible and infrared modal features. The input consists of visible light (RGB) and infrared (Thermal) image pairs, each with a width and height of 640 pixels and 3 channels. After passing through shallow feature extraction layers Fr1 and Ft1, the visible light-infrared image pair becomes 320 pixels in width and height and has 64 channels. After passing through shallow feature extraction layers Fr2 and Ft2, the visible light-infrared image pair becomes 160 pixels in width and height and has 128 channels. The output feature map is then input into the DFMDA, and after passing through deep feature extraction layers Fr3 and Ft3, the output feature map is... The feature map's length and width become 80, and the number of channels becomes 256. The obtained feature map is input into DDMDA, and after passing through deep feature extraction layers Fr4 and Ft4, the length and width of the output feature map become 40, and the number of channels becomes 512. The obtained feature map is input into DDMDA, and after passing through deep feature extraction layers Fr5 and Ft5, the length and width of the output feature map become 20, and the number of channels becomes 1024. Fr3, Fr4, Fr5 and Ft3, Ft4, Ft5 are added layer by layer to obtain P3, P4, P5.

Claims

1. A cross-modal image fusion method based on multi-scale dilated attention, characterized in that, Includes the following steps: Step S1: Acquire multiple sets of visible light-infrared image pairs; Step S2: Construct a two-stream network using the backbone network structure of YOLOv5; Step S3: Construct the Differential Modal Multiscale Hollow Attention Module (DFMDA); Step S4: Insert three constructed differential modal multiscale holed attention modules (DFMDA) into the constructed two-stream network; Step S5: Use the dual-stream network from step S4 to extract the layered features of the visible light-infrared image; Step S6: Use the three differential modal multi-scale dilated attention modules (DFMDA) from step S4 to enhance the extracted hierarchical features with multi-modal complementary information to obtain multi-scale semantic information. Step S7: Add the outputs of each Differential Modal Multiscale Hollow Attention Module (DFMDA) after feature extraction of visible light and infrared modal features to obtain cross-modal fusion image features at different scales; The construction of a dual-stream network using the YOLOv5 backbone in step S2 refers to the construction of a visible light mode branch and an infrared mode branch. Each branch uses the same YOLOv5 backbone network structure. Specifically, the five feature extraction layers Fr1, Fr2, Fr3, Fr4, and Fr5 serve as the visible light mode branch to extract the visible light mode feature Fr; and the five feature extraction layers Ft1, Ft2, Ft3, Ft4, and Ft5 serve as the infrared mode branch to extract the infrared mode feature Ft. The Differential Modal Multiscale Hollow Attention Module (DFMDA) mentioned in step S3 includes a feature difference structure and two multi-head hollow attention modules (MHDA). The step S5, which involves using a dual-stream network to extract layered features from visible light and infrared images, refers to inputting visible light and infrared images into the visible light modal branch and the infrared modal branch, respectively. The Fr1 and Fr2 feature extraction layers of the visible light modal branch and the Ft1 and Ft2 feature extraction layers of the infrared modal branch are selected to capture low-level information in the shallow stage, while the Fr3, Fr4, and Fr5 feature extraction layers of the visible light modal branch and the Ft3, Ft4, and Ft5 feature extraction layers of the infrared modal branch are selected to extract high-level semantic information. The enhancement of multimodal complementary features of the extracted hierarchical features by using the three differential modal multiscale holed attention modules (DFMDA) in step S4 in step S6 refers to using three DFMDAs to process the modal features extracted from the feature extraction layers of the visible light modal branches Fr2, Fr3, Fr4 and the infrared modal branches Ft2, Ft3, Ft4 respectively, with output layers of Fr3, Fr4, Fr5 and Ft3, Ft4, Ft5 respectively. Specifically, this includes first using the differential structure in the DFMDA to obtain the modal difference features of the same layer, and then using the two multi-head holed attention modules (MHDA) in the DFMDA to process the modal difference features.

2. The cross-modal image fusion method based on multi-scale dilated attention according to claim 1, characterized in that, Step S1 involves acquiring multiple sets of visible light-infrared image pairs, wherein the visible light image and the infrared image in each set of image pairs have the same number of channels.

3. The cross-modal image fusion method based on multi-scale dilated attention according to claim 2, characterized in that, The aforementioned method of obtaining modal difference features of the same layer using a differential structure refers to inputting the visible light modal features Fr and infrared modal features Ft of the deep feature layer into the DFMDA, and obtaining the visible light modal difference features by subtracting the visible light modal features Fr from the infrared modal features Ft. Infrared mode difference features are obtained by subtracting infrared mode features Ft from visible mode features Fr. .

4. The cross-modal image fusion method based on multi-scale dilated attention according to claim 3, characterized in that, The aforementioned use of two multi-headed hollow attention modules (MHDA) to process modal difference features refers to using two MHDA modules to process the difference features of visible light modes separately. Difference characteristics between infrared modes The two input matrices Q of the MHDA are respectively the outputs of the difference structure. and The input matrices K and V are the visible light mode feature Fr and the infrared mode feature Ft, respectively. The complementary feature maps of the two MHDA outputs are added to another mode to achieve multimodal information complementarity. The specific process is as follows: ； The output visible light mode features and infrared modal characteristics As input to the next deep feature layer; Specifically, the following steps are included: Step S301: The Multi-Head Hollow Attention Module (MHDA) is composed of multiple hollow sliding window attention modules (DSWA), given the visible light modal difference features as input. Infrared modal differential features Visible light modal characteristics Fr and infrared modal characteristics Ft; Step S302: ... , Linear mapping is performed on Fr and Ft, and the number of multiple heads is set accordingly. Divide the data into three DFMDA modules and determine the number of multi-heads. The values ​​were 4, 8, and 16, respectively; different hole coefficients were used in different attention heads. Perform a holed sliding window attention DSWA to obtain the output features for each head. ; indivual The features are concatenated together and then aggregated through linear layer mapping to obtain feature map Y. Step S303: Perform Layer Normalization (LayerNorm) and Multilayer Perceptron (MLP) operations on the feature map Y, and then perform residual connection with the previous input to obtain the output Z. The output Z is the final result after processing by the Multi-Head Diffuse Attention (MHDA) module. The MLP consists of two linear layers and a GELU activation function. The specific process is as follows: ； in, It is The rate of expansion of size, , and This represents a slice of the feature map input to the j-th head, and the slice of the feature map is expressed in matrix form; This represents the feature map output from the calculation of the visible light modal component. This represents the feature map output from the calculation of the infrared modal component.

5. The cross-modal image fusion method based on multi-scale dilated attention according to claim 4, characterized in that, Step S302 describes using different hole coefficients in different attention heads. The specific process of performing a hole-sliding window attention DSWA is as follows: Hollow Sliding Window Attention DSWA uses a sliding window approach to process all query matrices. Perform self-attention operation, given a voiding coefficient. With a size centered at position (a,b) In the sliding window, through the void coefficient The query vector Q will sparsely select positions in the surrounding key matrix K. To calculate self-attention, the DSWA formula is expressed as: , where, input Q, K, and V represent the query matrix, keyword matrix, and value matrix, respectively, with each row of the three matrices representing a single query vector, keyword vector, and value vector; C, and These are the height and width of the input matrices Q, K, and V; and Indicates from the key matrix Sum matrix In, through the void coefficient Selected key and value; in, and coordinates The calculation formula is as follows: 。 6. The cross-modal image fusion method based on multi-scale dilated attention according to claim 5, characterized in that, The step S7, which involves adding the outputs of each Differential Modal Multiscale Hollow Attention Module (DFMDA) after feature extraction of visible light and infrared modal features to obtain cross-modal fusion image features at different scales, refers to feeding the outputs of the three Differential Modal Multiscale Hollow Attention Modules (DFMDA) into deep feature extraction layers Fr3, Fr4, Fr5 and Ft3, Ft4, Ft5, respectively. The outputs of feature extraction layers Fr3 and Ft3 are added together to obtain P3, the outputs of feature extraction layers Fr4 and Ft4 are added together to obtain P4, and the outputs of feature extraction layers Fr5 and Ft5 are added together to obtain P5. P3, P4, and P5 are cross-modal image fusion features at different scales.

Images