Visible light and infrared image fusion method and system based on multi-scale attention

Abstract

The invention provides a visible light and infrared image fusion method and system based on multi-scale attention, and belongs to the field of computer vision, and the method comprises the steps: obtaining a to-be-fused infrared image and a to-be-fused visible light image; performing image fusion through the trained fusion model to obtain an image fusion result; the fusion model obtaining method comprises the steps of obtaining an infrared training set and a visible light training set; performing image fusion through the initialized fusion model to obtain an image fusion result; and according to the new loss function, training the fusion model by using infrared and visible light training set data, and performing parameter updating on the initialized fusion model to obtain a trained final fusion model. According to the method, the characteristics of the long-distance dependency relationship and the local information of the image are effectively extracted, different modal information characteristics are gradually and mutually fused, the high and low frequency characteristics, the global characteristics and the local characteristics of the infrared and visible light images are complementarily fused through an effective fusion strategy, and the distinction degree and the quality of the fused image are improved.

Description

Technical Field

[0001] The present invention provides a visible light and infrared image fusion method and system based on multi-scale attention, belonging to the technical field of visual information processing. Background Art

[0002] Visible light and infrared fusion technology combines image data from the visible and infrared spectra to produce richer visual information. Visible light images can provide rich color and detail under good lighting conditions. However, in low-light environments, such as at night or in dimly lit rooms, the performance of visible light cameras is significantly limited. Infrared imaging is unaffected by lighting conditions and can operate even in complete darkness. It can detect temperature differences and has good recognition capabilities for heat sources (such as the human body). However, infrared images typically lack color information and may not be clear enough when temperature differences are small. With the development of multimodal image fusion technology, multimodal image fusion technology is playing an important role in many fields. Visible light and infrared image fusion technology has achieved a series of achievements, such as traditional image fusion algorithms, methods based on autoencoders, methods based on convolutional neural networks, and methods based on generative adversarial neural networks.

[0003] Current image fusion methods are typically based on convolutional neural networks or traditional image fusion algorithms. These methods are not able to effectively extract features from infrared or visible light images. The extracted features have contrast that is inconsistent with human visual effects, resulting in poor fused image quality. Feature extraction fails to adequately consider the individual characteristics of infrared and visible light images, as well as the global and local features of the images. Consequently, the resulting fused image is of poor quality. Existing convolutional neural network-based methods primarily capture features through local receptive fields and may not be able to handle long-range dependencies well. Convolutional neural networks also achieve a certain degree of spatial invariance through operations such as convolution and pooling, but may not be as effective when handling unconventional perspectives or complex deformations. Furthermore, current attention mechanisms have high time complexity and redundancy, requiring significant computational resources. Therefore, the problems with existing fusion methods need to be further addressed and optimized. Summary of the Invention

[0004] To address the above problems, the present invention provides a visible light and infrared image fusion method and system based on multi-scale attention.

[0005] First, a visible light and infrared image fusion method based on multi-scale attention is proposed, which includes the following steps:

[0006] S1. Acquire the infrared image and visible light image to be fused;

[0007] S2. Based on the infrared image to be fused and the visible light image to be fused, perform image fusion using a trained fusion model to obtain an image fusion result;

[0008] The trained fusion model is obtained by the following steps:

[0009] S2.1. Obtain infrared training set and visible light training set;

[0010] S2.2. Based on the infrared training set and the visible light training set, perform image fusion using the initialized fusion model to obtain an image fusion result;

[0011] S2.3. According to the new loss function, the fusion model is trained using the infrared and visible light training set data, and the parameters of the initialized fusion model are updated to obtain the trained final fusion model.

[0012] The trained fusion model includes a shared network, a local network, an interactive network, a fusion network, and a reconstruction network;

[0013] (1) Shared network performs the following steps:

[0014] Perform global feature extraction on the input infrared image and visible light image to obtain the infrared global map and the visible light global map respectively;

[0015] Furthermore, performing global feature extraction on the input infrared image and visible light image includes:

[0016] The input infrared image and visible light image are respectively fed into a 3-layer global framework, a 1×1 normal convolution plus a Relu activation layer, and a 1×1 normal convolution plus a Tanh activation layer to obtain the infrared global image and the visible light global image;

[0017] The output of the first-layer infrared global framework is added to the first-layer infrared shared interaction module as the input of the second-layer infrared global framework. The output of the second-layer infrared global framework is added to the second-layer infrared shared interaction module as the input of the third-layer infrared global framework. The output of the third-layer infrared global framework is added to the third-layer infrared shared interaction module as the input of the first-layer 1×1 ordinary convolution plus ReLU activation layer.

[0018] Similarly, the output of the first-layer visible light global framework is added to the first-layer visible light shared interaction module as the input of the second-layer visible light global framework. The output of the second-layer visible light global framework is added to the second-layer visible light shared interaction module as the input of the third-layer visible light global framework. The output of the third-layer visible light global framework is added to the third-layer visible light shared interaction module as the input of the first-layer 1×1 ordinary convolution plus ReLU activation layer.

[0019] The global framework includes a feature module, an attention module, and two 1×1 ordinary convolutional layers in sequence;

[0020] The feature module sequentially includes a 3×3 depth convolution layer, a normalization layer, a 1×1 ordinary convolution plus Gelu activation layer; then the input infrared / visible light image is added to the output of the 1×1 ordinary convolution plus Gelu activation layer; then sequentially passes through a 3×3 ordinary convolution plus Relu activation layer, a 1×1 ordinary convolution plus Relu activation layer; then the input infrared / visible light image is added to the output of the 1×1 ordinary convolution plus Relu activation layer as the input of the attention module;

[0021] The attention module adopts the attention mechanism, and the specific operation is shown as follows:

[0022] Attention=Norm(Q)+L(K*Q)V

[0023] Where Norm(·) represents normalization, L(·) represents linear transformation, q represents the query matrix, k represents the matrix to be queried, v represents the value matrix, and Q is expressed as follows:

[0024] Q = Conv 1×1 (Concat(Conv 1×1 (α*q),Conv 3×3 (α*q),Conv 5×5 (α*q))

[0025]

[0026] where w a represents the learnable query matrix parameters, α represents the global attention query matrix, Concat(·) is the channel dimension splicing operation, Conv 1×1 (·) is a 1×1 convolution layer, Conv 3×3 (·) is a 3×3 convolution layer, Conv 5×5 (·) is a 5×5 convolutional layer;

[0027] K is expressed as follows:

[0028] K = Conv 1×1 (Concat(Conv 1×1 (β*k),Conv 3×3 (β*k),Conv 5×5 (β*k)

[0029]

[0030] where w brepresents the learnable query matrix parameters, β represents the global attention query matrix, Concat(·) is the channel dimension splicing operation, Conv 1×1 (·) is a 1×1 convolution layer, Conv 3×3 (·) is a 3×3 convolution layer, Conv 5×5 (·) is a 5×5 convolutional layer;

[0031] V is represented as follows:

[0032]

[0033] where w C Represents learnable value matrix parameters.

[0034] (2) passing the infrared image and the visible light image through a local network respectively to extract local features;

[0035] The local network includes three double residual modules in sequence;

[0036] The output of the first-layer infrared double residual module is added to the first-layer infrared local interaction module as the input of the second-layer infrared double residual module. The output of the second-layer infrared double residual module is added to the second-layer infrared local interaction module as the input of the third-layer infrared double residual module. The output of the third-layer infrared double residual module is added to the third-layer infrared local interaction module to obtain an infrared local map.

[0037] Similarly, the output of the first-layer visible light double residual module is added to the first-layer visible light local interaction module as the input of the second-layer visible light double residual module. The output of the second-layer visible light double residual module is added to the second-layer visible light local interaction module as the input of the third-layer visible light double residual module. The output of the third-layer visible light double residual module is added to the third-layer visible light local interaction module to obtain a visible light local map.

[0038] The dual residual module performs the following steps:

[0039] The double residual module includes 1×1 ordinary convolution + batch normalization + Tanh activation layer, 3 extraction modules, 1×1 ordinary convolution + batch normalization layer; then added to the infrared image and then activated by Relu; then passed through 1×1 ordinary convolution + batch normalization + Tanh activation layer, 3×3 ordinary convolution + batch normalization + Relu activation layer, 1×1 ordinary convolution + batch normalization layer; then added to the output of the skip module; finally, Tanh activation;

[0040] The extraction module includes 3×3 depth convolution + batch normalization + Relu activation layer, 3×3 normal convolution + batch normalization + Relu activation layer in sequence;

[0041] The jump module performs the following steps:

[0042] The infrared / visible light image is passed through a 1×1 normal convolution + batch normalization + Relu activation layer and a 1×1 normal convolution + batch normalization layer in sequence;

[0043] (3) The interaction network includes three layers of shared interaction modules and three layers of local interaction modules, and performs the following steps:

[0044] The shared interaction module performs the following steps:

[0045] The output of the infrared global frame of the i-th layer and the output of the i-th layer visible light global framework As input (i=1,2,3), the specific operation is as follows:

[0046]

[0047] in is the output of the infrared global framework at layer i, is the output of the visible light global framework at layer i, is the output of the infrared sharing interaction module at layer i, is the output of the visible light sharing interaction module in the i-th layer, δ(·) represents the ReLU function, ⊙ represents the Hadamard product, Conv 1x1 (·) is a 1×1 convolution, Conv 3x3 (·) is a 3×3 convolution, PConv 5x1 (·) is a 5×1 strip convolution, PConv 1x5 (·) is a 1×5 strip convolution, SConv 3x3 (·) is a 3×3 dilated convolution.

[0048] The local interaction module performs the following steps:

[0049] The output of the i-th layer infrared double residual module and the output of the i-th layer visible light double residual module As input (i=1,2,3), the specific operation is as follows:

[0050]

[0051] in is the output of the infrared double residual module at layer i, is the output of the i-th layer visible light double residual module, is the output of the infrared sharing interaction module at layer i, is the output of the visible light sharing interaction module at layer i, Conv 1×1is a 1×1 convolution layer, Conv 3×3 is a 3×3 convolutional layer, Conv 5×5 is a 5×5 convolutional layer, represents the element summation, R(·) represents the ReLU function, and ⊙ represents the Hadamard product.

[0052] (4) To integrate the network, perform the following steps:

[0053] The infrared global image and the visible light global image are subjected to a Hadamard product operation to obtain a global fusion image;

[0054] The infrared local image and the visible light local image are added together to form a local fusion image;

[0055] Performing a splicing operation on the global fusion map and the local fusion map to obtain a fusion map;

[0056] (5) Reconstruct the network and perform the following steps:

[0057] The reconstruction network includes 5 convolutional layers;

[0058] The fusion graph is sequentially fed into a 1×1 normal convolution layer with Leaky Relu activation, a 3×3 normal convolution layer with Leaky Relu activation, a 3×3 normal convolution layer with Leaky Relu activation, a 3×3 normal convolution layer with LeakyRelu activation, and a 1×1 normal convolution layer with Tanh activation to obtain the final fusion graph;

[0059] The loss function L in step S2.3 total Based on texture loss L text , correlation loss L corr , strength loss L in and the structural similarity index SSIM loss L ssim ;

[0060] Loss function L total As shown below:

[0061] L total =L text +αL corr +βL in +γL ssim

[0062] The weight parameters α, β, and γ are set as learnable parameters;

[0063]

[0064] Among them F f represents the final fused image, F v represents the input visible light image, FI represents the input infrared image, represents the gradient operator, ||·||1 represents L1 norm normalization, H represents the height of the input image, W represents the width of the input image, θ i (i=1,2) represents infrared and visible light measurement parameters;

[0065]

[0066] in represents the infrared global image, represents the global visible light map, represents the infrared local image, represents the local visible light map, O(·) represents the correlation coefficient operator;

[0067]

[0068] Among them F f represents the final fused image, F v represents the input visible light image, F I represents the input infrared image, ||·||1 represents L1 norm normalization, H represents the height of the input image, W represents the width of the input image, θ1 and θ2 represent the infrared and visible light measurement parameters respectively;

[0069]

[0070] Among them F f represents the final fused image, F V represents the input visible light image, F I represents the input infrared image, SSIM(·,·) represents the structural similarity index;

[0071] According to the loss function, the fusion model is trained using the infrared training set and the visible light training set data to obtain a trained final fusion model.

[0072] A visible light and infrared image fusion system based on multi-scale attention includes an infrared sensor, a visible light sensor, an image receiving module, a fusion algorithm module, a bus and a fusion image display.

[0073] First, the infrared sensor and visible light sensor respectively capture infrared and visible light images and transmit them to the signal receiving module. Then, the signal receiving module receives the input infrared and visible light images and transmits them to the fusion algorithm module through the bus. The infrared and visible light images fused by the fusion algorithm are then transmitted through the bus to the fusion image display. Finally, the fusion image display displays the fusion image results.

[0074] The beneficial effects of the present invention are:

[0075] By extracting global and local features, the global information of infrared and visible light can be effectively extracted, and the characteristics of long-range image dependencies and local information can be effectively extracted, allowing infrared and visible light images to effectively capture their respective local and global information. In addition, based on the different characteristics of global and local features of different modal images, through the interactive network, the global and local feature information of different modal images is fully and effectively extracted. At the same time, the information features of different modalities are gradually integrated with each other and through an effective fusion strategy, the high- and low-frequency features, global features, and local features of infrared and visible light images are well complemented and integrated, improving the discrimination and quality of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 Schematic diagram of the fusion model reasoning and fusion steps of the present invention;

[0077] Figure 2 Schematic diagram of the training fusion model of the present invention;

[0078] Figure 3 It is a schematic diagram of the overall model framework of the present invention;

[0079] Figure 4 A schematic diagram of a shared network according to the present invention;

[0080] Figure 5 This is a schematic diagram of a characteristic module of the present invention;

[0081] Figure 6 A schematic diagram of a local network of the present invention;

[0082] Figure 7 Schematic diagram of the dual residual module of the present invention;

[0083] Figure 8 Schematic diagram of the device and system of the present invention. DETAILED DESCRIPTION

[0084] The specific technical solution of the present invention is described with reference to the embodiments. It includes the following processes:

[0085] S1. Acquire the infrared image and visible light image to be fused;

[0086] S2, such as Figure 1 , based on the infrared image to be fused and the visible light image to be fused, performing image fusion through a trained fusion model to obtain an image fusion result;

[0087] Among them, such as Figure 2 , the trained fusion model is obtained by the following steps:

[0088] S2.1. Obtain infrared training set and visible light training set;

[0089] S2.2. Based on the infrared training set and the visible light training set, perform image fusion using the initialized fusion model to obtain an image fusion result;

[0090] S2.3. According to the new loss function, the fusion model is trained using the infrared and visible light training set data, and the parameters of the initialized fusion model are updated to obtain the trained final fusion model.

[0091] Specifically:

[0092] 1. Model Input: The fusion model input images are aligned visible light and infrared image pairs. Training datasets are available, including public visible light and infrared image datasets (such as Tno and RoadScene) and private datasets. Inference input data should be tailored to the actual use case and can be aligned visible light and infrared image pairs of various resolutions and sizes. Input images can be in either JPG or PNG format.

[0093] like Figure 3 ,The fusion model includes shared network, local network, interactive network, fusion network, and ,reconstructed network;

[0094] (1) Shared network:

[0095] like Figure 4 The shared network extracts global features of the input visible light and infrared images, and respectively enters a 3-layer global framework, a 1×1 ordinary convolution plus Relu activation layer, and a 1×1 ordinary convolution plus Tanh activation layer to obtain the infrared global map and the visible light global map.

[0096] The global framework consists of a feature module, an attention module, and two 1×1 ordinary convolutional layers. Figure 5 As shown, the feature module includes a 3×3 depthwise convolutional layer, a normalization layer, a 1×1 normal convolution plus Gelu activation layer, a 3×3 normal convolution plus Relu activation layer, and a 1×1 normal convolution plus Relu activation layer. The input infrared / visible light image is then summed with the output residual of the 1×1 normal convolution plus Gelu activation layer, and the input infrared / visible light image is summed with the output of the 1×1 normal convolution plus Relu activation layer. These two layers then enter the attention module and, finally, two layers of 1×1 normal convolution layers.

[0097] In addition, the output of the first-layer infrared / visible light global framework is added to the first-layer infrared / visible light shared interaction module as the input of the second-layer infrared / visible light global framework. The output of the second-layer infrared / visible light global framework is added to the second-layer infrared / visible light shared interaction module as the input of the third-layer infrared / visible light global framework. The output of the third-layer infrared / visible light global framework is added to the third-layer infrared / visible light shared interaction module as the input of the first-layer 1×1 ordinary convolution plus ReLU activation layer.

[0098] The attention module in the shared network takes the following actions:

[0099] Attention=Norm(Q)+L(K*Q)V

[0100] Where Norm(·) represents normalization, L(·) represents linear transformation, q represents the query matrix, k represents the matrix to be queried, v represents the value matrix, and Q is expressed as follows:

[0101] Q = Conv 1×1 (Concat(Conv 1×1 (α*q),Conv 3×3 (α*q),Conv 5×5 (α*q))

[0102]

[0103] where w a represents the learnable query matrix parameters, α represents the global attention query matrix, Concat(·) is the channel dimension splicing operation, Conv 1×1 (·) is a 1×1 convolution layer, Conv 3×3 (·) is a 3×3 convolution layer, Conv 5×5 (·) is a 5×5 convolutional layer;

[0104] K is expressed as follows:

[0105] K = Conv 1×1 (Concat(Conv 1×1 (β*k),Conv 3×3 (β*k),Conv 5×5 (β*k)

[0106]

[0107] where w b represents the learnable query matrix parameters, β represents the global attention query matrix, Concat(·) is the channel dimension splicing operation, Conv 1×1 (·) is a 1×1 convolution layer, Conv3×3 (·) is a 3×3 convolution layer, Conv 5×5 (·) is a 5×5 convolutional layer;

[0108] V is represented as follows:

[0109]

[0110] where w C Represents learnable value matrix parameters.

[0111] (2) Local network:

[0112] like Figure 6 As shown in , the input infrared image and visible light image are respectively passed through the local network to extract local features; the local network includes three double residual modules in sequence, such as Figure 7 As shown in the figure, the dual residual module sequentially includes a 1×1 normal convolution + batch normalization + Tanh activation layer, three extraction modules, and a 1×1 normal convolution + batch normalization layer. The image is then added to the input infrared image and activated by Relu. The image then passes through a 1×1 normal convolution + batch normalization + Tanh activation layer, a 3×3 normal convolution + batch normalization + Relu activation layer, and a 1×1 normal convolution + batch normalization layer. The image is then added to the output of the skip module and finally activated by Tanh. The extraction module sequentially includes a 3×3 depthwise convolution + batch normalization + Relu activation layer, a 3×3 normal convolution + batch normalization + Relu activation layer, and a 1×1 normal convolution + batch normalization layer. The skip module sequentially passes the infrared / visible light image through a 1×1 normal convolution + batch normalization + Relu activation layer, and a 1×1 normal convolution + batch normalization layer.

[0113] In addition, the output of the first-layer infrared / visible light double residual module is added to the first-layer infrared / visible light local interaction module as the input of the second-layer infrared / visible light double residual module. The output of the second-layer infrared / visible light double residual module is added to the second-layer infrared / visible light local interaction module as the input of the third-layer infrared / visible light double residual module. The output of the third-layer infrared / visible light double residual module is added to the third-layer infrared / visible light local interaction module to obtain the infrared / visible light local map.

[0114] (3) Interactive network:

[0115] Shared interaction module: The output of the infrared global frame of the i-th layer and the output of the i-th layer visible light global framework As input (i=1,2,3), perform the following operations:

[0116]

[0117] in is the output of the infrared global framework at layer i, is the output of the visible light global framework at layer i, is the output of the infrared sharing interaction module at layer i, is the output of the visible light sharing interaction module in the i-th layer, δ(·) represents the ReLU function, ⊙ represents the Hadamard product, Conv 1x1 (·) is a 1×1 convolution, Conv 3x3 (·) is a 3×3 convolution, PConv 5x1 (·) is a 5×1 strip convolution, PConv 1x5 (·) is a 1×5 strip convolution, SConv 3x3 (·) is a 3×3 dilated convolution.

[0118] Local interaction module: the output of the i-th layer infrared double residual module and the output of the i-th layer visible light double residual module As input (i=1,2,3), perform the following operations:

[0119]

[0120] in is the output of the infrared double residual module at layer i, is the output of the i-th layer visible light double residual module, is the output of the infrared sharing interaction module at layer i, is the output of the visible light sharing interaction module at layer i, Conv 1×1 is a 1×1 convolution layer, Conv 3×3 is a 3×3 convolutional layer, Conv 5×5 is a 5×5 convolutional layer, represents the element summation, R(·) represents the ReLU function, and ⊙ represents the Hadamard product.

[0121] (4) Converged Network:

[0122] Perform a Hadamard product operation on the infrared global image and the visible light global image to obtain a global fusion image. Then perform an addition operation on the infrared local image and the visible light local image to obtain a local fusion image. Finally, perform a splicing operation on the global fusion image and the local fusion image to obtain a fusion image.

[0123] (5) Reconstructing the network:

[0124] The fusion image is sequentially fed into a 1×1 normal convolution layer with Leaky Relu activation, a 3×3 normal convolution layer with Leaky Relu activation, a 3×3 normal convolution layer with Leaky Relu activation, a 3×3 normal convolution layer with Leaky Relu activation, and a 1×1 normal convolution layer with Tanh activation to obtain the final fusion image.

[0125] Among them, the loss function is to calculate the loss by combining the final fusion image with the input infrared image and visible light image through the loss function to train the model. The input infrared image and visible light image are respectively fed into the VGG-16 network to obtain the perceived infrared image G I and the perceived visible light image G V Infrared\visible light measurement parameter θ i (i=1,2) can be calculated as follows:

[0126]

[0127] Where MSE(·,·) represents the mean square error loss, F f is the final fused image.

[0128] 2. Model output: After the input image pair is fused through the above model, the corresponding final fused image is output. The output image format is consistent with the input image, for example, both the input image and the output image are JPG images or both are PNG images.

[0129] A visible light and infrared image fusion system based on multi-scale attention, the system includes an infrared sensor, a visible light sensor, an image receiving module, a fusion algorithm module, a bus and a fusion image display, such as Figure 8 shown.

[0130] First, the infrared sensor and visible light sensor respectively capture infrared and visible light images and transmit them to the signal receiving module. Then, the signal receiving module receives the input infrared and visible light images and transmits them to the fusion algorithm module through the bus. The infrared and visible light images fused by the fusion algorithm are then transmitted through the bus to the fusion image display. Finally, the fusion image display displays the fusion image results.

[0131] The key points of the present invention are:

[0132] 1. A multi-scale infrared and visible light fusion method combining attention mechanism and convolutional network is proposed, which enables infrared and visible light images to effectively capture their respective local and global information, and can capture the global information of the image while enhancing the local information of the image.

[0133] 2. A new multi-scale lightweight attention mechanism is proposed, which can effectively obtain the long-range dependency information of infrared and visible light images, while making the model more lightweight.

[0134] 3. A new convolutional residual network structure is proposed to more effectively extract local features of the image, enhance the local information of the fused image, and improve the quality of the fused image.

[0135] 4. An interactive network is proposed, which can gradually fuse the information features of different modalities of infrared and visible light when extracting global or local features, so that the infrared and visible light image features can complement each other well.

[0136] 5. A fusion strategy is proposed. First, the Hadamard product is performed on the infrared global features and the visible light global features, and an element-by-element addition fusion strategy is proposed for the infrared local features and the visible light local features. Finally, the fused global features and local features are spliced ​​and fused again, which can fully and effectively fuse the high and low frequency features of infrared and visible light images.

[0137] 6. A new loss function is proposed, which can dynamically adjust the weight values ​​of the loss function according to the respective image characteristics of infrared and visible light, better improve the quality of the fused image, and fully integrate the useful information of infrared and visible light.

Claims

1. A visible light and infrared image fusion method based on multi-scale attention, characterized in that: The following steps are involved: S1. Acquire the infrared image and visible light image to be fused; S2. Based on the infrared image to be fused and the visible light image to be fused, perform image fusion using a trained fusion model to obtain an image fusion result; The trained fusion model is obtained by the following steps: S2.

1. Obtain infrared training set and visible light training set; S2.

2. Based on the infrared training set and the visible light training set, perform image fusion using the initialized fusion model to obtain an image fusion result; S2.

3. According to the new loss function, the fusion model is trained using the infrared and visible light training set data, and the parameters of the initialized fusion model are updated to obtain the trained final fusion model.

2. The method for fusion of visible light and infrared images based on multi-scale attention according to claim 1, characterized in that: The trained fusion model includes a shared network, a local network, an interactive network, a fusion network, and a reconstruction network; (1) The shared network extracts global features from the input infrared image and visible light image to obtain the infrared global map and the visible light global map respectively; (2) extracting local features from the infrared image and the visible light image respectively through a local network; (3) The interaction network includes three layers of shared interaction modules and three layers of local interaction modules, and performs the following steps: The shared interaction module takes the output of the infrared global frame of the i-th layer and the output of the i-th layer visible light global framework As input (i=1,2,3); The local interaction module converts the output of the i-th layer infrared double residual module and the output of the i-th layer visible light double residual module As input (i=1,2,3); (4) To integrate the network, perform the following steps: Perform Hadamard product operation on the infrared global image and the visible light global image to obtain a global fusion image; The infrared local image and the visible light local image are added together to form a local fusion image; Perform a splicing operation on the global fusion map and the local fusion map to obtain a fusion map; (5) Reconstruct the network and perform the following steps: The reconstruction network includes 5 convolutional layers; The fusion graph is sequentially fed into a 1×1 normal convolution layer with Leaky Relu activation, a 3×3 normal convolution layer with Leaky Relu activation, a 3×3 normal convolution layer with Leaky Relu activation, a 3×3 normal convolution layer with Leaky Relu activation, and a 1×1 normal convolution layer with Tanh activation to obtain the final fusion graph.

3. The method for fusion of visible light and infrared images based on multi-scale attention according to claim 2, characterized in that: The global feature extraction of the input infrared image and visible light image in the shared network includes: The input infrared image and visible light image are respectively fed into a 3-layer global framework, a 1×1 ordinary convolution plus a Relu activation layer, and a 1×1 ordinary convolution plus a Tanh activation layer in sequence to obtain the infrared global image and the visible light global image; The output of the first-layer infrared global framework is added to the first-layer infrared shared interaction module as the input of the second-layer infrared global framework. The output of the second-layer infrared global framework is added to the second-layer infrared shared interaction module as the input of the third-layer infrared global framework. The output of the third-layer infrared global framework is added to the third-layer infrared shared interaction module as the input of the first-layer 1×1 ordinary convolution plus ReLU activation layer. Similarly, the output of the first-layer visible light global framework is added to the first-layer visible light shared interaction module as the input of the second-layer visible light global framework. The output of the second-layer visible light global framework is added to the second-layer visible light shared interaction module as the input of the third-layer visible light global framework. The output of the third-layer visible light global framework is added to the third-layer visible light shared interaction module as the input of the first-layer 1×1 ordinary convolution plus ReLU activation layer. The global framework includes a feature module, an attention module, and two 1×1 ordinary convolutional layers in sequence; The feature module sequentially includes a 3×3 depth convolution layer, a normalization layer, a 1×1 ordinary convolution plus Gelu activation layer; then the input infrared / visible light image is added to the output of the 1×1 ordinary convolution plus Gelu activation layer; then sequentially passes through a 3×3 ordinary convolution plus Relu activation layer, a 1×1 ordinary convolution plus Relu activation layer; then the input infrared / visible light image is added to the output of the 1×1 ordinary convolution plus Relu activation layer as the input of the attention module; The attention module adopts the attention mechanism, and the specific operation is shown as follows: Attention=Norm(Q)+L(K*Q)V Where Norm(·) represents normalization, L(·) represents linear transformation, q represents the query matrix, k represents the matrix to be queried, v represents the value matrix, and Q is expressed as follows: Q=Conv 1×1 (Concat(Conv 1×1 (α*q),Conv 3×3 (α*q),Conv 5×5 (α*q)) where w a represents the learnable query matrix parameters, α represents the global attention query matrix, Concat(·) is the channel dimension splicing operation, Conv 1×1 (·) is a 1×1 convolution layer, Conv 3×3 (·) is a 3×3 convolution layer, Conv 5×5 (·) is a 5×5 convolutional layer; K is expressed as follows: K=Conv 1×1 (Concat(Conv 1×1 (β*k),Conv 3×3 (β*k),Conv 5×5 (β*k) where w b represents the learnable query matrix parameters, β represents the global attention query matrix, Concat(·) is the channel dimension splicing operation, Conv 1×1 (·) is a 1×1 convolution layer, Conv 3×3 (·) is a 3×3 convolution layer, Conv 5×5 (·) is a 5×5 convolutional layer; V is represented as follows: where w C Represents the learnable value matrix parameters.

4. The method for fusion of visible light and infrared images based on multi-scale attention according to claim 2, characterized in that: The local network includes three double residual modules in sequence; The output of the first-layer infrared double residual module is added to the first-layer infrared local interaction module as the input of the second-layer infrared double residual module. The output of the second-layer infrared double residual module is added to the second-layer infrared local interaction module as the input of the third-layer infrared double residual module. The output of the third-layer infrared double residual module is added to the third-layer infrared local interaction module to obtain an infrared local map. Similarly, the output of the first-layer visible light double residual module is added to the first-layer visible light local interaction module as the input of the second-layer visible light double residual module. The output of the second-layer visible light double residual module is added to the second-layer visible light local interaction module as the input of the third-layer visible light double residual module. The output of the third-layer visible light double residual module is added to the third-layer visible light local interaction module to obtain a visible light local map. The dual residual module performs the following steps: The double residual module includes 1×1 ordinary convolution + batch normalization + Tanh activation layer, 3 extraction modules, 1×1 ordinary convolution + batch normalization layer; then added to the infrared image and then activated by Relu; then passed through 1×1 ordinary convolution + batch normalization + Tanh activation layer, 3×3 ordinary convolution + batch normalization + Relu activation layer, 1×1 ordinary convolution + batch normalization layer; then added to the output of the skip module; finally, Tanh activation; The extraction module includes 3×3 depth convolution + batch normalization + Relu activation layer, 3×3 normal convolution + batch normalization + Relu activation layer in sequence; The jump module performs the following steps: The infrared / visible light image is passed through a 1×1 normal convolution + batch normalization + Relu activation layer and a 1×1 normal convolution + batch normalization layer in sequence.

5. The method for fusion of visible light and infrared images based on multi-scale attention according to claim 2, characterized in that: The specific operations performed by the shared interaction module in the interaction network are as follows: in is the output of the infrared global framework at layer i, is the output of the visible light global framework at layer i, is the output of the infrared sharing interaction module at layer i, is the output of the visible light sharing interaction module in the i-th layer, δ(·) represents the ReLU function, ⊙ represents the Hadamard product, Conv 1x1 (·) is a 1×1 convolution, Conv 3x3 (·) is a 3×3 convolution, PConv 5x1 (·) is a 5×1 strip convolution, PConv 1x5 (·) is a 1×5 strip convolution, SConv 3x3 (·) is a 3×3 dilated convolution; The specific operations performed by the local interaction module are as follows: in is the output of the i-th layer infrared double residual module, is the output of the i-th layer visible light double residual module, is the output of the infrared sharing interaction module at layer i, is the output of the visible light sharing interaction module at layer i, Conv 1×1 is a 1×1 convolution layer, Conv 3×3 is a 3×3 convolutional layer, Conv 5×5 is a 5×5 convolutional layer, ⊕ represents element summation, R(·) represents the ReLU function, and ⊙ represents the Hadamard product; (4) To integrate the network, perform the following steps: A Hadamard product operation is performed on the infrared global image and the visible light global image to obtain a global fusion image; The infrared local image and the visible light local image are added together to form a local fusion image; Performing a splicing operation on the global fusion map and the local fusion map to obtain a fusion map; (5) Reconstruct the network and perform the following steps: The reconstruction network includes 5 convolutional layers; The fusion graph is sequentially fed into a 1×1 ordinary convolution layer with Leaky Relu activation, a 3×3 ordinary convolution layer with Leaky Relu activation, a 3×3 ordinary convolution layer with Leaky Relu activation, a 3×3 ordinary convolution layer with Leaky Relu activation, and a 1×1 ordinary convolution layer with Tanh activation to obtain the final fusion graph.

6. The method for fusion of visible light and infrared images based on multi-scale attention according to claim 1, characterized in that: The loss function L in step S2.3 total Based on texture loss L text , correlation loss L corr , strength loss L in and the structural similarity index SSIM loss L ssim ; Loss function L total As shown below: L total =L text +αL corr +βL in +γL ssim The weight parameters α, β, and γ are set as learnable parameters; Among them F f represents the final fused image, F v represents the input visible light image, F I represents the input infrared image, represents the gradient operator, ||·||1 represents L1 norm normalization, H represents the height of the input image, W represents the width of the input image, θ i (i=1,2) represents infrared and visible light measurement parameters; in represents the infrared global image, represents the global visible light map, represents the infrared local image, represents the local visible light map, O(·) represents the correlation coefficient operator; Among them F f represents the final fused image, F v represents the input visible light image, F I represents the input infrared image, ||·||1 represents L1 norm normalization, H represents the height of the input image, W represents the width of the input image, θ1 and θ2 represent infrared and visible light measurement parameters respectively; Among them F f represents the final fused image, F V represents the input visible light image, F I represents the input infrared image, SSIM(·,·) represents the structural similarity index; According to the loss function, the fusion model is trained using the infrared training set and the visible light training set data to obtain a trained final fusion model.

7. A visible light and infrared image fusion system based on multi-scale attention, characterized in that: Used to implement the method described in any one of claims 1 to 6; the system includes an infrared sensor, a visible light sensor, an image receiving module, a fusion algorithm module, a bus and a fusion image display; The infrared sensor and the visible light sensor respectively capture infrared and visible light images and transmit them to the signal receiving module; The signal receiving module receives the input infrared and visible light images and transmits them to the fusion algorithm module through the bus. The infrared and visible light images fused by the fusion algorithm are then transmitted to the fusion image display through the bus. The fused image display shows the fused image result.

Images