An infrared and visible light image fusion method based on attention and threshold segmentation

Abstract

The present application relates to the field of image fusion, an infrared and visible light image fusion method based on attention and threshold segmentation, in which the input ratio of infrared and visible light images is reasonably allocated through experimental analysis in the early stage; different levels of features extracted from two different types of pictures are reversely fused at the element level to reduce information loss, and deep features receive all shallow feature element information, so that image features are fully extracted, information exchange is carried out and enhancement is achieved by using an attention interaction module when deep level convolution is reversely transmitted in a double-flow path; a residual structure is applied in the feature reconstruction stage to weaken gradient disappearance and prevent degradation; finally, threshold segmentation is introduced to optimize the loss function and accelerate the fusion speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image fusion, and in particular to an infrared and visible light image fusion method based on attention and threshold segmentation. BACKGROUND

[0002] Infrared and visible light image fusion is an important branch of image fusion. The essence of image fusion is to combine the most important information in the source images to generate a single image with more information and better visual effects. Infrared imaging has strong detection and recognition ability for targets, is not affected by bad environments such as rain, snow, wind and frost, and has clear target imaging, high accuracy and strong anti-interference ability. However, it also has problems such as low pixel resolution, poor contrast and blurred background texture. Visible light images conform to human visual characteristics and have high resolution, which can reflect rich texture and scene information. However, visible light images are easily affected by environmental factors, and in the case of interference, visible light images cannot highlight the scene target. The images obtained by infrared and visible light sensors are complementary and fused, and the fused image has both the significant target information of the infrared image and the background texture information of the visible light image. At present, infrared and visible light fusion technology is widely used in image enhancement, target recognition, target detection, target tracking, agricultural automation, remote sensing detection and other fields.

[0003] Existing traditional methods usually use the same transformation or representation to extract features from source images without considering the inherent characteristics of infrared and visible images; in most methods, the measurement of activity level and the design of fusion rules are manual, which often becomes more complex. Unlike traditional image fusion algorithms that rely on human measurement and design of fusion rules, deep learning algorithms can train end-to-end fusion through autonomous learning of the network without human intervention. Convolutional neural network methods rely on the powerful fitting ability of neural networks to extract and reconstruct effective information under the guidance of a carefully designed loss function, and have great potential in the field of image fusion. Based on the particularity of the infrared and visible light image fusion task, the fused image needs to retain most of the information of the source images, and simple forward convolution operations are difficult to retain important information of the source images, and too abstract semantic information will bring difficulties to feature reconstruction. SUMMARY

[0004] The technical problem to be solved by the present application is to overcome the defects in the above related technologies, provide an infrared and visible light image fusion method based on attention and threshold segmentation, and reasonably allocate the input ratio of infrared and visible light images through early experimental analysis; the different level features extracted from two different types of pictures are reversely fused at the element level, information loss is reduced, and deep features receive all shallow feature element information, so that image features are fully extracted, attention interaction modules are used for information exchange and enhancement when deep level convolution is reversely transmitted in the double flow path; residual structure is applied in the feature reconstruction stage to weaken gradient disappearance and prevent degradation; finally, threshold segmentation is introduced to optimize the loss function and accelerate the fusion speed.

[0005] The technical solution adopted by the present application is: an infrared and visible light image fusion method based on attention and threshold segmentation, comprising the following steps:

[0006] S1: constructing an image fusion network, including: an input module, an infrared path, a visible light path, a channel attention interaction module, a spatial attention interaction module, a feature reconstruction module, and a threshold segmentation module; the input of the input module is an infrared image I ir and a visible light image I vi , in the input module, two same infrared images I ir and one corresponding visible light image I vi are spliced in the channel dimension to form a three-dimensional image as the input A of the infrared path, two same visible light images I vi and one corresponding infrared image I irThe three-dimensional image is formed as input B of a visible light path; the infrared path and the visible light path use a 5-layer convolutional neural network for forward feature extraction to obtain a feature vector, the feature vector of each layer includes an infrared feature vector A and a visible light feature vector B, and with the network layer number increasing from 1 to 5, a shallow layer convolutional feature vector of the first layer and the second layer gradually forms a deep layer convolutional feature vector of the fourth layer and the fifth layer; the fifth layer feature vector, that is, the infrared feature vector A1 and the visible light feature vector B1, are selected as inputs of a channel attention interaction module, then the outputs A1' and B1' of the channel attention interaction module are respectively element-level fused with A1 and B1 and the fourth layer feature vector to obtain A2 and B2, corresponding elements are added and the feature shape is unchanged, A2 and B2 are selected as inputs of a spatial attention interaction module, then the outputs A2' and B2' of the spatial attention interaction module are respectively element-level fused with A2 and B2 and the second layer feature vector to obtain the final infrared path output and the visible light path output; the infrared path and the visible light path first process the deep layer feature vector and then transmit it to the shallow layer feature vector, that is, the feature vector is transmitted in reverse; then the infrared path output and the visible light path output are fused, that is, the two outputs are spliced in the channel dimension to obtain a feature vector F; then a feature reconstruction module performs feature reconstruction, that is, the feature vector F is reduced in dimension by using a convolutional neural network and a residual structure, and finally a one-dimensional fused image I is output f ; a threshold segmentation module segments the infrared image I ir to obtain a threshold segmentation image I th , and a loss function is established around the fused image, the infrared image, the visible light image and the threshold segmentation image to control the output of the final fused image.

[0007] S2: using a plurality of pairs of infrared images and visible light images as a training set to train the image fusion network, using the trained image fusion network to perform image fusion to obtain a fused image.

[0008] The 5-layer convolutional neural network is used for forward feature extraction to obtain a feature vector, specifically: the 5-layer convolutional neural network is composed of 5×5 convolution kernels, 3×3 convolution kernels, 1×1 convolution kernels, 3×3 convolution kernels and 1×1 convolution kernels with a step of 1, and each layer of convolutional neural network operation is used in conjunction with a Batch Normalization batch normalization layer and a LeakyReLu activation function; the feature vector sizes of the first layer to the fifth layer of the convolutional neural network are H×W×16, H×W×64, H×W×32, H×W×64 and H×W×64, wherein H and W are the first dimension height and the second dimension width of the feature, and H and W remain unchanged during the convolution process.

[0009] Both the channel attention interaction module and the spatial attention interaction module have a two-input, two-output structure, and the feature dimensions of the inputs and outputs remain unchanged. Specifically, the channel attention interaction module first concatenates the two input feature vectors A1 and B1 along the channel dimension to obtain feature vector F1, and then performs the following operation on F1:

[0010]

[0011] In equation (1) f gapc (·) represents global average pooling with channels unchanged. H, W, and C refer to the first dimension height, the second dimension width, and the third dimension number of channels of F1, respectively. Input c This represents the number of channels for the input feature vector F1, i.e., the spatial dimension H×W of the C channels; Input c (i,j) represents the coordinates of an element point in each channel spatial dimension, where i∈(1,H), j∈(1,W). By compressing the C channel spatial dimensions H×W, a 1×1×C dimensional vector g is obtained. c Then for g c Perform the following operations:

[0012] f c =fc(g c )=σ(W2δ(W1g c (2)

[0013] In equation (2), fc(·) is the fully connected layer operation, which is the operation on the input g. c Attention weights are assigned, σ represents the Sigmoid activation function, δ represents the LeakyReLU activation function, W1 and W2 are the weights of the first and second fully connected layers, respectively, and the two fully connected layers are paired with g. c Adaptive recalibration, through the first fully connected layer, g c The number of channels is reduced to 1 / r of the original number, and the second fully connected layer will then reduce the number of channels by g. c The number of channels is increased to the same number as A1 and B1, and the attention weight parameter f is finally obtained. c Multiply by the inputs A1 and B1 respectively to obtain the two output feature vectors A1' and B1' of the channel attention interaction module, where r is 8;

[0014] The spatial attention interaction module is specifically described as follows: First, the two input feature vectors A2 and B2 of the spatial attention interaction module are concatenated along the channel dimension to obtain feature vector F2. Then, the following operations are performed on F2:

[0015]

[0016] In equation (3) f gaps(·) can be described as space-invariant global average pooling, where H, W, and C refer to the height, width, and number of channels of F2, respectively. H,W This refers to the spatial dimension of the input feature vector F2, specifically C channels of spatial dimension H×W. H,W (k) represents the k-th channel of the spatial dimension H×W, where k∈(1,C). A 1H×W element g is obtained by averaging the C channels of each element in H×W. s Then for g s Perform the following operations:

[0017] f s =f 7×7 (g s (4)

[0018] In equation (4), f 7×7 (·) represents a convolution operation with a 7×7 kernel, which is then activated by a sigmoid function to obtain f. s Multiplying these values ​​by inputs A2 and B2 respectively yields two output feature vectors, A2' and B2', from the spatial attention interaction module.

[0019] The feature reconstruction module performs feature reconstruction by using a convolutional neural network and a residual structure to reconstruct the feature vector F. First, it uses a 3×3 convolutional kernel with a stride of 1 and a 1×1 convolutional kernel to perform convolution operations on the feature vector F. Then, it uses a 1×1 convolutional kernel with a stride of 1 as a skip layer to perform convolution operations on the feature vector F, finally obtaining a fused image of H×W×1.

[0020] The threshold segmentation module uses threshold segmentation to segment infrared images I. ir Obtain threshold segmentation image I th The method is as follows:

[0021]

[0022] In equation (5), f(i,j) represents the infrared image I. ir The pixel value of a pixel, where thresh represents the threshold, is the average value of all pixels in the image. This formula means that if the pixel value of a pixel is greater than the threshold, the pixel value remains unchanged; otherwise, it is assigned 0.

[0023] The loss function includes the total loss L. total as follows:

[0024] L total =L content +L gradient (6)

[0025] In equation (6) L total For the total loss, L contentL is a content loss gradient L is a gradient loss.

[0026] The content loss L content As follows:

[0027]

[0028] In formula (7), ||·||2 indicates an L2 norm, I fusion I is a fusion image, ir I vi and I th are respectively an infrared image, a visible light image and a corresponding infrared threshold segmentation image, H and W are respectively a height and a width of the image, and a1, a2 and b are constants, and a1 < a2.

[0029] The gradient loss L gradient As follows:

[0030]

[0031] In formula (8), ||·||2 indicates an L2 norm, denotes a gradient operator, and are respectively a gradient form of the fusion image, the infrared image and the visible light image, H and W are respectively a height and a width of the image, and a and b are constants.

[0032] The beneficial effects of the present application are: (1) feature element level fusion is adopted in the fusion network, that is, corresponding elements of different feature vectors are added, and the dimension of the feature vector in the fusion process remains unchanged. The shallow network feature vector and the input are relatively close, and contain some fine-grained information, including image texture, edge and corner information, feature element level addition can add these fine-grained information to the feature vector, and the information amount under each dimension of the feature vector is increased, which obviously makes the semantic information of each picture more abundant, thereby enriching the detail information of the final fusion image, which is beneficial to the final image description. (2) Two different attention interaction modules, channel attention interaction module and spatial attention interaction module, better extract image features, improve the ability to retain important feature information, and enhance the information flow between the two paths. After channel attention interaction, the model focuses on the channel features with the largest global information amount of the image, and suppresses the unimportant channel features. The spatial attention interaction module is used to allocate attention on the feature space layer, reduce noise and retain the information required by the fusion image. (3) The threshold segmentation image is introduced to optimize the loss function. The threshold segmentation image can retain significant infrared targets, suppress the interference of non-infrared information, improve the information retention degree of the infrared target, enhance the contrast between infrared information and visible light information, speed up the loss reduction process and improve the training speed. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to make the technical solutions in the embodiments of the present application or the related art clearer, the accompanying drawings needed in the embodiments or the related art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0034] Figure 1 The method flowchart of the embodiment of the present application is shown in the figure.

[0035] Figure 2 The fusion algorithm framework of the embodiment of the present application is shown in the figure.

[0036] Figure 3 The channel attention interaction module structure diagram of the embodiment of the present application is shown in the figure.

[0037] Figure 4 The spatial attention interaction module structure diagram of the embodiment of the present application is shown in the figure.

[0038] Figure 5 The infrared and visible light image fusion result 1 of the embodiment of the present application is shown in the figure.

[0039] Figure 6 The infrared and visible light image fusion result 2 of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0040] In order to make the technical solutions in the embodiments of the present application or the related art clearer, the accompanying drawings needed in the embodiments or the related art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0041] The embodiments are described in detail below with reference to the accompanying drawings:

[0042] In the embodiments of the present application, Figure 1 and Figure 2 The flowchart and the fusion algorithm framework of the infrared and visible light image fusion method based on attention and threshold segmentation of the embodiment of the present application are shown in the figures, and the steps include:

[0043] Step 1, image acquisition: infrared images and visible light images under the same scene are captured by using a visible light camera and an infrared camera, a plurality of groups of image data under different light intensities are collected, 24 pairs are selected as training data, and 17 pairs are selected as test data.

[0044] Step 2, Image Preprocessing:

[0045] The 24 pairs of training images were then processed by thresholding the selected infrared images as follows:

[0046]

[0047] In the formula, f(i,j) represents the pixel value of the image pixel, and thresh represents the threshold, which is the average value of the pixels in the entire image. This formula means that if the pixel value of a certain pixel is greater than the threshold, the pixel value remains unchanged, otherwise it is assigned 0.

[0048] The visible light image, infrared image, and their corresponding threshold segmentation image were augmented. Each image was then cropped into an H×W×1 image with a step size of 14 in both dimensions, resulting in 3 sets of 20,036 images.

[0049] Step 3, Model Training Input: Single selection of infrared image I ir With visible light image I vi For each pair, two identical infrared images and one corresponding visible light image are stitched together along the channel dimension to form a 3D image, which serves as the input A for the infrared path. For each pair, two identical visible light images and one corresponding infrared image are stitched together along the channel dimension to form a 3D image, which serves as the input B for the visible light path. During training, 32 pairs of infrared and visible light images are selected simultaneously in each iteration.

[0050] Step 4, Feature Extraction: Both the infrared path and the visible light path are extracted using layers 1 to 5. The 5-layer network consists of 5×5 convolutional kernels, 3×3 convolutional kernels, 1×1 convolutional kernels, 3×3 convolutional kernels, and 1×1 convolutional kernels with a stride of 1. Each convolutional operation is used in conjunction with a Batch Normalization layer and a LeakyReLu activation function. The feature vector sizes of layers 1 to 5 are H×W×16, H×W×64, H×W×32, H×W×64, and H×W×64, respectively. H and W remain unchanged during the convolution process.

[0051] Step 5, Attention Interaction:

[0052] Step 1: Input the 5th layer feature vector A1 of the infrared path and the 5th layer feature vector B1 of the visible light path into the channel attention interaction module respectively.

[0053] like Figure 3 As shown in the channel attention interaction module, the two input feature vectors A1 and B1 are first concatenated along the channel dimension to obtain feature vector F1. The following operations are then performed on F1:

[0054]

[0055] In the formula f gapc (·) can be described as channel-invariant global average pooling, where H, W, and C refer to the height, width, and number of channels of F1, respectively. c This represents the spatial dimension H×W of the C channels of the input feature vector F1; Input c (i,j) represents the coordinates of an element point in each channel's spatial dimension, where i∈(1,H) and j∈(1,W). A 1×1×C dimensional vector g is obtained by compressing the C channel spatial dimensions H×W. c Then, regarding g... c Perform the following operations:

[0056] f c =fc(g c )=σ(W2δ(W1g c ))

[0057] In the formula, fc(·) represents the fully connected layer operation, which can be described as the operation on the input g. c Attention weights are assigned, where σ represents the Sigmoid activation function, δ represents the LeakyReLU activation function, and W1 and W2 are the weights of the first and second fully connected layers, respectively. The two fully connected layers are paired with g. c Adaptive recalibration, through the first fully connected layer, g c The number of channels is reduced to 1 / r of the original number, and the second fully connected layer will then reduce the number of channels by g. c The number of channels is increased to the same number as A1 and B1, and the attention weight parameter f is finally obtained. c The vectors are multiplied by the inputs A1 and B1 respectively to obtain the two output feature vectors A1' and B1' of the channel attention interaction module. In this method, r is set to 8.

[0058] Step 2: Element-wise fuse the 5th and 4th layer feature vectors of the infrared path with the output A1' of the channel attention interaction module to obtain A2; element-wise fuse the 5th and 4th layer feature vectors of the visible light path with the output B1' of the channel attention interaction module to obtain B2; and input both into the spatial attention interaction module.

[0059] like Figure 4 As shown in the spatial attention interaction module, the two input feature vectors A2 and B2 are first concatenated along the channel dimension to obtain feature vector F2. The following operations are then performed on F2:

[0060]

[0061] In the formula f gaps (·) can be described as space-invariant global average pooling, where H, W, and C refer to the height, width, and number of channels of F2, respectively.H,W represents the C channels of the spatial dimension for the input feature vector F2, i.e., the spatial dimension HxW, Input H,W (k) represents the kth channel of the spatial dimension HxW, where k e (1, C). An HxW dimensional element g is obtained by averaging the C channels of each element in HxW s . Then g s is operated as follows:

[0062] f s = f 7×7 (g s )

[0063] where f 7×7 (·) represents a convolution operation with a 7x7 convolution kernel, and f s is obtained after sigmoid activation. f 7×7 and f s are multiplied by input A2 and B2 respectively to obtain two output feature vectors A2' and B2' of the spatial attention interaction module.

[0064] Step 3, element-level fusion is performed on the second layer feature vector of the infrared path, the output A1' of the channel attention interaction module, and the output A2' of the spatial attention interaction module to obtain the final output of the infrared path; element-level fusion is performed on the second layer feature vector of the visible light path, the output B1' of the channel attention interaction module, and the output B2' of the spatial attention interaction module to obtain the final output of the visible light path.

[0065] Step 6, feature fusion: the final output of the infrared path and the final output of the visible light path are spliced in the channel to obtain a feature vector F of HxWx128.

[0066] Step 7, feature reconstruction: as shown in ResBlock in Figure 2 , a 3x3 convolution kernel with a step size of 1 and a 1x1 convolution kernel are used to reconstruct and compress the channels of the fused feature, a 1x1 convolution kernel with a step size of 1 is used as a skip layer to compress and match the channels, a residual structure is constructed to preserve feature information to a large extent, and finally a fused image of HxWx1 is obtained.

[0067] Step 8, use a loss function to control the feature preservation relationship between the final fused image and the source image:

[0068] The loss function includes:

[0069] L total = L content + L gradient

[0070] where L total is the total loss, L content is the content loss, and L gradientThe gradient loss is:

[0071] The content loss includes:

[0072]

[0073] where ||·||2 denotes the L2 norm, I fusion is a fusion image, I ir , I vi and I th are an infrared image, a visible light image and a corresponding infrared threshold segmentation image respectively, H and W are the height and width of the image respectively, a1, a2 and b are constants, and a1 < a2.

[0074] The gradient loss includes:

[0075]

[0076] where ||·||2 denotes the L2 norm, denotes a gradient operator, and are gradient forms of the fusion image, the infrared image and the visible light image respectively, H and W are the height and width of the image respectively, and a and b are constants.

[0077] Step 9, training the fusion network: the network parameters are updated by the Adam optimizer, and the learning rate is 0.0001. The network is built and trained based on the Tensorflow deep learning framework, and the hardware environment for model training is an NVIDIA TITAN XP 12GB.

[0078] Step 10, testing the fusion network: 17 pairs of infrared and visible light test images are input into the network, and finally 17 fusion images are obtained.

[0079] In this embodiment, Figure 5 and Figure 6 are infrared and visible light image fusion results of the method in the TNO data set and the MSRS data set; Figure 5 is a night image in a military scene, Figure 6 is a daytime and nighttime image with different brightness in a complex road scene.

[0080] Further, as shown in Figure 5 and Figure 6 , the fusion image information of the method is rich, the light and dark contrast is large, the definition is high, the visual information is retained relatively complete, and the fusion effect of the infrared target and the visible light background is the best. In particular, as shown in Figure 6As shown, the method of the present application performs certain data enhancement on the source image with low brightness, clearly displays environmental details, and the fused result has high visibility, thereby improving the visibility of the image.

[0081] In the description of the present specification, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0082] The above merely describes specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which shall be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for fusing infrared and visible light images based on attention and threshold segmentation, characterized in that, Includes the following steps: S1: Construct an image fusion network, including: an input module, an infrared path, a visible light path, a channel attention interaction module, a spatial attention interaction module, a feature reconstruction module, and a threshold segmentation module; the input to the input module is an infrared image I describing the same scene at the same time. ir With a visible light image I vi Inside the input module, two identical infrared images are stitched together along the channel dimension. ir And one corresponding visible light image I vi A 3D image is generated as input A for the infrared path, and two identical visible light images I are stitched together along the channel dimension. vi And 1 corresponding infrared image I ir A 3D image is formed as input B for the visible light path. For both the infrared and visible light paths, a 5-layer convolutional neural network is used for forward feature extraction to obtain feature vectors. Each layer's feature vector includes an infrared feature vector A and a visible light feature vector B. As the number of network layers increases from 1 to 5, the feature vectors from the 1st and 2nd shallow convolutional layers are gradually transformed into the 4th and 5th deep convolutional layers. The 5th layer feature vectors, namely the infrared feature vector A1 and the visible light feature vector B1, are selected as inputs to the channel attention interaction module. Then, the outputs A1' and B1' of the channel attention interaction module are element-wise fused with A1, B1, and the 4th layer feature vector to obtain A2 and B2. Corresponding elements are added together, and the feature vectors are... Without changing the shape, A2 and B2 are selected as inputs to the spatial attention interaction module. Then, the outputs A2' and B2' of the spatial attention interaction module are element-wise fused with A2, B2, and the second-layer feature vector to obtain the final infrared path output and visible light path output. The infrared and visible light paths first process the deep feature vectors before transmitting them to the shallow feature vectors, i.e., feature vector reverse transmission. Then, the infrared path output and the visible light path output are fused, i.e., the two outputs are concatenated along the channel dimension to obtain the feature vector F. Subsequently, the feature reconstruction module performs feature reconstruction, i.e., using a convolutional neural network and residual structure to reduce the dimensionality of the feature vector F, finally outputting a one-dimensional fused image I. f The threshold segmentation module uses threshold segmentation to segment infrared images. ir Obtain threshold segmentation image I th A loss function is established based on the fused image and the infrared image, visible light image, and threshold segmentation image to control the output of the final fused image; S2: The image fusion network is trained using multiple pairs of infrared and visible light images as training sets. The trained image fusion network is then used to perform image fusion to obtain a fused image.

2. The infrared and visible light image fusion method based on attention and threshold segmentation according to claim 1, characterized in that: The method of using a 5-layer convolutional neural network to extract feature vectors for forward feature extraction is as follows: the 5-layer convolutional neural network consists of 5×5 convolutional kernels, 3×3 convolutional kernels, 1×1 convolutional kernels, 3×3 convolutional kernels and 1×1 convolutional kernels with a stride of 1. Each layer of the convolutional neural network is used in conjunction with a batch normalization layer and a LeakyReLU activation function. The feature vector sizes of the 1st to 5th layers of the convolutional neural network are H×W×16, H×W×64, H×W×32, H×W×64 and H×W×64, respectively, where H and W are the first dimension height and the second dimension width of the feature. H and W remain unchanged during the convolution process.

3. The infrared and visible light image fusion method based on attention and threshold segmentation according to claim 1, characterized in that: Both the channel attention interaction module and the spatial attention interaction module have a two-input, two-output structure, and the feature dimensions of the inputs and outputs remain unchanged. Specifically, the channel attention interaction module first concatenates the two input feature vectors A1 and B1 along the channel dimension to obtain feature vector F1, and then performs the following operation on F1: In equation (1) f gapc (·) represents global average pooling with channels unchanged. H, W, and C refer to the first dimension height, the second dimension width, and the third dimension number of channels of F1, respectively. Input c This represents the number of channels for the input feature vector F1, i.e., the spatial dimension H×W of the C channels; Input c (i,j) represents the coordinates of an element point in each channel spatial dimension, where i∈(1,H), j∈(1,W). By compressing the C channel spatial dimensions H×W, a 1×1×C dimensional vector g is obtained. c Then for g c Perform the following operations: f c =fc(g c )=σ(W2δ(W1g c )) (2) In equation (2), fc(·) is the fully connected layer operation, which is the operation on the input g. c Attention weights are assigned, σ represents the Sigmoid activation function, δ represents the LeakyReLU activation function, W1 and W2 are the weights of the first and second fully connected layers, respectively, and the two fully connected layers are paired with g. c Adaptive recalibration, through the first fully connected layer, g c The number of channels is reduced to 1 / r of the original number, and the second fully connected layer will then reduce the number of channels by g. c The number of channels is increased to the same number as A1 and B1, and the attention weight parameter f is finally obtained. c Multiply by the inputs A1 and B1 respectively to obtain the two output feature vectors A1' and B1' of the channel attention interaction module, where r is 8; The spatial attention interaction module is specifically described as follows: First, the two input feature vectors A2 and B2 of the spatial attention interaction module are concatenated along the channel dimension to obtain feature vector F2. Then, the following operations are performed on F2: In equation (3) f gaps (·) can be described as space-invariant global average pooling, where H, W, and C refer to the height, width, and number of channels of F2, respectively. H,W This refers to the spatial dimension of the input feature vector F2, specifically C channels of spatial dimension H×W. H,W (k) represents the k-th channel of the spatial dimension H×W, where k∈(1,C). A 1H×W element g is obtained by averaging the C channels of each element in H×W. s Then for g s Perform the following operations: f s =f 7×7 (g s ) (4) In equation (4) f 7×7 (·) represents a convolution operation with a 7×7 kernel, which is then activated by a sigmoid function to obtain f. s Multiplying these values ​​by inputs A2 and B2 respectively yields two output feature vectors, A2' and B2', from the spatial attention interaction module.

4. The infrared and visible light image fusion method based on attention and threshold segmentation according to claim 1, characterized in that: The feature reconstruction module performs feature reconstruction by using a convolutional neural network and a residual structure to reconstruct the feature vector F. First, it uses a 3×3 convolutional kernel with a stride of 1 and a 1×1 convolutional kernel to perform convolution operations on the feature vector F. Then, it uses a 1×1 convolutional kernel with a stride of 1 as a skip layer to perform convolution operations on the feature vector F, finally obtaining a fused image of H×W×1.

5. The infrared and visible light image fusion method based on attention and threshold segmentation according to claim 1, characterized in that: The threshold segmentation module uses threshold segmentation to segment infrared images I. ir Obtain threshold segmentation image I th The method is as follows: In equation (5), f(i,j) represents the infrared image I. ir The pixel value of a pixel, where thresh represents the threshold, is the average value of all pixels in the image. This formula means that if the pixel value of a pixel is greater than the threshold, the pixel value remains unchanged; otherwise, it is assigned 0.

6. The infrared and visible light image fusion method based on attention and threshold segmentation according to claim 1, characterized in that: The loss function includes the total loss L. total as follows: L total =L content +L gradient (6) In equation (6), L total For the total loss, L content For content loss, L gradient This is the gradient loss.

7. The infrared and visible light image fusion method based on attention and threshold segmentation according to claim 6, characterized in that: The content loss L content as follows: In equation (7), ||·||2 refers to the L2 norm, I fusion To fuse images, I ir I vi and I th These are the infrared image, the visible light image, and the corresponding infrared threshold segmentation image, respectively. H and W are the height and width of the image, respectively, and a1, a2, and b are constants. <a2。 8. The infrared and visible light image fusion method based on attention and threshold segmentation according to claim 6, characterized in that: The gradient loss L gradient as follows: In equation (8), ||·||2 refers to the L2 norm. Represents the gradient operator. and These are the gradient forms of the fused image, infrared image, and visible light image, respectively, where H and W are the height and width of the image, respectively, and α and β are constants.

Images