A method for infrared and visible light image fusion based on depth-guided filtering

Abstract

The invention relates to the technical field of image processing, and discloses an infrared-visible light image fusion method based on deep guided filtering, comprising the following steps: obtaining an infrared image and a visible light image to be fused; preliminarily fusing the infrared image and the visible light image to obtain a preliminary fused image; performing deep feature extraction on the preliminary fused image through an encoder; performing layer-by-layer decoding by using a decoder based on the feature map of the last layer, wherein the layer-by-layer decoding comprises: before performing decoding of the next layer, fusing detail information of an output result of a current decoding layer with an output result of a corresponding encoding layer through deep guided filtering, and using the fused feature map as input of the next layer decoding until a final decoding result is output; and obtaining a fused image by passing the final decoding result through a convolution layer and an activation layer; the invention can prevent the loss of feature texture details by means of guided filtering while deepening the network depth, and improve the information complementarity of the generated fused image.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and more particularly to an infrared and visible light image fusion method based on depth guided filtering. Background Art

[0002] At present, infrared and visible light fusion methods are mainly divided into traditional fusion methods and deep learning-based fusion methods, but the core idea is to make the generated fusion image have the significant intensity information of the infrared image and the texture detail information of the visible light image as much as possible. However, how to more effectively select the information in the infrared and visible light images that is helpful for the fusion image is a problem that both methods have been trying to solve.

[0003] Traditional methods use operators in the field of image processing, such as filters, wavelet decomposition, etc., to decompose the input infrared and visible light, generally into detail layers and base layers. The detail layer has rich details of the original image, and the base layer contains more information about the overall structure of the original image. Then, the fusion rules are manually and explicitly designed to fuse the base layer and detail layer generated by the decomposition, and finally generate a fused image. However, the fusion rules are manually designed, and when the fusion scene is changed, the fusion rules often need to be redesigned.

[0004] In recent years, deep learning methods have gradually achieved good results in the infrared and visible light fields. By designing a fusion network and leveraging the powerful fitting ability of deep learning models, it is possible to find the fitting distribution of infrared and visible light from the training data, and gradually iterate to generate a fusion image that meets the expected effect with the constraints of the loss function. However, how to design a fusion network to ensure a better fusion effect is often a difficult problem.

[0005] Therefore, how to balance the intensity information in infrared images and the detail information in visible light and integrate the effective information of infrared and visible light is an urgent problem that technicians in this field need to solve. Summary of the invention

[0006] In view of this, the present invention provides an infrared-visible light image fusion method based on depth guided filtering, which can prevent the loss of feature texture details by guided filtering while deepening the network depth, and improve the information complementarity of the generated fused image.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] An infrared and visible light image fusion method based on depth-guided filtering comprises the following steps:

[0009] Acquire the infrared image and visible light image to be fused;

[0010] Preliminarily fusing the infrared image and the visible light image to obtain a preliminary fused image;

[0011] Performing deep feature extraction on the preliminary fusion graph through an encoder;

[0012] Decoding is performed layer by layer based on the feature map of the last layer. The layer by layer decoding includes: before decoding the next layer, the output result of the current decoding layer and the output result of the corresponding encoding layer are fused with detail information through depth-guided filtering, and the fused feature map is used as the input of the next layer decoding until the final decoding result is output;

[0013] The final decoding result is passed through a convolution layer and an activation layer to obtain a fused image.

[0014] Furthermore, the preliminary fusion step includes:

[0015] Feature extraction is performed through convolutional layers and activation layers to obtain infrared feature maps and visible light feature maps;

[0016] Connect infrared and visible light features in the channel dimension and calculate channel attention weights;

[0017] The infrared feature map and the visible light feature map are weighted by the attention weights respectively by element-by-element multiplication to obtain a first branch feature map obtained by weighting the infrared feature map and a second branch feature map obtained by weighting the visible light feature map;

[0018] Connecting the first branch feature map with the visible light feature map to obtain a first branch enhanced feature map; connecting the second branch feature map with the infrared feature map to obtain a second branch enhanced feature map;

[0019] The first branch enhanced feature map and the second branch enhanced feature map are connected in series; and input into the parallel attention module to generate the final fusion weight, and the fusion weight is weighted to the first branch enhanced feature map and the second branch enhanced feature map respectively, and then the two weighted feature maps are added and fused element by element to generate a preliminary fusion map.

[0020] Furthermore, the step of calculating the channel attention weight includes:

[0021] Get the feature map after channel connection and perform global average pooling;

[0022] According to the result after global average pooling, multiple point-by-point convolutions are performed and the channel attention weight is output through the activation function.

[0023] Furthermore, the parallel attention module is composed of a channel attention branch and a spatial attention branch, and the specific steps of generating the fusion weight include:

[0024] The concatenated feature maps are obtained and input into the channel attention branch and the spatial attention branch respectively after point-by-point convolution.

[0025] The results of the channel attention branch and the spatial attention branch are convolved and then the fusion weight is obtained through the activation function.

[0026] Furthermore, the depth-guided filtering step includes:

[0027] Get the output of the previous encoding layer and the corresponding decoding layer, and filter them through the filtering layer respectively.

[0028] After performing dilated convolution and point-by-point convolution on the filtering results corresponding to the encoding layer in turn, the filtering results corresponding to the decoding layer are input into the linear layer to obtain a fused feature map.

[0029] Furthermore, the encoder includes three channel residual attention modules and a downsampling module between the channel residual attention modules. The encoding process is: the residual channel attention module first convolves the input feature map to obtain features, and then performs channel attention calculation on the obtained features to generate channel attention weights; through the residual connection, the obtained features are multiplied element by element with the attention weights to generate a weighted attention feature map, and the attention feature map is then added element by element to the input feature map to generate a new feature map; the new feature map is downsampled and input into the second channel residual attention module; the output of the second channel residual attention module is downsampled and input into the third channel residual attention module to obtain the final decoding result.

[0030] Furthermore, the decoder includes three channel residual attention modules and an upsampling module arranged between the channel residual attention modules.

[0031] A neural network model adopts the above-mentioned image fusion method, including:

[0032] An input module, used for acquiring infrared images and visible light images to be fused;

[0033] A preliminary fusion module, used for performing preliminary fusion of the infrared image and the visible light image to obtain a preliminary fused image;

[0034] An encoder, used for performing deep feature extraction on the preliminary fusion graph;

[0035] The decoder is used to decode layer by layer according to the feature map output by the encoder to obtain the final decoding result

[0036] A guided filtering module, used for obtaining the output of the encoder and the output of the decoder during layer-by-layer decoding, and performing guided fusion according to the output of the encoder for decoding of the next layer;

[0037] The output module is used to perform convolution and activation on the final decoding result to obtain a fused image.

[0038] A training method for a neural network model, using the above neural network model, comprises the following steps:

[0039] Acquire training data, where the training data includes an infrared image dataset and a visible light image dataset;

[0040] configuring the infrared image data set and the visible light image data set to generate an image data pair;

[0041] Inputting the image data pair into the neural network model to obtain a fused image;

[0042] The model loss is calculated according to the fused image, and the parameters of the neural network model are optimized.

[0043] Furthermore, the loss function used to calculate the model loss is as follows:

[0044] L total =L int +λ ρ L chl +(1-λ ρ )L struct

[0045] Among them, L int represents the target intensity loss of the fused image and the infrared image, λ ρ Represents the weight coefficient, which is a hyperparameter set artificially; L chl represents the Charbonnier loss of the fused image and the original visible light image, L struct Represents the SSIM structural consistency loss of the fused image and the visible light image;

[0046]

[0047] Among them, H and W are the height and width of the picture respectively. is the fused image, I ir is an infrared image, ||·||1 represents the L1 norm, and its mathematical meaning is the addition of absolute values;

[0048]

[0049] in, is the fused image, I viis a visible light image, ∈ is a smoothing parameter used to keep the function differentiable, and its value is 10 -3 The constant.

[0050]

[0051] L struct =1-SSIM(X,Y)

[0052] Among them, X and Y represent two images, where X represents the visible light image and Y represents the fused image. μ and σ represent the mean and standard deviation respectively; σ XY Represents the covariance between X and Y; C1 and C2 are stability coefficients. It is used to measure the structural similarity between the fused image and the visible light.

[0053] It can be known from the above technical solution that, compared with the prior art, the present invention discloses an infrared visible light image fusion method based on deep guided filtering. By using a deep learning guided filtering method to enhance the features of each layer, the encoder output feature map and the decoder feature map can be better filtered for the first time, the texture features of each feature map can be retained and fused, and the redundant texture information between the two feature maps can be removed, so that the feature map input to the next layer of decoder contains richer information. Through preliminary fusion, the redundant information between the two modalities can be initially filtered, and the common feature information of the two modalities can be enhanced. The features of the two modalities can be shallowly interacted in space and channels. On the one hand, the convergence speed of the entire network can be accelerated, and on the other hand, the network design can be made more lightweight. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0055] Figure 1 A schematic diagram of an infrared and visible light image fusion method based on depth-guided filtering provided by the present invention;

[0056] Figure 2 A schematic diagram of the framework of an infrared-visible light image fusion method based on depth-guided filtering provided by the present invention;

[0057] Figure 3 This is a schematic diagram of the preliminary integration method architecture in the present invention;

[0058] Figure 4 It is a schematic diagram of the encoding and decoding process based on depth-guided filtering of the present invention;

[0059] Figure 5 It is a schematic diagram of the process of depth guided filtering in the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Example 1

[0062] like Figure 1 and Figure 2 The embodiment of the present invention discloses an infrared visible light image fusion method based on depth-guided filtering, comprising the following steps:

[0063] S1: Obtain the infrared and visible light image dataset to be fused; use the corresponding registered infrared and visible light pairs as network input.

[0064] S2: The registered infrared-visible light pair is simultaneously input into the preliminary fusion module. The preliminary fusion module obtains the infrared image and the visible light image feature map through the convolution layer and the activation layer. The shallow fusion module based on channel attention then fuses the two image feature maps to perform preliminary information fusion and obtain a preliminary fusion map.

[0065] The general method is to directly splice the infrared and visible light images in the channel dimension, and then input them into the subsequent network for processing. However, considering that there is a lot of redundancy and complementarity in the channel dimension of infrared and visible light images, simple channel dimension splicing increases the redundancy of information on the one hand, and does not more comprehensively integrate the complementary information between multimodal features on the other hand. Therefore, the present invention provides the following embodiments: Figure 2 and Figure 3 , the initial fusion steps include:

[0066] S21: After batch loading the dataset image pairs, resize all infrared and visible light image pairs to a uniform size of H*W*C. Perform feature extraction through 2D convolutional layer and RELU activation layer to obtain infrared feature map and visible light feature map, and input them into the shallow fusion module.

[0067] S22: The shallow fusion module connects the infrared and visible light features in the channel dimension and calculates the channel attention weights.

[0068] S23: Use element-by-element multiplication to weight the infrared feature map and the visible light feature map respectively with the attention weight to enhance the representation, thereby obtaining a first branch feature map obtained by weighting the infrared feature map and a second branch feature map obtained by weighting the visible light feature map.

[0069] S24: Connect the first branch feature map with the visible light feature map to obtain a first branch enhanced feature map; connect the second branch feature map with the infrared feature map to obtain a second branch enhanced feature map.

[0070] S25: Connect the first branch enhanced feature map and the second branch enhanced feature map in series; input them into the parallel attention module to generate the final fusion weight, weight the fusion weight to the first branch enhanced feature map and the second branch enhanced feature map respectively, and then add and fuse the two weighted feature maps element by element to generate a preliminary fusion map.

[0071] In this embodiment, step S22 may specifically include:

[0072] S221: Obtain the feature map after channel connection and perform global average pooling.

[0073] S222: Perform multiple point-by-point convolutions based on the results of global average pooling and output the channel attention weights through an activation function.

[0074] The formula for generating channel attention weights is as follows:

[0075]

[0076] in, represents the infrared and visible light features of the i-th layer, respectively. C(·) represents the connection operation in the channel dimension. CAP(·) and δ(·) represent the global average pooling and sigmoid activation function, respectively. Pw-Conv n (·) represents the cascaded point-wise convolutional layer, where n is 2 and W is attention Represents the generated attention weight.

[0077] Furthermore, the processing of the parallel attention module in S25 may include:

[0078] S251: Obtain the concatenated feature map, and input it into the channel attention branch and the spatial attention branch after point-by-point convolution. The channel attention branch includes a global average pooling layer and two point-by-point convolution layers; the spatial attention branch consists of two point-by-point convolution layers.

[0079] S252: Convolve the results of the channel attention branch and the spatial attention branch and obtain the fusion weight through the activation function.

[0080] The formula for generating fusion weights is as follows:

[0081]

[0082] in, and is the infrared and optical features weighted by attention weights, GAP(·) and δ(·) represent global average pooling and sigmoid activation function respectively, Pw-Conv(·) represents the point-by-point convolution layer, Pw-Conv n (·) represents the cascaded point-wise convolutional layer, where n is 2. and Represent the channel attention feature map and spatial attention feature map respectively. represents the element-wise multiplication operation, Represents the final fusion weight generated.

[0083] S3: Deep feature extraction of the preliminary fusion graph is performed through the encoder.

[0084] In one embodiment, if Figure 4 The encoder includes three channel residual attention modules and downsampling modules between the channel residual attention modules. Therefore, among the three encoding layers in the encoder, the first two encoding layers are composed of channel residual attention modules and downsampling modules, and the last encoding layer is composed of channel residual attention modules. Among them, the encoding layer is the feature extraction module, and the preliminary fusion module is input into the encoder network to obtain the feature map output by the corresponding feature extraction module.

[0085] The residual channel attention module first convolves the input feature map to extract features, then performs channel attention calculation on the extracted features to generate channel attention weights, and then multiplies the extracted features and the attention weights element by element through residual connections to generate weighted attention feature maps. The attention feature maps are then element-wise added to the input feature maps to generate new feature maps, and the generated original feature maps are downsampled to generate thumbnail feature maps, completing a layer of feature extraction for subsequent module calculations. The output feature maps of each layer of feature encoding modules will be retained, and the output feature maps of the last layer of feature encoding modules will be input into the decoder.

[0086] The first two layers of the encoder process the image using the following formula:

[0087] X input =Conv(ReLU(Conv((f input )))

[0088]

[0089] X inputFor the input of the first layer of the encoder, the dimension is H*W*C. If it is the output of the previous encoder, the dimension is equal to the output dimension of the previous encoding layer. Represents the output feature map of the i-th encoding layer, i can take values ​​of 1 and 2, where The dimension is H / (2*i)*W / (2*i)*C / 2.

[0090] The third layer is different from the first two layers in that there is no downsampling operation at the end. The intermediate encoded feature maps of the first and second layer encoding outputs in the encoder are retained, and the output of the third layer is input to the decoder.

[0091] S4: Perform layer-by-layer decoding based on the feature map of the last layer. The layer-by-layer decoding includes: before decoding the next layer, the output result of the current decoding layer and the output result of the corresponding encoding layer are fused with detail information through depth-guided filtering, and the fused feature map is used as the input of the next layer of decoding until the final decoding result is output.

[0092] In one embodiment, if Figure 4 and Figure 5 The decoder structure corresponds to the encoder structure. The decoder includes three channel residual attention modules and an upsampling module arranged between the channel residual attention modules. Therefore, the first two decoding layers are composed of a channel residual attention module and an upsampling module, and the third decoding layer is composed of only one channel residual attention module, where the decoding layer is the feature restoration module.

[0093] During decoding:

[0094] Starting from the second layer, the received inputs are all from the output of the depth-guided filter. The restored feature map output by the first decoding layer will be input into the depth-guided filter at the same time as the intermediate coded feature map output by the second coding layer to restore and enhance the detail information. Finally, the detail-enhanced feature map will be input into the feature restoration module of the second layer of the decoder. Similarly, the restored feature map output by the second layer of the decoder will be input into the depth-guided filter at the same time as the intermediate coded feature map output by the first layer of the encoder to restore and enhance the detail information. Then, the detail-enhanced feature map will be input into the feature restoration module of the third layer of the decoder to generate the final decoding result.

[0095] The calculation formula for generating feature maps by the 1st and 2nd layer feature restoration modules of the decoder is as follows:

[0096] X input =Conv(ReLU(Conv((f input )))

[0097]

[0098] Xinput For the input of the first layer of the decoder, the dimension is H / 4*W / 4*C / 2. If it is the output of the depth-guided filter module, the dimension is the output dimension of the depth-guided filter module. Represents the output feature map of the i-th layer encoder, i can take values ​​of 1 and 2, where The dimension is H*(2*i)*W*(2*i)*C / 2. The third layer is different from the first two layers in that there is no upsampling operation at the end.

[0099] The calculation process in the depth-guided filtering module is as follows:

[0100]

[0101]

[0102] In the above formula, represents the output feature map of the i-th layer of the encoder, with dimensions of H / (2*i)*W / (2*i)*C / 2, represents the output feature map of the i-1th layer of the decoder, where the dimension is H / (2*i)*W / (2*i)*C / 2, and They represent the corresponding feature guide maps generated. D-Conv(·) represents the dilated two-dimensional convolution operation, the convolution kernel is 3, and the dilation rate is 1. Pw-Conv(·) represents the point-by-point convolution operation. represents the generated intermediate feature map, Linear-Layer(·) represents the linear layer, is the output fusion filter feature map, with the dimension of H / (2*i)*W / (2*i)*C / 2; FI(·) represents a module composed of two-dimensional convolution and ReLU linear activation function, and its calculation process is as follows:

[0103] f output =Conv2d(ReLU(Conv2d(f input )))

[0104] where f input is the input feature map, f output To output the feature map, Conv2d(·) represents a 2D convolution with a kernel size of 1. ReLU(·) represents the ReLU linear activation function.

[0105] S5: The final decoding result is passed through a 2D convolution layer and a Tanh activation layer to obtain a fused image.

[0106] Example 2

[0107] Based on the same inventive concept, an embodiment of the present invention discloses a neural network model for implementing the image fusion method in Embodiment 1, including:

[0108] The input module is used to obtain the infrared image and visible light image to be fused.

[0109] The preliminary fusion module is used to perform preliminary fusion of the infrared image and the visible light image to obtain a preliminary fused image.

[0110] Encoder, used for deep feature extraction of preliminary fusion graph.

[0111] The decoder is used to decode layer by layer according to the feature map output by the encoder to obtain the final decoding result.

[0112] The guided filtering module is used to obtain the output of the encoder and the output of the decoder during layer-by-layer decoding, and to perform guided fusion according to the output of the encoder for decoding of the next layer.

[0113] The output module is used to convolve and activate the final decoding result to obtain a fused image.

[0114] Example 3

[0115] The embodiment of the present invention discloses a method for training a neural network model, which is used for training the neural network model provided in embodiment 2, and includes the following steps:

[0116] The training data includes an infrared image dataset and a visible light image dataset.

[0117] The infrared image data set and the visible light image data set are configured to generate image data pairs.

[0118] The image data pair is input into the neural network model to obtain a fused image.

[0119] The model loss is calculated based on the fused image, and the parameters of the neural network model are optimized.

[0120] In order to further implement the above technical solution, the loss function used to calculate the model loss is as follows:

[0121] L total =L int +λ ρ L chl +(1-λ ρ )L struct

[0122] Among them, L int represents the target intensity loss of the fused image and the infrared image, λ ρ Represents the weight coefficient, which is a hyperparameter set artificially; L chlrepresents the Charbonnier loss of the fused image and the original visible light image, L struct Represents the SSIM structural consistency loss of the fused image and the visible light image;

[0123]

[0124] Among them, H and W are the height and width of the picture respectively. is the fused image, I ir is an infrared image, ||·||1 represents l1-norm;

[0125]

[0126] in, is the fused image, I vi is a visible light image, ∈ is a value of 10 -3 The constant of

[0127]

[0128] L struct =1-SSIM(X,Y)

[0129] Among them, X and Y represent two images, where X represents the visible light image and Y represents the fused image. μ and σ represent the mean and standard deviation respectively; σ XY Represents the covariance between X and Y; C1 and C2 are stability coefficients. It is used to measure the structural similarity between the fused image and the visible light.

[0130] In the present invention, infrared and visible light belong to different modal information. Infrared images have rich intensity information and can highlight the features of objects in the dark according to the heat information. Visible light images have rich texture detail information, and visible light images are also the most consistent with the perception of the human eye. Multimodal information is complementary. After a carefully designed fusion module, a fusion image that simultaneously expresses infrared and visible light information can be generated. However, the fusion rules of the traditional method need to be manually designed and do not have good transferability. In the process of network propagation, deep learning methods often lose feature information, resulting in distortion of the network when information is transmitted. This embodiment adopts a deep guided filtering module, with the help of the commonly used guided filtering module in the traditional method, and the module is deeply learned, thereby forming a whole with the network gradient transmission update. Guided filtering can not only enhance the representation of feature details, but also ensure that while the network depth is deepened, a similar residual processing method is adopted, and the feature texture details will not be lost. At the same time, in the network preliminary fusion module, unlike the previous direct channel connection method, we use a shallow fusion module to perform the initial fusion of the two modal channel information through the channel attention module to ensure the quality of the input to the subsequent network feature map. The present invention not only solves the problem of loss of detail information as the network deepens in the current deep learning image fusion method, but also improves the information complementarity of the generated fused image. The finally generated fused image results achieve good results in both subjective and objective evaluation indicators.

[0131] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0132] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for infrared and visible light image fusion based on depth-guided filtering, characterized in that: The following steps are involved: Acquire the infrared image and visible light image to be fused; Preliminarily fusing the infrared image and the visible light image, the specific steps include: extracting features through a convolution layer and an activation layer to obtain an infrared feature map and a visible light feature map; Connect infrared and visible light features in the channel dimension and calculate channel attention weights; The infrared feature map and the visible light feature map are weighted by the attention weights respectively by element-by-element multiplication to obtain a first branch feature map obtained by weighting the infrared feature map and a second branch feature map obtained by weighting the visible light feature map; Connecting the first branch feature map with the visible light feature map to obtain a first branch enhanced feature map; connecting the second branch feature map with the infrared feature map to obtain a second branch enhanced feature map; The first branch enhanced feature map and the second branch enhanced feature map are connected in series; and input into the parallel attention module to generate the final fusion weight, and the fusion weight is weighted to the first branch enhanced feature map and the second branch enhanced feature map respectively, and then the two weighted feature maps are added and fused element by element to generate a preliminary fusion map; Performing deep feature extraction on the preliminary fusion graph through an encoder; Based on the deepest feature map extracted by the encoder, a decoder is used to perform layer-by-layer decoding, and the layer-by-layer decoding includes: before performing decoding of the next layer, an output result of the current decoding layer and an output result of the corresponding encoding layer are fused with detail information through a depth-guided filter, and the fused feature map is used as an input for decoding of the next layer until a final decoding result is output; The final decoding result is passed through a convolution layer and an activation layer to obtain a fused image.

2. The infrared-visible light image fusion method based on depth-guided filtering according to claim 1, characterized in that: The step of calculating the channel attention weight comprises: Get the feature map after channel connection and perform global average pooling; According to the result after global average pooling, multiple point-by-point convolutions are performed and the channel attention weight is output through the activation function.

3. The infrared-visible light image fusion method based on depth-guided filtering according to claim 1, characterized in that: The parallel attention module is composed of a channel attention branch and a spatial attention branch. The specific steps of generating fusion weights include: Obtain the concatenated feature maps, and input them into the channel attention branch and the spatial attention branch respectively after point-by-point convolution; The results of the channel attention branch and the spatial attention branch are convolved and then the fusion weight is obtained through the activation function.

4. The infrared-visible light image fusion method based on depth-guided filtering according to claim 1, characterized in that: The depth-guided filtering step comprises: Get the output of the previous encoding layer and the corresponding decoding layer, and filter them through the filtering layer respectively; After performing dilated convolution and point-by-point convolution on the filtering results corresponding to the encoding layer in turn, the filtering results corresponding to the decoding layer are input into the linear layer to obtain a fused feature map.

5. According to the infrared-visible light image fusion method based on depth-guided filtering in claim 1, the encoder comprises three channel residual attention modules and a downsampling module between the channel residual attention modules, The encoding process is: The residual channel attention module first convolves the input feature map to obtain features, then performs channel attention calculation on the obtained features to generate channel attention weights; through residual connection, the obtained features are element-wise multiplied with the attention weights to generate a weighted attention feature map, and the attention feature map is then element-wise added to the input feature map to generate a new feature map; The new feature map is downsampled and input into the second channel residual attention module; The output of the second channel residual attention module is downsampled and input into the third channel residual attention module to obtain the final decoding result.

6. The infrared-visible light image fusion method based on depth-guided filtering according to claim 5, characterized in that: The decoder includes three channel residual attention modules and an upsampling module arranged between the channel residual attention modules.

7. A neural network model, characterized in that: The image fusion method according to any one of claims 1 to 6 is adopted, comprising: An input module, used for acquiring infrared images and visible light images to be fused; A preliminary fusion module, used for performing preliminary fusion of the infrared image and the visible light image to obtain a preliminary fused image; An encoder, used for performing deep feature extraction on the preliminary fusion graph; The decoder is used to decode layer by layer according to the feature map output by the encoder to obtain the final decoding result; A guided filtering module, used for obtaining the output of the encoder and the output of the decoder during layer-by-layer decoding, and performing guided fusion according to the output of the encoder for decoding of the next layer; The output module is used to perform convolution and activation on the final decoding result to obtain a fused image.

8. A method for training a neural network model, characterized in that: Applied to the neural network model described in claim 7, the steps include: Acquire training data, where the training data includes an infrared image dataset and a visible light image dataset; configuring the infrared image data set and the visible light image data set to generate an image data pair; Inputting the image data pair into the neural network model to obtain a fused image; The model loss is calculated according to the fused image, and the parameters of the neural network model are optimized.

9. A training method for a neural network model according to claim 8, characterized in that: The loss function used to calculate the model loss is as follows: L total =L int +λ ρ L chl +(1-l ρ )L struct Among them, L int represents the target intensity loss of the fused image and the infrared image, λ ρ Represents the weight coefficient, which is a hyperparameter set artificially; L chl represents the Charbonnier loss of the fused image and the original visible light image, L struct Represents the SSIM structural consistency loss of the fused image and the visible light image; Among them, H and W are the height and width of the picture respectively. is the fused image, I ir is an infrared image, ||·||1 represents the L1 norm; l1-norm; in, is the fused image, I vi is a visible light image, ∈ is a smoothing parameter used to keep the function differentiable; L struct =1-SSIM(X,Y) Among them, X and Y represent two images, where X represents the visible light image and Y represents the fused image; μ and σ represent the mean and standard deviation respectively; σ XY represents the covariance between X and Y; C1 and C2 are stability coefficients used to measure the structural similarity between the fused image and the visible light.

Images