Image fusion method based on raindrop layer interference reduction and background detail information enhancement

Abstract

The present application discloses an image fusion method based on raindrop layer interference reduction and background detail information enhancement, and relates to the field of new generation information technology. The fusion method disclosed in the present application can obtain an enhanced feature map containing directional weights, a reconstructed infrared image feature map containing directional weights and enhanced thermal radiation information, and a raindrop layer feature map containing directional weights and having rich raindrop edge information and texture details, and can perform subtraction operations on the above-mentioned enhanced feature map and the above-mentioned raindrop layer feature map to obtain a background detail feature map with reduced raindrop layer interference and enhanced background detail information; and the background detail feature map and the reconstructed infrared image feature map can use a cross-modal complementary feature fusion network to achieve cross-modal complementary feature fusion to obtain a Y channel fused image. The color fused image obtained in the present application contains richer edge information, more background detail information and thermal radiation information in the source image, and there are fewer artifacts in the color fused image.

Description

Technical Field

[0001] The present invention belongs to the field of new generation information technology, and in particular relates to an image fusion method based on reducing raindrop layer interference and enhancing background detail information. Background Art

[0002] At present, the fusion methods of visible light images and infrared images are mainly divided into two categories: one is the traditional image fusion method, and the other is the method based on deep learning. However, both the traditional image fusion method and the existing deep learning-based image fusion method have poor effects on the image fusion of visible light images and infrared images in rainstorm scenes. This is mainly because the background detail information in the visible light image under rainstorm conditions is severely weakened by factors such as rain occlusion and light scattering, and although the infrared image can provide a clear target outline, it lacks in texture and background details. To this end, the present application proposes an image fusion method based on reducing the interference of the raindrop layer and enhancing the background detail information. Summary of the invention

[0003] In order to make up for the deficiencies of the prior art, the present invention provides an image fusion method based on reducing raindrop layer interference and enhancing background detail information.

[0004] The technical solution of the present invention is:

[0005] An image fusion method based on reducing raindrop layer interference and enhancing background detail information includes the following steps:

[0006] Extract visible light features and detail texture features of the raindrop layer from the Y channel image of the visible light image in the rainstorm scene, respectively, to obtain a feature map with visible light features and a feature map with detail texture features of the raindrop layer; then, perform multi-scale and multi-directional feature extraction on the global background features in the feature map with visible light features, calculate the horizontal direction weight and the vertical direction weight in the feature map, and obtain an enhanced feature map containing the direction weight; extract and enhance the edge information and detail features of the raindrop layer in the feature map with detail texture features of the raindrop layer, calculate the horizontal direction weight and the vertical direction weight in the feature map, and obtain a raindrop layer feature map containing the direction weight and having rich raindrop edge information and raindrop texture details; then, perform a subtraction operation on the enhanced feature map and the raindrop layer feature map, and obtain a background detail feature map with reduced raindrop layer interference and enhanced background detail information;

[0007] The thermal radiation information in the infrared image is extracted to obtain a feature map with thermal radiation information, and then multi-scale and multi-directional feature extraction is performed on the feature map with thermal radiation information, and the horizontal direction weight and vertical direction weight in the feature map are calculated to obtain a reconstructed infrared image feature map containing direction weights and enhanced thermal radiation information;

[0008] Then, the background detail feature map and the reconstructed infrared image feature map are fused with cross-modal complementary features using a cross-modal complementary feature fusion network to obtain a Y channel fused image.

[0009] An image fusion method based on reducing raindrop layer interference and enhancing background detail information specifically comprises the following steps:

[0010] S1, obtain training set and test set;

[0011] S2. Construct a background detail enhancement extraction network, and use the training set to train the background detail enhancement extraction network to obtain a background detail enhancement extraction network model;

[0012] The background detail enhancement extraction network includes a visible light information extraction branch, a global feature enhancement module connected to the visible light information extraction branch, a raindrop layer information extraction branch, a detail feature enhancement module connected to the raindrop layer information extraction branch, a subtraction module connected to the global feature enhancement module and the detail feature enhancement module, and a decoder connected to the subtraction module; it also includes an infrared information extraction branch, another global feature enhancement module connected to the infrared information extraction branch, and a decoder connected to the global feature enhancement module; wherein the global feature enhancement module connected to the visible light information extraction branch and the global feature enhancement module connected to the infrared information extraction branch have the same network structure;

[0013] Among them, the visible light information extraction branch and the raindrop layer information extraction branch are respectively used to extract visible light features and detailed texture features of the raindrop layer from the Y channel image of the visible light image in the rainstorm scene, and obtain a feature map with visible light features and a feature map with detailed texture features of the raindrop layer; then, the global feature enhancement module connected to the visible light information extraction branch is used to perform multi-scale and multi-directional feature extraction on the global background features in the feature map with visible light features, and calculate the horizontal direction weight and the vertical direction weight in the feature map to obtain an enhanced feature map containing directional weights; the detail feature enhancement module connected to the raindrop layer information extraction branch extracts and enhances the edge information and detail features of the raindrop layer in the feature map with detailed texture features of the raindrop layer, and calculates the horizontal direction weight and the vertical direction weight in the feature map to obtain a raindrop layer feature map containing directional weights and rich raindrop edge information and raindrop texture details; the subtraction module is used to perform a subtraction operation on the enhanced feature map and the raindrop layer feature map to obtain a background detail feature map with reduced raindrop layer interference and enhanced background detail information;

[0014] The infrared information extraction branch is used to extract the thermal radiation information in the infrared image to obtain a feature map with thermal radiation information. Another global feature enhancement module connected to the infrared information extraction branch is used to perform multi-scale and multi-directional feature extraction on the feature map with thermal radiation information, and calculate the horizontal direction weight and vertical direction weight in the feature map to obtain a reconstructed infrared image feature map containing direction weights and enhanced thermal radiation information.

[0015] S3. Construct a cross-modal complementary feature fusion network, and use the training set to train the cross-modal complementary feature fusion network to obtain a cross-modal complementary feature fusion network model; wherein the cross-modal complementary feature fusion network is used to fuse the background detail feature map with the reconstructed infrared image feature map to realize the cross-modal complementary feature, and obtain a Y channel fusion image;

[0016] S4. Input the Y channel images of the infrared image and the visible light image of the rainstorm scene into the background detail enhancement extraction network model and forward propagate once to obtain the background detail feature map of the Y channel image. And reconstruct infrared image feature map Then, the background detail feature map And reconstruct infrared image feature map The input is forward propagated once into the cross-modal complementary feature fusion network to obtain the Y channel fused image.

[0017] Preferably, in step S4, the Y channel fused image is spliced ​​with the Cb channel image and the Cr channel image of the visible light image in the channel dimension, and after splicing, the image is converted into RGB format to obtain a color fused image.

[0018] Preferably, in step S3, the cross-modal complementary feature fusion network includes branch I, branch II, a Concat layer connected to branch I and branch II, and a convolution layer connected to the Concat layer; wherein the branch and branch II have the same structure, and the infrared image feature map is reconstructed. and background detail feature map They serve as the input of branch I and branch II respectively; branch I includes a linear layer, a cross-attention layer, a Concat layer, a first normalization layer, a first Add layer, a feedforward network layer, a second normalization layer, and a second Add layer connected in sequence. The input of the linear layer is also connected to the input of the first Add layer, and the output of the linear layer is also connected to the input of the ReLu layer. The output of the ReLu layer is connected to the input of the Concat layer, and the output of the first Add layer is also connected to the second Add layer; the output of the linear layer in branch I is connected to the input of the cross-attention layer in branch II, and the output of the linear layer in branch II is connected to the input of the cross-attention layer in branch I. The output of the second Add layer of branch I and the output of the second Add layer of branch II are both connected to the Concat layer connected to the convolutional layer.

[0019] Preferably, in step S2, the network structures of the infrared information extraction branch, the visible light information extraction branch and the raindrop layer information extraction branch are the same, and the infrared information extraction branch, the raindrop layer information extraction branch and the infrared information extraction branch all include four convolution modules I connected in sequence, and the four convolution modules I all include convolution layers and LReLU layers connected in sequence; the weight parameters of the infrared information extraction branch, the visible light information extraction branch and the raindrop layer information extraction branch are different. This is because when the infrared information extraction branch, the visible light information extraction branch and the raindrop layer feature extraction branch are trained using the training set, the weight parameters are not shared, and when the infrared information extraction branch, the visible light information extraction branch and the raindrop layer feature extraction branch are trained, they can independently learn different weight parameters.

[0020] Preferably, in step S2, the global feature enhancement module includes two average pooling layers, the two average pooling layers are connected to the Concat layer, the Concat layer is respectively connected to the convolution layer and the global feature enhancement unit, the convolution layer and the global feature enhancement unit are both connected to the Add layer, the Add layer is sequentially connected to the normalization layer and the Swish layer, the Swish layer is respectively connected to the first branch and the second branch, the first branch and the second branch have the same structure, wherein the first branch includes a convolution layer and a global feature enhancement unit connected to the Swish layer, the convolution layer and the global feature enhancement unit are also connected to the Add layer, the Add layer is connected to the Sigmoid layer, the Sigmoid layer of the first branch, the Sigmoid layer of the second branch and the input end of the global feature enhancement module are all connected to the element-by-element multiplication unit.

[0021] Preferably, in step S2, the global feature enhancement unit includes three average pooling layers of different scales, and the three average pooling layers of different scales are respectively a 1×1 average pooling layer, a 2×2 average pooling layer and a 3×3 average pooling layer. The three average pooling layers are all connected to the connection layer, the connection layer is connected to the convolution layer, and the convolution layer is connected to the ReLU layer.

[0022] Preferably, in step S2, the network structure of the detail feature enhancement module differs from the network structure of the global feature enhancement module in that: the maximum pooling layer in the detail feature enhancement module replaces the average pooling layer in the global feature enhancement module, and the detail-aware texture enhancement unit in the detail feature enhancement module replaces the global feature enhancement unit in the global feature enhancement module.

[0023] Preferably, in step S2, the detail-aware texture enhancement unit includes a Scharr operator, a maximum pooling layer and a convolution layer (the convolution kernel size of the convolution layer is 3×3) connected in sequence; wherein the Scharr operator is used to perform edge detection on the feature map input to the detail-aware texture enhancement unit to obtain a feature map with relatively accurate edge information; the maximum pooling layer is used to perform a pooling operation on the feature map output by the Scharr operator to obtain a feature map with significant edge features; the convolution layer is used to extract and enhance the raindrop layer detail features in the feature map output by the maximum pooling layer to obtain a raindrop layer feature map with rich raindrop edge information and raindrop texture details.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The present invention provides an image fusion method based on reducing raindrop layer interference and enhancing background detail information; the convolution module I in the visible light information extraction branch of the present application can extract visible light feature information in a rainstorm scene in a Y channel image; the convolution module I in the raindrop layer information extraction branch can extract detail texture features of the raindrop layer in the Y channel image; the global feature enhancement module connected to the visible light information extraction branch can perform multi-scale and multi-directional feature extraction on the global background features in the feature map output by the visible light information extraction branch, and calculate the horizontal direction weight and the vertical direction weight in the feature map to obtain a reconstructed infrared image feature map containing direction weights and enhanced thermal radiation information; the detail feature enhancement module can extract and enhance the edge information and detail features of the raindrop layer in the feature map input therein, and calculate the horizontal direction weight and the vertical direction weight in the feature map to obtain a raindrop layer feature map containing direction weights and rich raindrop edge information and raindrop texture details; the subtraction module is used to perform a subtraction operation on the feature map output by the global feature enhancement module connected to the visible light information extraction branch and the feature map output by the detail feature enhancement module to obtain a background detail feature map with reduced raindrop layer interference and enhanced background detail information;

[0026] The convolution module I in the infrared information extraction branch can extract the thermal radiation information in the infrared image; the global feature enhancement module connected to the infrared information extraction branch can perform multi-scale and multi-directional feature extraction on the global thermal radiation features in the feature map output by the infrared information extraction branch, and calculate the horizontal and vertical weights in the feature map to obtain a reconstructed infrared image feature map containing directional weights and enhanced thermal radiation information;

[0027] The background detail feature map and the infrared feature map use a cross-modal complementary feature fusion network to achieve cross-modal complementary feature fusion, and obtain a Y channel fusion image. Then, the Y channel fusion image is spliced ​​with the Cb channel image and Cr channel image of the visible light image in the channel dimension. After splicing, the image is converted into RGB format to obtain a color fusion image. Through testing, it is found that the color fusion image obtained by the fusion method described in this application contains richer edge information, and also contains more background detail information and thermal radiation information in the source image. In addition, there are fewer artifacts in the color fusion image, which better reduces the interference caused by the raindrop layer. In addition, the color fusion image obtained by this application also has higher contrast and overall brightness, which is more in line with human visual perception. Obviously, this application can effectively solve the problem that the existing image fusion method is difficult to fully mine the background detail information in the visible light image in the rainstorm scene, resulting in weak background details and poor visual perception of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of the connection relationship between the background detail enhancement extraction network and the cross-modal complementary feature fusion network of the present invention;

[0029] Figure 2 This is a schematic diagram of a global feature enhancement module of the present invention;

[0030] Figure 3 Schematic diagram of a global feature enhancement unit in a global feature enhancement module of the present invention;

[0031] Figure 4 It is a schematic diagram of a detail feature enhancement module of the present invention;

[0032] Figure 5 Schematic diagram of a detail-aware texture enhancement unit in a detail feature enhancement module of the present invention;

[0033] Figure 6 is a schematic diagram of a cross-modal complementary feature fusion network in the present invention, Figure 6 In the figure, © represents the Concat layer. Indicates the Add layer;

[0034] Figure 7 It is the fused image obtained by the image fusion method of the present application and seven existing image fusion methods based on the test set, Figure 7 In the figure, (a) Infrared represents an infrared image used to test visual effects, and (b) Visible represents a visible light image of a rainstorm scene used to test visual effects; Figure 7(c) to (j) respectively represent the GTF image fusion method, NestFuse image fusion method, RFN-Nest image fusion method, MFEIF image fusion method, LRRNet image fusion method, DATFuse image fusion method, BTSFusion image fusion method and the fused image obtained by the image fusion method described in this application based on the test set. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] An image fusion method based on reducing raindrop layer interference and enhancing background detail information specifically comprises the following steps:

[0037] S1, obtain training set and test set;

[0038] This application is from M 3 The FD dataset randomly selects 500 pairs of paired infrared images and visible light images. The above visible light images are used as original visible light images. Then, raindrops are added to the original visible light images using the raindrop synthesis algorithm to obtain visible light images in heavy rain scenes. The infrared images are not processed.

[0039] This application randomly selects 450 pairs from the above 500 pairs of paired infrared images and visible light images in rainstorm scenes, and retains the corresponding original visible light images without raindrops to form the training set of this application; the remaining 50 pairs of infrared images and visible light images in rainstorm scenes, and retains the corresponding original visible light images without raindrops to form the test set of this application;

[0040] In this application 3 The FD dataset is an existing dataset and can be obtained from: https: / / github.com / JinyuanLiu-CV / TarDAL.

[0041] S2. Construct a background detail enhancement extraction network, and use the training set to train the background detail enhancement extraction network to obtain a background detail enhancement extraction network model;

[0042] The network structure of the background detail enhancement extraction network is as follows: Figure 1As shown, it includes a visible light information extraction branch, a global feature enhancement module connected to the visible light information extraction branch, a raindrop layer information extraction branch, a detail feature enhancement module connected to the raindrop layer information extraction branch, a subtraction module connected to the global feature enhancement module and the detail feature enhancement module, and a decoder connected to the subtraction module; it also includes an infrared information extraction branch, another global feature enhancement module connected to the infrared information extraction branch, and a decoder connected to the global feature enhancement module; wherein the network structure of the global feature enhancement module connected to the visible light information extraction branch and the global feature enhancement module connected to the infrared information extraction branch are the same;

[0043] In the present application, the network structures of the infrared information extraction branch, the visible light information extraction branch and the raindrop layer information extraction branch are the same. The infrared information extraction branch, the raindrop layer information extraction branch and the infrared information extraction branch all include four sequentially connected convolution modules I, and the four convolution modules I all include sequentially connected convolution layers (the convolution kernel size of the convolution layer is 3×3) and LReLU layers; however, the weight parameters of the infrared information extraction branch, the visible light information extraction branch and the raindrop layer information extraction branch are different. This is because when the infrared information extraction branch, the visible light information extraction branch and the raindrop layer feature extraction branch are trained using the training set, the weight parameters are not shared. When the infrared information extraction branch, the visible light information extraction branch and the raindrop layer feature extraction branch are trained, they can autonomously learn different weight parameters respectively;

[0044] In the visible light information extraction branch, the convolution module I is used to extract the visible light feature information in the Y channel image under the rainstorm scene;

[0045] In the raindrop layer information extraction branch, the convolution module I is used to extract the detailed texture features of the raindrop layer in the Y channel image;

[0046] In the infrared information extraction branch, convolution module I is used to extract thermal radiation information from infrared images;

[0047] The global feature enhancement module is used to perform multi-scale and multi-directional feature extraction on the global features in the feature map input therein, and calculate the horizontal direction weight and the vertical direction weight in the feature map to obtain a reconstructed infrared image feature map containing the direction weight and enhanced thermal radiation information;

[0048] The detail feature enhancement module is used to extract and enhance the raindrop layer edge information and detail features in the feature map input therein, and calculate the horizontal direction weight and the vertical direction weight in the feature map to obtain a raindrop layer feature map containing the direction weight and having rich raindrop edge information and raindrop texture details;

[0049] The subtraction module is used to perform a subtraction operation on the feature map output by the global feature enhancement module connected to the visible light information extraction branch and the feature map output by the detail feature enhancement module to obtain a background detail feature map with reduced raindrop layer interference and enhanced background detail information;

[0050] The decoder connected to the subtraction module in this application is used to upsample and reconstruct the feature map output by the subtraction module to obtain a scene detail map, which is used to calculate the visible light image deraining loss. The decoder connected to the global feature enhancement module is used to upsample and reconstruct the feature map of the reconstructed infrared image output by the global feature enhancement module to obtain a reconstructed infrared image, and the reconstructed infrared image is used to calculate the infrared image reconstruction loss. .

[0051] In this application, the structure of the global feature enhancement module is as follows: Figure 2 As shown, it includes two average pooling layers, the two average pooling layers are connected to the Concat layer, the Concat layer is respectively connected to the convolution layer (the convolution kernel size of the convolution layer is 3×3) and the global feature enhancement unit, the convolution layer and the global feature enhancement unit are both connected to the Add layer, the Add layer is sequentially connected to the normalization layer and the Swish layer, the Swish layer is respectively connected to the first branch and the second branch, the first branch and the second branch have the same structure, wherein the first branch includes a convolution layer (the convolution kernel size of the convolution layer is 3×3) and a global feature enhancement unit connected to the Swish layer, the convolution layer and the global feature enhancement unit are also connected to the Add layer, the Add layer is connected to the Sigmoid layer, the Sigmoid layer of the first branch and the Sigmoid layer of the second branch and the input end of the global feature enhancement module are all connected to the element-by-element multiplication unit;

[0052] One of the average pooling layers of the global feature enhancement module is used to perform an average pooling operation on the horizontal features in the feature map of the global feature enhancement module, so as to obtain a horizontal feature map with global information in the horizontal direction. Another average pooling layer is used to perform an average pooling operation on the vertical features in the feature map of the global feature enhancement module, so as to obtain a vertical feature map with global information in the vertical direction. The Concat layer is used to perform feature splicing on the feature maps output by the two average pooling layers, and output a fused feature map that combines the horizontal global features and the vertical global features. The convolution layer connected to the Concat layer is used to extract features from the feature map input therein. The global feature enhancement unit connected to the Concat layer is used to extract and enhance the global features in the feature map output by the Concat layer to obtain a feature map with global information enhancement; the Add layer connected to the Concat layer and the global feature enhancement unit is used to perform a pixel-by-pixel addition operation on the feature map output by the Concat layer and the global feature enhancement unit to obtain a fused global texture enhancement feature map; the normalization layer is used to normalize the feature map output by the Add layer to obtain a standardized feature map; the Swish layer is used to activate the feature map output by the normalization layer to obtain an activated feature map;

[0053] The convolution layer in the first branch is used to extract the horizontal direction features in the feature map output by the Swish layer, so as to obtain a feature map with significant horizontal direction features. The convolution layer in the second branch is used to extract the vertical direction features in the feature map output by the Swish layer, so as to obtain a feature map with significant vertical direction features. The global feature enhancement unit in the first branch is used to perform global feature extraction and enhancement on the horizontal direction features in the feature map output by the Swish layer, so as to obtain a feature map with enhanced global features in the horizontal direction. The global feature enhancement unit in the second branch is used to perform global feature extraction and enhancement on the vertical direction features in the feature map output by the Swish layer, so as to obtain a feature map with enhanced global features in the vertical direction. The Add layer is used to perform element-by-element addition of the feature maps output by the convolution layer and the global feature enhancement unit in the first branch to obtain an enhanced horizontal feature map. The Add layer in the second branch is used to perform element-by-element addition of the feature maps output by the convolution layer and the global feature enhancement unit in the first branch to obtain an enhanced vertical feature map. The Sigmoid layers in the first branch and the second branch are used to introduce nonlinearity. The Sigmoid layers in the first branch and the second branch output horizontal weights and vertical weights respectively. The element-by-element multiplication unit is used to multiply the feature map input to the global feature enhancement module as well as the horizontal weight and the vertical weight to obtain a reconstructed infrared image feature map containing directional weights and enhanced thermal radiation information.

[0054] In this application, the network structure of the global feature enhancement unit is as follows: Figure 3 As shown in the figure, it includes three average pooling layers of different scales, which are 1×1 average pooling layer, 2×2 average pooling layer and 3×3 average pooling layer respectively. The three average pooling layers are all connected to the connection layer, the connection layer is connected to the convolution layer (the convolution kernel size of the convolution layer is 2×2), and the convolution layer is connected to the ReLU layer; among them, the three average pooling layers of different scales are used to extract and enhance features of different scales in the feature map input to the global feature enhancement unit, the connection layer connected to the three average pooling layers is used to connect the features in the feature maps output by the three average pooling layers, the ReLU layer is used for activation operation, and finally the feature map with rich global information is output.

[0055] In this application, the network structure of the detail feature enhancement module is as follows: Figure 4 As shown, the network structure of the detail feature enhancement module is different from that of the global feature enhancement module in that the maximum pooling layer in the detail feature enhancement module replaces the average pooling layer in the global feature enhancement module, and the detail-aware texture enhancement unit in the detail feature enhancement module replaces the global feature enhancement unit in the global feature enhancement module.

[0056] In the present application, the detail feature enhancement module is used to obtain a raindrop layer feature map containing directional weights and having rich raindrop edge information and raindrop texture details. One of the maximum pooling layers of the detail feature enhancement module is used to perform a horizontal maximum pooling operation on the feature map of the input detail feature enhancement unit to obtain a feature map with prominent horizontal features. Another maximum pooling layer is used to perform a vertical maximum pooling operation on the feature map of the input detail feature enhancement unit to obtain a feature map with prominent vertical features. The detail-aware texture enhancement unit of the detail feature enhancement module is used to extract and enhance the raindrop layer detail features and edge features in the feature map input therein to obtain a raindrop layer feature map with rich raindrop edge information and raindrop texture details. In the present application, the detail feature enhancement module is the same module as the global feature enhancement module, and its function is also the same. The only difference is that the input feature map is different.

[0057] In this application, the network structure of the detail-aware texture enhancement unit is as follows: Figure 5As shown in the figure, it includes a Scharr operator, a maximum pooling layer and a convolutional layer (the convolution kernel size of the convolutional layer is 3×3) connected in sequence; wherein the Scharr operator is used to perform edge detection on the feature map of the input detail-aware texture enhancement unit to obtain a feature map with relatively accurate edge information; the maximum pooling layer is used to perform a pooling operation on the feature map output by the Scharr operator to obtain a feature map with significant edge features; the convolutional layer is used to extract and enhance the raindrop layer detail features in the feature map output by the maximum pooling layer to obtain a raindrop layer feature map with rich raindrop edge information and raindrop texture details.

[0058] In this application, the specific steps of using the training set to train the background detail enhancement extraction network are as follows:

[0059] S2-1, converting the visible light image under the rainstorm scene in the training set into a YCbCr image, and then separating the Y channel image of the visible light image, the Cb channel image of the visible light image, and the Cr channel image of the visible light image, respectively, to obtain the Y channel image of the visible light image, the Cb channel image of the visible light image, and the Cr channel image of the visible light image;

[0060] S2-2, the Y channel image of the visible light image obtained in step S2-1 of the training set is input into the visible light information extraction branch and the raindrop layer information extraction branch in the background detail enhancement extraction network respectively, and the global feature enhancement module connected to the visible light information extraction branch outputs the visible light feature map of the Y channel image At the same time, the detail feature enhancement module connected to the raindrop layer information extraction branch outputs the raindrop layer feature map of the Y channel image , and then the subtraction module converts the visible light feature map Subtract raindrop layer feature map , get the background detail feature map of the Y channel image ; Input the infrared image into the infrared information extraction branch in the background detail enhancement extraction network, and the global feature enhancement module connected to the infrared information extraction branch outputs a reconstructed infrared image feature map with thermal radiation information ; The background detail feature map of the Y channel image And reconstruct infrared image feature map The two decoders are input respectively for upsampling and feature reconstruction operations to obtain the background detail images of the Y channel image. and reconstruct infrared images ;

[0061] S2-3, based on rain loss Calculate the loss value of the background detail enhancement extraction network, and use the Adam optimizer to optimize the loss gradient and back-propagate to update the model parameters of the background detail enhancement extraction network;

[0062] Rain loss in this application Including infrared image reconstruction loss And visible light image deraining loss , rain loss The calculation formula is shown in formula (1):

[0063] (1)

[0064] In formula (1), the infrared image reconstruction loss The calculation formula is shown in formula (2):

[0065] (2)

[0066] In formula (2), represents the first norm calculation, is the infrared image in the training set, To reconstruct infrared images;

[0067] In formula (1), the visible light image deraining loss The calculation formula is shown in formula (3):

[0068] (3)

[0069] In formula (3), represents the original visible light image, Represents the scene detail feature map of the Y channel image , express Norm.

[0070] S2-4, repeating step S2-2 and step S2-3 until the number of iterations reaches a set threshold, and the background detail enhancement extraction network model can be obtained; wherein, completing the process of step S2-2 and step S2-3 once means realizing one iteration process, and the set threshold of the number of iterations in the background detail enhancement extraction network training process in this embodiment is 200; in this embodiment, the training of the background detail enhancement extraction network is based on the NVIDIA A100 GPU chip, and the training of the background detail enhancement extraction network adopts the Adam optimizer, the batch size is set to 16, and the initial learning rate is set to 1×10 -4 .

[0071] S3, construct a cross-modal complementary feature fusion network, use the training set to train the cross-modal complementary feature fusion network, and obtain a cross-modal complementary feature fusion network model;

[0072] In this application, the network structure of the cross-modal complementary feature fusion network is as follows: Figure 6As shown, it includes branch I, branch II, a Concat layer connected to branch I and branch II, and a convolution layer connected to the Concat layer; wherein the structure of branch I and branch II is the same, and the infrared image feature map is reconstructed. And the background detail feature map of the Y channel image They serve as the input of branch Ⅰ and branch Ⅱ respectively;

[0073] In the present application, branch I includes a linear layer, a cross attention layer, a Concat layer, a first normalization layer, a first Add layer, a feedforward network layer and a second normalization layer, and a second Add layer connected in sequence. The input end of the linear layer is also connected to the input end of the first Add layer, the output end of the linear layer is also connected to the input end of the ReLu layer, the output end of the ReLu layer is connected to the input end of the Concat layer, and the output end of the first Add layer is also connected to the second Add layer; the output end of the linear layer in branch I is connected to the input end of the cross attention layer in branch II, the output end of the linear layer in branch II is connected to the input end of the cross attention layer in branch I, and the output end of the second Add layer of branch I and the output end of the second Add layer of branch II are both connected to the Concat layer connected to the convolutional layer.

[0074] In this application, the linear layer in branch I is used to reconstruct the infrared image feature map Perform feature mapping to obtain a feature map with enhanced thermal radiation information. The linear layer in branch II is used to map the background detail feature map of the Y channel image. Feature mapping is performed to obtain a feature map with prominent background detail information; the cross attention layer in branch I is used to perform information association and feature enhancement operations on the feature map output by the linear layer in branch I and the feature map output by the linear layer in branch II, so as to obtain a feature map with thermal radiation information and background detail information; the cross attention layer in branch II is used to perform information association and feature enhancement operations on the feature map output by the linear layer in branch I and the feature map output by the linear layer in branch II, so as to obtain a feature map with rich thermal radiation information and background detail information. The above setting can realize the fusion of cross-modal complementary features in the process of fusion of the feature map of the reconstructed infrared image and the background detail feature map of the Y channel image using the cross-modal complementary feature fusion network;

[0075] In branch I, the ReLu layer is used to activate the feature map output by the linear layer, the Concat layer is used to perform channel splicing on the feature map output by the ReLu layer and the feature map output by the cross attention layer to obtain a feature map with background detail information and enhanced thermal radiation information, the first normalization layer is used to normalize the feature map output by the Concat layer, and the first Add layer is used to reconstruct the feature map of the infrared image. The feature maps output by the first Add layer and the normalization layer are added element by element to obtain a fused feature map with background detail information and richer thermal radiation information. The forward feedback network layer is used to perform a nonlinear feature enhancement operation on the feature map output by the first Add layer to obtain an enhanced fused feature map with background detail information and richer thermal radiation information. The second normalization layer is used to normalize the feature map output by the forward feedback network. The second Add layer is used to add the feature map output by the first Add layer and the feature map output by the second normalization layer element by element to obtain a fused feature map with rich background detail information and richer thermal radiation information. The ReLu layer, Concat layer, first normalization layer, and first The operations of the first Add layer, the feedforward network, the second normalization layer and the second normalization layer are respectively the same as the operations of the ReLu layer, the Concat layer, the first normalization layer, the first Add layer, the feedforward network, the second normalization layer and the second normalization layer in branch I, and the only difference is that different feature maps are processed; the Concat layer connected to the convolutional layer is used to perform channel splicing on the feature map output by the second Add layer in branch I and the feature map output by the second Add layer in branch II to obtain a multi-channel fused feature map, and the convolutional layer is used to perform channel splicing on the feature map output by the Concat layer connected to the convolutional layer to obtain a Y channel fused image with richer background detail information and richer thermal radiation information.

[0076] The training process of the cross-modal complementary feature fusion network in this application includes the following steps:

[0077] S3-1, input the infrared image in the training set and the Y channel image of the visible light image obtained in step S2-1 into the background detail enhancement extraction network model trained in step S2, and forward propagate once to obtain the background detail feature map of the Y channel image And reconstruct infrared image feature map ;

[0078] S3-2, the background detail feature map of the Y channel image obtained in step S3-1 And reconstruct infrared image feature map Input into the cross-modal complementary feature fusion network and forward propagate once to obtain the Y channel fusion image;

[0079] S3-3. Constructing fusion loss , and according to the fusion loss Calculate the loss value of the cross-modal complementary feature fusion network, use the Adam optimizer to optimize the loss gradient and back-propagate, and update the model parameters of the cross-modal complementary feature fusion network;

[0080] In this application, the fusion loss Including auxiliary strength loss and gradient loss , fusion loss The calculation formula is shown in formula (4):

[0081] (4)

[0082] In formula (4), To assist strength loss, is the gradient loss; and All are hyperparameters; in this embodiment, the hyperparameters and Set to 20 and 50 respectively.

[0083] In formula (4), auxiliary strength loss The calculation formula is shown in formula (5):

[0084] (5)

[0085] In formula (5), and Represents the height and width of the Y channel fused image, Indicates the Y channel fused image, The scene detail map representing the Y channel image, To reconstruct the infrared image, represents the maximum element selection, Represents the second norm calculation.

[0086] In formula (4), the gradient loss The calculation formula is shown in formula (6):

[0087] (6)

[0088] In formula (6), and Represents the height and width of the Y channel fused image, Represents the gradient value of the Y channel fused image, Scene detail map representing the Y channel image The gradient value of represents the gradient value of the reconstructed infrared image, represents the first norm calculation, represents the absolute value, Represents maximum element selection;

[0089] S3-4, repeating steps S3-1 to S3-3 until the number of iterations epoch reaches the set threshold, and the trained cross-modal complementary feature fusion network model can be obtained; wherein, completing the process from step S3-1 to step S3-3 once means realizing one iteration process; in this embodiment, the set threshold of the number of iterations in the cross-modal complementary feature fusion network training process is 200. In this embodiment, the training of the cross-modal complementary feature fusion network is based on the NVIDIA A100 GPU chip, and the Adam optimizer is used to optimize the loss gradient and back propagate. In the process of cross-modal complementary feature fusion network training, the batch size is set to 16, and the initial learning rate is set to 1×10 -4 .

[0090] S4. Input the Y channel images of the infrared image and the visible light image of the rainstorm scene into the trained background detail enhancement extraction network model and forward propagate once to obtain the background detail feature map of the Y channel image. And reconstruct infrared image feature map Then, the background detail feature map of the Y channel image And reconstruct infrared image feature map The image is input into the cross-modal complementary feature fusion network and forward propagated once to obtain the Y channel fusion image. Then, the Y channel fusion image is spliced ​​with the Cb channel image and Cr channel image of the visible light image in the channel dimension. After splicing, the image is converted into RGB format to obtain a color fusion image.

[0091] test:

[0092] In order to verify the fusion effect of the image fusion method of the present application, the present application first uses the image fusion method described in the present application and the GTF image fusion method (derived from "Infrared and visible image fusion via gradient transfer and total variation minimization"), the NEST-Fuse image fusion method (derived from "NestFuse: An infrared and visible image fusion architecture based on nest connection and spatial / channel attention models"), the RFN-Nest image fusion method (derived from "RFN-Nest: An endto-end residual fusion network for infrared and visible images"), the MFEIF image fusion method (derived from "Learning a deep multi-scale feature ensemble and an edge-attention guidance for image fusion"), the LRR image fusion method (derived from "LRRNet: A novel representation learning guided fusion framework for infrared and visible images"), the DAT image fusion method (derived from "DATFuse: Infrared and Visible ImageFusion via Dual Attention Transformer"), and the BTS image fusion method (derived from "Btsfusion: Fusion of infrared and visible image via a mechanism of balancing texture") The seven existing image fusion methods generate color fusion images based on the infrared images in the test set and the visible light images in the rainstorm scene. Then, the color fusion images are used to generate color fusion images using the existing Matlab test module (access URL

[0093] The visual effect test is performed on the image fusion (https: / / github.com / Linfeng-Tang / Evaluation-for-Image-Fusion). The test results are shown in Table 1 and Figure 7 As shown;

[0094] Table 1

[0095]

[0096] In Table 1, Q AB / F It is used to measure the edge information index transferred from the source image to the fused image. The higher its value means the better the image fusion effect. SCD characterizes the quality of the fusion method by measuring the difference between the fused image and the source image. The higher its value means the richer the information in the fused image including the source image. VIF is based on natural scene statistics and the human visual system to quantify the amount of information shared between the fused image and the source image. The higher its value means the more the fusion result conforms to human visual perception. FMI w It is used to measure the amount of feature information transferred from the source image to the fused image. The higher the value, the more features are transferred to the fused image. MS_SSIM is an indicator for comprehensively evaluating the distortion of the fused image based on SSIM combined with multi-scale structural similarity. The higher the value, the higher the image quality. AB / F It is an artifact-based indicator used to measure the artifacts introduced in the fusion process. The lower its value, the fewer artifacts are introduced in the fusion process. In addition, the image fusion method described in this application is represented by Ours method in Table 1.

[0097] It can be seen from Table 1 that:

[0098] 1) The image fusion method described in this application can achieve higher Q AB / F The Q value obtained by the image fusion method described in this application AB / F The value is compared with the NEST-Fuse image fusion method in the prior art (the Q obtained by this method in the above existing image fusion method is AB / F The value is the highest) increased by 33.48%, which shows that in the process of image fusion of infrared images and visible light images in rainstorm scenes by the image fusion method described in the present application, the color fused image obtained after fusion contains richer edge information and the fusion effect is better;

[0099] 2) The image fusion method described in the present application can obtain a higher SCD value. The SCD value obtained by the image fusion method described in the present application is 2.28% higher than that of the prior art RFN-NEST image fusion method (the SCD value obtained by this method is the highest among the above-mentioned existing image fusion methods). This shows that in the process of image fusion of infrared images and visible light images in rainstorm scenes by the image fusion method described in the present application, the color fused image obtained after fusion contains more background detail information and thermal radiation information in the source images;

[0100] 3) The image fusion method described in the present application can obtain a higher VIF value. The VIF value obtained by the image fusion method described in the present application is 28.7% higher than that of the prior art NEST-Fuse image fusion method (which has the highest VIF value among the above-mentioned existing image fusion methods). This shows that the image fusion method described in the present application is more in line with human visual perception in the process of image fusion of infrared images and visible light images in rainstorm scenes.

[0101] 4) The image fusion method described in this application can achieve a higher FMI w The FMI obtained by the image fusion method described in this application w Compared with the NEST-Fuse image fusion method in the prior art (the FMI obtained by this method in the above existing image fusion method) w The value is the highest) increased by 18.65%; this shows that in the process of image fusion of infrared images and visible light images in rainstorm scenes by the image fusion method described in the present application, the color fused image obtained after fusion obtains more background detail features and thermal radiation features of the source image;

[0102] 5) The image fusion method described in the present application can obtain a higher MS_SSIM value. The MS_SSIM value obtained by the image fusion method described in the present application is 9.95% higher than that of the MFEIF image fusion method in the prior art (the MS_SSIM value obtained by this method is the highest among the above-mentioned existing image fusion methods); this shows that the image fusion method described in the present application is used in the process of image fusion of infrared images and visible light images in rainstorm scenes. The color fused image obtained after fusion has higher image quality;

[0103] 6) The image fusion method described in this application can achieve a lower N AB / F value, N obtained by the image fusion method described in this application AB / F The value is compared with the BTS image fusion method in the prior art (the N obtained by this method in the above-mentioned existing image fusion method is AB / F The lowest value) is reduced by 25.62%; this shows that in the process of image fusion of infrared images and visible light images in rainstorm scenes by the image fusion method described in the present application, the color fusion image obtained after fusion has fewer artifacts, and the interference caused by the raindrop layer is better reduced;

[0104] In addition, from Figure 7 It can also be seen that the color fused image obtained by the image fusion method described in the present application also has higher contrast and overall brightness. Therefore, the fused image obtained by the image fusion method described in the present application is more in line with human visual perception.

Claims

1. An image fusion method based on reducing raindrop layer interference and enhancing background detail information, characterized by: The steps include: Extract visible light features and detail texture features of the raindrop layer from the Y channel image of the visible light image in the rainstorm scene, respectively, to obtain a feature map with visible light features and a feature map with detail texture features of the raindrop layer; perform multi-scale and multi-directional feature extraction on the global background features in the feature map with visible light features, calculate the horizontal direction weight and the vertical direction weight in the feature map, obtain an enhanced feature map containing the direction weight, extract and enhance the edge information and detail features of the raindrop layer in the feature map with detail texture features of the raindrop layer, calculate the horizontal direction weight and the vertical direction weight in the feature map, obtain a raindrop layer feature map containing the direction weight and rich raindrop edge information and texture details, and then perform a subtraction operation on the enhanced feature map and the raindrop layer feature map to obtain a background detail feature map with reduced raindrop layer interference and enhanced background detail information; The thermal radiation information in the infrared image is extracted to obtain a feature map with thermal radiation information. Multi-scale and multi-directional feature extraction is performed on the feature map with thermal radiation information, and the horizontal and vertical direction weights in the feature map are calculated to obtain a reconstructed infrared image feature map with directional weights and enhanced thermal radiation information. Then, the background detail feature map and the reconstructed infrared image feature map are fused with cross-modal complementary features using a cross-modal complementary feature fusion network to obtain a Y channel fused image.

2. The image fusion method based on reducing raindrop layer interference and enhancing background detail information according to claim 1, characterized in that: The specific steps include: S1, obtain training set and test set; S2. Construct a background detail enhancement extraction network, and use the training set to train the background detail enhancement extraction network to obtain a background detail enhancement extraction network model; The background detail enhancement extraction network includes a visible light information extraction branch, a global feature enhancement module connected to the visible light information extraction branch, a raindrop layer information extraction branch, a detail feature enhancement module connected to the raindrop layer information extraction branch, a subtraction module connected to the global feature enhancement module and the detail feature enhancement module, and a decoder connected to the subtraction module; it also includes an infrared information extraction branch, another global feature enhancement module connected to the infrared information extraction branch, and a decoder connected to the global feature enhancement module; wherein the global feature enhancement module connected to the visible light information extraction branch and the global feature enhancement module connected to the infrared information extraction branch have the same network structure; Among them, the visible light information extraction branch and the raindrop layer information extraction branch are respectively used to extract visible light features and detailed texture features of the raindrop layer from the Y channel image of the visible light image in the rainstorm scene, and obtain a feature map with visible light features and a feature map with detailed texture features of the raindrop layer; then, the global feature enhancement module connected to the visible light information extraction branch is used to perform multi-scale and multi-directional feature extraction on the global background features in the feature map with visible light features, and calculate the horizontal direction weight and the vertical direction weight in the feature map to obtain an enhanced feature map containing directional weights; the detail feature enhancement module connected to the raindrop layer information extraction branch extracts and enhances the edge information and detail features of the raindrop layer in the feature map with detailed texture features of the raindrop layer, and calculates the horizontal direction weight and the vertical direction weight in the feature map to obtain a raindrop layer feature map containing directional weights and rich raindrop edge information and raindrop texture details; the subtraction module is used to perform a subtraction operation on the enhanced feature map and the raindrop layer feature map to obtain a background detail feature map with reduced raindrop layer interference and enhanced background detail information; The infrared information extraction branch is used to extract the thermal radiation information in the infrared image to obtain a feature map with thermal radiation information. Another global feature enhancement module connected to the infrared information extraction branch is used to perform multi-scale and multi-directional feature extraction on the feature map with thermal radiation information, and calculate the horizontal direction weight and vertical direction weight in the feature map to obtain a reconstructed infrared image feature map containing direction weights and enhanced thermal radiation information. S3. Construct a cross-modal complementary feature fusion network, and use the training set to train the cross-modal complementary feature fusion network to obtain a cross-modal complementary feature fusion network model; wherein the cross-modal complementary feature fusion network is used to fuse the background detail feature map with the reconstructed infrared image feature map to realize the cross-modal complementary feature, and obtain a Y channel fusion image; S4. Input the Y channel images of the infrared image and the visible light image of the rainstorm scene into the background detail enhancement extraction network model and forward propagate once to obtain the background detail feature map of the Y channel image. And reconstruct infrared image feature map Then, the background detail feature map And reconstruct infrared image feature map The input is forward propagated once into the cross-modal complementary feature fusion network to obtain the Y channel fused image.

3. The image fusion method based on reducing raindrop layer interference and enhancing background detail information according to claim 2, characterized in that: In step S4, the Y channel fused image is spliced ​​with the Cb channel image and the Cr channel image of the visible light image in the channel dimension, and after splicing, the image is converted into RGB format to obtain a color fused image.

4. The image fusion method based on reducing raindrop layer interference and enhancing background detail information according to claim 2, characterized in that: In step S3, the cross-modal complementary feature fusion network includes branch I, branch II, a Concat layer connected to branch I and branch II, and a convolution layer connected to the Concat layer; wherein the branch and branch II have the same structure, and the infrared image feature map is reconstructed. and background detail feature map They serve as the input of branch I and branch II respectively; branch I includes a linear layer, a cross-attention layer, a Concat layer, a first normalization layer, a first Add layer, a feedforward network layer, a second normalization layer, and a second Add layer connected in sequence. The input of the linear layer is also connected to the input of the first Add layer, and the output of the linear layer is also connected to the input of the ReLu layer. The output of the ReLu layer is connected to the input of the Concat layer, and the output of the first Add layer is also connected to the second Add layer; the output of the linear layer in branch I is connected to the input of the cross-attention layer in branch II, and the output of the linear layer in branch II is connected to the input of the cross-attention layer in branch I. The output of the second Add layer of branch I and the output of the second Add layer of branch II are both connected to the Concat layer connected to the convolutional layer.

5. The image fusion method based on reducing raindrop layer interference and enhancing background detail information according to claim 2, characterized in that: In step S2, the network structures of the infrared information extraction branch, the visible light information extraction branch and the raindrop layer information extraction branch are the same. The infrared information extraction branch, the raindrop layer information extraction branch and the infrared information extraction branch all include four convolution modules I connected in sequence, and the four convolution modules I all include convolution layers and LReLU layers connected in sequence. The weight parameters of the infrared information extraction branch, the visible light information extraction branch and the raindrop layer information extraction branch are different. This is because when the infrared information extraction branch, the visible light information extraction branch and the raindrop layer feature extraction branch are trained using the training set, the weight parameters are not shared. When the infrared information extraction branch, the visible light information extraction branch and the raindrop layer feature extraction branch are trained, they can independently learn different weight parameters.

6. The image fusion method based on reducing raindrop layer interference and enhancing background detail information according to claim 2, characterized in that: In step S2, the global feature enhancement module includes two average pooling layers, the two average pooling layers are connected to the Concat layer, the Concat layer is respectively connected to the convolution layer and the global feature enhancement unit, the convolution layer and the global feature enhancement unit are both connected to the Add layer, the Add layer is sequentially connected to the normalization layer and the Swish layer, the Swish layer is respectively connected to the first branch and the second branch, the first branch and the second branch have the same structure, wherein the first branch includes a convolution layer and a global feature enhancement unit connected to the Swish layer, the convolution layer and the global feature enhancement unit are also connected to the Add layer, the Add layer is connected to the Sigmoid layer, the Sigmoid layer of the first branch and the Sigmoid layer of the second branch and the input end of the global feature enhancement module are all connected to the element-by-element multiplication unit.

7. The image fusion method based on reducing raindrop layer interference and enhancing background detail information according to claim 6, characterized in that: In step S2, the global feature enhancement unit includes three average pooling layers of different scales, which are 1×1 average pooling layer, 2×2 average pooling layer and 3×3 average pooling layer, respectively. The three average pooling layers are all connected to the connection layer, the connection layer is connected to the convolution layer, and the convolution layer is connected to the ReLU layer.

8. The image fusion method based on reducing raindrop layer interference and enhancing background detail information according to claim 2, characterized in that: In step S2, the network structure of the detail feature enhancement module differs from that of the global feature enhancement module in that the maximum pooling layer in the detail feature enhancement module replaces the average pooling layer in the global feature enhancement module, and the detail-aware texture enhancement unit in the detail feature enhancement module replaces the global feature enhancement unit in the global feature enhancement module.

9. The image fusion method based on reducing raindrop layer interference and enhancing background detail information according to claim 2, characterized in that: In step S2, the detail-aware texture enhancement unit includes a Scharr operator, a maximum pooling layer, and a convolutional layer connected in sequence.

Images