Infrared and visible light remote sensing image deep learning fusion method based on visual state space model

Abstract

The invention discloses an infrared and visible light remote sensing image deep learning fusion method based on a visual state space model. The method comprises the following steps: acquiring an infrared remote sensing image and a visible light remote sensing image to be fused; the infrared remote sensing image and the visible light remote sensing image are spliced and then input into a pre-constructed and trained image fusion network model, a fusion image output by the image fusion network model is obtained, and the image fusion network model comprises a first convolution module, a double-attention residual module, a multidirectional scanning module and a second convolution module which are connected in sequence. The first convolution module is used for extracting image shallow layer features, the double-attention residual module is used for extracting and fusing important features of an infrared remote sensing image and a visible light remote sensing image, the multi-direction scanning module is used for obtaining global features of the images in multiple directions, and the second convolution module is used for recovering image dimensions and obtaining a fused image; and outputting the fused image. According to the invention, infrared remote sensing image information and visible light remote sensing image information can be fully fused.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a deep learning fusion method of infrared and visible light remote sensing images based on a visual state space model. Background Art

[0002] Single-modal remote sensing image sensors are often unable to independently extract target information from the scene. Multimodal remote sensing image fusion can combine complementary information from multi-source remote sensing images, and is an important basis for downstream tasks such as target detection and object segmentation. Infrared remote sensing image sensors mainly image based on the thermal radiation characteristics of objects, and can obtain thermal radiation targets, but the image background is blurred and texture details are missing; visible light remote sensing image sensors mainly image based on the spectral reflectance characteristics of objects, and can well describe scene information and have strong visual interpretation capabilities, but the imaging quality is poor in dark environments. The fusion of infrared remote sensing images and visible light remote sensing images is to extract features from two types of heterogeneous images, perform image fusion, and generate a fused image with prominent infrared thermal radiation targets and rich background details.

[0003] Existing image fusion methods mainly include traditional fusion methods and deep learning fusion methods. According to existing theories, traditional fusion methods are mainly divided into fusion methods based on multi-scale transformation, fusion methods based on sparse representation, fusion methods based on compressed sensing, fusion methods based on subspace technology, and fusion methods based on saliency. Deep learning fusion methods mainly include fusion methods based on autoencoders, fusion methods based on convolutional neural networks, and fusion methods based on generative adversarial networks. Among them, the fusion method based on autoencoders uses the encoder part to extract low-level features and high-level features, and the decoder part restores the spatial dimension to fuse the extracted features to achieve image fusion; the fusion method based on convolutional neural networks uses convolutional neural networks for feature extraction, fusion, and image reconstruction; the fusion method based on generative adversarial networks uses the generator to obtain the fusion result, uses the discriminator to distinguish the fused image from the source image, and enriches the texture and background details of the fused image through adversarial games.

[0004] In practical applications, existing traditional fusion methods usually require more manual intervention, and the fusion rules are relatively complex, with problems such as high computational cost and low fusion efficiency. Existing deep learning fusion methods do not fully consider the physical laws of spatial features of natural images and remote sensing images. The spatial features of natural images are often horizontal and vertical, while the spatial features of remote sensing images may be in any direction due to satellite overhead shooting. Existing deep learning fusion methods use convolution operations to extract only local features and cannot represent long-distance dependencies. The extraction of global important context information is insufficient, and the spatial features of a single object and the spatial dependencies between multiple objects will be lost, resulting in poor fusion effect and fusion accuracy. Summary of the invention

[0005] In order to solve some or all of the technical problems existing in the above-mentioned prior art, the present invention provides a deep learning fusion method of infrared and visible light remote sensing images based on a visual state space model.

[0006] The technical solution of the present invention is as follows:

[0007] A method for deep learning fusion of infrared and visible light remote sensing images based on a visual state space model is provided, the method comprising:

[0008] Acquire infrared remote sensing images and visible light remote sensing images to be fused;

[0009] The infrared remote sensing image and the visible light remote sensing image are spliced ​​and input into a pre-built and trained image fusion network model to obtain a fused image output by the image fusion network model, wherein the image fusion network model comprises a first convolution module, a dual attention residual module, a multi-directional scanning module and a second convolution module connected in sequence, wherein the first convolution module is used to extract shallow features of the image, the dual attention residual module is used to extract important features of the infrared remote sensing image and the visible light remote sensing image and fuse them, the multi-directional scanning module is used to obtain global features of the image in multiple directions, and the second convolution module is used to restore the image dimension to obtain a fused image;

[0010] The fused image is output.

[0011] In some optional embodiments, the first convolution module includes: a 5×5 convolution layer, a first batch normalization layer, and a first ReLU activation function layer;

[0012] The input end of the 5×5 convolution layer is connected to the input end of the first convolution module, the output end of the 5×5 convolution layer is connected to the input end of the first batch normalization layer, the output end of the first batch normalization layer is connected to the input end of the first ReLU activation function layer, and the output end of the first ReLU activation function layer is connected to the output end of the first convolution module.

[0013] In some optional embodiments, the dual attention residual module includes: a residual module, a spatial attention module, a channel attention module, a first multiplication unit, a second multiplication unit, a first addition unit, and a second ReLU activation function layer;

[0014] The input end of the residual module is connected to the output end of the first convolution module, and the output end of the residual module is respectively connected to the input end of the spatial attention module, the input end of the channel attention module, the input end of the first multiplication unit and the input end of the second multiplication unit, the output end of the spatial attention module is connected to the input end of the first multiplication unit, the output end of the channel attention module is connected to the input end of the second multiplication unit, the output end of the first multiplication unit is connected to the input end of the first addition unit, the output end of the second multiplication unit is connected to the input end of the first addition unit, the input end of the first addition unit is also connected to the output end of the first convolution module, the output end of the first addition unit is connected to the input end of the second ReLU activation function layer, and the output end of the second ReLU activation function layer is connected to the output end of the dual attention residual module.

[0015] In some optional implementations, the residual module includes: a first 3×3 convolutional layer, a second batch normalization layer, a third ReLU activation function layer, a second 3×3 convolutional layer, and a third batch normalization layer;

[0016] The input end of the first 3×3 convolutional layer is connected to the input end of the residual module, the output end of the first 3×3 convolutional layer is connected to the input end of the second batch normalization layer, the output end of the second batch normalization layer is connected to the input end of the third ReLU activation function layer, the output end of the third ReLU activation function layer is connected to the input end of the second 3×3 convolutional layer, the output end of the second 3×3 convolutional layer is connected to the input end of the third batch normalization layer, and the output end of the third batch normalization layer is connected to the output end of the residual module.

[0017] In some optional embodiments, the spatial attention module includes: a third 3×3 convolution layer, a fourth ReLU activation function layer, a fourth 3×3 convolution layer and a first Sigmoid activation function layer;

[0018] The input end of the third 3×3 convolutional layer is connected to the input end of the spatial attention module, the output end of the third 3×3 convolutional layer is connected to the input end of the fourth ReLU activation function layer, the output end of the fourth ReLU activation function layer is connected to the input end of the fourth 3×3 convolutional layer, the output end of the fourth 3×3 convolutional layer is connected to the input end of the first Sigmoid activation function layer, and the output end of the first Sigmoid activation function layer is connected to the output end of the spatial attention module.

[0019] In some optional embodiments, the channel attention submodule includes: a global average pooling layer, a first fully connected layer, a fifth ReLU activation function layer, a second fully connected layer, and a second Sigmoid activation function layer;

[0020] The input end of the global average pooling layer is connected to the input end of the channel attention submodule, the output end of the global average pooling layer is connected to the input end of the first fully connected layer, the output end of the first fully connected layer is connected to the input end of the fifth ReLU activation function layer, the output end of the fifth ReLU activation function layer is connected to the input end of the second fully connected layer, the output end of the second fully connected layer is connected to the input end of the second Sigmoid activation function layer, and the output end of the second Sigmoid activation function layer is connected to the output end of the channel attention module.

[0021] In some optional embodiments, the multi-directional scanning module includes: a layer normalization layer, a first linear layer, a second linear layer, a depth-separable convolution layer, a selective scanning module, a third linear layer, a third multiplication unit, a fourth linear layer, and a second addition unit;

[0022] The input end of the layer normalization layer is connected to the output end of the dual attention residual module, the output end of the layer normalization layer is respectively connected to the input ends of the first linear layer and the second linear layer, the output end of the first linear layer is connected to the input end of the third multiplication unit, the output end of the second linear layer is connected to the input end of the depthwise separable convolution layer, the output end of the depthwise separable convolution layer is connected to the input end of the selective scanning module, the output end of the selective scanning module is connected to the input end of the third linear layer, the output end of the third linear layer is connected to the input end of the third multiplication unit, the output end of the third multiplication unit is connected to the input end of the fourth linear layer, the output end of the fourth linear layer is connected to the input end of the second addition unit, the input end of the second addition unit is also connected to the output end of the dual attention residual module, and the output end of the second addition unit is connected to the output end of the multi-directional scanning module.

[0023] In some optional implementations, the second convolution module includes: a 1×1 convolution layer and a Tanh activation function layer;

[0024] The input end of the 1×1 convolution layer is connected to the output end of the multi-directional scanning module, the output end of the 1×1 convolution layer is connected to the input end of the Tanh activation function layer, and the output end of the Tanh activation function layer is connected to the output end of the second convolution module.

[0025] In some optional implementations, the image fusion network model is trained in the following manner:

[0026] Acquire a training sample set, wherein the training samples include matched infrared remote sensing images and visible light remote sensing images;

[0027] The training samples in the training sample set are used as inputs of the image fusion network model, and the image fusion network model is iteratively updated using a preset image fusion loss function.

[0028] In some optional implementations, the image fusion loss function is:

[0029] Loss = L int +αL g +βL s ;

[0030] Among them, Loss represents the image fusion loss function, L int Indicates strength loss, L g represents the gradient loss, L s represents the structural loss, α represents the weight corresponding to the gradient loss, and β represents the weight corresponding to the structural loss;

[0031] L int , L g and L s Respectively expressed as:

[0032]

[0033] L s =1-SSIM(I F ,max(I ir ,I vis ));

[0034] H and W represent the height and width of the image respectively, I F represents the fused image, I ir represents infrared remote sensing image, I vis represents a visible light remote sensing image, I F Perform gradient calculations, I ir Perform gradient calculations, I vis Perform gradient calculations, SSIM represents structural similarity calculations, ||·||1 represents the L1 norm, and ||·||2 represents the L2 norm.

[0035] The main advantages of the technical solution of the present invention are as follows:

[0036] The deep learning fusion method of infrared and visible light remote sensing images based on the visual state space model of the present invention introduces a convolution module, a dual attention residual module and a multi-directional scanning module into the image fusion network model. The dual attention residual module can fully extract high-level semantic information in the remote sensing image, and at the same time can avoid the loss of important information in the remote sensing image as much as possible; the multi-directional scanning module can enhance the global effective receptive field of the remote sensing image in multiple directions, obtain the spatial characteristics of a single object and the spatial dependency relationship between multiple objects, and enrich the detailed characteristics of the fused image; and can fully fuse the infrared remote sensing image information and the visible light remote sensing image information to obtain a fused image with prominent infrared thermal radiation targets and rich background details. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the embodiments of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0038] Figure 1 A flowchart of a method for deep learning fusion of infrared and visible light remote sensing images based on a visual state space model provided by an embodiment of the present invention;

[0039] Figure 2 A structural block diagram of an image fusion network model provided by an embodiment of the present invention;

[0040] Figure 3 for Figure 2 The structural block diagram of the first convolution module in the image fusion network model shown;

[0041] Figure 4 for Figure 2 The structural block diagram of the dual attention residual module in the image fusion network model shown;

[0042] Figure 5 for Figure 2 The structural block diagram of the multi-directional scanning module in the image fusion network model shown;

[0043] Figure 6 for Figure 2 The structural block diagram of the second convolution module in the image fusion network model shown. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0045] The technical solution provided by the embodiments of the present invention is described in detail below with reference to the accompanying drawings.

[0046] See also Figure 1-2 The embodiment of the present invention provides a method for deep learning fusion of infrared and visible light remote sensing images based on a visual state space model, the method comprising the following steps 1-3:

[0047] Step 1, obtaining the infrared remote sensing image and the visible light remote sensing image to be fused;

[0048] In the embodiment of the present invention, the infrared remote sensing image and the visible light remote sensing image to be fused are acquired according to actual needs.

[0049] It should be noted that the infrared remote sensing image and the visible light remote sensing image to be fused are registered with each other.

[0050] Step 2, splicing the infrared remote sensing image and the visible light remote sensing image and inputting them into a pre-built and trained image fusion network model to obtain a fused image output by the image fusion network model;

[0051] In an embodiment of the present invention, an image fusion network model is constructed based on a convolutional neural network (CNN) and a visual state space model (VMamba), and specifically includes a first convolution module, a dual attention residual module, a multi-directional scanning module and a second convolution module connected in sequence. The first convolution module is used to extract shallow features of the image, the dual attention residual module is used to extract important features of infrared remote sensing images and visible light remote sensing images and fuse them, the multi-directional scanning module is used to obtain global features of the image in multiple directions, and the second convolution module is used to restore the image dimension and obtain a fused image.

[0052] In an embodiment of the present invention, the infrared remote sensing image and the visible light remote sensing image are spliced ​​in the channel dimension and then input into the image fusion network model. After being processed by each module in the image fusion network model, a fused image output by the image fusion network model can be obtained.

[0053] In an embodiment of the present invention, by introducing a convolution module, a dual attention residual module and a multi-directional scanning module into an image fusion network model, the dual attention residual module can be used to fully extract high-level semantic information in remote sensing images, while avoiding the loss of important information in remote sensing images as much as possible; the multi-directional scanning module can be used to enhance the global effective receptive field of remote sensing images in multiple directions, obtain the spatial characteristics of a single object and the spatial dependencies between multiple objects, and enrich the detail features of the fused image.

[0054] Step 3: Output the fused image.

[0055] In the embodiment of the present invention, the fused image output by the image fusion network model is the fused image corresponding to the infrared remote sensing image to be fused and the visible light remote sensing image.

[0056] The deep learning fusion method of infrared and visible light remote sensing images based on the visual state space model provided by the embodiment of the present invention can fully fuse the infrared remote sensing image information and the visible light remote sensing image information by introducing a convolution module, a dual attention residual module and a multi-directional scanning module into the image fusion network model, so as to obtain a fused image with prominent infrared thermal radiation targets and rich background details.

[0057] refer to Figure 3 , Further, in an optional implementation of the embodiment of the present invention, in order to realize the function of the first convolution module defined above, the first convolution module includes: a 5×5 convolution layer, a first batch normalization layer and a first ReLU activation function layer;

[0058] The input end of the 5×5 convolution layer is connected to the input end of the first convolution module, the output end of the 5×5 convolution layer is connected to the input end of the first batch normalization layer, the output end of the first batch normalization layer is connected to the input end of the first ReLU activation function layer, and the output end of the first ReLU activation function layer is connected to the output end of the first convolution module.

[0059] In an embodiment of the present invention, the 5×5 convolution layer is used to perform a 5×5 convolution operation on the input features, the batch normalization layer is used to perform batch normalization processing on the input features, and the ReLU activation function layer is used to perform nonlinear transformation processing on the input features using the ReLU activation function.

[0060] It should be noted that in the attached Figure 3 In the figure, 5×5Conv represents a 5×5 convolutional layer, BN represents a batch normalization layer, and ReLU represents a ReLU activation function layer.

[0061] Furthermore, based on the above-set structure, the output of the first convolution module can be expressed as:

[0062]

[0063] in, represents the output of the first convolutional module, I vis represents the visible light remote sensing image, I ir represents infrared remote sensing image, represents a 5×5 convolution operation, BN represents batch normalization, and ReLU represents the ReLU activation function calculation.

[0064] refer to Figure 4 , Further, in an optional implementation of an embodiment of the present invention, in order to realize the function of the dual attention residual module defined above, the dual attention residual module includes: a residual module, a spatial attention module, a channel attention module, a first multiplication unit, a second multiplication unit, a first addition unit and a second ReLU activation function layer;

[0065] The input end of the residual module is connected to the output end of the first convolution module, and the output end of the residual module is respectively connected to the input end of the spatial attention module, the input end of the channel attention module, the input end of the first multiplication unit and the input end of the second multiplication unit, the output end of the spatial attention module is connected to the input end of the first multiplication unit, the output end of the channel attention module is connected to the input end of the second multiplication unit, the output end of the first multiplication unit is connected to the input end of the first addition unit, the output end of the second multiplication unit is connected to the input end of the first addition unit, the input end of the first addition unit is also connected to the output end of the first convolution module, the output end of the first addition unit is connected to the input end of the second ReLU activation function layer, and the output end of the second ReLU activation function layer is connected to the output end of the dual attention residual module.

[0066] In an embodiment of the present invention, a residual module is used to extract image features, and by jump-connecting the residual module to a first multiplication unit and a second multiplication unit, the loss of important information in the remote sensing image can be avoided; a spatial attention module and a channel attention module are used to extract high-level features of the image; the first multiplication unit is used to multiply features output by the residual module and features output by the spatial attention module; the second multiplication unit is used to multiply features output by the residual module and features output by the channel attention module; the first addition unit is used to add features output by the first multiplication unit, features output by the second multiplication unit, and features output by the first convolution module; feature fusion can be achieved by setting the first multiplication unit, the second multiplication unit, and the first addition unit, and jump-connecting the first addition unit to the output end of the first convolution module can further avoid the loss of important information in the remote sensing image; the ReLU activation function layer is used to perform nonlinear transformation processing on input features using the ReLU activation function.

[0067] It should be noted that in the attached Figure 4 middle, represents the multiplication unit, represents an addition unit, and ReLU represents a ReLU activation function layer.

[0068] Furthermore, based on the above-set structure, the output of the dual attention residual module can be expressed as:

[0069]

[0070] in, represents the output of the dual attention residual module, represents the output of the residual module, represents the output of the spatial attention module, represents the output of the channel attention module, Represents the output of the first convolutional module, and ReLU represents the ReLU activation function calculation.

[0071] refer to Figure 4 , Further, in an optional implementation of the embodiment of the present invention, in order to realize the function of the residual module defined above, the residual module includes: a first 3×3 convolution layer, a second batch normalization layer, a third ReLU activation function layer, a second 3×3 convolution layer and a third batch normalization layer;

[0072] The input end of the first 3×3 convolutional layer is connected to the input end of the residual module, the output end of the first 3×3 convolutional layer is connected to the input end of the second batch normalization layer, the output end of the second batch normalization layer is connected to the input end of the third ReLU activation function layer, the output end of the third ReLU activation function layer is connected to the input end of the second 3×3 convolutional layer, the output end of the second 3×3 convolutional layer is connected to the input end of the third batch normalization layer, and the output end of the third batch normalization layer is connected to the output end of the residual module.

[0073] In an embodiment of the present invention, the 3×3 convolution layer is used to perform a 3×3 convolution operation on the input features, the batch normalization layer is used to perform batch normalization processing on the input features, and the ReLU activation function layer is used to perform nonlinear transformation processing on the input features using the ReLU activation function.

[0074] It should be noted that in the attached Figure 4 In the figure, 3×3Conv represents a 3×3 convolutional layer, BN represents a batch normalization layer, and ReLU represents a ReLU activation function layer.

[0075] Furthermore, based on the above-set structure, the output of the residual module can be expressed as:

[0076]

[0077] in, represents the output of the residual module, represents the output of the first convolutional module, represents a 3×3 convolution operation, BN represents batch normalization, and ReLU represents the ReLU activation function calculation.

[0078] refer to Figure 4 , Further, in an optional implementation of the embodiment of the present invention, in order to realize the function of the spatial attention module defined above, the spatial attention module includes: a third 3×3 convolution layer, a fourth ReLU activation function layer, a fourth 3×3 convolution layer and a first Sigmoid activation function layer;

[0079] The input end of the third 3×3 convolutional layer is connected to the input end of the spatial attention module, the output end of the third 3×3 convolutional layer is connected to the input end of the fourth ReLU activation function layer, the output end of the fourth ReLU activation function layer is connected to the input end of the fourth 3×3 convolutional layer, the output end of the fourth 3×3 convolutional layer is connected to the input end of the first Sigmoid activation function layer, and the output end of the first Sigmoid activation function layer is connected to the output end of the spatial attention module.

[0080] In an embodiment of the present invention, the 3×3 convolution layer is used to perform a 3×3 convolution operation on the input features, the ReLU activation function layer is used to perform a nonlinear transformation processing on the input features using the ReLU activation function, and the Sigmoid activation function layer is used to perform a nonlinear transformation processing on the input features using the Sigmoid activation function.

[0081] It should be noted that in the attached Figure 4 In the figure, 3×3Conv represents a 3×3 convolutional layer, ReLU represents a ReLU activation function layer, and Sigmoid represents a Sigmoid activation function layer.

[0082] Furthermore, based on the above-set structure, the output of the spatial attention module can be expressed as:

[0083]

[0084] in, represents the output of the spatial attention module, represents the output of the residual module, ReLU represents the ReLU activation function calculation, and Sigmoid represents the Sigmoid activation function calculation.

[0085] refer to Figure 4, Further, in an optional implementation of the embodiment of the present invention, in order to realize the function of the channel attention module defined above, the channel attention module includes: a global average pooling layer, a first fully connected layer, a fifth ReLU activation function layer, a second fully connected layer, and a second Sigmoid activation function layer;

[0086] The input end of the global average pooling layer is connected to the input end of the channel attention submodule, the output end of the global average pooling layer is connected to the input end of the first fully connected layer, the output end of the first fully connected layer is connected to the input end of the fifth ReLU activation function layer, the output end of the fifth ReLU activation function layer is connected to the input end of the second fully connected layer, the output end of the second fully connected layer is connected to the input end of the second Sigmoid activation function layer, and the output end of the second Sigmoid activation function layer is connected to the output end of the channel attention module.

[0087] In an embodiment of the present invention, the global average pooling layer is used to perform global average pooling processing on the input features, the fully connected layer is used to integrate all features and reduce the dimensionality of the features, the ReLU activation function layer is used to perform nonlinear transformation processing on the input features using the ReLU activation function, and the Sigmoid activation function layer is used to perform nonlinear transformation processing on the input features using the Sigmoid activation function.

[0088] It should be noted that in the attached Figure 4 In the figure, GAP represents the global average pooling layer, FC represents the fully connected layer, ReLU represents the ReLU activation function layer, and Sigmoid represents the Sigmoid activation function layer.

[0089] Furthermore, based on the above-set structure, the output of the channel attention module can be expressed as:

[0090]

[0091] in, represents the output of the channel attention module, Represents the output of the residual module, GAP represents global average pooling, FC represents full connection, ReLU represents ReLU activation function calculation, and Sigmoid represents Sigmoid activation function calculation.

[0092] refer to Figure 5 , Further, in an optional implementation of the embodiment of the present invention, in order to realize the function of the multi-directional scanning module defined above, the multi-directional scanning module includes: a layer normalization layer, a first linear layer, a second linear layer, a depth-separable convolution layer, a selective scanning module, a third linear layer, a third multiplication unit, a fourth linear layer and a second addition unit;

[0093] The input end of the layer normalization layer is connected to the output end of the dual attention residual module, the output end of the layer normalization layer is respectively connected to the input ends of the first linear layer and the second linear layer, the output end of the first linear layer is connected to the input end of the third multiplication unit, the output end of the second linear layer is connected to the input end of the depthwise separable convolution layer, the output end of the depthwise separable convolution layer is connected to the input end of the selective scanning module, the output end of the selective scanning module is connected to the input end of the third linear layer, the output end of the third linear layer is connected to the input end of the third multiplication unit, the output end of the third multiplication unit is connected to the input end of the fourth linear layer, the output end of the fourth linear layer is connected to the input end of the second addition unit, the input end of the second addition unit is also connected to the output end of the dual attention residual module, and the output end of the second addition unit is connected to the output end of the multi-directional scanning module.

[0094] In an embodiment of the present invention, the layer normalization layer is used to perform layer normalization processing on the input features, the linear layer is used to perform linear transformation processing on the input features, the depthwise separable convolution layer is used to perform depthwise separable convolution operations on the input features, the selective scanning module is used to selectively scan the input features, the third multiplication unit is used to multiply the output of the first linear layer and the output of the third linear layer, and the second addition unit is used to add the output of the dual attention residual module and the output of the fourth linear layer.

[0095] In an embodiment of the present invention, based on the above structure, the multi-directional scanning module can flatten the input tokens in the horizontal, vertical, oblique, reverse oblique and reverse directions into 8 image token sequences, and independently input them into the selective scanning module for selective scanning. The scanning results in all directions are finally added together to obtain the output tokens. Because the output tokens summarize the features of selective scanning in 8 directions, the remote sensing images can be globally modeled in multiple directions, thereby extracting large-scale spatial features in multiple directions.

[0096] It should be noted that in the attached Figure 5 In the figure, LN represents the layer normalization layer, Linear represents the linear layer, Depth-wiseConv represents the depth-wise separable convolutional layer, and SSM represents the selective scanning module. represents the multiplication unit, Represents an addition unit.

[0097] refer to Figure 6 , Further, in an optional implementation of the embodiment of the present invention, in order to realize the function of the second convolution module defined above, the second convolution module includes: a 1×1 convolution layer and a Tanh activation function layer;

[0098] The input end of the 1×1 convolution layer is connected to the output end of the multi-directional scanning module, the output end of the 1×1 convolution layer is connected to the input end of the Tanh activation function layer, and the output end of the Tanh activation function layer is connected to the output end of the second convolution module.

[0099] In the embodiment of the present invention, the 1×1 convolution layer is used to perform a 1×1 convolution operation on the input features, and the Tanh activation function layer is used to perform nonlinear transformation processing on the input features using the Tanh activation function.

[0100] It should be noted that in the attached Figure 6 In the figure, 1×1Conv represents a 1×1 convolutional layer, and Tanh represents a Tanh activation function layer.

[0101] Furthermore, based on the above-set structure, the output of the second convolution module can be expressed as:

[0102]

[0103] Among them, I F represents the output of the second convolutional module, Represents the output of the multi-directional scanning module, represents a 1×1 convolution operation, Tanh represents the Tanh activation function calculation, and the output of the second convolution module is the output of the image fusion network model, that is, the fused image.

[0104] Furthermore, in an optional implementation of the embodiment of the present invention, the image fusion network model is trained in the following manner:

[0105] Acquire a training sample set, the training sample including matched infrared remote sensing images and visible light remote sensing images;

[0106] The training samples in the training sample set are used as the input of the image fusion network model, and the image fusion network model is iteratively updated using the preset image fusion loss function.

[0107] In an embodiment of the present invention, when obtaining a training sample set, multiple visible light remote sensing images and multiple infrared remote sensing images are obtained from visible light remote sensing images and infrared remote sensing images taken by existing satellites, and image registration is performed using ENVI software. The registered remote sensing image pairs are then cropped to a size of 256*256 pixels to obtain multiple pairs of remote sensing image pair slices, and each pair of remote sensing image pair slices is used as a training sample to obtain a training sample set including multiple training samples.

[0108] Furthermore, in one embodiment of the present invention, the training samples in the training sample set are used as inputs of the image fusion network model, and the image fusion network model is iteratively updated using a preset image fusion loss function, further comprising the following steps 201 to 203:

[0109] Step 201, after splicing the infrared remote sensing images and visible light remote sensing images in all training samples in the channel dimension, the images are input into the image fusion network model in batches to obtain a fused image corresponding to each training sample output by the image fusion network model;

[0110] In one embodiment of the present invention, the infrared remote sensing image and the visible light remote sensing image in the training sample are spliced ​​in the channel dimension and input from the input end of the image fusion network model, processed by each layer in the image fusion network model in sequence, and output from the output end of the image fusion network model. The information output by the image fusion network model is the fused image corresponding to the input infrared remote sensing image and the visible light remote sensing image.

[0111] In one embodiment of the present invention, the image fusion network model is a model to be trained, and the model parameters are initialization parameters. During the training process of the model, the model parameters are continuously updated and learned.

[0112] Step 202, calculating an image fusion loss function according to the infrared remote sensing image and the visible light remote sensing image in the training sample and the fused image corresponding to the training sample;

[0113] In one embodiment of the present invention, the image fusion loss function is:

[0114] Loss = L int +αL g +βL s ;

[0115] Among them, Loss represents the image fusion loss function, L int Indicates strength loss, L g represents the gradient loss, L s represents the structural loss, α represents the weight corresponding to the gradient loss, and β represents the weight corresponding to the structural loss;

[0116] L int , L g and L s Respectively expressed as:

[0117]

[0118] L s =1-SSIM(I F ,max(I ir ,I vis ));

[0119] H and W represent the height and width of the image respectively, I F represents the fused image, I ir represents infrared remote sensing image, I vis represents a visible light remote sensing image, I F Perform gradient calculations, I ir Perform gradient calculations, I vis Perform gradient calculations, SSIM represents structural similarity calculations, ||·||1 represents the L1 norm, and ||·||2 represents the L2 norm.

[0120] It should be noted that the weights corresponding to the gradient loss and the weights corresponding to the structural loss are set according to the actual situation.

[0121] Step 203, determine whether the preset training stop condition is reached, if so, use the current image fusion network model as the trained image fusion network model, if not, use the image fusion loss function to update the parameters of the image fusion network model, and return to step 201.

[0122] In one embodiment of the present invention, the training stop condition is specifically set according to the actual situation, for example, the number of training iterations reaches a set iteration number or the optimization index reaches a set threshold. Among them, the image fusion loss function can be used as the optimization index.

[0123] Furthermore, in one embodiment of the present invention, a stochastic gradient descent method is used to train and update the parameters of the image fusion network model.

[0124] Specifically, the following formula is used to update the parameters of the image fusion network model:

[0125]

[0126] Among them, θ represents the parameter set of the image fusion network model, Δ[·] represents the optimizer, η represents the learning rate, and Loss represents the image fusion loss function. Among them, the optimizer is set according to the actual situation, for example, Adam, and the learning rate is pre-set to control the speed of updating the parameters of the image fusion network model.

[0127] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep learning fusion method for infrared and visible light remote sensing images based on a visual state space model, characterized in that: include: Acquire infrared remote sensing images and visible light remote sensing images to be fused; The infrared remote sensing image and the visible light remote sensing image are spliced ​​and input into a pre-built and trained image fusion network model to obtain a fused image output by the image fusion network model, wherein the image fusion network model comprises a first convolution module, a dual attention residual module, a multi-directional scanning module and a second convolution module connected in sequence, wherein the first convolution module is used to extract shallow features of the image, the dual attention residual module is used to extract important features of the infrared remote sensing image and the visible light remote sensing image and fuse them, the multi-directional scanning module is used to obtain global features of the image in multiple directions, and the second convolution module is used to restore the image dimension to obtain a fused image; The fused image is output.

2. The method for deep learning fusion of infrared and visible light remote sensing images based on a visual state space model according to claim 1, characterized in that: The first convolution module includes: a 5×5 convolution layer, a first batch normalization layer and a first ReLU activation function layer; The input end of the 5×5 convolution layer is connected to the input end of the first convolution module, the output end of the 5×5 convolution layer is connected to the input end of the first batch normalization layer, the output end of the first batch normalization layer is connected to the input end of the first ReLU activation function layer, and the output end of the first ReLU activation function layer is connected to the output end of the first convolution module.

3. The method for deep learning fusion of infrared and visible light remote sensing images based on a visual state space model according to claim 2 is characterized in that: The dual attention residual module includes: a residual module, a spatial attention module, a channel attention module, a first multiplication unit, a second multiplication unit, a first addition unit and a second ReLU activation function layer; The input end of the residual module is connected to the output end of the first convolution module, and the output end of the residual module is respectively connected to the input end of the spatial attention module, the input end of the channel attention module, the input end of the first multiplication unit and the input end of the second multiplication unit, the output end of the spatial attention module is connected to the input end of the first multiplication unit, the output end of the channel attention module is connected to the input end of the second multiplication unit, the output end of the first multiplication unit is connected to the input end of the first addition unit, the output end of the second multiplication unit is connected to the input end of the first addition unit, the input end of the first addition unit is also connected to the output end of the first convolution module, the output end of the first addition unit is connected to the input end of the second ReLU activation function layer, and the output end of the second ReLU activation function layer is connected to the output end of the dual attention residual module.

4. The method for deep learning fusion of infrared and visible light remote sensing images based on a visual state space model according to claim 3 is characterized in that: The residual module includes: a first 3×3 convolution layer, a second batch normalization layer, a third ReLU activation function layer, a second 3×3 convolution layer and a third batch normalization layer; The input end of the first 3×3 convolutional layer is connected to the input end of the residual module, the output end of the first 3×3 convolutional layer is connected to the input end of the second batch normalization layer, the output end of the second batch normalization layer is connected to the input end of the third ReLU activation function layer, the output end of the third ReLU activation function layer is connected to the input end of the second 3×3 convolutional layer, the output end of the second 3×3 convolutional layer is connected to the input end of the third batch normalization layer, and the output end of the third batch normalization layer is connected to the output end of the residual module.

5. The method for deep learning fusion of infrared and visible light remote sensing images based on a visual state space model according to claim 4 is characterized in that: The spatial attention module includes: a third 3×3 convolution layer, a fourth ReLU activation function layer, a fourth 3×3 convolution layer and a first Sigmoid activation function layer; The input end of the third 3×3 convolutional layer is connected to the input end of the spatial attention module, the output end of the third 3×3 convolutional layer is connected to the input end of the fourth ReLU activation function layer, the output end of the fourth ReLU activation function layer is connected to the input end of the fourth 3×3 convolutional layer, the output end of the fourth 3×3 convolutional layer is connected to the input end of the first Sigmoid activation function layer, and the output end of the first Sigmoid activation function layer is connected to the output end of the spatial attention module.

6. The method for deep learning fusion of infrared and visible light remote sensing images based on a visual state space model according to claim 5, characterized in that: The channel attention submodule includes: a global average pooling layer, a first fully connected layer, a fifth ReLU activation function layer, a second fully connected layer, and a second Sigmoid activation function layer; The input end of the global average pooling layer is connected to the input end of the channel attention submodule, the output end of the global average pooling layer is connected to the input end of the first fully connected layer, the output end of the first fully connected layer is connected to the input end of the fifth ReLU activation function layer, the output end of the fifth ReLU activation function layer is connected to the input end of the second fully connected layer, the output end of the second fully connected layer is connected to the input end of the second Sigmoid activation function layer, and the output end of the second Sigmoid activation function layer is connected to the output end of the channel attention module.

7. The method for deep learning fusion of infrared and visible light remote sensing images based on a visual state space model according to claim 3, characterized in that: The multi-directional scanning module includes: a layer normalization layer, a first linear layer, a second linear layer, a depth-separable convolutional layer, a selective scanning module, a third linear layer, a third multiplication unit, a fourth linear layer, and a second addition unit; The input end of the layer normalization layer is connected to the output end of the dual attention residual module, the output end of the layer normalization layer is respectively connected to the input ends of the first linear layer and the second linear layer, the output end of the first linear layer is connected to the input end of the third multiplication unit, the output end of the second linear layer is connected to the input end of the depthwise separable convolution layer, the output end of the depthwise separable convolution layer is connected to the input end of the selective scanning module, the output end of the selective scanning module is connected to the input end of the third linear layer, the output end of the third linear layer is connected to the input end of the third multiplication unit, the output end of the third multiplication unit is connected to the input end of the fourth linear layer, the output end of the fourth linear layer is connected to the input end of the second addition unit, the input end of the second addition unit is also connected to the output end of the dual attention residual module, and the output end of the second addition unit is connected to the output end of the multi-directional scanning module.

8. The method for deep learning fusion of infrared and visible light remote sensing images based on a visual state space model according to claim 4, characterized in that: The second convolution module includes: a 1×1 convolution layer and a Tanh activation function layer; The input end of the 1×1 convolution layer is connected to the output end of the multi-directional scanning module, the output end of the 1×1 convolution layer is connected to the input end of the Tanh activation function layer, and the output end of the Tanh activation function layer is connected to the output end of the second convolution module.

9. The deep learning fusion method of infrared and visible light remote sensing images based on a visual state space model according to any one of claims 1 to 8, characterized in that: The image fusion network model is trained in the following way: Acquire a training sample set, wherein the training samples include matched infrared remote sensing images and visible light remote sensing images; The training samples in the training sample set are used as inputs of the image fusion network model, and the image fusion network model is iteratively updated using a preset image fusion loss function.

10. The method for deep learning fusion of infrared and visible light remote sensing images based on a visual state space model according to claim 9, characterized in that: The image fusion loss function is: Loss=L int +αL g +βL s ; Among them, Loss represents the image fusion loss function, L int Indicates strength loss, L g represents the gradient loss, L s represents the structural loss, α represents the weight corresponding to the gradient loss, and β represents the weight corresponding to the structural loss; L int , L g and L s Respectively expressed as: L s =1-SSIM(I F ,max(I ir ,I vis )); H and W represent the height and width of the image respectively, I F represents the fused image, I ir represents infrared remote sensing image, I vis represents the visible light remote sensing image, ▽I F I F Perform gradient calculations, I ir Perform gradient calculations, I vis Perform gradient calculations, SSIM represents structural similarity calculations, ||·||1 represents the L1 norm, and ||·||2 represents the L2 norm.

Images