Sar and visible light remote sensing image fusion method based on deep learning

Abstract

The application discloses a SAR and visible light remote sensing image fusion method based on deep learning, and a fusion model adopted by the method comprises a feature extraction part, a fusion part and an image reconstruction part. An overall loss function used for training the fusion model at least comprises a similarity loss and a brightness distribution loss; the similarity loss uses a preprocessing model to extract features of each mode, and the degree of important information reservation in visible light and SAR images is quantified by using entropy and standard deviation calculation; the brightness distribution loss uses a Gram matrix to measure feature correlation, and then obtains the degree of inconsistency between a fusion result image and an original SAR image in feature distribution. The application not only realizes the complementation of different resolution features, guarantees the effective combination of reflection intensity information in the SAR image and detail information in the visible light image, but also can highlight the SAR information in the fusion result, reserve key features, and improve visual quality and information integrity.

Description

Technical Field

[0001] The present invention belongs to the field of image data processing, and in particular relates to a method for fusing SAR images with visible light remote sensing images. Background Art

[0002] The fusion of synthetic aperture radar (SAR) and visible light imagery is a key technology in remote sensing image processing and analysis. The core of this technique lies in combining the characteristics of the two data types to achieve complementary advantages, thereby providing more comprehensive and accurate information. This fusion technology plays a vital role in numerous remote sensing applications, such as land use change monitoring, disaster assessment, urban planning, and agricultural monitoring. By integrating different types of imagery, more comprehensive information on the morphology, structure, and dynamic changes of ground targets can be obtained, thereby improving the accuracy and reliability of information extraction.

[0003] In recent years, deep learning technology has achieved breakthroughs in the field of image fusion. Compared to traditional methods that rely on manually designed features and rules, deep learning can automatically learn data features, achieving more intelligent image fusion, thereby improving image quality and information extraction capabilities. Existing image fusion methods often focus on preserving pixel brightness and detail information through loss functions or adding residual connections and dense connections to the network architecture to capture texture details. However, existing methods still have the following shortcomings:

[0004] 1. Existing deep learning methods are based on dual attention fusion of original scale features, ignoring the complementarity of features at different resolution levels, resulting in the inability to effectively combine the reflection intensity information in SAR images with the detail information in visible light images.

[0005] 2. In target detection or nighttime recognition tasks, it is necessary to rely on SAR modal information in the fusion results to improve detection or recognition accuracy. However, existing deep learning methods only adjust the proportion of SAR modalities, which is not precise enough and cannot fully retain and utilize the important SAR modal information.

[0006] 3. Existing methods do not determine the degree of retention of important information based on the inherent properties of each modality, and cannot effectively integrate complementary information, resulting in insufficient retention of key features in the fusion results and suboptimal information integrity. Summary of the Invention

[0007] The present invention proposes a deep learning-based SAR and visible light remote sensing image fusion method, the objectives of which are: 1. to solve the problem that the reflection intensity information in the SAR image cannot be effectively combined with the detail information in the visible light image; 2. to solve the problem that important information of the SAR modality is not sufficiently retained; 3. to solve the problem of insufficient information integrity of the fusion result.

[0008] The technical solutions of the present invention are as follows:

[0009] A deep learning-based SAR and visible light remote sensing image fusion method is proposed. A visible light remote sensing image and a SAR image that have been registered at the same time and in the same area are input into a fusion model to obtain a fused result image. The fusion model includes a feature extraction part, a fusion part, and an image reconstruction part.

[0010] The feature extraction part includes a first branch corresponding to the visible light remote sensing image and a second branch corresponding to the SAR image, which are respectively used to obtain visible light feature maps of different resolutions and SAR feature maps of different resolutions;

[0011] The fusion part includes multiple fusion blocks based on channel attention and spatial attention. The fusion blocks include same-resolution dual-attention fusion blocks and different-resolution dual-attention fusion blocks. The same-resolution dual-attention fusion blocks are used to fuse visible light feature maps and SAR feature maps of the same resolution. The different-resolution dual-attention fusion blocks are used to fuse visible light feature maps and SAR feature maps of different resolutions. The fusion part cross-fuses the visible light feature map and SAR feature map output by the feature extraction part to obtain a fused feature map corresponding to the fusion block one by one. The obtained multiple fused feature maps are distributed at multiple resolution levels.

[0012] The image reconstruction part is used to reconstruct the multiple fusion feature maps output by the fusion part to the resolution of the original visible light remote sensing image and the SAR image, and then splice all the fusion feature maps with the same resolution, and then obtain the grayscale fusion image by convolution of the splicing result, and finally superimpose the grayscale fusion image with the CrCb information separated from the visible light remote sensing image to obtain the fusion result image. ;

[0013] The overall loss function used to train the fusion model includes at least similarity loss and brightness distribution loss. The similarity loss uses a preprocessing model to extract features from each modality and quantifies the degree of retention of important information in visible light and SAR images using entropy and standard deviation calculations. The brightness distribution loss uses the Gram matrix to measure feature correlation and then obtains the degree of inconsistency in feature distribution between the fusion result image and the original SAR image.

[0014] As a further improvement of the deep learning-based SAR and visible light remote sensing image fusion method: in the feature extraction part, after the visible light remote sensing image is input into the first branch, it is first converted from RGB format to YCrCb format, and then the Y channel is extracted from the YCrCb format image to obtain the image , and then through convolution and Activation function gets features ;in, express activation function, Represents the convolution operation; after that, the features Input to the first resolution interpretation module Get the first visible light feature map, and Input to the second resolution interpretation module Get the second visible light characteristic map, and transform the characteristic Input to the third resolution interpretation module Obtaining a third visible light characteristic map;

[0015] After the SAR image is input into the second branch, it is first converted into a grayscale image , and then through convolution and The activation function performs preliminary extraction and expands its channel dimension to obtain features ;in, express activation function, Represents the convolution operation; then the features Input to the first resolution interpretation module Get the first SAR feature map, and Input to the second resolution interpretation module Get the second SAR feature map, and transform the feature Input to the third resolution interpretation module The third SAR characteristic map is obtained.

[0016] As a further improvement of the deep learning-based SAR and visible light remote sensing image fusion method: the first resolution interpretation module and the second resolution interpretation module Each consists of a basic residual downsampling module, and a third resolution interpretation module It is composed of two and second resolution interpretation modules The same basic residual downsampling is connected in sequence;

[0017] The basic residual downsampling module includes a residual branch and a downsampling branch. The features input to the basic residual downsampling module are added together after passing through the residual branch and the downsampling branch to obtain output features.

[0018] As a further improvement of the deep learning-based SAR and visible light remote sensing image fusion method: the fusion block first generates a channel feature vector and a spatial feature vector based on the input visible light feature map and the SAR feature map, and then obtains the fusion feature map output by the fusion block based on the channel feature vector and the spatial feature vector.

[0019] As a further improvement of the deep learning-based SAR and visible light remote sensing image fusion method, the channel feature vector is generated as follows:

[0020] Step A-1: ​​If the input visible light feature map and the SAR feature map have the same resolution, directly proceed to step A-2; otherwise, convert the feature map with the lower resolution between the visible light feature map and the SAR feature map to the same resolution as the other feature map;

[0021] Step A-2: Visible light feature maps with the same resolution and SAR characteristic diagram First, perform element-by-element summation, then pass the summed feature tensor through a global average pooling layer for dimensionality reduction, and then pass it through a downsampling layer Compress to obtain channel feature vector , Indicates the resolution at which the two feature maps are consistent;

[0022] The spatial feature vector is generated as follows:

[0023] Step B-1: If the input visible light feature map and the SAR feature map have the same resolution, directly proceed to step B-2; otherwise, convert the feature map with the lower resolution between the visible light feature map and the SAR feature map to the same resolution as the other feature map;

[0024] Step B-2: Visible light feature maps with the same resolution and SAR characteristic diagram First, splice and then input the spliced ​​feature map into the spatial downsampling layer , and then reduce the spatial dimension through convolution, further compress the important spatial information to obtain the spatial feature tensor ;

[0025] Based on channel feature vector and spatial eigenvectors Get the fusion feature map output by the fusion block The process is:

[0026] ;

[0027] ;

[0028] ;

[0029] in, and Represent the spatial information representation diagrams of visible light remote sensing images and SAR images respectively, and The visible light feature map and SAR feature map input by the current fusion block are converted into the same resolution as the visible light feature map and SAR feature map. represents the spatial upsampling operation, Represents the channel upsampling operation, Represents the sigmoid function activation operation, Represents element-wise multiplication.

[0030] As a further improvement of the deep learning-based SAR and visible light remote sensing image fusion method, the cross fusion refers to: the fusion part includes 5 fusion blocks, the first fusion block, the third fusion block and the fifth fusion block are same-resolution dual-attention fusion blocks, and the second fusion block and the fourth fusion block are different-resolution dual-attention fusion blocks;

[0031] The first visible light feature map and the first SAR feature map are input into the first fusion block to obtain a first fused feature map, the first visible light feature map and the second SAR feature map are input into the second fusion block to obtain a second fused feature map, the second visible light feature map and the second SAR feature map are input into the third fusion block to obtain a third fused feature map, the second visible light feature map and the third SAR feature map are input into the fourth fusion block to obtain a fourth fused feature map, and the third visible light feature map and the third SAR feature map are input into the fifth fusion block to obtain a fifth fused feature map.

[0032] As a further improvement of the SAR and visible light remote sensing image fusion method based on deep learning: in the five fusion feature maps output by the fusion part, the resolution of the first feature map and the second feature map is consistent with the resolution of the original visible light remote sensing image and the SAR image, and both are , the resolution of the third and fourth feature maps are , reconstruct the resolution to , the resolution of the fifth feature map is , the resolution is reconstructed to ;

[0033] First low-resolution reconstruction module It consists of a basic residual upsampling block, which includes a residual branch and an upsampling branch. The features input to the basic residual upsampling module are added after passing through the residual branch and the upsampling branch to obtain the output features; the second low-resolution reconstruction module It consists of two sequentially connected low-resolution reconstruction modules The basic residual upsampling blocks with the same structure.

[0034] As a further improvement of the deep learning-based SAR and visible light remote sensing image fusion method, a loss function is constructed based on a pre-trained model; the pre-trained model includes a first convolutional unit layer, a first maximum pooling layer, a second convolutional unit layer, and a second maximum pooling layer connected in sequence;

[0035] The image corresponding to the current training sample Input into the pre-trained model, and get a set of images through the first convolution unit layer. The feature maps corresponding to each channel , a set of images are obtained through the second convolution unit layer The feature maps corresponding to each channel , is the sequence number of the channel;

[0036] The grayscale image corresponding to the current training sample Input into the pre-trained model, and get a set of grayscale images through the first convolution unit layer. The feature maps corresponding to each channel As well as grayscale images The overall corresponding feature map , through the second convolution unit layer to obtain a set of grayscale images The feature maps corresponding to each channel As well as grayscale images The overall corresponding feature map ;

[0037] The fusion result image obtained by the training sample through the fusion model Input into the pre-trained model and obtain the feature map through the first convolutional unit layer , the feature map is obtained through the second convolution unit layer .

[0038] As a further improvement of the SAR and visible light remote sensing image fusion method based on deep learning: the fusion model is trained by similarity loss. , multi-layer perception loss and brightness distribution loss The overall loss function is: ,in , Control and The weight of .

[0039] As a further improvement of the deep learning-based SAR and visible light remote sensing image fusion method:

[0040] 1. The calculation method of similarity loss is:

[0041] First, the standard deviation and entropy are combined to calculate the weight of the constrained light intensity term and :

[0042] ;

[0043] ;

[0044] in, is the sequence number of the convolutional unit layer in the pre-trained model, Indicates the total number of channels output by the current convolutional unit layer, and are two weights used to control the entropy value and standard deviation ratio, and are the calculation formulas for entropy and standard deviation respectively;

[0045] Then calculate the gradient term and :

[0046] ;

[0047] ;

[0048] in, represents the Laplace of Gaussian operator, Represents the L2 norm operation;

[0049] Then calculate the brightness intensity similarity loss and gradient similarity loss :

[0050] ;

[0051] ;

[0052] in, express and The average structural similarity between

[0053] Finally, calculate the similarity loss : , To control the balance weight of the two losses;

[0054] 2. The calculation method of multi-layer perception loss is:

[0055] First, calculate the multi-layer perception loss of high-resolution visible light remote sensing images :

[0056] ;

[0057] in, is the number of fusion blocks in the fusion part, It is The constraint weights of the fusion blocks, It is The fusion feature map obtained by the fusion blocks, Is input to The visible light characteristics of the fusion block, represents the Laplace of Gaussian operator, Represents the L2 norm operation;

[0058] Then calculate the multi-layer perception loss of the low-resolution SAR image :

[0059] ;

[0060] Finally, calculate the multi-layer perception loss :

[0061] ;

[0062] Indicates balance and The weights of the two loss functions;

[0063] 3. The calculation method of brightness distribution loss function is:

[0064] ;

[0065] in, Represents the Gram matrix that measures feature correlation.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] 1. The present invention first generates feature maps of different resolutions, then performs dual-attention cross-fusion on the two feature maps of different resolutions, and finally unifies the resolution and completes image reconstruction, thereby achieving the complementarity of features of different resolutions and ensuring the effective combination of the reflection intensity information in the SAR image and the detail information in the visible light image.

[0068] 2. The brightness distribution loss function of the present invention uses the Gram matrix to measure feature correlation, and then obtains the degree of inconsistency in feature distribution between the fusion result image and the original SAR image. Training the model based on this loss function can make the feature distribution of the fusion result image as close as possible to that of the original SAR image, thereby accurately highlighting the SAR information in the fusion result.

[0069] 3. The similarity loss function of the present invention uses a preprocessing model to extract the features of each modality, and uses entropy and standard deviation calculations to quantify the degree of retention of important information in visible light and SAR images. This not only ensures that the fusion result can retain key features, but also improves the visual quality and information integrity.

[0070] 4. The dual-path framework of the present invention has a lighter structure and higher training efficiency while ensuring the fusion quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 Schematic diagram of the framework of the fusion model of the present invention;

[0072] Figure 2 is a schematic diagram of a first-resolution interpretation module;

[0073] Figure 3 is a schematic diagram of a second resolution interpretation module;

[0074] Figure 4 is a schematic diagram of a third-resolution interpretation module;

[0075] Figure 5 Schematic diagram of the process of generating channel feature vectors based on feature maps of different resolutions;

[0076] Figure 6 Schematic diagram of the process of generating channel feature vectors based on feature maps of the same resolution;

[0077] Figure 7 Schematic diagram of the process of generating spatial feature vectors based on feature maps of different resolutions;

[0078] Figure 8 Schematic diagram of the process of generating spatial feature vectors based on feature maps of the same resolution;

[0079] Figure 9 Schematic diagram of the dual attention fusion process;

[0080] Figure 10 Schematic diagram of the first low-resolution reconstruction module;

[0081] Figure 11 Schematic diagram of the second low-resolution reconstruction module;

[0082] Figure 12 Schematic diagram of the pre-trained feature extraction model. DETAILED DESCRIPTION

[0083] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only a part of the embodiments of the present invention, rather than all the embodiments.

[0084] A SAR and visible light remote sensing image fusion method based on deep learning is to input the visible light remote sensing image and SAR image that have been registered in the same area at the same time into the image. Figure 1 In the fusion model shown, the fusion result image is obtained.

[0085] The fusion model includes a feature extraction part, a fusion part and an image reconstruction part.

[0086] 1. Feature extraction part.

[0087] The feature extraction part includes a first branch corresponding to the visible light remote sensing image and a second branch corresponding to the SAR image, which are respectively used to obtain visible light feature maps of different resolutions and SAR feature maps of different resolutions.

[0088] After the visible light remote sensing image is input into the first branch, it is first converted from RGB format to YCrCb format, and then the Y channel is extracted from the YCrCb format image to obtain the image , and then input to a layer of convolution kernel size 3×3, stride 1×1, and use The convolution layer as the activation function extracts the features preliminarily and expands the channel dimension to obtain the features. .in, express activation function, Represents the convolution operation. Afterwards, the feature Input to the first resolution interpretation module Get the first visible light feature map, and Input to the second resolution interpretation module Get the second visible light characteristic map, and transform the characteristic Input to the third resolution interpretation module A third visible light characteristic map is obtained.

[0089] After the SAR image is input into the second branch, it is first converted into a grayscale image , and then input to a layer of convolution kernel size 3×3, stride 1, and use The convolution layer as the activation function extracts the features preliminarily and expands the channel dimension to obtain the features. .in, express activation function, Represents the convolution operation. Afterwards, the feature Input to the first resolution interpretation module Get the first SAR feature map, and Input to the second resolution interpretation module Get the second SAR feature map, and transform the feature Input to the third resolution interpretation module The third SAR characteristic map is obtained.

[0090] First resolution interpretation module , second resolution interpretation module and third resolution interpretation module The resolution of the output feature maps decreases successively.

[0091] Specifically, if Figures 2 to 4 , the first resolution interpretation module and the second resolution interpretation module Each consists of a basic residual downsampling module, and a third resolution interpretation module It is composed of two and second resolution interpretation modules The same basic residual downsampling is connected in sequence.

[0092] The basic residual downsampling module includes a residual branch and a downsampling branch. The features input to the basic residual downsampling module are added after passing through the residual branch and the downsampling branch to obtain the output features. In this embodiment, the residual branch includes two sequentially connected As the activation function of the convolution layer (convolution kernel size is 3×3, stride is 1), a convolution kernel size of 3×3, using The convolutional layer as an activation function ( Here the convolutional layer stride is 1. Here, the convolution layer has a stride of 2) and a convolution layer with a kernel size of 3×3 and a stride of 1. The downsampling branch is used to reduce the spatial dimension of the feature map, including a convolution layer with a kernel size of 3×3 and a stride of 1 and a convolution layer with a kernel size of 3×3 and a stride of 1. The convolutional layer as an activation function ( Here the convolutional layer stride is 1. Here the convolutional layer stride is 2). Finally, the resolution of the first visible light feature map and the first SAR feature map is , the resolution of the second visible light feature map and the second SAR feature map is , the resolution of the third visible light feature map and the third SAR feature map is , and are the height and width of the original visible light remote sensing image and SAR image.

[0093] 2. Integration part.

[0094] like Figure 1The fusion part includes multiple fusion blocks based on channel attention and spatial attention. The fusion blocks include same-resolution dual-attention fusion blocks and different-resolution dual-attention fusion blocks. The same-resolution dual-attention fusion blocks are used to fuse visible light feature maps and SAR feature maps of the same resolution. The different-resolution dual-attention fusion blocks are used to fuse visible light feature maps and SAR feature maps of different resolutions. The fusion part cross-fuses the visible light feature map and SAR feature map output by the feature extraction part to obtain a fused feature map corresponding to the fusion block one by one. The obtained multiple fused feature maps are distributed on multiple resolution levels.

[0095] Specifically, the fusion block first generates a channel feature vector and a spatial feature vector based on the input visible light feature map and the SAR feature map, and then obtains the fusion feature map output by the fusion block based on the channel feature vector and the spatial feature vector.

[0096] like Figure 5 and 6 , the channel feature vector is generated as follows:

[0097] Step A-1: ​​If the input visible light feature map and SAR feature map have the same resolution, then directly execute step A-2; otherwise, the feature map with the lower resolution among the visible light feature map and SAR feature map is converted to the same resolution as the other feature map. The conversion method in this embodiment is to sequentially input the low-resolution feature map into a convolution kernel with a size of 3×3, a step size of 2, and a In the transposed convolution layer as the activation function. represents the higher resolution of the two feature maps, represents the lower resolution of the two feature maps.

[0098] Step A-2: Visible light feature maps with the same resolution and SAR characteristic diagram First, perform element-by-element summation, then pass the summed feature tensor through a global average pooling layer for dimensionality reduction, and then pass it through a downsampling layer Compress to obtain channel feature vector .

[0099] Therefore: ,in represents the channel downsampling operation, represents the global pooling operation, Represents element-by-element addition. The downsampling layer Contains a 1×1 convolutional layer to reduce the number of channels and a activation function, Indicates the resolution at which the two feature maps are consistent.

[0100] like Figure 7 and 8 , the spatial feature vector is generated as follows:

[0101] Step B-1: If the input visible light feature map and SAR feature map have the same resolution, then directly execute step B-2; otherwise, the feature map with the lower resolution among the visible light feature map and SAR feature map is converted to the same resolution as the other feature map. The conversion method in this embodiment is to sequentially input the low-resolution feature map into a convolution kernel with a size of 3×3, a step size of 2, and a In the transposed convolutional layer as the activation function.

[0102] Step B-2: Visible light feature maps with the same resolution and SAR characteristic diagram First, splice and then input the spliced ​​feature map into the spatial downsampling layer Then, a convolution layer with a convolution kernel size of 3×3 and a step size of 1 is used to reduce the spatial dimension and further compress the important spatial information to obtain the spatial feature tensor. .

[0103] Therefore: ,in, Represents a splicing operation, represents the spatial downsampling operation, Represents the above convolution operation. The spatial downsampling layer Contains two uses connected in sequence Convolutional layers as activation functions (the first convolutional layer has a kernel size of 1×1 and a stride of 1, and the second convolutional layer has a kernel size of 3×3 and a stride of 2).

[0104] like Figure 9 , based on the channel feature vector and spatial eigenvectors Get the fusion feature map output by the fusion block The process is:

[0105] ;

[0106] ;

[0107] ;

[0108] in, and Represent the spatial information representation diagrams of visible light remote sensing images and SAR images respectively, and The visible light feature map and SAR feature map input by the current fusion block are converted into the same resolution as the visible light feature map and SAR feature map. Represents a spatial upsampling operation (specifically, a convolution layer with a convolution kernel size of 3×3 and a stride of 2 and a activation function), Represents a channel upsampling operation (specifically, a 1×1 convolutional layer that increases the number of channels and a activation function), Represents the sigmoid function activation operation, Represents element-wise multiplication.

[0109] Furthermore, the cross fusion means that the fusion part includes five fusion blocks, the first fusion block, the third fusion block, and the fifth fusion block are dual-attention fusion blocks of the same resolution, and the second fusion block and the fourth fusion block are dual-attention fusion blocks of different resolutions. The first visible light feature map and the first SAR feature map are input into the first fusion block to obtain a first fused feature map, the first visible light feature map and the second SAR feature map are input into the second fusion block to obtain a second fused feature map, the second visible light feature map and the second SAR feature map are input into the third fusion block to obtain a third fused feature map, the second visible light feature map and the third SAR feature map are input into the fourth fusion block to obtain a fourth fused feature map, and the third visible light feature map and the third SAR feature map are input into the fifth fusion block to obtain a fifth fused feature map.

[0110] 3. Image reconstruction part.

[0111] The image reconstruction part is used to reconstruct the multiple fusion feature maps output by the fusion part to the resolution of the original visible light remote sensing image and SAR image, and then splice all the fusion feature maps with the same resolution, and then pass the splicing results through two layers of convolution layers in sequence to obtain a grayscale fusion image. The Tanh activation function is used after each convolution layer, and the convolution kernel size is 3×3 and the step size is 1. Finally, the reconstructed grayscale fusion image is superimposed with the CrCb information separated from the visible light remote sensing image to obtain the fusion result image .

[0112] Specifically, in the five fusion feature maps output by the fusion part in this embodiment, the resolution of the first feature map and the second feature map is consistent with the resolution of the original visible light remote sensing image and the SAR image. , the resolution of the third and fourth feature maps are , reconstruct the resolution to , the resolution of the fifth feature map is , the resolution is reconstructed to .

[0113] like Figure 10 , the first low-resolution reconstruction module It is composed of a basic residual upsampling block, which includes a residual branch and an upsampling branch. The features input to the basic residual upsampling module are added after passing through the residual branch and the upsampling branch to obtain the output features. In this embodiment, the residual branch includes two sequentially connected As the activation function, there is a convolution layer (convolution kernel size is 3×3, stride is 1), a transposed convolution layer with convolution kernel size 3×3 and stride 2, and a convolution layer with convolution kernel size 3×3 and stride 1. The upsampling branch consists of a convolution layer with convolution kernel size 3×3 and stride 1 and a convolution layer with convolution kernel size 3×3 and stride 2 connected in sequence. A transposed convolutional layer with θ as the activation function.

[0114] like Figure 11 , the second low-resolution reconstruction module It consists of two sequentially connected low-resolution reconstruction modules The basic residual upsampling blocks with the same structure.

[0115] 4. Loss function.

[0116] The fusion model is trained using the similarity loss , multi-layer perception loss and brightness distribution loss The overall loss function is expressed as: ,in , Control and The weight of .

[0117] The loss function is based on Figure 12 The pre-training model shown in the figure is constructed, and the pre-training model includes a first convolution unit layer, a first maximum pooling layer, a second convolution unit layer, and a second maximum pooling layer connected in sequence. The first convolution unit layer and the second convolution layer each include two sequentially connected and using Convolutional layer as activation function.

[0118] Specifically, the first convolutional unit layer includes a convolution kernel of 3×3, a stride of 1, 3 input channels, 64 output channels, and a As the activation function, the convolution layer has a convolution kernel of 3×3, a stride of 1, 64 input channels, 64 output channels, and uses The convolution layer as the activation function. The second convolution unit layer includes a convolution kernel of 3×3, a stride of 1, 64 input channels, 128 output channels, and a As the activation function, the convolution layer has a convolution kernel of 3×3, a stride of 1, 128 input channels, 128 output channels, and uses Convolutional layer as activation function.

[0119] The image corresponding to the current training sample Input into the pre-trained model, and get a set of images through the first convolution unit layer. The feature maps corresponding to each channel , a set of images are obtained through the second convolution unit layer The feature maps corresponding to each channel , The sequence number of the channel.

[0120] The grayscale image corresponding to the current training sample Input into the pre-trained model, and get a set of grayscale images through the first convolution unit layer. The feature maps corresponding to each channel As well as grayscale images The overall corresponding feature map , through the second convolution unit layer to obtain a set of grayscale images The feature maps corresponding to each channel As well as grayscale images The overall corresponding feature map .

[0121] The fusion result image obtained by the training sample through the fusion model Input into the pre-trained model and obtain the feature map through the first convolutional unit layer , the feature map is obtained through the second convolution unit layer .

[0122] The training samples are a group of visible light remote sensing images and SAR images that are of the same time and the same area and have completed registration.

[0123] 1. Similarity loss.

[0124] The similarity loss function uses the preprocessing model to extract the features of each modality, and quantifies the degree of retention of important information in visible light and SAR images using entropy and standard deviation calculations.

[0125] First, the standard deviation and entropy are combined to calculate the weight of the constrained light intensity term and :

[0126] ;

[0127] ;

[0128] in, is the sequence number of the convolutional unit layer in the pre-trained model, Indicates the total number of channels output by the current convolutional unit layer, and are two weights used to control the entropy value and standard deviation ratio, and are the calculation formulas for entropy and standard deviation respectively.

[0129] Then calculate the gradient term and :

[0130] ;

[0131] ;

[0132] in, represents the Laplace of Gaussian operator, Represents the L2 norm operation.

[0133] Then calculate the brightness intensity similarity loss and gradient similarity loss :

[0134] ;

[0135] ;

[0136] in, express and The average structural similarity between .

[0137] Finally, calculate the similarity loss : , To control the balance weight of the two losses.

[0138] 2. Multi-layer perception loss.

[0139] First, calculate the multi-layer perception loss of high-resolution visible light remote sensing images :

[0140] ;

[0141] in, is the number of fusion blocks in the fusion part, which is 5 in this embodiment. It is The constraint weights of the fusion blocks, It is The fusion feature map obtained by the fusion blocks, Is input to The visible light characteristics of the fusion block, represents the Laplace of Gaussian operator, Represents the L2 norm operation.

[0142] Then calculate the multi-layer perception loss of the low-resolution SAR image :

[0143] ;

[0144] Finally, calculate the multi-layer perception loss :

[0145] ;

[0146] Indicates balance and The weights of the two loss functions.

[0147] 3. Brightness distribution loss function.

[0148] The Gram matrix is ​​used in the brightness distribution loss function to measure the feature correlation, and then the degree of inconsistency between the fusion result image and the original SAR image in feature distribution is obtained.

[0149] Brightness distribution loss function Calculation method:

[0150] ;

[0151] in, Represents the Gram matrix that measures feature correlation.

[0152] It should be noted that it is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. The scope of the present invention is defined by the claims rather than the foregoing description.

Claims

1. A deep learning-based SAR and visible light remote sensing image fusion method, which inputs a visible light remote sensing image and a SAR image that have been registered at the same time and in the same area into a fusion model to obtain a fused result image, characterized by: The fusion model includes a feature extraction part, a fusion part and an image reconstruction part; The feature extraction part includes a first branch corresponding to the visible light remote sensing image and a second branch corresponding to the SAR image, which are respectively used to obtain visible light feature maps of different resolutions and SAR feature maps of different resolutions; The fusion part includes multiple fusion blocks based on channel attention and spatial attention. The fusion blocks include same-resolution dual-attention fusion blocks and different-resolution dual-attention fusion blocks. The same-resolution dual-attention fusion blocks are used to fuse visible light feature maps and SAR feature maps of the same resolution. The different-resolution dual-attention fusion blocks are used to fuse visible light feature maps and SAR feature maps of different resolutions. The fusion part cross-fuses the visible light feature map and SAR feature map output by the feature extraction part to obtain a fused feature map corresponding to the fusion block one by one. The obtained multiple fused feature maps are distributed at multiple resolution levels. The image reconstruction part is used to reconstruct the multiple fusion feature maps output by the fusion part to the resolution of the original visible light remote sensing image and the SAR image, and then splice all the fusion feature maps with the same resolution, and then obtain the grayscale fusion image by convolution of the splicing result, and finally superimpose the grayscale fusion image with the CrCb information separated from the visible light remote sensing image to obtain the fusion result image. ; The overall loss function used to train the fusion model includes at least similarity loss and brightness distribution loss. The similarity loss uses a preprocessing model to extract features from each modality and quantifies the degree of retention of important information in visible light and SAR images using entropy and standard deviation calculations. The brightness distribution loss uses the Gram matrix to measure feature correlation and then obtains the degree of inconsistency in feature distribution between the fusion result image and the original SAR image.

2. The method for fusion of SAR and visible light remote sensing images based on deep learning according to claim 1, wherein: In the feature extraction part, after the visible light remote sensing image is input into the first branch, it is first converted from RGB format to YCrCb format, and then the Y channel is extracted from the YCrCb format image to obtain the image , and then through convolution and Activation function gets features ;in, express activation function, Represents the convolution operation; after that, the features Input to the first resolution interpretation module Get the first visible light feature map, and Input to the second resolution interpretation module Get the second visible light characteristic map, and transform the characteristic Input to the third resolution interpretation module Obtaining a third visible light characteristic map; After the SAR image is input into the second branch, it is first converted into a grayscale image , and then through convolution and The activation function performs preliminary extraction and expands its channel dimension to obtain features ;in, express activation function, Represents the convolution operation; then the features Input to the first resolution interpretation module Get the first SAR feature map, and Input to the second resolution interpretation module Get the second SAR feature map, and transform the feature Input to the third resolution interpretation module The third SAR characteristic map is obtained.

3. The method for fusion of SAR and visible light remote sensing images based on deep learning according to claim 2, characterized in that: The first resolution interpretation module and the second resolution interpretation module Each consists of a basic residual downsampling module, and a third resolution interpretation module It is composed of two and second resolution interpretation modules The same basic residual downsampling is connected in sequence; The basic residual downsampling module includes a residual branch and a downsampling branch. The features input to the basic residual downsampling module are added together after passing through the residual branch and the downsampling branch to obtain output features.

4. The deep learning-based SAR and visible light remote sensing image fusion method according to any one of claims 1 to 3, characterized in that: The fusion block first generates a channel feature vector and a spatial feature vector based on the input visible light feature map and the SAR feature map, and then obtains the fusion feature map output by the fusion block based on the channel feature vector and the spatial feature vector.

5. The method for fusion of SAR and visible light remote sensing images based on deep learning according to claim 4, wherein: The channel feature vector is generated as follows: Step A-1: ​​If the input visible light feature map and the SAR feature map have the same resolution, directly proceed to step A-2; otherwise, convert the feature map with the lower resolution between the visible light feature map and the SAR feature map to the same resolution as the other feature map; Step A-2: Visible light feature maps with the same resolution and SAR characteristic diagram First, perform element-by-element summation, then pass the summed feature tensor through a global average pooling layer for dimensionality reduction, and then pass it through a downsampling layer Compress to obtain channel feature vector , Indicates the resolution at which the two feature maps are consistent; The spatial feature vector is generated as follows: Step B-1: If the input visible light feature map and the SAR feature map have the same resolution, directly proceed to step B-2; otherwise, convert the feature map with the lower resolution between the visible light feature map and the SAR feature map to the same resolution as the other feature map; Step B-2: Visible light feature maps with the same resolution and SAR characteristic diagram First, splice and then input the spliced ​​feature map into the spatial downsampling layer , and then reduce the spatial dimension through convolution, further compress the important spatial information to obtain the spatial feature tensor ; Based on channel feature vector and spatial eigenvectors Get the fusion feature map output by the fusion block The process is: ； ； ； in, and Represent the spatial information representation diagrams of visible light remote sensing images and SAR images respectively, and The visible light feature map and SAR feature map input by the current fusion block are converted into the same resolution as the visible light feature map and SAR feature map. represents the spatial upsampling operation, Represents the channel upsampling operation, Represents the sigmoid function activation operation, Represents element-wise multiplication.

6. The method for fusion of SAR and visible light remote sensing images based on deep learning according to claim 2, wherein: The cross fusion means that the fusion part includes 5 fusion blocks, the first fusion block, the third fusion block and the fifth fusion block are dual attention fusion blocks with the same resolution, and the second fusion block and the fourth fusion block are dual attention fusion blocks with different resolutions; The first visible light feature map and the first SAR feature map are input into the first fusion block to obtain a first fused feature map, the first visible light feature map and the second SAR feature map are input into the second fusion block to obtain a second fused feature map, the second visible light feature map and the second SAR feature map are input into the third fusion block to obtain a third fused feature map, the second visible light feature map and the third SAR feature map are input into the fourth fusion block to obtain a fourth fused feature map, and the third visible light feature map and the third SAR feature map are input into the fifth fusion block to obtain a fifth fused feature map.

7. The method for fusion of SAR and visible light remote sensing images based on deep learning according to claim 6, characterized in that: Among the five fusion feature maps output by the fusion part, the resolution of the first feature map and the second feature map is consistent with the resolution of the original visible light remote sensing image and SAR image. , the resolution of the third and fourth feature maps are , reconstruct the resolution to , the resolution of the fifth feature map is , the resolution is reconstructed to ; First low-resolution reconstruction module It consists of a basic residual upsampling block, which includes a residual branch and an upsampling branch. The features input to the basic residual upsampling module are added together after passing through the residual branch and the upsampling branch to obtain the output features. Second low-resolution reconstruction module It consists of two sequentially connected low-resolution reconstruction modules The basic residual upsampling blocks with the same structure.

8. The method for fusion of SAR and visible light remote sensing images based on deep learning according to claim 2, wherein: The loss function is constructed based on a pre-trained model; the pre-trained model includes a first convolutional unit layer, a first maximum pooling layer, a second convolutional unit layer, and a second maximum pooling layer connected in sequence; The image corresponding to the current training sample Input into the pre-trained model, and get a set of images through the first convolution unit layer. The feature maps corresponding to each channel , a set of images are obtained through the second convolution unit layer The feature maps corresponding to each channel , is the sequence number of the channel; The grayscale image corresponding to the current training sample Input into the pre-trained model, and get a set of grayscale images through the first convolution unit layer. The feature maps corresponding to each channel As well as grayscale images The overall corresponding feature map , through the second convolution unit layer to obtain a set of grayscale images The feature maps corresponding to each channel As well as grayscale images The overall corresponding feature map ; The fusion result image obtained by the training sample through the fusion model Input into the pre-trained model and obtain the feature map through the first convolutional unit layer , the feature map is obtained through the second convolution unit layer .

9. The method for fusion of SAR and visible light remote sensing images based on deep learning according to claim 8, characterized in that: The fusion model is trained using the similarity loss , multi-layer perception loss and brightness distribution loss The overall loss function is: ,in , Control and The weight of .

10. The method for fusion of SAR and visible light remote sensing images based on deep learning according to claim 9, characterized in that:

1. The calculation method of similarity loss is: First, the standard deviation and entropy are combined to calculate the weight of the constrained light intensity term and : ； ； in, is the sequence number of the convolutional unit layer in the pre-trained model, Indicates the total number of channels output by the current convolutional unit layer, and are two weights used to control the entropy value and standard deviation ratio, and are the calculation formulas for entropy and standard deviation respectively; Then calculate the gradient term and : ； ； in, represents the Laplace of Gaussian operator, Represents the L2 norm operation; Then calculate the brightness intensity similarity loss and gradient similarity loss : ； ； in, express and The average structural similarity between Finally, calculate the similarity loss : , To control the balance weight of the two losses; 2. The calculation method of multi-layer perception loss is: First, calculate the multi-layer perception loss of high-resolution visible light remote sensing images : ； in, is the number of fusion blocks in the fusion part, It is The constraint weights of the fusion blocks, It is The fusion feature map obtained by the fusion blocks, Is input to The visible light characteristics of the fusion block, represents the Laplace of Gaussian operator, Represents the L2 norm operation; Then calculate the multi-layer perception loss of the low-resolution SAR image : ； Finally, calculate the multi-layer perception loss : ； Indicates balance and The weights of the two loss functions; 3. The calculation method of brightness distribution loss function is: ； in, Represents the Gram matrix that measures feature correlation.

Images