SAR and visible light remote sensing image fusion method based on deep learning

Abstract

The invention discloses a deep learning-based SAR (Synthetic Aperture Radar) and visible light remote sensing image fusion method. An adopted fusion model comprises a feature extraction part, a fusion part and an image reconstruction part. An overall loss function used for training the fusion model at least comprises similarity loss and brightness distribution loss; for similarity loss, each modal feature is extracted by using a preprocessing model, and the retention degree of important information in visible light and SAR images is quantified by using entropy and standard deviation calculation; according to the brightness distribution loss, the feature correlation is measured by using a Gram matrix, and then the inconsistency degree of the fusion result image and the original SAR image in feature distribution is obtained. According to the method, complementation of different resolution characteristics is realized, effective combination of reflection intensity information in an SAR image and detail information in a visible light image is ensured, SAR information in a fusion result can be highlighted, key characteristics are reserved, and visual quality and information integrity are improved.

Description

Technical Field

[0001] The invention belongs to the field of image data processing, and in particular relates to a method for fusing a SAR image with a visible light remote sensing image. Background Art

[0002] The fusion of synthetic aperture radar (SAR) images and visible light images is one of the key technologies in the field of remote sensing image processing and analysis. The core of this technology is to combine the characteristics of the two data types to achieve complementary advantages and provide more comprehensive and accurate information. This fusion technology plays a vital role in many remote sensing application scenarios, such as land use change monitoring, disaster assessment, urban planning, agricultural monitoring, etc. By integrating different types of images, the morphology, structure and dynamic change information of ground targets can be more comprehensively obtained, thereby improving the accuracy and reliability of information extraction.

[0003] In recent years, deep learning technology has made breakthrough progress in the field of image fusion. Compared with traditional methods that rely on manually designed features and rules, deep learning can automatically learn data features and achieve smarter image fusion, thereby improving image quality and information extraction capabilities. Existing image fusion methods often focus on maintaining pixel brightness and detail information through loss functions, or adding residual connections and dense connections in the network architecture to capture texture details. However, existing methods still have the following defects: 1. The existing deep learning method is based on the dual attention fusion of original scale features, which ignores the complementarity of features at different resolution levels, resulting in the inability to effectively combine the reflection intensity information in SAR images with the detail information in visible light images.

[0004] 2. In target detection or night recognition tasks, it is necessary to rely on the SAR modal information in the fusion results to improve the detection or recognition accuracy. However, the existing deep learning method only adjusts the proportion of the SAR modality, which is not precise enough and cannot fully retain and utilize the important information of the SAR modality.

[0005] 3. The existing methods do not determine the degree of retention of important information based on the properties of each modality, and cannot effectively integrate complementary information, resulting in insufficient retention of key features in the fusion results and less than ideal information integrity. Summary of the invention

[0006] The present invention proposes a SAR and visible light remote sensing image fusion method based on deep learning, the purposes of which are: 1. To solve the problem that the reflection intensity information in the SAR image and the detail information in the visible light image cannot be effectively combined; 2. To solve the problem that the important information of the SAR modality is not sufficiently retained; 3. To solve the problem of insufficient information integrity of the fusion result.

[0007] The technical solution of the present invention is as follows: A SAR and visible light remote sensing image fusion method based on deep learning, inputting a visible light remote sensing image and a SAR image registered at the same time and in the same area into a fusion model to obtain a fusion result image, wherein the fusion model includes a feature extraction part, a fusion part and an image reconstruction part; The feature extraction part includes a first branch corresponding to the visible light remote sensing image and a second branch corresponding to the SAR image, which are respectively used to obtain visible light feature maps with different resolutions and SAR feature maps with different resolutions; The fusion part includes a plurality of fusion blocks based on channel attention and spatial attention, and the fusion blocks include a same-resolution dual-attention fusion block and a different-resolution dual-attention fusion block. The same-resolution dual-attention fusion block is used to fuse the visible light feature map and the SAR feature map of the same resolution, and the different-resolution dual-attention fusion block is used to fuse the visible light feature map and the SAR feature map of different resolutions. The fusion part cross-fuses the visible light feature map and the SAR feature map output by the feature extraction part to obtain a fusion feature map corresponding to the fusion block one by one, and the obtained multiple fusion feature maps are distributed at multiple resolution levels; The image reconstruction part is used to reconstruct the multiple fusion feature maps output by the fusion part to the resolution of the original visible light remote sensing image and the SAR image, and then splice all the fusion feature maps with the same resolution, and then obtain the grayscale fusion image by convolution of the splicing result, and finally superimpose the grayscale fusion image with the CrCb information separated from the visible light remote sensing image to obtain the fusion result image ; The overall loss function used to train the fusion model includes at least similarity loss and brightness distribution loss; the similarity loss uses a preprocessing model to extract features of each modality, and quantifies the degree of retention of important information in visible light and SAR images using entropy and standard deviation calculations; the brightness distribution loss uses a Gram matrix to measure feature correlation, and then obtains the degree of inconsistency between the fusion result image and the original SAR image in feature distribution.

[0008] As a further improvement of the deep learning-based SAR and visible light remote sensing image fusion method: in the feature extraction part, after the visible light remote sensing image is input into the first branch, it is first converted from RGB format to YCrCb format, and then the Y channel is extracted from the YCrCb format image to obtain the image , and then through convolution and Activation function gets features ;in, express Activation function, represents the convolution operation; then, the feature Input to the first resolution interpretation module Get the first visible light feature map, and convert the feature Input to the second resolution interpretation module Get the second visible light characteristic map, and transform the characteristic Input to the third resolution interpretation module Obtaining a third visible light characteristic map; After the SAR image is input into the second branch, it is first converted into a grayscale image , and then through convolution and The activation function performs preliminary extraction and expands its channel dimension to obtain features ;in, express Activation function, Represents the convolution operation; then the feature Input to the first resolution interpretation module Get the first SAR feature map and convert the feature Input to the second resolution interpretation module Get the second SAR feature map and convert the feature Input to the third resolution interpretation module The third SAR characteristic map is obtained.

[0009] As a further improvement of the deep learning-based SAR and visible light remote sensing image fusion method: the first resolution interpretation module and the second resolution interpretation module Each consists of a basic residual downsampling module and a third resolution interpretation module. It consists of two modules with the second resolution interpretation The same basic residual downsampling is connected in sequence; The basic residual downsampling module includes a residual branch and a downsampling branch. The features input to the basic residual downsampling module are added after passing through the residual branch and the downsampling branch respectively to obtain output features.

[0010] As a further improvement of the deep learning-based SAR and visible light remote sensing image fusion method: the fusion block first generates a channel feature vector and a spatial feature vector based on the input visible light feature map and the SAR feature map, and then obtains the fusion feature map output by the fusion block based on the channel feature vector and the spatial feature vector.

[0011] As a further improvement of the deep learning-based SAR and visible light remote sensing image fusion method, the channel feature vector is generated as follows: Step A-1: ​​If the input visible light feature map and the SAR feature map have the same resolution, directly execute step A-2; otherwise, convert the feature map with lower resolution between the visible light feature map and the SAR feature map to have the same resolution as the other feature map; Step A-2: Visible light feature maps with the same resolution And SAR characteristic diagram First, the element-by-element summation is performed, and then the feature tensor obtained by the summation is reduced in dimension through a global average pooling layer, and then through a downsampling layer Compress to get the channel feature vector , Indicates the resolution at which two feature maps are consistent; The spatial feature vector is generated as follows: Step B-1: if the input visible light feature map and the SAR feature map have the same resolution, directly execute step B-2; otherwise, convert the feature map with lower resolution between the visible light feature map and the SAR feature map to have the same resolution as the other feature map; Step B-2: Visible light feature maps with the same resolution And SAR characteristic diagram First, splice, and then input the spliced ​​feature map into the spatial downsampling layer , and then use convolution to reduce the spatial dimension and further compress the important spatial information to obtain the spatial feature tensor ; Based on channel feature vector and spatial eigenvectors Get the fusion feature map output by the fusion block The process is: ; ; ; in, and Represent the spatial information representation diagrams of visible light remote sensing images and SAR images respectively, and They are the visible light feature map and SAR feature map input by the current fusion block, which are obtained by resolution conversion and have the same resolution as the visible light feature map and SAR feature map. represents the spatial upsampling operation, represents the channel upsampling operation, Represents the sigmoid function activation operation, Represents element-wise multiplication.

[0012] As a further improvement of the SAR and visible light remote sensing image fusion method based on deep learning, the cross fusion means that: the fusion part includes 5 fusion blocks, the first fusion block, the third fusion block and the fifth fusion block are double attention fusion blocks with the same resolution, and the second fusion block and the fourth fusion block are double attention fusion blocks with different resolutions; The first visible light feature map and the first SAR feature map are input to the first fusion block to obtain a first fused feature map, the first visible light feature map and the second SAR feature map are input to the second fusion block to obtain a second fused feature map, the second visible light feature map and the second SAR feature map are input to the third fusion block to obtain a third fused feature map, the second visible light feature map and the third SAR feature map are input to the fourth fusion block to obtain a fourth fused feature map, and the third visible light feature map and the third SAR feature map are input to the fifth fusion block to obtain a fifth fused feature map.

[0013] As a further improvement of the deep learning-based SAR and visible light remote sensing image fusion method: in the five fusion feature maps output by the fusion part, the resolution of the first feature map and the second feature map is consistent with the resolution of the original visible light remote sensing image and the SAR image, and both are , the resolution of the third and fourth feature maps are , reconstruct the resolution to , the resolution of the fifth feature map is , the resolution is reconstructed to ; The first low-resolution reconstruction module The basic residual upsampling block includes a residual branch and an upsampling branch. The features input to the basic residual upsampling module are added after passing through the residual branch and the upsampling branch to obtain the output features. The second low-resolution reconstruction module The first low-resolution reconstruction module consists of two sequentially connected The basic residual upsampling blocks with the same structure.

[0014] As a further improvement of the SAR and visible light remote sensing image fusion method based on deep learning: the loss function is constructed based on a pre-trained model; the pre-trained model includes a first convolution unit layer, a first maximum pooling layer, a second convolution unit layer and a second maximum pooling layer connected in sequence; The image corresponding to the current training sample Input into the pre-trained model, and obtain a set of images through the first convolution unit layer The feature maps corresponding to each channel , through the second convolutional unit layer to obtain a set of images The feature maps corresponding to each channel , is the sequence number of the channel; The grayscale image corresponding to the current training sample Input into the pre-trained model, and obtain a set of grayscale images through the first convolution unit layer The feature maps corresponding to each channel As well as grayscale images The overall corresponding feature map , through the second convolution unit layer to obtain a set of grayscale images The feature maps corresponding to each channel As well as grayscale images The overall corresponding feature map ; The fusion result image obtained by the training sample through the fusion model Input into the pre-trained model and obtain the feature map through the first convolutional unit layer , the feature map is obtained through the second convolutional unit layer .

[0015] As a further improvement of the deep learning-based SAR and visible light remote sensing image fusion method: the fusion model is trained using a similarity loss , multi-layer perceptual loss and brightness distribution loss The overall loss function is: ,in , Control and The weight of .

[0016] As a further improvement of the deep learning-based SAR and visible light remote sensing image fusion method: 1. The calculation method of similarity loss is: First, the standard deviation and entropy value are combined to calculate the weight of the constrained light intensity term and : ; ; in, is the serial number of the convolutional unit layer in the pre-trained model, Indicates the total number of channels output by the current convolutional unit layer, and are two weights used to control the entropy value and standard deviation ratio. and They are the calculation formulas for entropy value and standard deviation respectively; Recalculate the gradient term and : ; ; in, represents the Laplacian of Gaussian, represents the L2 norm operation; Then calculate the brightness intensity similarity loss and gradient similarity loss : ; ; in, express and The average structural similarity between Finally, calculate the similarity loss : , To control the balance weight of the two losses; 2. The calculation method of multi-layer perception loss is: First, the multi-layer perceptual loss of high-resolution visible light remote sensing images is calculated : ; in, is the number of fusion blocks in the fusion part, It is The constraint weights of the fusion blocks, It is The fusion feature map obtained by the fusion blocks is Is input to The visible light characteristics of the fusion blocks, represents the Laplacian of Gaussian, represents the L2 norm operation; Then the multi-layer perceptual loss of the low-resolution SAR image is calculated : ; Finally, calculate the multi-layer perceptual loss : ; Indicates balance and The weights of the two loss functions; 3. The calculation method of brightness distribution loss function is: ; in, Represents the Gram matrix measuring feature correlations.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention first generates feature maps with different resolutions, then performs cross-fusion based on dual attention on two feature maps with different resolutions, and finally unifies the resolution and completes image reconstruction, thereby achieving the complementarity of features with different resolutions and ensuring the effective combination of the reflection intensity information in the SAR image and the detail information in the visible light image.

[0018] 2. The Gram matrix is ​​used in the brightness distribution loss function of the present invention to measure feature correlation, and then the inconsistency between the fusion result image and the original SAR image in feature distribution is obtained. The model trained based on the loss function can make the feature distribution of the fusion result image as close as possible to the feature distribution of the original SAR image, thereby accurately highlighting the SAR information in the fusion result.

[0019] 3. The similarity loss function of the present invention uses a preprocessing model to extract the features of each modality, and uses entropy and standard deviation calculations to quantify the degree of retention of important information in visible light and SAR images, which not only ensures that the fusion result can retain key features, but also improves the visual quality and information integrity.

[0020] 4. The dual-path framework of the present invention has a lighter structure and higher training efficiency while ensuring the fusion quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the framework of the fusion model of the present invention; Figure 2 is a schematic diagram of a first resolution interpretation module; Figure 3 is a schematic diagram of a second resolution interpretation module; Figure 4 is a schematic diagram of a third resolution interpretation module; Figure 5 Schematic diagram of the process of generating channel feature vectors based on feature maps of different resolutions; Figure 6 Schematic diagram of the process of generating channel feature vectors based on feature maps of the same resolution; Figure 7 Schematic diagram of the process of generating spatial feature vectors based on feature maps of different resolutions; Figure 8 Schematic diagram of the process of generating spatial feature vectors based on feature maps of the same resolution; Fig. 9 Schematic diagram of the dual attention fusion process; Fig.10 is a schematic diagram of a first low-resolution reconstruction module; Fig.11 is a schematic diagram of a second low-resolution reconstruction module; Fig.12 Schematic diagram of the pre-trained feature extraction model. DETAILED DESCRIPTION

[0022] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0023] A SAR and visible light remote sensing image fusion method based on deep learning is to input the visible light remote sensing image and SAR image that have been registered in the same area at the same time into the Figure 1 In the fusion model shown, a fusion result image is obtained.

[0024] The fusion model includes a feature extraction part, a fusion part and an image reconstruction part.

[0025] 1. Feature extraction part.

[0026] The feature extraction part includes a first branch corresponding to the visible light remote sensing image and a second branch corresponding to the SAR image, which are respectively used to obtain visible light feature maps with different resolutions and SAR feature maps with different resolutions.

[0027] After the visible light remote sensing image is input into the first branch, it is first converted from RGB format to YCrCb format, and then the Y channel is extracted from the YCrCb format image to obtain the image , and then input to a layer of convolution kernel size 3×3, stride 1×1, and use The convolution layer as an activation function performs preliminary feature extraction and expands its channel dimension to obtain the feature .in, express Activation function, represents the convolution operation. After that, the feature Input to the first resolution interpretation module Get the first visible light feature map, and convert the feature Input to the second resolution interpretation module Get the second visible light characteristic map, and transform the characteristic Input to the third resolution interpretation module A third visible light characteristic graph is obtained.

[0028] After the SAR image is input into the second branch, it is first converted into a grayscale image , and then input to a layer of convolution kernel size 3×3, stride 1, and use The convolution layer as an activation function performs preliminary feature extraction and expands its channel dimension to obtain the feature .in, express Activation function, represents the convolution operation. After that, the feature Input to the first resolution interpretation module Get the first SAR feature map and convert the feature Input to the second resolution interpretation module Get the second SAR feature map and convert the feature Input to the third resolution interpretation module The third SAR characteristic map is obtained.

[0029] First resolution interpretation module , Second resolution interpretation module and third resolution interpretation module The resolution of the output feature maps decreases successively.

[0030] Specifically, Figures 2 to 4 , the first resolution interpretation module and the second resolution interpretation module Each consists of a basic residual downsampling module and a third resolution interpretation module. It consists of two modules with the second resolution interpretation The same basic residual downsampling is constructed by connecting them sequentially.

[0031] The basic residual downsampling module includes a residual branch and a downsampling branch. The features input to the basic residual downsampling module are added after passing through the residual branch and the downsampling branch to obtain the output features. As the activation function of the convolution layer (convolution kernel size is 3×3, stride is 1), a convolution kernel size is 3×3, using The convolutional layer as an activation function ( Here the convolutional layer stride is 1. Here, the convolution layer has a stride of 2) and a convolution layer with a kernel size of 3×3 and a stride of 1. The downsampling branch is used to reduce the spatial dimension of the feature map, including a convolution layer with a kernel size of 3×3 and a stride of 1 and a convolution layer with a kernel size of 3×3 and a stride of 1. The convolutional layer as an activation function ( Here the convolutional layer stride is 1. Here the convolutional layer step size is 2). Finally, the resolution of the first visible light feature map and the first SAR feature map is , the resolution of the second visible light feature map and the second SAR feature map is , the resolution of the third visible light feature map and the third SAR feature map is , and are the height and width of the original visible light remote sensing image and SAR image.

[0032] 2. Integration part.

[0033] like Figure 1 The fusion part includes multiple fusion blocks based on channel attention and spatial attention. The fusion blocks include same-resolution double-attention fusion blocks and different-resolution double-attention fusion blocks. The same-resolution double-attention fusion blocks are used to fuse visible light feature maps and SAR feature maps of the same resolution. The different-resolution double-attention fusion blocks are used to fuse visible light feature maps and SAR feature maps of different resolutions. The fusion part cross-fuses the visible light feature map and SAR feature map output by the feature extraction part to obtain a fused feature map corresponding to the fusion block one by one. The obtained multiple fused feature maps are distributed on multiple resolution levels.

[0034] Specifically, the fusion block first generates a channel feature vector and a spatial feature vector based on the input visible light feature map and the SAR feature map, and then obtains the fusion feature map output by the fusion block based on the channel feature vector and the spatial feature vector.

[0035] like Figure 5 and 6 , the channel feature vector is generated as follows: Step A-1: ​​If the input visible light feature map and the SAR feature map have the same resolution, then directly execute step A-2; otherwise, the feature map with the lower resolution among the visible light feature map and the SAR feature map is converted to have the same resolution as the other feature map. The conversion method in this embodiment is to sequentially input the low-resolution feature map into a convolution kernel with a size of 3×3, a step size of 2, and a convolution kernel with a step size of 1. In the transposed convolutional layer as an activation function. represents the higher resolution of the two feature maps, represents the lower resolution of the two feature maps.

[0036] Step A-2: Visible light feature maps with the same resolution And SAR characteristic diagram First, the element-by-element summation is performed, and then the feature tensor obtained by the summation is reduced in dimension through a global average pooling layer, and then through a downsampling layer Compress to get the channel feature vector .

[0037] Therefore, there are: ,in represents the channel downsampling operation, represents the global pooling operation, represents element-by-element addition. The downsampling layer Contains a 1×1 convolutional layer to reduce the number of channels and a Activation function, Indicates the resolution at which two feature maps are consistent.

[0038] like Figure 7 and 8 , the spatial feature vector is generated as follows: Step B-1: If the input visible light feature map and the SAR feature map have the same resolution, then directly execute step B-2; otherwise, the feature map with the lower resolution among the visible light feature map and the SAR feature map is converted to have the same resolution as the other feature map. The conversion method in this embodiment is to sequentially input the low-resolution feature map into a convolution kernel with a size of 3×3, a step size of 2, and a convolution kernel with a step size of 1. In a transposed convolutional layer as an activation function.

[0039] Step B-2: Visible light feature maps with the same resolution And SAR characteristic diagram First, splice, and then input the spliced ​​feature map into the spatial downsampling layer Then, a convolution layer with a convolution kernel size of 3×3 and a step size of 1 is used to reduce the spatial dimension and further compress the important spatial information to obtain the spatial feature tensor. .

[0040] Therefore, there are: ,in, Represents a splicing operation, represents the spatial downsampling operation, Represents the above convolution operation. The spatial downsampling layer Contains two uses connected in sequence Convolutional layers as activation functions (the first convolutional layer has a kernel size of 1×1 and a stride of 1, the second convolutional layer has a kernel size of 3×3 and a stride of 2).

[0041] like Fig. 9 , based on the channel feature vector and spatial eigenvectors Get the fusion feature map output by the fusion block The process is: ; ; ; in, and Represent the spatial information representation diagrams of visible light remote sensing images and SAR images respectively, and They are the visible light feature map and SAR feature map input by the current fusion block, which are obtained by resolution conversion and have the same resolution as the visible light feature map and SAR feature map. Represents a spatial upsampling operation (specifically, a convolution layer with a convolution kernel size of 3×3 and a stride of 2 and a activation function), Represents a channel upsampling operation (specifically including a 1×1 convolutional layer that increases the number of channels and a activation function), Represents the sigmoid function activation operation, Represents element-wise multiplication.

[0042] Further, the cross fusion means that: the fusion part includes 5 fusion blocks, the first fusion block, the third fusion block and the fifth fusion block are dual attention fusion blocks with the same resolution, and the second fusion block and the fourth fusion block are dual attention fusion blocks with different resolutions. The first visible light feature map and the first SAR feature map are input to the first fusion block to obtain the first fusion feature map, the first visible light feature map and the second SAR feature map are input to the second fusion block to obtain the second fusion feature map, the second visible light feature map and the second SAR feature map are input to the third fusion block to obtain the third fusion feature map, the second visible light feature map and the third SAR feature map are input to the fourth fusion block to obtain the fourth fusion feature map, and the third visible light feature map and the third SAR feature map are input to the fifth fusion block to obtain the fifth fusion feature map.

[0043] 3. Image reconstruction part.

[0044] The image reconstruction part is used to reconstruct the multiple fusion feature maps output by the fusion part to the resolution of the original visible light remote sensing image and the SAR image, and then splice all the fusion feature maps with the same resolution, and then pass the splicing results through two layers of convolution layers in sequence to obtain a grayscale fusion image. The Tanh activation function is used after each layer of convolution, and the convolution kernel size is 3×3 and the step size is 1. Finally, the reconstructed grayscale fusion image is superimposed with the CrCb information separated from the visible light remote sensing image to obtain the fusion result image .

[0045] Specifically, among the five fusion feature maps output by the fusion part in this embodiment, the resolution of the first feature map and the second feature map is consistent with the resolution of the original visible light remote sensing image and the SAR image, and both are , the resolution of the third and fourth feature maps are , reconstruct the resolution to , the resolution of the fifth feature map is , the resolution is reconstructed to .

[0046] like Fig.10 , the first low-resolution reconstruction module The basic residual upsampling block includes a residual branch and an upsampling branch. The features input to the basic residual upsampling module are added after passing through the residual branch and the upsampling branch to obtain the output features. In this embodiment, the residual branch includes two sequentially connected As the activation function, there is a convolutional layer (convolution kernel size is 3×3, stride is 1), a transposed convolutional layer with a convolution kernel size of 3×3 and stride is 2, and a convolutional layer with a convolution kernel size of 3×3 and stride is 1. The upsampling branch consists of a convolutional layer with a convolution kernel size of 3×3 and stride of 1 and a convolution kernel size of 3×3 and stride of 2 connected in sequence. A transposed convolutional layer with θ as the activation function.

[0047] like Fig.11 , the second low-resolution reconstruction module The first low-resolution reconstruction module consists of two sequentially connected The basic residual upsampling blocks with the same structure.

[0048] 4. Loss function.

[0049] The fusion model is trained using the similarity loss , multi-layer perceptual loss and brightness distribution loss The overall loss function is expressed as: ,in , Control and The weight of .

[0050] The loss function is based on Fig.12 The pre-training model shown in the figure is constructed, and the pre-training model includes a first convolution unit layer, a first maximum pooling layer, a second convolution unit layer, and a second maximum pooling layer connected in sequence. The first convolution unit layer and the second convolution layer each include two sequentially connected and using Convolutional layer as activation function.

[0051] Specifically, the first convolution unit layer includes a convolution kernel of 3×3, a step size of 1, 3 input channels, 64 output channels, and a convolution kernel of 3×3, a step size of 1, and a step size of 64. As an activation function, the convolution layer has a convolution kernel of 3×3, a stride of 1, 64 input channels, 64 output channels, and uses The second convolution unit layer includes a convolution kernel of 3×3, a stride of 1, 64 input channels, 128 output channels, and a As the activation function, the convolution layer has a convolution kernel of 3×3, a stride of 1, 128 input channels, 128 output channels, and uses Convolutional layer as activation function.

[0052] The image corresponding to the current training sample Input into the pre-trained model, and obtain a set of images through the first convolution unit layer The feature maps corresponding to each channel , through the second convolutional unit layer to obtain a set of images The feature maps corresponding to each channel , The sequence number of the channel.

[0053] The grayscale image corresponding to the current training sample Input into the pre-trained model, and obtain a set of grayscale images through the first convolution unit layer The feature maps corresponding to each channel As well as grayscale images The overall corresponding feature map , through the second convolution unit layer to obtain a set of grayscale images The feature maps corresponding to each channel As well as grayscale images The overall corresponding feature map .

[0054] The fusion result image obtained by the training sample through the fusion model Input into the pre-trained model and obtain the feature map through the first convolutional unit layer , the feature map is obtained through the second convolutional unit layer .

[0055] The training samples are a group of visible light remote sensing images and SAR images of the same time and the same area and completed with registration.

[0056] 1. Similarity loss.

[0057] The similarity loss function uses the preprocessing model to extract the features of each modality, and quantifies the degree of retention of important information in visible light and SAR images using entropy and standard deviation calculations.

[0058] First, the standard deviation and entropy value are combined to calculate the weight of the constrained light intensity term and : ; ; in, is the serial number of the convolutional unit layer in the pre-trained model, Indicates the total number of channels output by the current convolutional unit layer, and are two weights used to control the entropy value and standard deviation ratio. and are the calculation formulas for entropy value and standard deviation respectively.

[0059] Recalculate the gradient term and : ; ; in, represents the Laplacian of Gaussian, Represents the L2 norm operation.

[0060] Then calculate the brightness intensity similarity loss and gradient similarity loss : ; ; in, express and The average structural similarity between .

[0061] Finally, calculate the similarity loss : , To control the balance weight of the two losses.

[0062] 2. Multi-layer perception loss.

[0063] First, the multi-layer perceptual loss of high-resolution visible light remote sensing images is calculated : ; in, is the number of fusion blocks in the fusion part, which is 5 in this embodiment. It is The constraint weights of the fusion blocks, It is The fusion feature map obtained by the fusion blocks is Is input to The visible light characteristics of the fusion blocks, represents the Laplacian of Gaussian, Represents the L2 norm operation.

[0064] Then the multi-layer perceptual loss of the low-resolution SAR image is calculated : ; Finally, calculate the multi-layer perceptual loss : ; Indicates balance and Weights of the two loss functions.

[0065] 3. Brightness distribution loss function.

[0066] The Gram matrix is ​​used in the brightness distribution loss function to measure the feature correlation, and then the inconsistency between the fusion result image and the original SAR image in feature distribution is obtained.

[0067] Brightness distribution loss function Calculation method: ; in, Represents the Gram matrix measuring feature correlations.

[0068] It should be noted that it is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. The scope of the present invention is defined by the claims rather than the above description.

Claims

1. A SAR and visible light remote sensing image fusion method based on deep learning, inputting a visible light remote sensing image and a SAR image registered at the same time and in the same area into a fusion model to obtain a fusion result image, characterized in that: The fusion model includes a feature extraction part, a fusion part and an image reconstruction part; The feature extraction part includes a first branch corresponding to the visible light remote sensing image and a second branch corresponding to the SAR image, which are respectively used to obtain visible light feature maps with different resolutions and SAR feature maps with different resolutions; The fusion part includes a plurality of fusion blocks based on channel attention and spatial attention, and the fusion blocks include a same-resolution dual-attention fusion block and a different-resolution dual-attention fusion block. The same-resolution dual-attention fusion block is used to fuse the visible light feature map and the SAR feature map of the same resolution, and the different-resolution dual-attention fusion block is used to fuse the visible light feature map and the SAR feature map of different resolutions. The fusion part cross-fuses the visible light feature map and the SAR feature map output by the feature extraction part to obtain a fusion feature map corresponding to the fusion block one by one, and the obtained multiple fusion feature maps are distributed at multiple resolution levels; The image reconstruction part is used to reconstruct the multiple fusion feature maps output by the fusion part to the resolution of the original visible light remote sensing image and the SAR image, and then splice all the fusion feature maps with the same resolution, and then obtain the grayscale fusion image by convolution of the splicing result, and finally superimpose the grayscale fusion image with the CrCb information separated from the visible light remote sensing image to obtain the fusion result image ; The overall loss function used to train the fusion model includes at least similarity loss and brightness distribution loss; the similarity loss uses a preprocessing model to extract features of each modality, and quantifies the degree of retention of important information in visible light and SAR images using entropy and standard deviation calculations; the brightness distribution loss uses a Gram matrix to measure feature correlation, and then obtains the degree of inconsistency between the fusion result image and the original SAR image in feature distribution.

2. The SAR and visible light remote sensing image fusion method based on deep learning as claimed in claim 1, characterized in that: In the feature extraction part, after the visible light remote sensing image is input into the first branch, it is first converted from RGB format to YCrCb format, and then the Y channel is extracted from the YCrCb format image to obtain the image , and then through convolution and Activation function gets features ;in, express Activation function, represents the convolution operation; then, the feature Input to the first resolution interpretation module Get the first visible light feature map, and convert the feature Input to the second resolution interpretation module Get the second visible light characteristic map, and convert the characteristic Input to the third resolution interpretation module Obtaining a third visible light characteristic map; After the SAR image is input into the second branch, it is first converted into a grayscale image , and then through convolution and The activation function performs preliminary extraction and expands its channel dimension to obtain features ;in, express Activation function, Represents the convolution operation; then the feature Input to the first resolution interpretation module Get the first SAR feature map and convert the feature Input to the second resolution interpretation module Get the second SAR feature map and convert the feature Input to the third resolution interpretation module The third SAR characteristic map is obtained.

3. The SAR and visible light remote sensing image fusion method based on deep learning as claimed in claim 2, characterized in that: The first resolution interpretation module and the second resolution interpretation module Each consists of a basic residual downsampling module and a third resolution interpretation module. It consists of two modules with the second resolution interpretation The same basic residual downsampling is connected in sequence; The basic residual downsampling module includes a residual branch and a downsampling branch. The features input to the basic residual downsampling module are added after passing through the residual branch and the downsampling branch respectively to obtain output features.

4. The SAR and visible light remote sensing image fusion method based on deep learning according to any one of claims 1 to 3, characterized in that: The fusion block first generates a channel feature vector and a spatial feature vector based on the input visible light feature map and the SAR feature map, and then obtains the fused feature map output by the fusion block based on the channel feature vector and the spatial feature vector.

5. The SAR and visible light remote sensing image fusion method based on deep learning as claimed in claim 4, characterized in that: The channel feature vector is generated as follows: Step A-1: ​​If the input visible light feature map and the SAR feature map have the same resolution, directly execute step A-2; otherwise, convert the feature map with lower resolution between the visible light feature map and the SAR feature map to have the same resolution as the other feature map; Step A-2: Visible light feature maps with the same resolution And SAR characteristic diagram First, the element-by-element summation is performed, and then the feature tensor obtained by the summation is reduced in dimension through a global average pooling layer, and then through a downsampling layer Compress to get the channel feature vector , Indicates the resolution at which two feature maps are consistent; The spatial feature vector is generated as follows: Step B-1: if the input visible light feature map and the SAR feature map have the same resolution, directly execute step B-2; otherwise, convert the feature map with lower resolution between the visible light feature map and the SAR feature map to have the same resolution as the other feature map; Step B-2: Visible light feature maps with the same resolution And SAR characteristic diagram First, splice, and then input the spliced ​​feature map into the spatial downsampling layer , and then use convolution to reduce the spatial dimension and further compress the important spatial information to obtain the spatial feature tensor ; Based on channel feature vector and spatial eigenvectors Get the fusion feature map output by the fusion block The process is: ； ； ； in, and Represent the spatial information representation diagrams of visible light remote sensing images and SAR images respectively, and They are the visible light feature map and SAR feature map input by the current fusion block, which are obtained by resolution conversion and have the same resolution as the visible light feature map and SAR feature map. represents the spatial upsampling operation, represents the channel upsampling operation, Represents the sigmoid function activation operation, Represents element-wise multiplication.

6. The SAR and visible light remote sensing image fusion method based on deep learning as claimed in claim 2, characterized in that: The cross fusion means that: the fusion part includes 5 fusion blocks, the first fusion block, the third fusion block and the fifth fusion block are double attention fusion blocks with the same resolution, and the second fusion block and the fourth fusion block are double attention fusion blocks with different resolutions; The first visible light feature map and the first SAR feature map are input to the first fusion block to obtain a first fused feature map, the first visible light feature map and the second SAR feature map are input to the second fusion block to obtain a second fused feature map, the second visible light feature map and the second SAR feature map are input to the third fusion block to obtain a third fused feature map, the second visible light feature map and the third SAR feature map are input to the fourth fusion block to obtain a fourth fused feature map, and the third visible light feature map and the third SAR feature map are input to the fifth fusion block to obtain a fifth fused feature map.

7. The SAR and visible light remote sensing image fusion method based on deep learning as claimed in claim 6, characterized in that: Among the five fusion feature maps output by the fusion part, the resolution of the first feature map and the second feature map is consistent with the resolution of the original visible light remote sensing image and SAR image. , the resolution of the third and fourth feature maps are , reconstruct the resolution to , the resolution of the fifth feature map is , the resolution is reconstructed to ; The first low-resolution reconstruction module It is composed of a basic residual upsampling block, which includes a residual branch and an upsampling branch. The features input to the basic residual upsampling module are added after passing through the residual branch and the upsampling branch to obtain output features. Second low-resolution reconstruction module The first low-resolution reconstruction module consists of two sequentially connected The basic residual upsampling blocks with the same structure.

8. The SAR and visible light remote sensing image fusion method based on deep learning as claimed in claim 2, characterized in that: The loss function is constructed based on a pre-trained model; the pre-trained model includes a first convolutional unit layer, a first maximum pooling layer, a second convolutional unit layer, and a second maximum pooling layer connected in sequence; The image corresponding to the current training sample Input into the pre-trained model, and obtain a set of images through the first convolution unit layer The feature maps corresponding to each channel , through the second convolutional unit layer to obtain a set of images The feature maps corresponding to each channel , is the sequence number of the channel; The grayscale image corresponding to the current training sample Input into the pre-trained model, and obtain a set of grayscale images through the first convolution unit layer The feature maps corresponding to each channel As well as grayscale images The overall corresponding feature map , through the second convolution unit layer to obtain a set of grayscale images The feature maps corresponding to each channel As well as grayscale images The overall corresponding feature map ; The fusion result image obtained by the training sample through the fusion model Input into the pre-trained model and obtain the feature map through the first convolution unit layer , the feature map is obtained through the second convolutional unit layer .

9. The SAR and visible light remote sensing image fusion method based on deep learning as claimed in claim 8, characterized in that: The fusion model is trained using the similarity loss , multi-layer perceptual loss and brightness distribution loss The overall loss function is: ,in , Control and The weight of .

10. The method for fusion of SAR and visible light remote sensing images based on deep learning according to claim 9, characterized in that:

1. The calculation method of similarity loss is: First, the standard deviation and entropy value are combined to calculate the weight of the constrained light intensity term and : ； ； in, is the serial number of the convolutional unit layer in the pre-trained model, Indicates the total number of channels output by the current convolutional unit layer, and are two weights used to control the entropy value and standard deviation ratio. and They are the calculation formulas for entropy value and standard deviation respectively; Recalculate the gradient term and : ； ； in, represents the Laplacian of Gaussian, represents the L2 norm operation; Then calculate the brightness intensity similarity loss and gradient similarity loss : ； ； in, express and The average structural similarity between Finally, calculate the similarity loss : , To control the balance weight of the two losses.

2. The calculation method of multi-layer perception loss is: First, the multi-layer perceptual loss of high-resolution visible light remote sensing images is calculated : ； in, is the number of fusion blocks in the fusion part, It is The constraint weights of the fusion blocks, It is The fusion feature map obtained by the fusion blocks is Is input to The visible light characteristics of the fusion blocks, represents the Laplacian of Gaussian, represents the L2 norm operation; Then the multi-layer perceptual loss of the low-resolution SAR image is calculated : ； Finally, calculate the multi-layer perceptual loss : ； Indicates balance and The weights of the two loss functions; 3. The calculation method of brightness distribution loss function is: ； in, Represents the Gram matrix measuring feature correlations.

Images