A remote sensing image fusion method based on deep learning model with dense residual

Abstract

The present invention discloses a remote sensing image fusion method based on a dense residual deep learning model, which belongs to the field of image fusion. The method comprises the following steps: obtaining a GF-6 remote sensing image training dataset and preprocessing the dataset; performing model training using a basic module based on a Transformer and an image fusion module based on a dense residual; obtaining a GF-6 remote sensing image test dataset, constructing an image restoration module, constructing a loss function optimization generation process, and finally performing a test based on the trained deep learning model to evaluate the generated high-resolution multispectral image; and comparing the results with those of different fusion methods. The remote sensing image fusion method based on a dense residual deep learning model of the present invention performs excellently on the GF-6 test dataset, and the generated fused image shows stronger richness in spatial details and texture features, and achieves significant advantages in retaining spectral information.

Description

Technical Field

[0001] The present invention belongs to the field of image fusion, and specifically relates to a remote sensing image fusion method based on a deep learning model of dense residual. Background Art

[0002] In recent years, with the widespread development of machine learning, remote sensing image fusion methods based on deep learning have become a research hotspot and are developing rapidly. Deep learning models, such as convolutional neural networks and generative adversarial networks, have demonstrated significant advantages over traditional models. Furthermore, new Transformer models have recently emerged in the field of computer image generation. Existing techniques have introduced the Swin Transformer architecture from computer image generation into the field of remote sensing image fusion, with targeted innovations and redesigns based on the multimodal nature of remote sensing images.

[0003] The PNN model first introduced deep learning to the field of remote sensing image fusion. Using three convolutional layers, the PNN model convolves the input high-resolution panchromatic image and low-resolution multispectral image to extract their spectral and spatial features. However, its limited shallow feature representation results in incomplete capture of high-frequency spatial details, and its ability to model the deep nonlinear relationships between spectral and spatial features is limited. Subsequently, the Pix2Pix migration application first implemented adversarial feature optimization, leading to the rise of remote sensing image fusion models based on generative adversarial networks, such as PSGAN (a panchromatic sharpening image fusion model based on a generative adversarial network) and UCGAN (an image fusion network based on cycle consistency). These models, which utilize the interaction between the generator and the discriminator to constrain adversarial training, produce high-quality, high-resolution images, resulting in more realistic details in image fusion. However, generative adversarial networks themselves suffer from issues such as pattern collapse and training instability, which are particularly prominent in remote sensing image fusion. The original Transformer model was used for natural language processing and has gradually been expanded to sequence data. It currently shows potential in the field of image generation. Its unique self-attention mechanism breaks through the local receptive field limitations of convolutional neural networks and realizes long-range dependency modeling in the entire image domain. In particular, after the method of the present invention introduces dense residuals, it realizes the flow of the entire feature chain from shallow edges to deep semantics through densely connected band-level residual compensation, greatly enriching spectral and spatial information.

[0004] In existing technologies, the PanFormer model extracts modality-specific features through a two-stream network and fuses them with a novel cross-attention module. The cross-attention module can capture the redundant and complementary information of the PAN and MS modalities, thereby achieving good image fusion performance. However, PanFormer has limitations in terms of long-range feature dependency and gradient propagation efficiency. It also has weak spectral continuity and lacks the ability to preserve nonlinear spectral responses through dense interaction paths between bands. Its effectiveness in applying to large-scale remote sensing image training datasets needs to be optimized. Furthermore, the PanFormer model has low computational complexity, but its feature richness is average, local details are weak, and there is no cross-layer interaction. Summary of the Invention

[0005] In response to the problems mentioned in the background technology, the present invention proposes a remote sensing image fusion method based on a deep learning model of dense residuals. For the first time, dense residuals are fused to construct a remote sensing image fusion model based on Transformer, which optimizes the spectral feature preservation of multispectral images and the reconstruction of panchromatic image spatial information, and generates multispectral remote sensing images with both high-resolution spectral and high-resolution spatial information.

[0006] Technical solution: In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0007] A remote sensing image fusion method based on a dense residual deep learning model includes obtaining a remote sensing image training dataset and preprocessing it, using a basic Transformer module and an image fusion module based on dense residual to train the model; obtaining a remote sensing image test dataset, building an image restoration module, and constructing a loss function to optimize the generation process; and finally testing the trained deep learning Transformer model to evaluate the generated high-resolution multispectral image.

[0008] As a preferred embodiment, the specific implementation steps are:

[0009] S1: Obtain a remote sensing image dataset and divide it into a remote sensing image training dataset and a remote sensing image test dataset, then preprocess the low-resolution multispectral image and panchromatic image in the remote sensing image training dataset;

[0010] S2: Build the basic module of Transformer and perform corresponding processing on the pre-processed panchromatic image and low-resolution multispectral image;

[0011] S3: Construct a dense residual-based image fusion module, use the spectral and spatial information in S2 for modeling, and connect the outputs of different attentions to form a fused representation;

[0012] S4: Extract the modal information of the panchromatic image and the low-resolution multispectral image in S3 and send them to the image restoration module for fusion processing, and output the fused high-resolution multispectral image;

[0013] S5: After testing the results in S4 by constructing a loss function, the loss is calculated, and the generation process is iteratively optimized to output the final high-resolution multispectral remote sensing image and evaluate the generated high-resolution multispectral image.

[0014] Preferably, in S1, the specific implementation process is:

[0015] S11: Segment the panchromatic image and low-resolution multispectral image in the remote sensing image training dataset separately;

[0016] S12: Divide the full-color image into blocks and project the full-color image into a hidden dimension through linear projection;

[0017] S13: Linear embedding of low-resolution multispectral images.

[0018] Preferably, in S2, the specific implementation process is:

[0019] S21: The full-color image after linear projection in S1 is processed by self-attention module-block merging layer-self-attention module to extract the spatial information of the full-color image;

[0020] S22: Process the low-resolution multispectral image after linear embedding in S1 with the self-attention module to extract the spectral information of the low-resolution multispectral image.

[0021] Preferably, in S3, the specific implementation content is:

[0022] S31: Receive the spectral information and spatial information in S2 to establish a cross-modal fusion module, and adaptively inject the spatial detail information of the panchromatic image into the spectral feature map of the multispectral image;

[0023] S32: Residual feature calculation and accumulation: Perform residual connection on the spectral features of the initial multispectral image and the fused features obtained in S31;

[0024] S33: Dense residual feature fusion, densely connecting the multiple residual feature maps processed in S32 to generate the final output feature map.

[0025] Preferably, in S4, the specific implementation content is:

[0026] S41: establishing a convolutional feature extraction layer group;

[0027] S42: Establish a pixel rearrangement layer, integrate it at the output of the first convolutional layer and the second convolutional layer, and sample the output feature map;

[0028] S43: Channel dimension reduction and feature filtering are achieved through the pixel rearrangement layer, ReLu activation function and convolution layer in sequence. Finally, the concentrated C channel features are reconstructed into a 4-channel multispectral image through the ReLu activation function and convolution layer.

[0029] As a preferred embodiment, the specific implementation content of S41 is:

[0030] The first and second convolutional layers use Convolution kernel, output 4-channel feature map, used to capture multi-scale spatial details of the input image;

[0031] The third convolutional layer uses Convolution kernel compresses the number of feature channels to C channels to achieve feature dimensionality reduction and information concentration;

[0032] The fourth convolutional layer uses Convolution kernel is used to output the 4-channel fusion result to generate the final high-resolution image.

[0033] Preferably, in S5, the specific implementation content is:

[0034] S51: Convert the input information into a feature map, save the network, and calculate the L1 loss for the image generated by each iteration;

[0035] S52: Feedback the network model to iterate the model according to the L1 loss. After the iteration is completed, the final high-resolution multispectral remote sensing image is output and the generated high-resolution multispectral image is evaluated.

[0036] Preferably, in S51, the specific content of calculating the L1 loss is:

[0037] The trained model is tested using the loss function and the loss is calculated, and the generation process is iteratively optimized; the calculation formula for L1 loss is:

[0038]

[0039] Among them, N represents the number of training samples, I ps represents the high-resolution multispectral image generated by this model, I gt denotes the corresponding ground truth used as validation.

[0040] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0041] (1) The present invention integrates dense residuals for the first time to construct a Transformer-based remote sensing image fusion model. Different from traditional convolutional neural networks and generative adversarial networks, the shift window strategy and hierarchical feature design of the Swin Transformer of the present invention provide new ideas for remote sensing image fusion. At the same time, dense residuals are introduced. When processing each layer, the current layer input is superimposed with the accumulated residual features to form a dense connection path to solve the limitations of the traditional Swin Transformer module in long-distance feature dependence and gradient propagation efficiency, and promote multi-level feature reuse.

[0042] (2) The remote sensing image fusion method based on the dense residual deep learning model of the present invention is applied to the remote sensing image fusion of the GF-6 satellite. The results are compared with those of different fusion methods. The remote sensing image fusion method based on the dense residual deep learning model has excellent overall performance on the GF-6 test dataset. The generated fused image shows stronger richness in spatial details and texture features, and has achieved significant advantages in retaining spectral information. At the same time, compared with traditional image fusion algorithms, it has significant advantages in spatial-spectral collaborative optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of the remote sensing image fusion method based on the deep learning model of dense residual of the present invention;

[0044] Figure 2 is a schematic diagram of the self-attention module of the present invention;

[0045] Figure 3 is a schematic diagram of the image fusion module of the present invention;

[0046] Figure 4 is a schematic diagram of an image restoration module of the present invention;

[0047] Figure 5 It is a comparison chart of the fusion results of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further illustrated below with reference to specific examples. The examples are implemented based on the technical solutions of the present invention. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0049] like Figure 1-Figure 5As shown, the remote sensing image fusion method based on a dense residual deep learning model provided in this embodiment includes obtaining a GF-6 remote sensing image training dataset and preprocessing it, using the basic module of Transformer and the image fusion module based on dense residual to train the model using the preprocessed dataset; obtaining a GF-6 remote sensing image test dataset, constructing an image restoration module that generates high-resolution multispectral images, constructing a loss function to optimize the generation process, and finally testing based on the trained deep learning Transformer model to evaluate the generated high-resolution multispectral images. The specific implementation steps are as follows:

[0050] S1: Obtain the GF-6 remote sensing image dataset and divide it into a GF-6 remote sensing image training dataset and a GF-6 remote sensing image test dataset. Then, preprocess the low-resolution multispectral images and high-resolution panchromatic images in the GF-6 remote sensing image training dataset.

[0051] S11: Segment the panchromatic image and low-resolution multispectral image in the GF-6 remote sensing image training dataset separately;

[0052] And divide it into two parts: training set and test set.

[0053] S12: For high-resolution panchromatic images ( ) is divided into blocks and the full-color image is projected into a hidden dimension (denoted as C) through linear projection;

[0054] The block size of the high-resolution panchromatic image is set to , and convert it to a size of Tensor of .

[0055] S13: Low-resolution multispectral images ( ) for linear embedding;

[0056] Through linear embedding, the shape of the low-resolution multispectral image is .

[0057] Construct a deep learning dataset of remote sensing images including panchromatic and multispectral images.

[0058] S2: Build the basic module of Transformer and perform corresponding processing on the pre-processed panchromatic image and low-resolution multispectral image;

[0059] S21: Receive the full-color image after linear projection in S1 and perform self-attention module ( )-Block Merging Layer-Self-Attention Module ( ) processing to extract high-quality spatial information from panchromatic images;

[0060] Input the panchromatic image tensor after linear projection. First, receive the panchromatic image tensor after linear projection and perform two self-attention module processes: generate the query vector Q, key vector K and value vector V through linear transformation, where Q is generated by the panchromatic image (PAN) and K and V are generated by the multispectral image (MS). The formula is:

[0061]

[0062]

[0063] Among them, Linear() represents a linear transformation operation.

[0064] To calculate scaled dot-product attention, we perform a dot product of the query matrix (Q) and the transpose of the key matrix (K) to compute attention scores. We apply a SoftMax function to normalize these scores to obtain attention weights. These attention weights are then multiplied by the value matrix (V) to obtain a weighted value representation.

[0065] Then, a block merging layer operation is performed to concatenate the features of adjacent image blocks, compress the channel dimension through a linear layer, and expand the receptive field to capture a wide range of spatial structures.

[0066] Finally, the merged features are processed twice again with self-attention to further refine the global spatial dependencies.

[0067] S22: Receive the low-resolution multispectral image after linear embedding in S1 and perform self-attention module (×4) processing to extract high-quality spectral information of the low-resolution multispectral image;

[0068] The input is a linearly embedded multispectral tensor. Four self-attention modules are executed consecutively. A linear transformation is performed to generate a query vector Q, a key vector K, and a value vector V. Scaled dot product attention is then computed, taking the dot product of the query matrix (Q) and the transpose of the key matrix (K) to calculate the attention. A softmax function is applied to normalize these scores to obtain attention weights. The resulting attention weights are multiplied by the value matrix (V) to obtain a weighted value representation. Attention weights are calculated for the spectral dimension, and four layers of self-attention are stacked sequentially to deeply extract inter-band correlations.

[0069] S3: Construct a dense residual-based image fusion module, use the spectral and spatial information in S2 for modeling, and connect the outputs of different attentions to form a fused representation;

[0070] Receive the spectral information and spatial information in S2 to establish a cross-modal fusion module, and then use the residual calculation module and dense residual connection module to connect and fuse. Specifically:

[0071] S31: Construction of cross-modal feature fusion module: Receive the spatial feature map extracted from the panchromatic image (as the query vector Q) and the spectral feature map extracted from the multispectral image (as the value vector V and key vector K), and build a cross-modal fusion module. Specifically, it includes:

[0072] An adaptive mutual enhancement attention mechanism is adopted: first, a cross-attention calculation is performed, with the spatial feature map of the panchromatic image as the query vector Q, and the spectral feature map of the multispectral image as both the key vector K and the value vector V. Through the cross-attention calculation, the spatial detail information of the panchromatic image is adaptively injected into the spectral feature map of the multispectral image.

[0073] The formula for CA cross attention calculation is:

[0074]

[0075] in, is a normalization factor used to scale the dot product calculation result so that its variance remains at 1; K represents the key vector; V represents the value vector; Q represents the query vector; T represents the transpose. When matrices are multiplied, the dimensions of Q and K must match in order to perform the dot product, so K needs to be transposed; SoftMax is a normalization function whose core function is to convert the "query-key" similarity score into an attention weight so that the weight satisfies the "non-negativity" and "sums to 1".

[0076] S32: Residual feature calculation and accumulation: The fused feature map F_fused obtained by S31 is residually connected with the initial multi-spectral input feature map X_m; before performing the residual connection, use The convolution kernel performs dimension alignment on the fused feature map F_fused to generate a residual feature map F_res, which is calculated by the following formula:

[0077]

[0078] Where n is the convolution kernel size; Represents the convolution kernel convolution.

[0079] Initialize the zero tensor dense_skip with the same dimension as the initial multispectral input feature map X_m as the residual accumulation variable; process the residual feature map F_res layer by layer, and When the input feature map , the formula is:

[0080]

[0081] in, is the output feature map of the previous layer. For the first layer block1, x0= F_res; the feature map after each layer is processed Continuously accumulate it into the residual accumulation variable dense_skip, thereby building a chain feature reuse structure.

[0082] S33: Dense residual feature fusion, multiple residual feature maps processed in S32 Perform dense connection; the dense connection operation splices multiple residual feature maps in the channel dimension to generate the final output feature map F_out, the formula is:

[0083]

[0084] Among them, [;] represents the tensor concatenation operation in the channel dimension, which realizes feature reuse and gradient optimization.

[0085] S4: Extract the modal information of the panchromatic image and the low-resolution multispectral image in S3 and send them to the image restoration module for fusion processing, and output the fused high-resolution multispectral image;

[0086] S41: Based on the commonly used image restoration process of Transformer vision, the multi-band characteristics of remote sensing images are combined with the common upsampling method of super-resolution reconstruction to improve it. First, a convolutional feature extraction layer group is established. The first convolutional layer and the second convolutional layer are used. Convolution kernel, output 4-channel feature map, used to capture the multi-scale spatial details of the input image; the third convolution layer uses Convolution kernel compresses the number of feature channels to C channels (C is the preset hidden layer dimension) to achieve feature dimensionality reduction and information concentration; the fourth convolution layer uses Convolution kernel is used to output the 4-channel fusion result to generate the final high-resolution image.

[0087] S42: Establish a pixel rearrangement layer, integrate it into the output of the first and second convolutional layers, and sample the output feature maps.

[0088] S43: Channel dimension reduction and feature filtering are achieved through the pixel rearrangement layer, ReLu activation function and convolution layer in sequence. Finally, the concentrated C channel features are reconstructed into a 4-channel multispectral image through the ReLu activation function and convolution layer.

[0089] S5: After testing the results in S4 by constructing a loss function, the loss is calculated, and the generation process is iteratively optimized to output the final high-resolution multispectral remote sensing image and evaluate the generated high-resolution multispectral image.

[0090] S51: Save the network and calculate the L1 loss for the images generated at each iteration in S4;

[0091] The specific content of calculating image generation loss is:

[0092] A construction loss function is introduced as the image generation loss. The trained model is tested using the construction loss function and then the loss is calculated, and the generation process is iteratively optimized.

[0093] The calculation formula of loss L is:

[0094]

[0095] Where N is the number of training samples, I ps represents the full-color sharpened image generated by this model, I gt denotes the corresponding ground truth used as validation.

[0096] S52: Feedback the network model to iterate the model according to the L1 loss. After the iteration is completed, the final high-resolution multispectral remote sensing image is output and the generated high-resolution multispectral image is evaluated.

[0097] The evaluation index quantitatively analyzes the fusion results from the dimensions of spectral fidelity, spatial detail retention and information richness. λ 、D s The fusion results were compared and evaluated with the most advanced domestic and foreign fusion models using six commonly used evaluation indicators: QNR, SD, SF, and FCC. The evaluation content and optimal conditions of each parameter are shown in Tables 1 and 2 below.

[0098] Table 1 Remote sensing image fusion parameter indicators

[0099]

[0100] The low-resolution multispectral images and high-resolution panchromatic images in the test dataset are preprocessed, and the constructed remote sensing image test dataset including panchromatic images and multispectral images is input into the model to output the test results.

[0101] Table 2 Index evaluation results of remote sensing image generation using the dense residual-based Transformer model

[0102]

[0103] like Figure 5 As shown in Table 2, PanFormer, UCGAN and the Transformer model remote sensing image fusion method based on dense residual are compared. The method proposed in this application has excellent comprehensive performance on the Gaofen-6 test dataset. From the visual effect, the color spectrum performance is closer to low-resolution multispectral images, and the texture space details are closer to full-color images. As shown in Table 2, this model has a very low Dλ and D s The highest QNR (0.9418) achieved, demonstrating optimal performance in the core fusion objective. Detailed image quality was also outstanding: significant advantages in SD and SF (409.19 and 48.29, respectively) demonstrated the richness of information and sharp edges in the fused image, making it suitable for high-precision scenarios. This proposed fusion method has moderate computational complexity, high feature richness, strong local detail, and cross-layer interaction.

[0104] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A remote sensing image fusion method based on a dense residual deep learning model, characterized by: The process includes obtaining and preprocessing a remote sensing image training dataset, using the basic Transformer module and the dense residual-based image fusion module to train the model using the preprocessed training dataset; obtaining a remote sensing image test dataset, building an image restoration module, and constructing a loss function to optimize the generation process. Finally, the trained deep learning Transformer model is used for testing and evaluating the generated high-resolution multispectral image. The specific implementation steps are as follows: S1: Obtain a remote sensing image dataset and divide it into a remote sensing image training dataset and a remote sensing image test dataset, then preprocess the low-resolution multispectral image and panchromatic image in the remote sensing image training dataset; S2: Build the basic module of Transformer and perform corresponding processing on the pre-processed panchromatic image and low-resolution multispectral image; S21: The full-color image after linear projection in S1 is processed by self-attention module-block merging layer-self-attention module to extract the spatial information of the full-color image; S22: Process the low-resolution multispectral image after linear embedding in S1 with a self-attention module to extract the spectral information of the low-resolution multispectral image; S3: Construct an image fusion module based on dense residual, use the spectral and spatial information in S2 for modeling, and connect the outputs of different attentions to form a fused representation; S31: Receive the spectral information and spatial information in S2 to establish a cross-modal fusion module, and adaptively inject the spatial detail information of the panchromatic image into the spectral feature map of the multispectral image; S32: Residual feature calculation and accumulation: Perform residual connection on the spectral features of the initial multispectral image and the fused features obtained in S31; S33: Dense residual feature fusion, densely connecting the multiple residual feature maps processed in S32 to generate the final output feature map; S4: Extract the modal information of the panchromatic image and the low-resolution multispectral image in S3 and send them to the image restoration module for fusion processing, and output the fused high-resolution multispectral image; S5: After testing the results in S4 by constructing a loss function, the loss is calculated, and the generation process is iteratively optimized to output the final high-resolution multispectral remote sensing image and evaluate the generated high-resolution multispectral image.

2. The remote sensing image fusion method based on a dense residual deep learning model according to claim 1, characterized in that: In S1, the specific implementation process is: S11: Segment the panchromatic image and low-resolution multispectral image in the remote sensing image training dataset separately; S12: Divide the full-color image into blocks and project the full-color image into a hidden dimension through linear projection; S13: Linear embedding of low-resolution multispectral images.

3. The remote sensing image fusion method based on a dense residual deep learning model according to claim 1, characterized in that: In S4, the specific implementation content is: S41: establishing a convolutional feature extraction layer group; S42: Establish a pixel rearrangement layer, integrate it at the output of the first convolutional layer and the second convolutional layer, and sample the output feature map; S43: Channel dimension reduction and feature filtering are achieved through the pixel rearrangement layer, ReLu activation function and convolution layer in sequence. Finally, the concentrated C channel features are reconstructed into a 4-channel multispectral image through the ReLu activation function and convolution layer.

4. The remote sensing image fusion method based on a dense residual deep learning model according to claim 3, characterized in that: The specific implementation content of S41 is: The first and second convolutional layers use 3×3 convolution kernels and output 4-channel feature maps to capture multi-scale spatial details of the input image; The third convolutional layer uses a 3×3 convolution kernel to compress the number of feature channels to C channels, achieving feature dimensionality reduction and information concentration; The fourth convolutional layer uses a 3×3 convolution kernel to output the 4-channel fusion result to generate the final high-resolution image.

5. The remote sensing image fusion method based on a dense residual deep learning model according to claim 1, characterized in that: In S5, the specific implementation content is: S51: Convert the input information into a feature map, save the network, and calculate the L1 loss for the image generated by each iteration; S52: Feedback the network model to iterate the model according to the L1 loss. After the iteration is completed, the final high-resolution multispectral remote sensing image is output and the generated high-resolution multispectral image is evaluated.

6. The remote sensing image fusion method based on a dense residual deep learning model according to claim 5, characterized in that: In S51, the specific content of calculating L1 loss is: The trained model is tested using the loss function to calculate the loss and iteratively optimize the generation process; The calculation formula of L1 loss is: Among them, N represents the number of training samples, I ps represents the high-resolution multispectral image generated by this model, I gt denotes the corresponding ground truth used as validation.

Images