Remote sensing image time-space spectrum integrated fusion method, medium, equipment and product

Abstract

The invention provides a remote sensing image time-space spectrum integrated fusion method, a medium, equipment and a product, and relates to the technical field of remote sensing image processing, and the method comprises the steps: obtaining high-spatial-resolution remote sensing images # imgabs0 # and # imgabs1 # and low-spatial-resolution remote sensing images # imgabs2 # and # imgabs3 # at reference and target moments; constructing a space-time spectrum integrated fusion model, splicing the images # imgabs4 #, and extracting a time difference feature # imgabs5 # between a reference moment and a target moment; the method comprises the following steps: performing spectral unmixing on remote sensing images # imgabs6 # and # imgabs7 # to obtain abundance # imgabs10 # and # imgabs11 # of remote sensing images # imgabs8 # and # imgabs9 #, performing spatial degradation operation on abundance # imgabs12 # to obtain # imgabs13 #, # imgabs14 #, # imgabs15 # and # imgabs16 #, and inputting the abundance # imgabs13 #, # imgabs14 #, # imgabs15 # and # imgabs16 # into a convolutional layer to obtain a high-altitude spectral image # imgabs17 # at a reference moment and two high-time spectral images # imgabs18 # and # Adding the # imgabs20 # and the # imgabs21 # to obtain a time-space spectrum integrated fusion image # imgabs22 #; performing spectrum and space degradation on the images # imgabs23 # and # imgabs24 #, and then constructing a loss function training model; and performing time-space spectrum integrated fusion by using the trained model.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a remote sensing image time-space spectrum integrated fusion method, medium, equipment and product. Background Art

[0002] Due to the physical performance constraints of satellite sensors (such as the optical diffraction limit, signal-to-noise ratio threshold, and payload power consumption), the spatial, spectral, and temporal resolutions of individual satellite sensors are still mutually constrained. For example, the Sentinel-3 OLCI (Ocean and Land Colour Imager) sensor has 21 spectral channels and a 1.4-day revisit period, but its 300-meter spatial resolution is insufficient for detailed analysis of heterogeneous features. While the Sentinel-2 MSI (Multispectral Imager) sensor can achieve a 10-meter spatial resolution, it is limited by 13 spectral bands and a 5-day revisit period. Furthermore, the quality of data collected by satellite sensors is easily affected by weather factors such as clouds and fog, further impacting the spatiotemporal continuity of data acquisition.

[0003] Spatiotemporal fusion technology aims to integrate spatial, temporal, and spectral information in remote sensing imagery to improve image quality and information richness. However, existing methods often employ a stepwise fusion approach, resulting in error propagation and reduced accuracy. Furthermore, while deep learning methods employ end-to-end processing, they lack interpretability of their underlying mechanisms, and some methods are limited in their generalization capabilities due to training on simulated data. Therefore, fully considering the correlation constraints between spatiotemporal and spectral features, establishing an effective integrated fusion framework, and achieving joint modeling and processing of multidimensional and multiscale data are of great theoretical and practical value. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem of generating remote sensing images with high spatiotemporal and spectral resolution under an integrated modeling framework, and to propose a remote sensing image spatiotemporal and spectral integrated fusion method, comprising the following steps: S1. Acquire remote sensing images with high spatial resolution, low temporal resolution, and low spectral resolution at the reference and target times. , remote sensing images with low spatial resolution, high temporal resolution, and high spectral resolution and ; S2. Constructing a spatiotemporal-spectral integrated fusion model, including: spatiotemporal reconstruction module, spatial-spectral reconstruction module, and fusion module; The spatiotemporal reconstruction module is used to transform remote sensing images Stitching, extracting the time difference features between the reference time and the target time in the stitched image ; The spatial spectrum reconstruction module is used to convert remote sensing images and Perform spectral unmixing to obtain remote sensing images and Abundance and , abundance Perform spatial degradation operations to obtain , and Input a convolution layer to obtain the first high temporal spectrum image at the reference moment , the second highest time spectrum image at the reference time and the reference time high-altitude spectrum image ; The fusion module is used to and Fusion is performed to obtain a spatiotemporal integrated fusion image ; S3, the image and After performing spectral degradation and spatial degradation operations, a loss function is constructed and the model is trained based on the loss function; S4: Input the image to be fused into the trained model for integrated spatiotemporal fusion, and output the fused image.

[0005] Furthermore, the spatiotemporal reconstruction module includes an encoder and a decoder built on the Swin Transformer; The encoder consists of 4 sequentially connected stages, each of which consists of 2, 2, 6, and 2 consecutive SwinTransformer blocks and a patch merging layer. The input of each encoder stage first passes through the SwinTransformer block, and then the patch merging layer reduces the resolution of the input by downsampling. The decoder is the inverse process of the encoder, consisting of four sequentially connected stages. The four stages are composed of 2, 6, 2, and 2 consecutive Swin Transformer blocks and a patch expansion layer. The input of each decoder stage is first upsampled by the patch expansion layer to increase the input resolution, and then passes through the Swin Transformer block. The features output by different stages of the encoder are connected to the decoder through jump connections.

[0006] Furthermore, the spatial spectrum reconstruction module achieves the purpose of unmixing based on the autoencoder network, including an unmixing encoder and an unmixing decoder. The unmixing encoder is used to transform the spectral vector into an abundance vector, which is input into the unmixing decoder. The role of the unmixing decoder is to reconstruct the pixel based on the end member set extracted by the unmixing encoder. The unmixing encoder is composed of five consecutive 1×1 convolutional layers stacked together. The first three convolutional layers are followed by ReLU activation functions, and the last convolutional layer is followed by SENet. The unmixing decoder includes a 1×1 convolutional layer.

[0007] Furthermore, the fusion module is used to and Fusion is performed to obtain a spatiotemporal integrated fusion image , expressed as: .

[0008] Furthermore, the spectral degradation network contains a convolutional layer with a convolution kernel size of , the step size is 1; the spatial degradation network is implemented by a layer of convolution, and the convolution kernel size is , the step size is , is the scaling factor of the spatial resolution of the image before and after degradation.

[0009] Furthermore, the loss function is expressed as: , , , , , , , in, represents the total loss function, represents the reconstruction loss based on unmixing, Indicates space loss, represents the spectral loss, represents the abundance sum to 1 loss, represents the abundance sparsity loss, represents the spectral angle loss, ε represents the parameter used to stabilize the back propagation of the error, N represents the number of pixels, Indicates abundance The i-th band of Indicates abundance The i-th band, p represents the number of end members, a represents the coefficient parameter, express The (i, j)th element of the abundance matrix, Indicates the number of bands, PSF indicates spatial degradation operation, SRF indicates spectral degradation operation, It means taking the i-th pixel and the j-th band.

[0010] The present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned remote sensing image spatiotemporal spectral integrated fusion method.

[0011] The present invention also proposes an electronic device, comprising a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program includes computer-readable instructions, and the processor is configured to call the computer-readable instructions to execute the above-mentioned remote sensing image spatiotemporal spectral integrated fusion method.

[0012] The present invention also provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned remote sensing image spatiotemporal spectral integrated fusion method.

[0013] The beneficial effects brought about by the technical solution provided by the present invention are: The present invention constructs a space-time spectrum integrated fusion model including a space-time reconstruction module, a space-spectrum reconstruction module and a fusion module. The space-time reconstruction module is used to extract and reconstruct space-time features, and the space-spectrum reconstruction module is used to reconstruct space-spectrum features. Through the coordinated and parallel operation of the two subtasks of space-time reconstruction and space-spectrum reconstruction, effective correlation learning of space-time spectrum information in data is realized, and a loss function is constructed based on a set of parameters for simulating data degradation. The parameters of the space-time spectrum integrated fusion network are iteratively optimized. The present invention has significant advantages in both spatial resolution improvement and spectral high fidelity, and realizes the generation of remote sensing images with high space-time spectrum resolution under an integrated modeling framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Flowchart of the method for integrated spatiotemporal fusion of remote sensing images according to an embodiment of the present invention; Figure 2 is a structural diagram of a spatiotemporal reconstruction module according to an embodiment of the present invention, wherein Figure 2 (a) is the encoder structure diagram of the spatiotemporal reconstruction module based on Swin Transformer. Figure 2 (b) is the decoder structure diagram of the spatiotemporal reconstruction module based on SwinTransformer. Figure 2 (c) is the Swin Transformer block structure diagram; Figure 3 is a structural diagram of a spatial spectrum reconstruction module according to an embodiment of the present invention; Figure 4 It is a block diagram of an electronic device in an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0015] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0016] The flowchart of the remote sensing image spatiotemporal spectral integrated fusion method according to the embodiment of the present invention is as follows: Figure 1 , specifically including the following steps: S1. Obtain multi-source remote sensing images after atmospheric correction and registration, specifically including 4 groups of remote sensing images with complementary characteristics: (1) Reference time Remote sensing images with low spatial resolution, high temporal resolution, and high spectral resolution ; (2) Reference time Remote sensing images with high spatial resolution, low temporal resolution, and low spectral resolution ; (3) Target time Remote sensing images with low spatial resolution, high temporal resolution, and high spectral resolution ; (4) Target time Remote sensing images with high spatial resolution, low temporal resolution, and low spectral resolution .

[0017] S2. Construct a space-time spectrum integrated fusion model, including: space-time reconstruction module, space-spectrum reconstruction module, and fusion module.

[0018] The structural diagram of the spatiotemporal reconstruction module of the embodiment of the present invention is shown in FIG. Figure 2 ,in Figure 2 (a) is the encoder structure diagram of the spatiotemporal reconstruction module based on Swin Transformer. Figure 2 (b) is the decoder structure diagram of the spatiotemporal reconstruction module based on SwinTransformer. Figure 2 (c) is the Swin Transformer block structure diagram. The spatiotemporal reconstruction module is used to transform remote sensing images 、 and Perform stitching in the channel dimension and extract the time difference feature between the reference time and the target time in the stitched image The spatiotemporal reconstruction module includes an encoder and a decoder built on Swin Transformer. The encoder consists of 4 stages connected in sequence. The 4 stages are composed of 2, 2, 6, and 2 consecutive Swin Transformer blocks and a patch merging layer. The input of each stage of the encoder first calculates the self-attention through the Swin Transformer block, and then the patch merging layer reduces the resolution of the input by downsampling. The 4 stages form a hierarchical structure, which reduces the resolution layer by layer and increases the receptive field layer by layer to obtain global information. The encoder first converts the image of size H W The input image of B is expanded to the size of After 4 stages, the dimension of the vector is Feature vector. H represents the height of the input image, W represents the width of the input image, and B and C represent the dimensions of the vector.

[0019] The decoder is the inverse of the encoder, consisting of four sequentially connected stages, each consisting of 2, 6, 2, and 2 consecutive Swin Transformer blocks, followed by a patch expansion layer. The input to each decoder stage is first upsampled by the patch expansion layer to increase the input resolution. It then passes through a Swin Transformer block, establishing long-range dependencies and global context connections, restoring the global temporal differential features to the input resolution. Features output from different encoder stages are fed to the decoder via jump connections.

[0020] The core component of the Swin Transformer block is the multi-head self-attention module. In the self-attention calculation, the entire image is divided into multiple non-overlapping windows, each containing M×M blocks of size 2×2. The image blocks are then used as input to several stacked transformer blocks. The self-attention within the local window is calculated by introducing relative position information, which is expressed as: , , in, is the vector matrix of the input patch, n is the number of patches, and d is the feature dimension. With parameter matrix and Multiply them together to get the query matrix , key matrix , value matrix Then use and Multiply to get the normalized weight, and Multiply together to get the feature of interest. is the relative position bias, which indicates the relative position between patches within a single window.

[0021] The structure diagram of the spatial spectrum reconstruction module of the embodiment of the present invention is shown in FIG. Figure 3The core goal of the spatial-spectral reconstruction module is to generate high-spatial-resolution hyperspectral data at the reference moment through high-precision reconstruction of the spectral dimension, providing fundamental support for multi-dimensional spatiotemporal-spectral collaborative optimization. This module introduces the physical model of the spectral linear mixing process into the deep learning framework through an autoencoder network architecture based on spectral unmixing theory. The unmixing encoder uses a five-layer deep convolutional network for feature extraction and introduces a channel attention mechanism, SENet, to improve the accuracy of abundance extraction. The unmixing encoder consists of five consecutive 1×1 convolutional layers. The first three convolutional layers are followed by a ReLU activation function, and the last convolutional layer is followed by an SENet (Squeeze-and-Excitation Network). The unmixing decoder includes a single 1×1 convolutional layer.

[0022] The spatial spectrum reconstruction module is used to convert remote sensing images and To perform spectral unmixing, first and Input into two unmixing encoders (high spatial encoder and hyperspectral encoder) to obtain the abundance of high spatial resolution multispectral data and low spatial resolution hyperspectral data respectively. and , the process can be expressed as: , , in, 、 Represent the high spatial encoder and the high spectral encoder, respectively. 、 Respectively represent and The abundance of Represent the length and width of the image and the number of end members respectively.

[0023] abundance Perform spatial degradation operations to obtain , 、 and Input a weight-shared unbiased 1×1 convolutional layer to obtain the first high-frequency spectrum image at the reference moment , the second highest time spectrum image at the reference time and the reference time high-altitude spectrum image The parameters of the shared convolutional layer are based on the endmembers We get, where 1×1 represents the size of the convolution kernel space, represents the convolution kernel size, Indicates the number of convolution kernels. It can be obtained through the decoder later. Two reconstructed images of and 、 The reconstructed image This process can be expressed as: , , , in, Represents the decoder layer.

[0024] The fusion module is used to and Fusion is performed to obtain a spatiotemporal integrated fusion image , expressed as: .

[0025] S3, the image and After performing spectral degradation and spatial degradation operations, a loss function is constructed and the model is trained based on the loss function. The spectral degradation network contains a convolution layer with a convolution kernel size of , the number of convolution kernels is , the step size is 1, and the weights are used as parameters of the spectral degradation network to construct the integration process; the spatial degradation network is implemented by a layer of convolution, and the convolution kernel size is , the step size is , is the scaling factor of the spatial resolution of the image before and after degradation.

[0026] The images obtained by spectrally degrading a remote sensing image with low spatial resolution, high temporal resolution and high spectral resolution are consistent with the images obtained by spatially degrading a remote sensing image with high spatial resolution, low temporal resolution and low spectral resolution.

[0027] The loss function is expressed as: , , , , , , , in, represents the total loss function, represents the reconstruction loss based on unmixing, Indicates space loss, represents the spectral loss, represents the abundance sum to 1 loss, represents the abundance sparsity loss, represents the spectral angle loss, ε represents the parameter used to stabilize the back propagation of the error, N represents the number of pixels, Indicates abundance The i-th band of Indicates abundance The i-th band, p represents the number of end members, a represents the coefficient parameter, which is a number close to 0, and is set to 1 in the present invention. express The (i, j)th element of the abundance matrix, Indicates the number of bands, PSF indicates spatial degradation operation, SRF indicates spectral degradation operation, It means taking the i-th pixel and the j-th band.

[0028] S4. Input the remote sensing images with low spatial resolution, high temporal resolution, and high spectral resolution to be fused and the remote sensing images with high spatial resolution, low temporal resolution, and low spectral resolution at the same location into the trained model for spatiotemporal and spectral integrated fusion, and output remote sensing images with high spatiotemporal and spectral resolution.

[0029] In an exemplary embodiment, a computer-readable storage medium is included, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned remote sensing image spatiotemporal spectral integrated fusion method is implemented.

[0030] See also Figure 4 In an exemplary embodiment, an electronic device is also included, including at least one processor, at least one memory, and at least one communication bus.

[0031] The memory stores a computer program including computer-readable instructions. The processor calls the computer-readable instructions stored in the memory through a communication bus to execute the above-mentioned remote sensing image spatiotemporal spectral integrated fusion method.

[0032] In an exemplary embodiment, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned remote sensing image spatiotemporal spectral integrated fusion method.

[0033] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A remote sensing image spatiotemporal spectral integrated fusion method, characterized in that: The method comprises the following steps: S1. Acquire remote sensing images with high spatial resolution, low temporal resolution, and low spectral resolution at the reference and target times. , remote sensing images with low spatial resolution, high temporal resolution, and high spectral resolution and ; S2. Constructing a space-time spectrum integrated fusion model, including: space-time reconstruction module, space-spectrum reconstruction module, and fusion module; The spatiotemporal reconstruction module is used to transform remote sensing images Stitching, extracting the time difference features between the reference time and the target time in the stitched image ; The spatial spectrum reconstruction module is used to convert remote sensing images and Perform spectral unmixing to obtain remote sensing images and Abundance and , abundance Perform spatial degradation operations to obtain , 、 and Input a convolution layer to obtain the first high temporal spectrum image at the reference moment , the second highest time spectrum image at the reference time and the reference time high-altitude spectrum image ; The fusion module is used to and Fusion is performed to obtain a spatiotemporal integrated fusion image ; S3, the image and After performing spectral degradation and spatial degradation operations, a loss function is constructed and the model is trained based on the loss function; S4: Input the image to be fused into the trained model for integrated spatiotemporal fusion, and output the fused image.

2. The method for integrating time, space and spectrum of remote sensing images according to claim 1, characterized in that: The spatiotemporal reconstruction module includes an encoder and decoder built on Swin Transformer; The encoder consists of 4 sequentially connected stages, each of which consists of 2, 2, 6, and 2 consecutive SwinTransformer blocks and a patch merging layer. The input of each encoder stage first passes through the SwinTransformer block, and then the patch merging layer reduces the resolution of the input by downsampling. The decoder is the inverse process of the encoder, consisting of four sequentially connected stages. The four stages are composed of 2, 6, 2, and 2 consecutive Swin Transformer blocks and a patch expansion layer. The input of each decoder stage is first upsampled by the patch expansion layer to increase the input resolution, and then passes through the Swin Transformer block. The features output by different stages of the encoder are connected to the decoder through jump connections.

3. The method for integrating time, space and spectrum of remote sensing images according to claim 1, characterized in that: The spatial spectrum reconstruction module achieves the purpose of unmixing based on the autoencoder network, including an unmixing encoder and an unmixing decoder. The unmixing encoder is used to transform the spectral vector into an abundance vector, which is input into the unmixing decoder. The role of the unmixing decoder is to reconstruct the pixel based on the end member set extracted by the unmixing encoder. The unmixing encoder is composed of five consecutive 1×1 convolutional layers stacked together. The first three convolutional layers are followed by ReLU activation functions, and the last convolutional layer is followed by SENet. The unmixing decoder includes a 1×1 convolutional layer.

4. The method for integrating time, space and spectrum of remote sensing images according to claim 1, characterized in that: The fusion module is used to and Fusion is performed to obtain a spatiotemporal integrated fusion image , expressed as: 。 5. The method for integrating time, space and spectrum of remote sensing images according to claim 1, characterized in that: The spectral degradation network consists of a convolutional layer with a convolution kernel size of , the step size is 1; the spatial degradation network is implemented by a layer of convolution, and the convolution kernel size is , the step size is , is the scaling factor of the spatial resolution of the image before and after degradation.

6. The method for integrating time, space and spectrum of remote sensing images according to claim 1, characterized in that: The loss function is expressed as: ， ， ， ， ， ， ， in, represents the total loss function, represents the reconstruction loss based on unmixing, Indicates space loss, represents the spectral loss, represents the abundance sum to 1 loss, represents the abundance sparsity loss, represents the spectral angle loss, ε represents the parameter used to stabilize the back propagation of the error, N represents the number of pixels, Indicates abundance The i-th band of Indicates abundance The i-th band, p represents the number of end members, a represents the coefficient parameter, express The (i, j)th element of the abundance matrix, Indicates the number of bands, PSF indicates spatial degradation operation, SRF indicates spectral degradation operation, It means taking the i-th pixel and the j-th band.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program includes computer-readable instructions, and the processor is configured to call the computer-readable instructions to execute the method according to any one of claims 1 to 6.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images