An unsupervised generative adversarial remote sensing image hyperspectral fusion method and system

Abstract

This invention relates to the field of image processing technology, specifically to an unsupervised generative adversarial remote sensing image spatial-spectral fusion method and system. The method includes: inputting the remote sensing images to be fused into a trained spatial-spectral fusion model; extracting multi-scale local-global features from MS and PAN images respectively using a dual-stream convolution-Transformer generator in the spatial-spectral fusion model; firstly performing prior cross-fusion on the same-scale local-global features of the MS and PAN images; then performing cross-scale fusion on the prior cross-fusion results of different scales to form a recursive cascaded cross-fusion structure to obtain key fusion information; reconstructing injected information based on supplementary information to generate injected information; performing nonlinear correction on the upsampled MS image and injected information to generate a high spatial resolution multispectral image corresponding to the remote sensing image; this invention can improve the spatial resolution of multispectral remote sensing images and reduce spectral distortion.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an unsupervised generative adversarial remote sensing image spatial-spectral fusion method and system. Background Technology

[0002] Remote sensing imagery is widely used in geological exploration, land cover classification, agricultural yield prediction, pest and disease detection, and disaster prediction. These applications require high spatial resolution, high spectral resolution, or high temporal resolution imagery. However, due to limitations in sensor technology, the resulting images are typically low spatial resolution multispectral (MS) or hyperspectral images, low temporal resolution multispectral or hyperspectral images, and low spectral resolution panchromatic (PAN) images. Therefore, it is necessary to aggregate complementary information from low spatial resolution multispectral images and high spatial resolution panchromatic images to generate high spatial resolution multispectral images, providing high-precision imagery for remote sensing missions. This technique is called multi-source remote sensing image spatial-spectral fusion.

[0003] Although deep learning has made rapid progress in the spatial-spectral fusion of multi-source remote sensing images and achieved certain fusion results, it still faces the following problems: when models trained on down-resolution simulation data are tested on full-resolution remote sensing data, the resolution changes, causing spectral distortion and loss of detail information in the fusion results. Directly adding detail information to low-resolution multispectral data ignores the nonlinear relationship between the bands of multispectral and panchromatic images, which can easily lead to spectral distortion or ringing phenomena. Summary of the Invention

[0004] The purpose of this invention is to provide an unsupervised generative adversarial remote sensing image spatial-spectral fusion method and system that can improve spatial resolution and reduce spectral distortion.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] In a first aspect, embodiments of the present invention provide an unsupervised generative adversarial remote sensing image spatial-spectral fusion method, the method comprising the following steps:

[0007] S100, acquire the remote sensing images to be fused and the trained spatial-spectral fusion model; the remote sensing images include MS images and PAN images, and the spatial-spectral fusion model is trained based on a two-stream convolutional-Transformer generator and a convolutional-Transformer dual discriminator.

[0008] S200: Input the remote sensing images to be fused into the trained spatial-spectral fusion model. Extract multi-scale local-global features from MS and PAN images using the dual-stream convolution-Transformer generator in the spatial-spectral fusion model. First, perform prior cross-fusion on the same-scale local-global features of MS and PAN images. Then, perform cross-scale fusion on the prior cross-fusion results of different scales to form a recursive cascaded cross-fusion structure to obtain key fusion information. Based on the supplementary information, reconstruct the key fusion information to generate injection information. Perform nonlinear correction on the upsampled MS image and the injection information to generate a high spatial resolution multispectral image corresponding to the remote sensing image.

[0009] Optionally, the dual-stream convolutional-transformer generator includes a dual-stream multi-scale local-global feature extraction module; the dual-stream multi-scale local-global feature extraction module includes two weighted independent feature extraction branches, and the extraction of multi-scale local-global features of MS images and PAN images respectively through the dual-stream convolutional-transformer generator in the spatial-spectral fusion model includes:

[0010] S211, take the PAN image and the upsampled MS image as input images, input the PAN image into one feature extraction branch, and input the upsampled MS image into another feature extraction branch;

[0011] S212, after the input image is normalized by layer, the normalized features are obtained. The normalized features are then aggregated by 3-branch 1×1 pixel convolution to aggregate cross-channel context information and 3×3 depth convolution to encode channel space aggregated context information, generating query vector Q, key vector K and value vector V.

[0012] S213, the query vector Q, key vector K and value vector V are deformed respectively. The deformed query vector Q and key vector K are multiplied and Softmaxed in sequence to obtain attention A. Attention A is then multiplied, deformed and 1×1 pixel convolved with the deformed value vector V and added to the input image to obtain the correction feature.

[0013] S214, after the correction features are normalized by the layer, normalized correction features are generated. Then, the feature dimension is increased by 1×1 pixel convolution projection. Local information is obtained by 3×3 depth convolution. Then, after activation by nonlinear Gaussian error linear unit, cross-channel local context information is aggregated by 1×1 pixel convolution and the number of channels is adjusted to obtain local enhancement correction features.

[0014] S215 connects the correction features and the residuals of the local enhancement correction features to generate local-global features corresponding to the input image;

[0015] S216, using the local-global features as the input image, repeating S212 to S215, and combining it with the downsampling operation of convolution to obtain multi-scale local-global features corresponding to the input image.

[0016] Optionally, the prior cross-fusion of local-global features of the same scale from MS and PAN images includes:

[0017] S221, for each local-global feature of the PAN image and the local-global feature of the MS image at the same scale, the local-global feature of the PAN image is generated as a key vector through layer normalization, 1×1 cell convolution and 3×3 depth convolution projection. Sum value vector The local-global features of MS images are normalized by layers, and then projected by 1×1 pixel convolution and 3×3 depth convolution to generate query vectors. ;

[0018] S222, respectively for the query vector Key vector Sum value vector The vector is transformed, and after multiplication and softmax operations, the cross-attention vector (CA) is obtained. The cross-attention CA is then compared with the transformed value vector. After multiplication, deformation, and 1×1 pixel convolution, the cross-fusion feature is obtained by adding it to the local-global features of the PAN image.

[0019] S223, after the cross-fusion feature is normalized by the layer, the normalized cross-fusion feature is generated. Then, the feature dimension is increased by 1×1 pixel convolution projection. Local information is obtained by 3×3 depth convolution. Then, non-linear Gaussian error linear unit activation is used. Cross-channel local context information is aggregated by 1×1 pixel convolution and the number of channels is adjusted to obtain the locally enhanced cross-fusion feature.

[0020] S224 connects the cross-fusion feature and the residual of the locally enhanced cross-fusion feature to generate the prior cross-fusion result at this scale.

[0021] Optionally, the step of fusing prior cross-fusion results at different scales across scales and forming a recursive cascaded cross-fusion structure to obtain key fusion information includes:

[0022] S225, for each pair of scale-adjacent fine-scale prior cross-fusion results and coarse-scale prior cross-fusion results, the fine-scale prior cross-fusion results are downsampled to generate a downsampled image of the same scale as the coarse-scale prior cross-fusion results; the coarse-scale prior cross-fusion results are normalized by layers, and a query vector is generated by 1×1 pixel convolution and 3×3 depth convolution projection. The downsampled image is normalized by layers, and then projected by 1×1 pixel convolution and 3×3 depth convolution to generate a key vector. Sum value vector ;

[0023] S226, respectively for the query vector Key vector Sum value vector Transform the query vector. and key vector After multiplication and softmax operations, it is then compared with the transformed value vector. After multiplication, deformation, and 1×1 pixel convolution, the cross-scale cross-fusion feature is obtained by adding it to the downsampled image.

[0024] S227, the cross-scale cross-fusion features are then normalized by layers and a local enhancement feedforward network to obtain locally enhanced cross-scale fusion features, which are then connected with the cross-scale cross-fusion feature residuals to generate cross-scale fusion results of the same scale as the coarse-scale prior cross-fusion results;

[0025] S228, following the order from fine scale to coarse scale, the cross-scale fusion results of adjacent scales are recursively cascaded and fused step by step according to S225 to S227 to obtain key fusion information.

[0026] Optionally, the step of reconstructing the key fusion information based on supplementary information to generate injection information includes:

[0027] S231, the key fusion information is used as the input information of the dual-stream multi-scale reconstruction network. The dual-stream multi-scale reconstruction network includes a reconstruction module and two branches. The ends of the reconstruction module are respectively connected to the two branches. Each of the two branches includes multiple cascaded reconstruction modules.

[0028] S232, for this reconstruction module, the current scale of the information to be reconstructed is determined, the supplementary information corresponding to the current scale is obtained, the supplementary information is converted to the current scale, and then concatenated with the input information, followed by 3×3 convolution and a convolutional Transformer module to generate the reconstruction features at the current scale; the supplementary information includes key fusion information, cross-scale fusion results at the current scale, prior cross-fusion results at the current scale, and prior cross-fusion results at a finer scale;

[0029] S233, input the reconstruction features of the current scale into the two branches of the two-stream multi-scale reconstruction network respectively;

[0030] S234, S232 is executed sequentially for multiple cascaded reconstruction modules in the two branches. The reconstruction features of the current scale are used as the input information of the reconstruction module of the next scale. The reconstruction features corresponding to each reconstruction module in the two branches are generated respectively. The lower-level reconstruction features of one branch are reused to the higher-level reconstruction features of the other branch through skip connections to generate injection information.

[0031] Optionally, the step of performing nonlinear correction on the upsampled MS image and the injected information to generate a high spatial resolution multispectral image corresponding to the remote sensing image includes:

[0032] After stitching the injected information and the upsampled MS image together, a nonlinear mapping is performed sequentially through 1×1 convolution and 3×3 convolution to generate a high spatial resolution multispectral image.

[0033] Optionally, the spatial-spectral fusion model is trained in the following manner:

[0034] S101, Obtain a sample set containing multiple satellite remote sensing image samples, wherein the satellite remote sensing image samples include MS images and PAN images;

[0035] S102, Establish a dual-stream convolutional-transformer generator and a convolutional-transformer dual discriminator consistent with the spatial-spectral fusion model architecture, input the sample set into the spatial-spectral fusion model for training, and process the remote sensing image samples through the dual-stream convolutional-transformer generator to generate HMS images.

[0036] S103, the spectral and detail information of the HMS image is discriminated by the convolution-Transformer dual discriminator to obtain the adversarial loss of the two-stream convolution-Transformer generator and the convolution-Transformer dual discriminator; the composite loss of the spatial-spectral fusion model is obtained by combining the adversarial loss, the no-reference loss, the spatial consistency loss and the spectral consistency loss; the spatial-spectral fusion model is iteratively trained based on the composite loss until the optimization objective of the composite loss is met, and the trained spatial-spectral fusion model is obtained.

[0037] Secondly, embodiments of the present invention provide an unsupervised generative adversarial remote sensing image spatial-spectral fusion system, the system comprising:

[0038] At least one processor;

[0039] At least one memory for storing at least one program;

[0040] When the at least one program is executed by the at least one processor, the at least one processor implements the unsupervised generative adversarial remote sensing image spatial-spectral fusion method as described in any of the preceding claims.

[0041] The beneficial effects of this invention are as follows: This invention discloses an unsupervised generative adversarial remote sensing image spatial-spectral fusion method and system. This invention acquires spatial structure and spectral information at different resolutions through a dual-stream multi-scale local-global feature extraction branch. Then, a recursive hybrid-scale feature fusion sub-network with cross-attention performs prior cross-fusion of the extracted local-global features of the MS and PAN images at the same scale. The fine-scale prior cross-fusion result is then fused with the coarse-scale prior cross-fusion result across scales to generate coarse-resolution features. This process is repeated sequentially to form a recursive cascaded cross-fusion structure, ultimately generating key fusion information. Then, a supplementary information structure is used to reconstruct the key fusion information to compensate for information loss caused by downsampling. Next, an injection information correction module is constructed to address distortion caused by ignoring the nonlinear relationship between the bands of the MS and PAN images. Finally, a convolutional-Transformer dual discriminator is designed, and an unsupervised composite loss function is defined to optimize the objective. This invention can improve the spatial resolution of multispectral remote sensing images and reduce spectral distortion. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart of an unsupervised generative adversarial remote sensing image spatial-spectral fusion method according to an embodiment of the present invention is shown.

[0044] Figure 2 A structural diagram of the spatial spectrum fusion model in an embodiment of the present invention is shown.

[0045] Figure 3 The branch structure diagram of the dual-stream multi-scale local-global feature extraction module in an embodiment of the present invention is shown.

[0046] Figure 4 The diagram illustrates the multi-depth convolution head transposed attention and local enhancement feedforward network structure in an embodiment of the present invention.

[0047] Figure 5 The diagram shows the recursive cascaded cross-fusion network structure in an embodiment of the present invention.

[0048] Figure 6The diagram shows the structure of the prior cross-fusion module and the cross-scale fusion module in an embodiment of the present invention.

[0049] Figure 7 A structural diagram of the scale conversion module in an embodiment of the present invention is shown;

[0050] Figure 8 The diagram shows the structure of the dual-stream multi-scale reconstruction network in an embodiment of the present invention.

[0051] Figure 9 The diagram shows the structure of the multi-scale reconstruction module in an embodiment of the present invention.

[0052] Figure 10 The network structure diagram of the convolution-Transformer dual discriminator in an embodiment of the present invention is shown;

[0053] Figure 11 A block diagram of an unsupervised generative adversarial remote sensing image spatial-spectral fusion system according to an embodiment of the present invention is shown. Detailed Implementation

[0054] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present invention can be combined with each other.

[0055] Spatial-spectral fusion of remote sensing images is an important technique for improving the spatial resolution of multispectral images. Spectral distortion and detail loss remain key problems that need to be addressed in spatial-spectral fusion. Down-resolution models in related techniques produce severely distorted fusion results on full-resolution images, and detail injection ignores the nonlinear relationship between low-resolution multispectral images and panchromatic images.

[0056] To address the technical problems in related technologies, this invention provides an unsupervised generative adversarial remote sensing image spatial-spectral fusion method and system, which can improve the spatial resolution of multispectral remote sensing images and reduce spectral distortion. The spatial-spectral fusion model includes a convolutional-Transformer generator and a convolutional-Transformer dual discriminator. The convolutional-Transformer generator is designed with a dual-stream multi-scale local-global feature extraction branch to acquire spatial structure information and spectral information at different resolutions. A recursive hybrid-scale feature fusion sub-network with cross-attention is then designed to perform prior cross-fusion of the extracted local-global features of the MS and PAN images at the same scale. The fine-scale prior cross-fusion result is then fused with the coarse-scale prior cross-fusion result to generate coarse-resolution features. This process is repeated sequentially to form a recursive cascaded cross-fusion structure, ultimately generating key fusion information. Then, a supplementary information structure is used to reconstruct the key fusion information to compensate for information loss caused by downsampling. Finally, an injection information correction module is constructed to address the distortion caused by ignoring the nonlinear relationship between the bands of the MS and PAN images. Finally, a convolutional-Transformer dual discriminator was designed, and an unsupervised composite loss function was defined to optimize the objective. The proposed model was validated on Gaofen-2 and QuickBird satellite data. Experimental results demonstrate that the fusion results of this invention are superior to current mainstream methods in both visual evaluation and objective metrics.

[0057] Combination Figure 1 and Figure 2 This invention provides an unsupervised generative adversarial remote sensing image spatial-spectral fusion method, which includes the following steps:

[0058] S100, acquire the remote sensing images to be fused and the trained spatial-spectral fusion model; the remote sensing images include MS images and PAN images, and the spatial-spectral fusion model is trained based on a two-stream convolutional-Transformer generator and a convolutional-Transformer dual discriminator.

[0059] S200: Input the remote sensing images to be fused into the trained spatial-spectral fusion model. Extract multi-scale local-global features from MS and PAN images using the dual-stream convolution-Transformer generator in the spatial-spectral fusion model. First, perform prior cross-fusion on the same-scale local-global features of MS and PAN images. Then, perform cross-scale fusion on the prior cross-fusion results of different scales to form a recursive cascaded cross-fusion structure to obtain key fusion information. Based on the supplementary information, reconstruct the key fusion information to generate injection information. Perform nonlinear correction on the upsampled MS image and the injection information to generate a high spatial resolution multispectral image corresponding to the remote sensing image.

[0060] Specifically, this invention proposes an unsupervised convolutional-Transformer generative adversarial spatial-spectral fusion method. The spatial-spectral fusion model is trained directly using the original full-resolution remote sensing data, reducing the impact of resolution variations on the full-resolution data fusion results, thereby improving spatial resolution and reducing spectral distortion. The overall structure of the unsupervised convolutional-Transformer generative adversarial spatial-spectral fusion method is as follows: Figure 2 As shown, the system includes one two-stream convolutional-transformer generator and two convolutional-transformer discriminators. The MS image is an upsampled MS image, and the HMS image is a fused high spatial resolution multispectral image. In the two-stream convolutional-transformer generator, a two-stream multi-scale local-global feature extraction branch is first designed to extract low-level and high-level semantic information from the MS and PAN images at different scales, representing spatial structure and spectral information at different modalities and resolutions. Secondly, a recursive cascaded cross-fusion network is designed using the convolutional-transformer. First, prior cross-fusion is performed on the extracted features of the MS and PAN images at the same scale. Then, the prior cross-fusion result at this scale is combined with the prior cross-fusion result at the next scale to perform mixed-scale cross-fusion, generating features at a lower resolution scale. This process is repeated sequentially to form a recursive cascaded cross-fusion structure, generating key fusion information. Thirdly, a two-stream multi-scale reconstruction network reconstructs the key fusion information to generate injection information, and an injection information correction module is designed to generate the high spatial resolution multispectral image. The reconstruction of information at each scale uses the reconstructed features from the previous scale and supplementary information as input. The supplementary information compensates for the information loss caused by downsampling during the fusion stage, thereby reducing spectral distortion and loss of detail. Finally, a spectral information discriminator is constructed using a convolutional-transformer. ) and detail information discriminator ( ). The discriminator is used to distinguish the detail information in the generated HMS image from that in the PAN image, so as to ensure that the detail information in the generated HMS image is consistent with that in the PAN image. The discriminator is used to distinguish the spectral information of the generated HMS image from that of the original MS image, so as to ensure that the spectral information of the generated HMS image is consistent with that of the original MS image.

[0061] In some embodiments, the dual-stream multi-scale local-global feature extraction module includes two weighted independent feature extraction branches. The extraction of multi-scale local-global features from MS and PAN images using the dual-stream convolutional-Transformer generator in the spatial-spectral fusion model includes:

[0062] S211, take the PAN image and the upsampled MS image as input images, input the PAN image into one feature extraction branch, and input the upsampled MS image into another feature extraction branch;

[0063] S212, after the input image is normalized by layer, the normalized features are obtained. The normalized features are then aggregated by 3-branch 1×1 pixel convolution to aggregate cross-channel context information and 3×3 depth convolution to encode channel space aggregated context information, generating query vector Q, key vector K and value vector V.

[0064] S213, the query vector Q, key vector K and value vector V are deformed respectively. The deformed query vector Q and key vector K are multiplied and Softmaxed in sequence to obtain attention A. Attention A is then multiplied, deformed and 1×1 pixel convolved with the deformed value vector V and added to the input image to obtain the correction feature.

[0065] S214, after the correction features are normalized by the layer, normalized correction features are generated. Then, the feature dimension is increased by 1×1 pixel convolution projection. Local information is obtained by 3×3 depth convolution. Then, after activation by nonlinear Gaussian error linear unit, cross-channel local context information is aggregated by 1×1 pixel convolution and the number of channels is adjusted to obtain local enhancement correction features.

[0066] S215 connects the correction features and the residuals of the local enhancement correction features to generate local-global features corresponding to the input image;

[0067] S216, using the local-global features as the input image, repeating S212 to S215, and combining it with the downsampling operation of convolution to obtain multi-scale local-global features corresponding to the input image.

[0068] It should be noted that the time and computational complexity of ordinary visual Transformers increase quadratically with the input spatial resolution, making them unsuitable for large spatial resolution images and hindering the extraction of local features. Convolution, on the other hand, has weak ability to extract long-range dependencies. Therefore, this invention utilizes a convolutional Transformer to extract features from large-scale remote sensing images. The convolutional Transformer combines the advantages of convolution in extracting local features and Transformer in extracting long-range dependencies, enabling the acquisition of both local and global features. The structural diagram of the two-stream multi-scale local-global feature extraction module in the two-stream convolutional-Transformer generator is shown below. Figure 3As shown, it includes two independent feature extraction branches. The upper branch is used to extract local-global features at four different scales from the PAN image (taking a width-height-channel number of 256×256×1 as an example), and the lower branch is used to extract features at four different scales from the ↑MS image (taking a width-height-channel number of 256×256×4 as an example). Figure 3 In These represent the four different scales of local-global features extracted from the PAN image. These represent four different scales of local-global features extracted from MS images, with dimensions of 256×256×32, 128×128×64, 64×64×128, and 32×32×256, respectively. Figure 3 As shown, the dual-stream feature extraction branch structures are similar but do not share weights. They are composed of 4 convolutional Transformer modules and 3 convolutions for downsampling, extracting low-level and high-level semantic information from different modal MS and PAN images. The implementation process is shown in Equations (1) and (2).

[0069] (1);

[0070] (2);

[0071] Here, PAN and ↑MS represent the full-resolution PAN image and the upsampled MS image, respectively. This represents the i-th scale feature extracted from the MS image. This represents the i-th scale feature extracted from the PAN image, where i = 1, 2, 3, 4. This represents a function from the Convolution-Transformer module. This represents the convolution function used for downsampling.

[0072] The Convolutional Transformer module includes layer normalization, multi-depth convolutional head transpose attention, local enhancement feedforward network, and residual connections. For example... Figure 3 and Figure 4As shown, taking the first layer of the model as an example, firstly, the input PAN image (↑MS image) is normalized by the layer to obtain the layer normalized feature RP (RM). Then, it is generated by aggregating cross-channel context information through 3-branch 1×1 pixel convolution and encoding channel-aggregated spatial context information through 3×3 depth convolution to generate Q (Query), K (Key), and V (Value). Then, Q, K, and V are deformed. The deformed Q and K matrices are multiplied and Softmaxed to obtain attention A. Attention A is then multiplied, deformed, and 1×1 pixel convolutioned with the deformed V and added to the input PAN image (↑MS image) to obtain the correction feature RF (that is, the layer normalized features RP and RM). The implementation of the above process is expressed as in Equation (3). After the correction features are normalized by layer, normalized correction features are generated. Then, the feature dimension is increased by 1×1 pixel convolution projection. Local information is obtained by 3×3 depth convolution. Then, nonlinear Gaussian error linear unit activation is used. Cross-channel local context information is aggregated and the number of channels is adjusted by 1×1 pixel convolution to extract useful local information. The theoretical expression of the above process is shown in Equation (4).

[0073] (3);

[0074] (4);

[0075] in, The input images for the convolutional Transformer module are represented by PAN and ↑MS images, respectively. I represents the input image. Image after layer normalization , and These represent the 1×1 cell convolutions used to generate Q, K, and V through projection, respectively. , and These represent the 3×3 depthwise convolutions used for projection to generate Q, K, and V, respectively. It is a transformation operation. These are learnable scaling parameters, and SA is the generated self-attention map. These are enhanced features after attention weighting. These are the corrected features after convolutional self-attention. It is a correction feature The corrected image after layer normalization, EF is the corrected image Features enhanced locally It is a 1×1 pixel convolution. It is a Gaussian error linear unit function. It is a 3×3 depthwise convolution; This represents the local-global features output by the convolutional Transformer module. and ).

[0076] In some embodiments, the prior cross-fusion of local-global features of MS and PAN images at the same scale includes:

[0077] S221, for the local-global features of each PAN image and the local-global features of the MS image at the same scale, the local-global features of the PAN image are generated as key vectors through layer normalization, 1×1 cell convolution, and 3×3 depth convolution projection. Sum value vector The local-global features of MS images are normalized by layers, and then projected by 1×1 pixel convolution and 3×3 depth convolution to generate query vectors. ;

[0078] S222, respectively for the query vector Key vector Sum value vector Transform the query vector. and key vector After multiplication and softmax operations, the cross-attention vector (CA) is obtained. The cross-attention CA is then combined with the transformed value vector. After multiplication, deformation, and 1×1 pixel convolution, the cross-fusion feature is obtained by adding it to the local-global features of the PAN image.

[0079] S223, after the cross-fusion feature is normalized by the layer, the normalized cross-fusion feature is generated. Then, the feature dimension is increased by 1×1 pixel convolution projection. Local information is obtained by 3×3 depth convolution. Then, non-linear Gaussian error linear unit activation is used. Cross-channel local context information is aggregated by 1×1 pixel convolution and the number of channels is adjusted to obtain the locally enhanced cross-fusion feature.

[0080] S224 connects the cross-fusion feature and the residual of the locally enhanced cross-fusion feature to generate the prior cross-fusion result at this scale.

[0081] Specifically, based on the features of MS and PAN images at four scales generated during the dual-stream multi-scale local-global feature extraction stage, a recursive cascaded cross-fusion network is constructed, such as... Figure 5 As shown, the recursive cascaded cross-fusion network includes a priori cross-fusion module and a cross-scale fusion module. Among them... Figure 5 The structure of the prior cross-fusion module in the middle is as follows: Figure 6 As shown in (a), the structure of the cross-scale fusion module is as follows: Figure 6 As shown in (b). The scale conversion module in the cross-scale fusion module is as follows: Figure 7 As shown.

[0082] For inputs of MS and PAN images at different scales, a priori cross-fusion module is constructed to aggregate information from MS and PAN images across different modalities. This priori cross-fusion module facilitates the learning of multimodal information and initial feature fusion from MS and PAN images. Priori cross-fusion utilizes information from different modalities... and Feature-based cross-convolutional attention is used to aggregate modality-specific and complementary information, such as Figure 6 As shown in (a), the prior cross-fusion module consists of layer normalization, a multi-cross convolutional head transposed attention network, a local enhancement feedforward network, and residual connections. First, Generated by layer normalization, 1×1 cell convolution and 3×3 depth convolution projection , , Generated by layer normalization, 1×1 cell convolution and 3×3 depth convolution projection Then... , , To deform, to deform , The matrix is ​​multiplied and then subjected to a softmax operation to obtain cross attention (CA). CA is then combined with the transformed matrix. After multiplication, transformation, and 1×1 pixel convolution, the cross-fusion feature PF is obtained by adding it to the input. The implementation of the above process is expressed as Equation (5). The cross-fusion feature is then normalized by layer to generate normalized cross-fusion feature, which is then projected by 1×1 pixel convolution to increase the feature dimension. Local information is obtained by 3×3 depth convolution. Then, nonlinear Gaussian error linear unit activation is used, and cross-channel local context information is aggregated and the number of channels is adjusted by 1×1 pixel convolution to enhance useful local information. The theoretical expression of the specific implementation of the above process is expressed as Equation (6).

[0083] (5);

[0084] (6);

[0085] in, and Indicates the first Scale features and Normalized features These represent the components used for projection generation. , , 1×1 cell convolution, These represent the components used for projection generation. , , A 3×3 depthwise convolution, where CA is the generated cross-attention. It is a cross-enhanced feature after cross-attention weighting. These are cross-fusion features after cross-convolutional attention. It is a cross-fusion feature Image after layer normalization yes Features enhanced locally This indicates the prior cross-fusion result output by the prior cross-fusion module.

[0086] In some embodiments, the cross-scale fusion of prior cross-fusion results at different scales, forming a recursive cascaded cross-fusion structure to obtain key fusion information, includes:

[0087] S225, for each pair of scale-adjacent fine-scale prior cross-fusion results and coarse-scale prior cross-fusion results, the fine-scale prior cross-fusion results are downsampled to generate a downsampled image of the same scale as the coarse-scale prior cross-fusion results; the coarse-scale prior cross-fusion results are normalized by layers, and a query vector is generated by 1×1 pixel convolution and 3×3 depth convolution projection. The downsampled image is normalized by layers, and then projected by 1×1 pixel convolution and 3×3 depth convolution to generate a key vector. Sum value vector ;

[0088] S226, respectively for the query vector Key vector Sum value vector Transform the query vector. and key vector After multiplication and softmax operations, it is then compared with the transformed value vector. After multiplication, deformation, and 1×1 pixel convolution, the cross-scale cross-fusion feature is obtained by adding it to the downsampled image.

[0089] S227, the cross-scale cross-fusion features are then normalized by layers and a local enhancement feedforward network to obtain locally enhanced cross-scale fusion features, which are then connected with the cross-scale cross-fusion feature residuals to generate cross-scale fusion results of the same scale as the coarse-scale prior cross-fusion results;

[0090] S228, following the order from fine scale to coarse scale, the cross-scale fusion results of adjacent scales are recursively cascaded and fused step by step according to S225 to S227 to obtain key fusion information.

[0091] Specifically, the cross-scale fusion module consists of a scale conversion module and a cross-fusion module, such as... Figure 6As shown in (b). In the cross-scale fusion module Represents a fine-scale image. Represents a coarse-scale image. A fine-scale image. The generation is achieved through downsampling processing in the scale conversion module. Images of the same scale are then fused. Cross-scale fusion generates a total of three scale fusion results, namely... .

[0092] like Figure 5 In the middle, first extract the same scale and Feature cross-fusion generation The cross-scale fusion module then cross-fused the prior results. Cross-fusion results with the next scale As input, it is fused to generate and Features of the same scale Cross-scale information fusion is achieved sequentially in the manner described above, with recursive cascading fusion proceeding from large scale to small scale to generate key fused information. The entire fusion network constitutes a recursive cascaded cross-fusion network. This network utilizes information from MS and PAN images of different modalities and scales, reducing information loss in both PAN and MS images and thus improving spatial resolution.

[0093] The prior cross-fusion module expression can be represented as (7):

[0094] (7);

[0095] in, Indicates by and The results of prior cross-fusion It is a priori cross-fusion function.

[0096] The function of the cross-scale feature fusion module is expression (8):

[0097] (8);

[0098] in, Indicates the cross-scale fusion results. This represents a function that achieves cross-scale feature fusion using cross-convolutional attention.

[0099] In some embodiments, the step of reconstructing the key fusion information based on supplementary information to generate injection information includes:

[0100] S231, the key fusion information is used as the input information of the dual-stream multi-scale reconstruction network. The dual-stream multi-scale reconstruction network includes a reconstruction module and two branches. The ends of the reconstruction module are respectively connected to the two branches. Each of the two branches includes multiple cascaded reconstruction modules.

[0101] S232, for this reconstruction module, the current scale of the information to be reconstructed is determined, the supplementary information corresponding to the current scale is obtained, the supplementary information is converted to the current scale, and then concatenated with the input information, followed by 3×3 convolution and a convolutional Transformer module to generate the reconstruction features at the current scale; the supplementary information includes key fusion information, cross-scale fusion results at the current scale, prior cross-fusion results at the current scale, and prior cross-fusion results at a finer scale;

[0102] S233, input the reconstruction features of the current scale into the two branches of the two-stream multi-scale reconstruction network respectively;

[0103] S234, S232 is executed sequentially for multiple cascaded reconstruction modules in the two branches. The reconstruction features of the current scale are used as the input information of the reconstruction module of the next scale. The reconstruction features corresponding to each reconstruction module in the two branches are generated respectively. The lower-level reconstruction features of one branch are reused to the higher-level reconstruction features of the other branch through skip connections to generate injection information.

[0104] To obtain more accurate injection information, a dual-stream multi-scale reconstruction network was designed to fuse key information. Refactor, such as Figure 8 As shown. To reduce information loss during the downsampling process, an information supplementation module (containing key fusion information in the cross-scale fusion stage) was added during information reconstruction at each scale. (The fusion results at the same scale and finer scale as the reconstructed information) are used as input, such as... Figure 8 The red arrows in the diagram indicate supplementary information. Multi-scale information is progressively processed by the multi-scale reconstruction module to generate the final injected information. Multi-scale reconstruction modules, such as Figure 9 As shown, compared to the scale of the reconstructed information, H represents finer-scale information, S represents information at the same scale, and L represents coarser-scale information. These are the corresponding reconstructed features. Before reconstruction, multi-scale information needs to be converted into information of the same scale through a scale transformation module. The scale transformation method is as follows: Figure 7 As shown, coarse-scale information is transformed into fine-scale information through deconvolution, and fine-scale information is transformed back into coarse-scale information through downsampling. Then, multi-scale information reconstruction is achieved through concatenation, 3×3 convolution, and a convolutional Transformer module. The reconstructed features of the upper branch are shown below. for , for The reconstruction provides low-level semantic information. The multi-scale reconstruction structure reuses the extracted low-level semantic features in the reconstruction stage through multi-scale skip connections. The low-level semantic features contain rich detailed features, such as edges and contours, which can reduce the loss of details.

[0105] In some embodiments, the step of performing nonlinear correction on the upsampled MS image and the injected information to generate a high spatial resolution multispectral image corresponding to the remote sensing image includes:

[0106] After stitching the injected information and the upsampled MS image together, a nonlinear mapping is performed sequentially through 1×1 convolution and 3×3 convolution to generate a high spatial resolution multispectral image.

[0107] Because the spectral ranges of PAN and MS images do not completely overlap, and their relationship is not linear, if the detail injection model directly adds the injected details to the upsampled MS image, the nonlinear relationship between the PAN and MS image bands will be ignored during detail injection, resulting in spectral distortion. Therefore, this invention, based on the injection information obtained in the multi-scale reconstruction stage and the nonlinear relationship between the MS and PAN images, constructs an injection information correction module using stitching operations and two convolutional layers, and utilizes the MS image and injection information... The structure of the nonlinear mapping-generated HMS image and the injected information correction module is as follows: Figure 8 The convolution (1×1) has a kernel size of 1 and 12 kernels, followed by an LReLU function; the convolution (3×3) has a kernel size of 3 and 4 kernels. This injected information correction module considers the nonlinear relationship between the MS and PAN images in the bands, reducing spectral information distortion and spatial structure warping.

[0108] The expression for the spatial spectrum fusion model generator of this invention is (9):

[0109] (9);

[0110] HMS imagery is a generated high spatial resolution multispectral image. These are the parameters of the generated network. It is a two-stream convolution-Transformer generator function.

[0111] In some embodiments, the spatial-spectral fusion model is trained in the following manner:

[0112] S101, Obtain a sample set containing multiple satellite remote sensing image samples, wherein the satellite remote sensing image samples include MS images and PAN images;

[0113] S102, Establish a dual-stream convolutional-transformer generator and a convolutional-transformer dual discriminator consistent with the spatial-spectral fusion model architecture, input the sample set into the spatial-spectral fusion model for training, and process the remote sensing image samples through the dual-stream convolutional-transformer generator to generate HMS images.

[0114] S103, the spectral and detail information of the HMS image is discriminated by the convolution-Transformer dual discriminator to obtain the adversarial loss of the two-stream convolution-Transformer generator and the convolution-Transformer dual discriminator; the composite loss of the spatial-spectral fusion model is obtained by combining the adversarial loss, the no-reference loss, the spatial consistency loss and the spectral consistency loss; the spatial-spectral fusion model is iteratively trained based on the composite loss until the optimization objective of the composite loss is met, and the trained spatial-spectral fusion model is obtained.

[0115] To improve the performance and stability of the spatial-spectral fusion model, a relative average discriminator is used to determine the relative probability of the generated image and the real image, and then optimized using a least squares function, namely the Relative Average Least Squares Discriminator (RaLSD). This invention utilizes a convolutional-transformer architecture to enhance feature representation capabilities; the network structure of the convolutional-transformer dual discriminator is as follows... Figure 10 As shown, the discriminator is composed of spectral information convolution - Transformer. Convolution with detailed information - Transformer discriminator constitute, and The discriminators are similar in structure, but because PAN images have a higher resolution than MS images, The number of downsampling units is reduced. For The discriminator uses a spectral dimension down-resolution image of the original PAN image or an HMS image. As input, a relatively realistic probability is generated. First, a 3×3 kernel convolution and a convolution-transformer module are used to extract primary local-global features. Then, three 4×4 kernel convolutions with a stride of 2 and a convolution-transformer module are used to extract deeper features, and the feature size is reduced sequentially. Finally, two 4×4 kernel convolutions with a stride of 1 and a sigmoid function are used to output the relatively realistic probability.

[0116] for The discriminator uses the spatial dimensions of the original MS image or a down-resolution image of the HMS image. As input, a relatively realistic probability is generated. First, a 3×3 kernel convolution and a convolution-transformer module are used to extract primary local-global features. Then, two 4×4 kernel convolutions with a stride of 2 and a convolution-transformer module are used to extract deeper features, progressively reducing the feature size. Finally, two 4×4 kernel convolutions with a stride of 1 and a sigmoid function are used to output the relatively realistic probability. Simultaneously, to accelerate model convergence and improve stability and performance, a spectral normalization operation is introduced after each convolution operation.

[0117] The mathematical expressions for CT-RaLSD are shown in equations (10) and (11).

[0118] (10);

[0119] (11);

[0120] in, This represents the relative probability of the CT-RaLSD output. This represents the Sigmoid activation function. This represents the output of the untransformed discriminator. Representing real data Distribution of (MS or PAN images) Represents forecast data ( or The distribution of ) and This indicates an operation that takes the average of all real and merged data in a batch of data.

[0121] To optimize the spatial-spectral fusion model and improve its stability, a composite loss function for unsupervised learning was constructed, consisting of a no-reference loss function, a spatial consistency loss function, a spectral consistency loss function, and two adversarial loss functions.

[0122] The loss function for spatial consistency is shown in equation (12):

[0123] (12);

[0124] This represents the spatial consistency loss function, where M is the batch size. It is the first generated by the spatial-spectral fusion model A high spatial resolution multispectral image. and This indicates that high-pass filtering and gradient operations are used to extract high-frequency and gradient information from an image.

[0125] In order to generate The spectral information is consistent with that of the original MS image, and the spectral loss function is as shown in Equation (13).

[0126] (13);

[0127] Represents the spectral consistency loss function. This indicates a resolution reduction operation, including blurring and downsampling. It is the original multispectral image.

[0128] Since there is no reference image, the no-reference image metric QNR is used to measure the quality of the generated image, and the no-reference loss function is used. The expression is as shown in equation (14).

[0129] (14);

[0130] QNR is related to the spectral loss index and the spatial loss index, and the expression is (15).

[0131] (15);

[0132] in The spectral loss index is expressed as (16). The space loss index is expressed as formula (17), where t and v are constants, typically 1.

[0133] (16);

[0134] Where B is the band number, It consists of a low-resolution MS image from the nth band and a fused MS image. It is the low-resolution MS image of the y-th band and the fused MS image. Q is the image quality index, and its expression is (17).

[0135] (17);

[0136] in, It is a low-resolution version of the PAN image.

[0137] (18);

[0138] Where h and k are inputs, This represents the covariance between h and k. and This represents the variances of h and k. and This represents the mean of h and k.

[0139] To improve the performance and stability of the spatial-spectral fusion model, a relative average least squares loss function is used to optimize the adversarial network and the generator's adversarial loss. It consists of two parts: the generator and... and The adversarial loss of the discriminator is expressed as in formula (19).

[0140] (19);

[0141] in, Indicates and The adversarial loss function of the discriminator Indicates and The adversarial loss function of the discriminator.

[0142] and The expressions are as shown in formulas (20) and (21).

[0143] (20);

[0144] (twenty one);

[0145] in, This indicates a spatially down-resolution version of the HMS image, i.e. Figure 10 In ; This indicates a spectral down-resolution version of the HMS image, i.e. Figure 10 In . This indicates an operation to average the actual MS batch data. Indicates the integration Batch data averaging operation This indicates an operation to average the actual batch PAN data. Indicates the integration Average value operation for batch data.

[0146] Discriminator and The relative average least squares loss function is expressed as in formulas (22) and (23).

[0147] (twenty two);

[0148] (twenty three);

[0149] In some embodiments, the composite loss of the spatial-spectral fusion model is calculated using the following formula:

[0150] (twenty four);

[0151] in, It is a composite loss function. This represents the adversarial loss function between the generator and the spectral information discriminator. This represents the adversarial loss function between the generator and the detail information discriminator. Indicates the no-reference loss function. Represents the spatial consistency loss function. Represents the spectral consistency loss function. These are the weighting coefficients for each loss.

[0152] To verify the effectiveness of this invention, the following experiments were conducted:

[0153] Experimental setup:

[0154] The deep learning-based contrastive model implementation framework uses PyTorch and Tensorflow, while the traditional method is implemented in MATLAB. The workstation equipment includes two NVIDIA GeForce RTX 4090 GPUs with 24 GiB of video memory. The image patch size for training the network is 256×256×C, the batch size is 8, and the training epochs are 40. Based on experience and experimental testing, the hyperparameters in the loss function are set to... , and The model is optimized using Adam with an initial learning rate of... To more fairly compare the spatial-spectral fusion performance of the various comparison methods, experiments based on deep learning models were implemented on GPUs, while experiments based on traditional models were implemented on CPUs.

[0155] Comparative experiment:

[0156] To verify the effectiveness and fusion effect of the proposed method, comparative experiments were conducted with mainstream methods on both down-resolution and full-resolution test sets. The experimental results were evaluated subjectively using visual metrics, and objectively using reference metrics Q4, Q, SCC, SSIM, SAM, and ERGAS to evaluate the down-resolution experiment. No reference evaluation metrics were also used. , The results of the full-resolution experiments were objectively evaluated using QNR. The comparison methods included mainstream traditional methods and GAN-based methods. The traditional methods compared were GSA, BDSD, SFIM, and MTF-GLP, while the GAN-based methods compared were RED-cGAN, PsGAN, PGMAN, and DIGAN. Comparative experiments were conducted on down-resolution and full-resolution data from Gaofen-2 and QuickBird.

[0157] For Gaofen-2 data, the spatial resolution of the panchromatic image is 1m, and the spatial resolution of the multispectral image is 4m, with a resolution ratio of 4:1. The radiometric resolution is 10 bits. By performing spatial-spectral fusion on the original panchromatic and multispectral remote sensing images, a multispectral image with a spatial resolution of 1m is generated, improving the accuracy of surface monitoring. For QuickBird data, the spatial resolution of the panchromatic image is 0.6m, and the spatial resolution of the multispectral image is 2.4m, with a resolution ratio of 4:1. The radiometric resolution is 11 bits. By performing spatial-spectral fusion on the original panchromatic and multispectral remote sensing images, a multispectral image with a spatial resolution of 0.6m is generated, improving the accuracy of surface monitoring. The mean values ​​of the quantitative evaluation of the experimental results on the Gaofen-2 down-resolution data are shown in Table 1, and the mean values ​​of the quantitative evaluation of the experimental results on the QuickBird full-resolution data are shown in Table 2. From Tables 1 and 2, it can be observed that the method proposed in this invention performs better than the method in terms of reference evaluation metrics Q4, Q, SCC, SSIM, SAM, ERGAS, and no reference evaluation metrics. , It is superior in both QNR and QNR, indicating better integration capabilities.

[0158] Table 1: Mean values ​​of the quantitative evaluation of experimental results for Gaofen-2 down-resolution data

[0159]

[0160] Table 2: Mean values ​​of quantitative evaluation of QuickBird full-resolution experimental results

[0161]

[0162] in conclusion:

[0163] This invention proposes an unsupervised convolutional-transformer generative adversarial spatial-spectral fusion method to improve spatial resolution and reduce spectral information distortion. The model consists of a convolutional-transformer generator and two convolutional-transformer discriminators. The convolutional-transformer generator acquires spatial structure and spectral information at different resolutions through a two-stream multi-scale local-global feature extraction branch, constructing a recursive hybrid-scale feature fusion sub-network with cross-attention. Prior cross-fusion is performed on the extracted local-global features of the MS and PAN at the same scale. The fine-scale prior cross-fusion result is then fused with the coarse-scale prior cross-fusion result to generate coarse-resolution features. This process is repeated to form a recursive cascaded cross-fusion structure, ultimately generating key fused information. A supplementary information structure is used to reconstruct the key fused information to compensate for information loss caused by downsampling. An injection information correction module is constructed to address distortion caused by ignoring the nonlinear relationship between the bands of the MS and PAN images. A convolutional-transformer dual discriminator is constructed, and an unsupervised composite loss function is defined to optimize the objective. Validation results of the proposed method on Gaofen-2 and QuickBird satellite data demonstrate that, in terms of both visual evaluation and objective indicators, the results of this invention are superior to current mainstream methods. The proposed method exhibits superior performance in improving spatial resolution and preserving spectral information, thus enhancing fusion quality and providing more accurate remote sensing data for surface monitoring.

[0164] Compared with related technologies, the present invention has at least the following improvements:

[0165] A dual-stream, multi-scale local-global feature extraction branch was designed. A convolutional-transformer module was used to extract low-level and high-level semantic information from MS and PAN images, representing spatial structure and spectral information at different modalities and resolutions.

[0166] A recursive cascaded cross-fusion network was designed. Prior cross-fusion was performed on the local-global features of the extracted MS and PAN images at the same scale. The prior cross-fusion result at the fine scale was then fused with the prior cross-fusion result at the coarse scale to generate coarse-resolution features. The above steps were repeated to form a recursive cascaded cross-fusion structure to generate key fusion information.

[0167] A reconstruction module with supplementary information was constructed to generate injected information, which was then injected into the MS image via a correction module. The reconstruction of information at each scale used the reconstruction features of the previous scale and supplementary information as input to reduce spectral distortion and loss of detail.

[0168] A convolutional-Transformer dual discriminator was designed and an unsupervised composite loss function was defined to better optimize the model, learn adversarial details, and acquire spectral information.

[0169] and Figure 1 The corresponding method is referenced. Figure 11 This invention provides an unsupervised generative adversarial remote sensing image spatial-spectral fusion system, comprising:

[0170] At least one processor;

[0171] At least one memory for storing at least one program;

[0172] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0173] It is evident that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0174] Furthermore, embodiments of the present invention also disclose a computer program product or computer program stored in a computer-readable storage medium. A processor of a computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the described method. Similarly, the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0175] It will be understood by those skilled in the art that all or some of the methods and systems disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0176] The above is a detailed description of the preferred embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. An unsupervised generative adversarial remote sensing image spatial-spectral fusion method, characterized in that, The method includes the following steps: S100, acquire the remote sensing images to be fused and the trained spatial-spectral fusion model; the remote sensing images include MS images and PAN images, and the spatial-spectral fusion model is trained based on a two-stream convolutional-Transformer generator and a convolutional-Transformer dual discriminator. S200: Input the remote sensing images to be fused into the trained spatial-spectral fusion model. Extract multi-scale local-global features from MS and PAN images using the dual-stream convolution-Transformer generator in the spatial-spectral fusion model. First, perform prior cross-fusion on the same-scale local-global features of MS and PAN images. Then, perform cross-scale fusion on the prior cross-fusion results of different scales to form a recursive cascaded cross-fusion structure to obtain key fusion information. Based on the supplementary information, reconstruct the key fusion information to generate injection information. Perform nonlinear correction on the upsampled MS image and the injection information to generate a high spatial resolution multispectral image corresponding to the remote sensing image. The dual-stream convolutional-Transformer generator includes a dual-stream multi-scale local-global feature extraction module; the dual-stream multi-scale local-global feature extraction module includes two weighted independent feature extraction branches, and the extraction of multi-scale local-global features from MS images and PAN images through the dual-stream convolutional-Transformer generator in the spatial-spectral fusion model includes: S211, take the PAN image and the upsampled MS image as input images, input the PAN image into one feature extraction branch, and input the upsampled MS image into another feature extraction branch; S212, after the input image is normalized by layer, the normalized features are obtained. The normalized features are then aggregated by 3-branch 1×1 pixel convolution to aggregate cross-channel context information and 3×3 depth convolution to encode channel space aggregated context information, generating query vector Q, key vector K and value vector V. S213, the query vector Q, key vector K and value vector V are deformed respectively. The deformed query vector Q and key vector K are multiplied and Softmaxed in sequence to obtain attention A. Attention A is then multiplied, deformed and 1×1 pixel convolved with the deformed value vector V and added to the input image to obtain the correction feature. S214, after the correction features are normalized by the layer, normalized correction features are generated. Then, the feature dimension is increased by 1×1 pixel convolution projection. Local information is obtained by 3×3 depth convolution. Then, after activation by nonlinear Gaussian error linear unit, cross-channel local context information is aggregated by 1×1 pixel convolution and the number of channels is adjusted to obtain local enhancement correction features. S215 connects the correction features and the residuals of the local enhancement correction features to generate local-global features corresponding to the input image; S216, using the local-global features as the input image, repeating S212 to S215, and combining it with the downsampling operation of convolution to obtain multi-scale local-global features corresponding to the input image.

2. The method according to claim 1, characterized in that, The prior cross-fusion of local-global features of the same scale from MS and PAN images includes: S221, for each local-global feature of the PAN image and the local-global feature of the MS image at the same scale, the local-global feature of the PAN image is generated as a key vector through layer normalization, 1×1 cell convolution and 3×3 depth convolution projection. Sum value vector The local-global features of MS images are normalized by layers, and then projected by 1×1 pixel convolution and 3×3 depth convolution to generate query vectors. ; S222, respectively for the query vector Key vector Sum value vector The vector is transformed, and after multiplication and softmax operations, the cross-attention vector (CA) is obtained. The cross-attention CA is then compared with the transformed value vector. After multiplication, deformation, and 1×1 pixel convolution, the cross-fusion feature is obtained by adding it to the local-global features of the PAN image. S223, after the cross-fusion feature is normalized by the layer, the normalized cross-fusion feature is generated. Then, the feature dimension is increased by 1×1 pixel convolution projection. Local information is obtained by 3×3 depth convolution. Then, non-linear Gaussian error linear unit activation is used. Cross-channel local context information is aggregated by 1×1 pixel convolution and the number of channels is adjusted to obtain the locally enhanced cross-fusion feature. S224 connects the cross-fusion feature and the residual of the locally enhanced cross-fusion feature to generate the prior cross-fusion result at this scale.

3. The method according to claim 2, characterized in that, The process involves fusing prior cross-fusion results from different scales across scales to form a recursive cascaded cross-fusion structure, yielding key fusion information, including: S225, for each pair of scale-adjacent fine-scale prior cross-fusion results and coarse-scale prior cross-fusion results, the fine-scale prior cross-fusion results are downsampled to generate a downsampled image of the same scale as the coarse-scale prior cross-fusion results; the coarse-scale prior cross-fusion results are normalized by layers, and a query vector is generated by 1×1 pixel convolution and 3×3 depth convolution projection. The downsampled image is normalized by layers, and then projected by 1×1 pixel convolution and 3×3 depth convolution to generate a key vector. Sum value vector ; S226, respectively for the query vector Key vector Sum value vector Transform the query vector. and key vector After multiplication and softmax operations, it is then compared with the transformed value vector. After multiplication, deformation, and 1×1 pixel convolution, the cross-scale cross-fusion feature is obtained by adding it to the downsampled image. S227, the cross-scale cross-fusion features are then normalized by layers and a local enhancement feedforward network to obtain locally enhanced cross-scale fusion features, which are then connected with the cross-scale cross-fusion feature residuals to generate cross-scale fusion results of the same scale as the coarse-scale prior cross-fusion results; S228, following the order from fine scale to coarse scale, the cross-scale fusion results of adjacent scales are recursively cascaded and fused step by step according to S225 to S227 to obtain key fusion information.

4. The method according to claim 1, characterized in that, The process of reconstructing the key fusion information based on supplementary information to generate injected information includes: S231, the key fusion information is used as the input information of the dual-stream multi-scale reconstruction network. The dual-stream multi-scale reconstruction network includes a reconstruction module and two branches. The ends of the reconstruction module are respectively connected to the two branches. Each of the two branches includes multiple cascaded reconstruction modules. S232, for this reconstruction module, the current scale of the information to be reconstructed is determined, the supplementary information corresponding to the current scale is obtained, the supplementary information is converted to the current scale, and then concatenated with the input information, followed by 3×3 convolution and a convolutional Transformer module to generate the reconstruction features at the current scale; the supplementary information includes key fusion information, cross-scale fusion results at the current scale, prior cross-fusion results at the current scale, and prior cross-fusion results at a finer scale; S233, input the reconstruction features of the current scale into the two branches of the two-stream multi-scale reconstruction network respectively; S234, S232 is executed sequentially for multiple cascaded reconstruction modules in the two branches. The reconstruction features of the current scale are used as the input information of the reconstruction module of the next scale. The reconstruction features corresponding to each reconstruction module in the two branches are generated respectively. The lower-level reconstruction features of one branch are reused to the higher-level reconstruction features of the other branch through skip connections to generate injection information.

5. The method according to claim 1, characterized in that, The step of performing nonlinear correction on the upsampled MS image and injected information to generate a high spatial resolution multispectral image corresponding to the remote sensing image includes: After stitching the injected information and the upsampled MS image together, a nonlinear mapping is performed sequentially through 1×1 convolution and 3×3 convolution to generate a high spatial resolution multispectral image.

6. The method according to claim 1, characterized in that, The spatial-spectral fusion model is trained in the following manner: S101, Obtain a sample set containing multiple satellite remote sensing image samples, wherein the satellite remote sensing image samples include MS images and PAN images; S102, establish a dual-stream convolutional-transformer generator and a convolutional-transformer dual discriminator consistent with the spatial-spectral fusion model architecture, input the sample set into the spatial-spectral fusion model for training, and process the remote sensing image samples through the dual-stream convolutional-transformer generator to generate HMS images. S103, the spectral and detail information of the HMS image is discriminated by the convolution-Transformer dual discriminator to obtain the adversarial loss of the two-stream convolution-Transformer generator and the convolution-Transformer dual discriminator; the composite loss of the spatial-spectral fusion model is obtained by combining the adversarial loss, the no-reference loss, the spatial consistency loss and the spectral consistency loss; the spatial-spectral fusion model is iteratively trained based on the composite loss until the optimization objective of the composite loss is met, and the trained spatial-spectral fusion model is obtained.

7. An unsupervised generative adversarial remote sensing image spatial-spectral fusion system, characterized in that, The system includes: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1 to 6.

Images