Spatio-Spectral Fusion Method for Unpaired Hyperspectral and SAR Images with Dual-Domain Alignment

Abstract

The present invention relates to a spatio-spectral fusion method for non-paired hyperspectral and SAR images with dual-domain alignment, including: upsampling the hyperspectral image and performing feature extraction, performing Gaussian blur and feature extraction on the SAR image; approximating the radiation distribution of the SAR features to the radiation distribution features of the hyperspectral image; adjusting the spatial geometric information of the SAR features; extracting the change information between the image features, and predicting the low-spatial-resolution hyperspectral image through information injection; fusing the low-spatial-resolution hyperspectral image and the original SAR image after alignment to obtain the final high-spatial-resolution hyperspectral image. The beneficial effects of the present invention are: the present invention improves the spatio-spectral fusion accuracy of hyperspectral and SAR images, providing reliable support for subsequent applications.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and more specifically, to a spatio-temporal-spectral fusion method for non-paired hyperspectral and SAR images with dual-domain alignment. Background Art

[0002] The spectral coverage range of hyperspectral sensors is usually between 400 - 2500 nm, which can obtain more than 100 bands and has fine spectral resolution. Benefiting from the superior target recognition ability, hyperspectral data is widely used in environmental monitoring, resource exploration, agricultural monitoring, etc. However, limited surface energy is captured by narrow spectral channels, and hyperspectral data has low spatial resolution and low temporal resolution, and the satellite revisit period is usually greater than 30 days. Multispectral and SAR satellite data provide important auxiliary information for improving the resolution of hyperspectral data, and the temporal and spatial resolution of hyperspectral images can be effectively improved through image fusion. Currently, the methods for enhancing the resolution of hyperspectral images with auxiliary data can be roughly divided into optical auxiliary methods and SAR auxiliary methods.

[0003] Optical auxiliary methods mainly use multispectral data to improve the resolution of hyperspectral data, and spatio-temporal-spectral fusion aims to use hyperspectral and multispectral data at different times to predict the high-spatial-resolution hyperspectral data at the missing time. These methods can be divided into iterative optimization methods and deep learning methods. Among them, iterative optimization methods usually construct the spatio-temporal-spectral conversion relationship between images based on the variational framework and obtain the fused image through regular constraints. However, these methods need to construct a large number of constraint terms, with a large number of parameters and low robustness of model performance. Deep learning fusion methods construct a nonlinear mapping mechanism between auxiliary information and predicted values and obtain the final fusion parameters through continuous optimization. These methods can effectively handle the nonlinear changes existing between images. In addition, SAR auxiliary methods mainly reconstruct optical images by establishing a conversion mechanism between images. These methods are mainly used for reconstructing multispectral data and have not been used for reconstructing hyperspectral images.

[0004] However, 1) optical images are easily affected by weather factors. When the images are contaminated by clouds and fog, etc., optical auxiliary methods cannot effectively reconstruct hyperspectral images. 2) Compared with optical images, SAR is not affected by weather factors. However, there are obvious differences in the designs of optical and SAR sensors, and the imaging angles and mechanisms are different. There are obvious radiometric differences and spatial geometric differences between SAR images and optical images. 3) Spatio-temporal-spectral fusion methods can establish more prior constraints compared with image translation methods, and the image reconstruction accuracy is higher. However, there is currently no spatio-temporal-spectral fusion method for hyperspectral and SAR images. Summary of the Invention

[0005] The object of the present invention is to propose a spatio-temporal spectral fusion method for non-paired hyperspectral and SAR images with dual-domain alignment in view of the deficiencies of the prior art.

[0006] In a first aspect, a spatio-temporal spectral fusion method for non-paired hyperspectral and SAR images with dual-domain alignment is provided, including:

[0007] S1. Obtain the hyperspectral image at time T1 and perform spatial upsampling on the hyperspectral image; obtain the SAR image at time T2 and perform Gaussian blur processing on the SAR image;

[0008] S2. Construct a feature extraction network to extract features from the spatially upsampled hyperspectral image at time T1 and the Gaussian-blurred SAR image at time T2, and obtain the hyperspectral features at time T1 and the SAR features at time T2;

[0009] S3. Construct a radiation domain alignment network, use the hyperspectral features at time T1 as the style image, use the SAR features at time T2 as the content image, and continuously approximate the radiation distribution of the SAR features to the radiation distribution of the hyperspectral features in the convolution operation;

[0010] S4. Construct a spatial domain alignment network, calculate the affine matrix of the hyperspectral features at time T1 and the SAR features at time T2 after radiation alignment, and adjust the spatial geometric information of the SAR features through grid construction and sampler interpolation;

[0011] S5. Construct a change reconstruction network, extract the change information between the hyperspectral features at time T1 and the SAR features after radiation-spatial alignment at time T2, and inject the change information into the hyperspectral features to predict the low-spatial-resolution hyperspectral image at time T2;

[0012] S6. Extract parameters from the SAR features at time T2 after Gaussian blur processing, share the radiation-spatial domain alignment parameters with the original SAR image at time T2, and construct a spatial reconstruction network to fuse the low-spatial-resolution hyperspectral image at time T2 and the original SAR image after parameter sharing to obtain the high-spatial-resolution hyperspectral image at time T2.

[0013] Preferably, it further includes:

[0014] S7. Construct a radiation domain alignment loss function, a spatial domain alignment loss function, and a root mean square error loss function to jointly optimize and train the parameters of the feature extraction network, the radiation domain alignment network, the spatial domain alignment network, the change reconstruction network, and the spatial reconstruction network.

[0015] Preferably, in S4, the spatial domain alignment network includes a position network, a grid generator, and a sampler;

[0016] The position network is used to output parameters defining a spatial transformation; the grid generator is used to receive the transformation parameters predicted by the positioning network and generate a coordinate grid; the sampler is used to sample pixel values from the input image according to the coordinate grid provided by the grid generator to generate a transformed output image.

[0017] Preferably, in S4, a convolutional operation is used to calculate the affine matrix of the hyperspectral features at time T1 and the SAR features at time T2 after radiation alignment.

[0018] Preferably, in S5, a gating operation is used to extract the change information between the hyperspectral image features at time T1 and the SAR features at time T2 after radiation-spatial alignment; the gating operation is constructed by a Mamba structure.

[0019] Preferably, in S6, the spatial reconstruction network consists of a first branch and a second branch. The first branch combines an empty-spectrum convolutional block and a Mamba block, and the second branch combines a texture feature convolutional block and a Mamba block; there is a feature interaction branch between the empty-spectrum convolutional block and the texture feature convolutional block.

[0020] In a second aspect, a spatio-temporal spectral fusion system for dual-domain alignment of unpaired hyperspectral and SAR images is provided for performing any of the methods in the first aspect, including:

[0021] An acquisition module, configured to acquire the hyperspectral image at time T1 and perform spatial upsampling processing on the hyperspectral image; acquire the SAR image at time T2 and perform Gaussian blur processing on the SAR image;

[0022] A first construction module, configured to construct a feature extraction network, extract features from the spatially upsampled hyperspectral image at time T1 and the Gaussian-blurred SAR image at time T2, and obtain the hyperspectral features at time T1 and the SAR features at time T2;

[0023] A second construction module, configured to construct a radiation domain alignment network, use the hyperspectral features at time T1 as the style image and the SAR features at time T2 as the content image, and continuously approximate the radiation distribution of the SAR features to the radiation distribution of the hyperspectral features in the convolutional operation;

[0024] A third construction module, configured to construct a spatial domain alignment network, calculate the affine matrix of the hyperspectral features at time T1 and the SAR features at time T2 after radiation alignment, and adjust the spatial geometric information of the SAR features through grid construction and sampler interpolation;

[0025] The fourth construction module is used to construct a change reconstruction network, extract the change information between the hyperspectral features at time T1 and the SAR features after radiation-spatial alignment at time T2, and inject the change information into the hyperspectral features to predict the low-spatial-resolution hyperspectral image at time T2.

[0026] The fifth construction module is used to extract parameters from the SAR features of time T2 after Gaussian blur processing, share the radiation-spatial domain alignment parameters with the original SAR image at time T2, and construct a spatial reconstruction network to fuse the low-spatial-resolution hyperspectral image at time T2 and the original SAR image after parameter sharing to obtain the high-spatial-resolution hyperspectral image at time T2.

[0027] In a third aspect, a computer storage medium is provided. The computer storage medium stores a computer program. When the computer program runs on a computer, the computer is enabled to execute the method according to any one of the first aspects.

[0028] In a fourth aspect, an electronic device is provided, which is characterized by including:

[0029] A memory for storing a computer program;

[0030] A processor for executing the computer program to implement the method according to any one of the first aspects.

[0031] The beneficial effects of the present invention are as follows: The present invention first upsamples the hyperspectral image and performs feature extraction, and performs Gaussian blur and feature extraction on the SAR image. Aiming at the radiation difference problem between different images, a radiation domain alignment network is constructed. The hyperspectral image features are used as the style image, and the features of the Gaussian-blurred SAR image are used as the content image. In the convolution operation, the radiation distribution of the SAR features is continuously approximated to the radiation distribution features of the hyperspectral image. In order to weaken the spatial geometric difference between features, a spatial domain alignment network is constructed. The affine matrix of the hyperspectral features and the SAR features after radiation alignment is calculated by convolution operation, and the spatial geometric information of the SAR features is adjusted through grid construction and sampler interpolation. After that, a change reconstruction network is constructed to extract the change information between the image features through convolution operation, and the low-spatial-resolution hyperspectral image at time T2 is predicted by information injection. In order to improve the spatial resolution, the feature extraction parameters of the Gaussian-blurred SAR are shared with the original SAR image, and the radiation-spatial domain alignment parameters are shared, and a spatial reconstruction network is constructed to fuse And the original SAR images after alignment are used to obtain the final high - spatial - resolution hyperspectral images at T2. Finally, a low - rank loss function, a radiation alignment and spatial alignment loss function, and a root - mean - square error loss function are constructed to continuously optimize the network parameters. In summary, the method proposed in the present invention greatly improves the spatio - spectral fusion accuracy of hyperspectral and SAR images, providing reliable support for subsequent applications. Therefore, the method proposed in the present invention has important practical application significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 FIG. is the overall flow chart of the spatio - spectral fusion method for unpaired hyperspectral and SAR images with dual - domain alignment;

[0033] Figure 2 FIG. is the structural schematic diagram of the feature extraction network;

[0034] Figure 3 FIG. is the structural schematic diagram of the radiation - domain alignment network;

[0035] Figure 4 FIG. is the structural schematic diagram of the spatial - domain alignment network;

[0036] Figure 5 FIG. is the structural schematic diagram of the change reconstruction network;

[0037] Figure 6 FIG. is the structural schematic diagram of the spatial reconstruction network;

[0038] Figure 7 FIG. is the schematic diagram of the spatio - spectral fusion result. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] The following further describes the present invention with reference to the embodiments. The description of the following embodiments is only for helping to understand the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

[0040] Embodiment 1:

[0041] To solve the problems of the existing technology, Embodiment 1 of the present application provides a spatio - spectral fusion method for unpaired hyperspectral and SAR images with dual - domain alignment. By designing a radiation - domain alignment style transfer network to weaken the gray - scale difference between images, designing a spatial - domain alignment network based on convolution operation to eliminate the spatial position difference between images, adopting a multi - scale gating mechanism to construct a change information extraction network for change information extraction, and constructing an attention fusion network to integrate the spatial difference information features between images. This method realizes the high - quality fusion of unpaired hyperspectral and SAR images and has strong practicability.

[0042] Specifically, as Figure 1As shown in the figure, the method includes:

[0043] S1. Obtain the hyperspectral image at time T1, and perform spatial upsampling processing on the hyperspectral image; obtain the SAR image at time T2, and perform Gaussian blur processing on the SAR image.

[0044] S2. As Figure 2 shown in the figure, construct a feature extraction network, perform feature extraction on the spatially upsampled hyperspectral image at time T1 and the Gaussian-blurred SAR image at time T2, and obtain the hyperspectral features at time T1 and the SAR features at time T2.

[0045] The formula for feature extraction in S2 is:

[0046] HS f = FEM(φ(HS))

[0047] SAR f = FEM(ψ(SAR))

[0048] where HS is the hyperspectral image obtained at time T1, SAR is the SAR image obtained at time T2, φ is the upsampling operation, ψ is the Gaussian blur operation, HS f and SAR f are the features of the upsampled hyperspectral image and the Gaussian-blurred SAR image respectively, and FEM is the feature extraction network, which can be specifically expressed as:

[0049]

[0050] where I = [φ(HS), ψ(SAR)] represents the input data, C3, C5, and C7 respectively represent convolution operations with 3×3, 5×5, and 7×7 convolution kernels, M represents the Mamba operation, is the output feature. At this time, these features are multi-scale fused, and the final image features are output through a 1×1 convolution operation after feature stacking:

[0051]

[0052] where C1 represents a convolution operation with a 1×1 convolution kernel, is the final image feature.

[0053] S3. As Figure 3 shown in the figure, construct a radiation domain alignment network, use the hyperspectral features at time T1 as the style image, use the SAR features at time T2 as the content image, and continuously approximate the radiation distribution of the SAR features to the radiation distribution of the hyperspectral features in the convolution operation.

[0054] Due to different imaging mechanisms, hyperspectral and SAR images have obvious radiometric differences, which seriously affect the extraction of differential features between images. Considering that the style transfer algorithm can approximate the gray-scale distribution of the content image to the style image while ensuring the unchanged details of the content image, S3 introduces the idea of style transfer to design a radiometric domain alignment network to weaken the radiometric differences between hyperspectral and SAR images. Specifically, HS f is used as the style image, and after convolutional operations, average pooling operations, and linear operations, a style vector is obtained, which can be expressed as:

[0055]

[0056] where i represents the i-th convolutional operation, ap represents the average pooling operation, l represents the linear operation, and w is the style vector. At this time, the SAR image is used as the content image, and the AdaIN algorithm is used to achieve style control after convolutional operations:

[0057]

[0058] where μ and σ represent the mean and standard deviation respectively. It should be noted that the radiometric domain alignment network sets 9 convolutional operations when calculating w, and a total of 6 convolutional operations are set when performing style transfer on the SAR f image, and the AdaIN algorithm is used for style regulation after each convolutional operation.

[0059] S4. As Figure 4 shown, a spatial domain alignment network is constructed. The affine matrix of the hyperspectral features at time T1 and the SAR features at time T2 after radiometric alignment is calculated using convolutional operations, and the spatial geometric information of the SAR features is adjusted through grid construction and sampler interpolation.

[0060] SAR and hyperspectral sensors have different imaging platforms and angles, which results in obvious spatial misalignment between the images. To eliminate the spatial deviation between the images, a spatial domain alignment network is constructed. The spatial domain alignment network mainly includes three sub-modules: the position network, the grid generator, and the sampler. For the position network, the main purpose is to predict the parameters of the spatial transformation. The position network can output a set of parameters that define a spatial transformation, including affine transformation parameters such as translation, rotation, and scaling. The spatial domain alignment network first stacks HS f and the after radiometric alignment in the channel dimension, and performs 6 convolutional operations on the stacked features. A Max Pool module is embedded between every 3 convolutional operations. After passing through the fully connected layer, a reshape operation is performed to obtain the final affine matrix, and the formula can be expressed as:

[0061]

[0062] where represents the affine matrix, represents the SAR after radiation domain alignment, f is the feature of, v is the operation of the fully connected layer, is the reshape operation, mp represents the Max Pool operation. After that, a grid generator is defined. The grid generator is mainly used to receive the transformation parameters predicted by the localization network and generate a coordinate grid, which represents the new position of each pixel in the input image mapped to the output image. The formula can be defined as:

[0063]

[0064] where represents the transformed pixel coordinates, represents the pixel coordinates before transformation, G represents the grid, and i represents the i-th pixel. Finally, a bilinear interpolation method is used to construct a sampler, and pixel values are sampled from the input image according to the coordinate grid provided by the grid generator to generate the transformed output image.

[0065] S5. As Figure 5 shown, a change reconstruction network is constructed to extract the change information between the hyperspectral features at time T1 and the SAR features after radiation-spatial alignment at time T2, and the change information is injected into the hyperspectral features to predict the low-spatial-resolution hyperspectral image at time T2.

[0066] S6. Extract the parameters of the SAR features processed by Gaussian blur at time T2, share the radiation-spatial domain alignment parameters with the original SAR image at time T2, and as Figure 6 shown, construct a spatial reconstruction network to fuse the low-spatial-resolution hyperspectral image at time T2 and the original SAR image after parameter sharing to obtain the high-spatial-resolution hyperspectral image at time T2.

[0067] Example 2:

[0068] Based on Example 1, Example 2 of the present application provides a more specific spatio-temporal spectral fusion method for non-paired hyperspectral and SAR images with dual-domain alignment, including:

[0069] S1. Obtain the hyperspectral image at time T1 and perform spatial upsampling on the hyperspectral image; obtain the SAR image at time T2 and perform Gaussian blur processing on the SAR image.

[0070] S2. Construct a feature extraction network to extract features from the spatially upsampled hyperspectral image at time T1 and the Gaussian-blurred SAR image at time T2, and obtain the hyperspectral features at time T1 and the SAR features at time T2.

[0071] S3. Construct a radiation domain alignment network, use the hyperspectral features at time T1 as the style image, and the SAR features at time T2 as the content image, and continuously approximate the radiation distribution of the SAR features to the radiation distribution of the hyperspectral features during the convolution operation.

[0072] S4. Construct a spatial domain alignment network, calculate the affine matrix of the hyperspectral features at time T1 and the SAR features at time T2 after radiation alignment, and adjust the spatial geometric information of the SAR features through grid construction and sampler interpolation.

[0073] S5. Construct a change reconstruction network, use gated operation to extract the change information between the hyperspectral features at time T1 and the SAR features after radiation-spatial alignment at time T2, and inject the change information into the hyperspectral features to predict the hyperspectral image with low spatial resolution at time T2.

[0074] The change reconstruction network first designs a difference information extraction network, and uses and HS f as input data to obtain the initial change information. After that, a sigmoid activation function is used to analyze the distribution law of the change information. To optimize the initialized change information, eliminate and HS f in the change information, and extract the residual change information from the residual image again according to the distribution law of the change information and add it to the initialized change information. Finally, the change information is stacked, a 1×1 convolution operation is used to obtain the final change information, and the extracted change information is injected into HS f to obtain features. The specific formula can be expressed as:

[0075]

[0076]

[0077] where represents the initial change information extraction network, E in is the initial change information, and represent the optimized change information. It is mainly a two-stage operation composed of two gating operations, and the gating operation is specifically constructed by the Mamba structure. In the first stage, first take the features of an image as input data for the gating operation. After obtaining the hidden state and cell, update the initialized hidden state and cell. Then replace the initialized hidden state and cell with the updated ones for network iterative optimization. After the iteration is completed, take another image, the hidden state obtained in the previous stage, and the cell as input data for the second-stage iterative optimization. Each stage can be specifically expressed as follows:

[0078]

[0079] H t-1 = O t ⊙ tanh(C t-1 )

[0080] where w is the weight, b is the bias, F is the input feature, H is the hidden output, C is the cell output, f is the forget gate, O is the output gate, δ is the sigmoid function, tanh is the tanh function, t represents the t-th iteration, is the convolution operation, ⊙ is the multiplication operation, and ssm is the state space model.

[0081] S6. Extract the SAR feature extraction parameters and radiation-space domain alignment parameters for the SAR image at T2 after Gaussian blur processing, share them with the original SAR image at T2, and construct a spatial reconstruction network to fuse the hyperspectral image with low spatial resolution at T2 and the original SAR image after parameter sharing to obtain the hyperspectral image with high spatial resolution at T2.

[0082] Specifically, first share the feature extraction parameters, radiation domain alignment parameters, and spatial domain alignment parameters of ψ(SAR) with SAR to obtain the corrected After that, construct a spatial reconstruction network, and directly use and as input data to obtain the hyperspectral image with high spatial resolution at T2 The spatial reconstruction network mainly includes two branches. One branch is composed of a spectral-spatial convolution block and a Mamba block to extract features from Another branch is composed of a texture feature convolution block and a Mamba block to extract features from In addition, to better fuse the features, there is a feature interaction branch between the spectral-spatial convolution block and the texture feature convolution block. The specific formula can be expressed as:

[0083]

[0084]

[0085] Among them denotes an empty spectral convolution block, which consists of 3 convolutional layers. denotes a texture feature convolution block, which consists of 3 central difference convolutional layers. denotes the empty spectral feature, denotes the texture feature. IN represents the input data. After extracting the features of and these features are stacked to obtain and a three-branch network is designed by combining convolution with Mamba to extract features from as follows:

[0086]

[0087] where Q, K, and V respectively represent the output features, and mlp is a multi-layer perceptron. After that, Q and K are dot-producted, and the dot product with V obtains the attention feature:

[0088] A = V ⊙ ((Q ⊙ K))

[0089] where A is the attention feature. Finally, after the attention feature undergoes mlp and convolution operations, it is added to the convolutional feature to obtain the final

[0090]

[0091] S7. Construct a joint loss function of the radiation domain alignment loss function, the spatial domain alignment loss function, and the root mean square error loss function to optimize and train the parameters of the feature extraction network, the radiation domain alignment network, the spatial domain alignment network, the change reconstruction network, and the spatial reconstruction network.

[0092] Specifically, the network parameters in S2 - S6 are effectively trained to obtain high-quality Design the following loss function for network parameter optimization:

[0093]

[0094] where denotes the overall loss. Considering that the SAR image contains a large amount of noise signals, is designed to remove the noise signals of the SAR image, denotes the radiation alignment loss function. is the spatial alignment loss function, is the root mean square error loss between the fusion result and the ground truth. can be expressed as follows:

[0095]

[0096] The first term of the formula represents the reconstruction loss, and the second term is the low-rank loss. n represents the number of pixels, represents the to be reconstructed A, B, C, and D represent the factor matrices obtained after tensor decomposition, W is the weight vector, and ‖·‖ F is the Frobenius norm. is defined as follows:

[0097]

[0098] where K represents the number of bands, represents the f covariance matrix of HS represents the covariance matrix of. In addition, and can be expressed as follows:

[0099]

[0100] To reduce the spatial geometric error between images, a spatial domain alignment loss function is designed The specific expression is as follows:

[0101]

[0102] where and represent the means of HS f and respectively. Finally, a root mean square error loss function is designed to constrain the fusion result:

[0103]

[0104] This method is the first method for non-paired hyperspectral and SAR spatio-temporal spectral fusion. By using a simulated dataset to quantitatively verify this method, the peak signal-to-noise ratio PSNR of this method is 25.36, the spectral angle SAM is 9.18, and the root mean square error RMSE is 0.137; as Figure 7As shown in the figure, the comparison methods adopt typical CNN, GAN, and Transformer networks in the image translation method. Among them, the PSNR of the CNN translation method is 9.24, SAM is 168.18, and RMSE is 0.997; the PSNR of the GAN translation method is 12.67, SAM is 96.54, and RMSE is 0.989; the PSNR of the Transformer translation method is 13.24, SAM is 105.09, and RMSE is 0.991. The PSNR of this method is significantly higher than that of all comparison methods, and SAM and RMSE are significantly lower than those of all comparison methods, with reliability and practicality.

[0105] It should be noted that the same or similar parts in this embodiment and Embodiment 1 can be referred to each other and will not be elaborated in this application.

[0106] Embodiment 3:

[0107] Based on Embodiment 1, Embodiment 3 of this application provides a spatio-spectral fusion system for dual-domain alignment of unpaired hyperspectral and SAR images, including:

[0108] An acquisition module, configured to acquire a hyperspectral image at time T1 and perform spatial upsampling processing on the hyperspectral image; acquire a SAR image at time T2 and perform Gaussian blur processing on the SAR image;

[0109] A first construction module, configured to construct a feature extraction network, extract features from the spatially upsampled hyperspectral image at time T1 and the Gaussian-blurred SAR image at time T2, and obtain hyperspectral features at time T1 and SAR features at time T2;

[0110] A second construction module, configured to construct a radiation domain alignment network, use the hyperspectral features at time T1 as the style image, use the SAR features at time T2 as the content image, and continuously approximate the radiation distribution of the SAR features to the radiation distribution of the hyperspectral features in the convolution operation;

[0111] A third construction module, configured to construct a spatial domain alignment network, calculate the affine matrix of the hyperspectral features at time T1 and the SAR features at time T2 after radiation alignment, and adjust the spatial geometric information of the SAR features through grid construction and sampler interpolation;

[0112] A fourth construction module, configured to construct a change reconstruction network, extract the change information between the hyperspectral features at time T1 and the SAR features after radiation-spatial alignment at time T2, and inject the change information into the hyperspectral features to predict the low-spatial-resolution hyperspectral image at time T2;

[0113] The fifth construction module is used to share the SAR feature extraction parameters and radiation-spatial domain alignment parameters, which are Gaussian blurred at time T2, with the original SAR image at time T2, and construct a spatial reconstruction network to fuse the hyperspectral image with low spatial resolution at time T2 and the original SAR image after parameter sharing, so as to obtain the hyperspectral image with high spatial resolution at time T2.

[0114] Specifically, the system provided in this embodiment is the system corresponding to the method provided in Embodiment 1. Therefore, for the parts that are the same or similar in this embodiment and Embodiment 1, reference can be made to each other and will not be elaborated in this application.

Claims

1. A dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion method, characterized in that: include: S1, obtaining a hyperspectral image at time T1, and performing spatial upsampling processing on the hyperspectral image; Acquire a SAR image at time T2, and perform Gaussian blur processing on the SAR image; S2, construct a feature extraction network, extract features from the spatially upsampled hyperspectral image at time T1 and the Gaussian blurred SAR image at time T2, and obtain the hyperspectral features at time T1 and the SAR features at time T2; S3. Construct a radiation domain alignment network, use the hyperspectral features at time T1 as the style image, and the SAR features at time T2 as the content image, and continuously approach the radiation distribution of the SAR features to the radiation distribution of the hyperspectral features in the convolution operation; S4, constructing a spatial domain alignment network, calculating the affine matrix of the SAR features at T2 after the hyperspectral features at T1 are aligned with the radiation, and adjusting the spatial geometric information of the SAR features through grid construction and sampler interpolation; S5, construct a change reconstruction network, extract the change information between the hyperspectral features at time T1 and the SAR features after the radiation-spatial alignment at time T2, inject the change information into the hyperspectral features to predict the hyperspectral image with low spatial resolution at time T2; S6. The SAR feature extraction parameters processed by Gaussian blur at time T2 and the radiation-spatial domain alignment parameters are shared with the original SAR image at time T2, and a spatial reconstruction network is constructed to fuse the hyperspectral image with low spatial resolution at time T2 and the original SAR image after parameter sharing to obtain a hyperspectral image with high spatial resolution at time T2.

2. The dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion method according to claim 1, characterized in that: Also includes: S7. Construct the radiation domain alignment loss function, the spatial domain alignment loss function, and the root mean square error loss function to jointly optimize the parameters of the feature extraction network, the radiation domain alignment network, the spatial domain alignment network, the variation reconstruction network, and the spatial reconstruction network.

3. The dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion method according to claim 2, characterized in that: In S4, the spatial domain alignment network includes a position network, a grid generator, and a sampler; The position network is used to output parameters defining spatial transformation; the grid generator is used to receive the transformation parameters predicted by the positioning network and generate a coordinate grid; the sampler is used to sample pixel values ​​from the input image according to the coordinate grid provided by the grid generator to generate a transformed output image.

4. The dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion method according to claim 3, characterized in that: In S4, a convolution operation is used to calculate the affine matrix of the SAR features at time T2 after the hyperspectral features at time T1 are aligned with the radiation.

5. The dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion method according to claim 4, characterized in that: In S5, a gating operation is used to extract the change information between the hyperspectral image features at time T1 and the SAR features at time T2 after the radiation-space alignment; the gating operation is constructed by the Mamba structure.

6. The dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion method according to claim 5, characterized in that: In S6, the spatial reconstruction network consists of a first branch and a second branch, the first branch is a combination of a spatial spectrum convolution block and a Mamba block, and the second branch is a combination of a texture feature convolution block and a Mamba block; there is a feature interaction branch between the spatial spectrum convolution block and the texture feature convolution block.

7. Dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion system, characterized by: Used to perform the method according to any one of claims 1 to 6, comprising: The acquisition module is used to acquire the hyperspectral image at time T1 and perform spatial upsampling processing on the hyperspectral image; acquire the SAR image at time T2 and perform Gaussian blur processing on the SAR image; The first construction module is used to construct a feature extraction network to extract features from the spatially upsampled hyperspectral image at time T1 and the Gaussian blurred SAR image at time T2, so as to obtain the hyperspectral features at time T1 and the SAR features at time T2; The second construction module is used to construct a radiation domain alignment network, which uses the hyperspectral features at time T1 as the style image and the SAR features at time T2 as the content image. In the convolution operation, the radiation distribution of the SAR features is continuously approached to the radiation distribution of the hyperspectral features. The third building module is used to build a spatial domain alignment network, calculate the affine matrix of the SAR features at T2 after the hyperspectral features at T1 are aligned with the radiation, and adjust the spatial geometric information of the SAR features through grid construction and sampler interpolation; The fourth construction module is used to construct a change reconstruction network, extract the change information between the hyperspectral features at time T1 and the SAR features after the radiation-spatial alignment at time T2, and inject the change information into the hyperspectral features to predict the hyperspectral image with low spatial resolution at time T2; The fifth construction module is used to extract the parameters of the SAR features processed by Gaussian blur at time T2, share the radiation-spatial domain alignment parameters with the original SAR image at time T2, and build a spatial reconstruction network to fuse the hyperspectral image with low spatial resolution at time T2 and the original SAR image after parameter sharing, so as to obtain the hyperspectral image with high spatial resolution at time T2.

8. A computer storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed on a computer, the computer executes any one of the methods described in claims 1 to 6.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 6.

Images