Dual-domain aligned non-paired hyperspectrum and SAR image space-time spectrum fusion method

Abstract

The invention relates to a dual-domain aligned non-paired hyperspectrum and SAR image space-time spectrum fusion method, which comprises the following steps: carrying out up-sampling on a hyperspectrum and carrying out feature extraction, and carrying out Gaussian blur and feature extraction on an SAR image; enabling the radiation distribution of the SAR features to approach the radiation distribution features of the hyperspectral image; adjusting the spatial geometric information of the SAR features; extracting change information between image features, and predicting a hyperspectral image with low spatial resolution through information injection; and fusing the hyperspectral image with the low spatial resolution and the aligned original SAR image to obtain a final hyperspectral image with the high spatial resolution. The method has the beneficial effects that the fusion precision of the hyperspectrum and the SAR image space-time spectrum is improved, and reliable support is provided for subsequent application.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image processing, and more specifically, to a dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal spectral fusion method. Background Art

[0002] The spectral coverage of hyperspectral sensors is usually between 400-2500nm, and more than 100 bands can be obtained with fine spectral resolution. Thanks to its superior target recognition capability, hyperspectral data is widely used in environmental monitoring, resource exploration, agricultural monitoring, etc. However, the limited surface energy is captured by narrow spectral channels, and hyperspectral data has low spatial resolution and low temporal resolution, with a satellite revisit period of usually more than 30 days. Multispectral and SAR satellite data provide important auxiliary information for the resolution improvement of hyperspectral data, and image fusion can effectively improve the temporal and spatial resolution of hyperspectral images. At present, the methods for resolution enhancement of hyperspectral images through auxiliary data can be roughly divided into optical-assisted methods and SAR-assisted methods.

[0003] Optical-assisted methods mainly use multispectral data to improve the resolution of hyperspectral data, and spatiotemporal spectral fusion aims to use hyperspectral and multispectral data at different times to predict high spatial resolution hyperspectral data at the missing time. These methods can be divided into iterative optimization methods and deep learning methods. Among them, iterative optimization methods usually construct the spatiotemporal spectral conversion relationship between images based on the variational framework, and obtain fused images through regularization constraints. However, these methods need to construct a large number of constraints, the number of parameters is huge, and the robustness of model performance is low. Deep learning fusion methods construct a nonlinear mapping mechanism between auxiliary information and predicted values, and obtain the final fusion parameters through continuous optimization. These methods can effectively handle the nonlinear changes between images. In addition, SAR-assisted methods mainly reconstruct optical images by establishing a conversion mechanism between images. These methods are mainly used to reconstruct multispectral data and have not been used to reconstruct hyperspectral images.

[0004] However, 1) Optical images are easily affected by weather factors. When the images are polluted by clouds and fog, optical-assisted methods cannot effectively reconstruct hyperspectral images. 2) Compared with optical images, SAR is not affected by weather factors. However, there are obvious differences in the design of optical and SAR sensors, and the imaging angles and mechanisms are different. There are obvious radiation differences and spatial geometry differences between SAR images and optical images. 3) Compared with the image translation method, the spatiotemporal spectral fusion method can establish more prior constraints and has higher image reconstruction accuracy. However, there is currently no spatiotemporal spectral fusion method for hyperspectral and SAR images. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the prior art and propose a dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion method.

[0006] First, a dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion method is provided, including:

[0007] S1, obtaining a hyperspectral image at time T1, and performing spatial upsampling processing on the hyperspectral image; obtaining a SAR image at time T2, and performing Gaussian blur processing on the SAR image;

[0008] S2, construct a feature extraction network, extract features from the spatially upsampled hyperspectral image at time T1 and the Gaussian blurred SAR image at time T2, and obtain the hyperspectral features at time T1 and the SAR features at time T2;

[0009] S3. Construct a radiation domain alignment network, use the hyperspectral features at time T1 as the style image, and the SAR features at time T2 as the content image, and continuously approach the radiation distribution of the SAR features to the radiation distribution of the hyperspectral features in the convolution operation;

[0010] S4, constructing a spatial domain alignment network, calculating the affine matrix of the SAR features at T2 after the hyperspectral features at T1 are aligned with the radiation, and adjusting the spatial geometric information of the SAR features through grid construction and sampler interpolation;

[0011] S5, construct a change reconstruction network, extract the change information between the hyperspectral features at time T1 and the SAR features after the radiation-spatial alignment at time T2, inject the change information into the hyperspectral features to predict the hyperspectral image with low spatial resolution at time T2;

[0012] S6. The SAR feature extraction parameters processed by Gaussian blur at time T2 and the radiation-spatial domain alignment parameters are shared with the original SAR image at time T2, and a spatial reconstruction network is constructed to fuse the hyperspectral image with low spatial resolution at time T2 and the original SAR image after parameter sharing to obtain a hyperspectral image with high spatial resolution at time T2.

[0013] As a preference, it also includes:

[0014] S7. Construct the radiation domain alignment loss function, the spatial domain alignment loss function, and the root mean square error loss function to jointly optimize the parameters of the feature extraction network, the radiation domain alignment network, the spatial domain alignment network, the variation reconstruction network, and the spatial reconstruction network.

[0015] Preferably, in S4, the spatial domain alignment network includes a position network, a grid generator and a sampler;

[0016] The position network is used to output parameters defining spatial transformation; the grid generator is used to receive the transformation parameters predicted by the positioning network and generate a coordinate grid; the sampler is used to sample pixel values ​​from the input image according to the coordinate grid provided by the grid generator to generate a transformed output image.

[0017] Preferably, in S4, a convolution operation is used to calculate the affine matrix of the SAR features at time T2 after the hyperspectral features at time T1 are aligned with the radiation.

[0018] Preferably, in S5, a gating operation is used to extract the change information between the hyperspectral image features at time T1 and the SAR features at time T2 after the radiation-space alignment; the gating operation is constructed by a Mamba structure.

[0019] Preferably, in S6, the spatial reconstruction network consists of a first branch and a second branch, the first branch is a combination of a spatial-spectral convolution block and a Mamba block, and the second branch is a combination of a texture feature convolution block and a Mamba block; there is a feature interaction branch between the spatial-spectral convolution block and the texture feature convolution block.

[0020] In a second aspect, a dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal spectral fusion system is provided, which is used to execute any of the methods described in the first aspect, including:

[0021] The acquisition module is used to acquire the hyperspectral image at time T1 and perform spatial upsampling processing on the hyperspectral image; acquire the SAR image at time T2 and perform Gaussian blur processing on the SAR image;

[0022] The first construction module is used to construct a feature extraction network to extract features from the spatially upsampled hyperspectral image at time T1 and the Gaussian blurred SAR image at time T2, so as to obtain the hyperspectral features at time T1 and the SAR features at time T2;

[0023] The second construction module is used to construct a radiation domain alignment network, which uses the hyperspectral features at time T1 as the style image and the SAR features at time T2 as the content image. In the convolution operation, the radiation distribution of the SAR features is continuously approached to the radiation distribution of the hyperspectral features.

[0024] The third building module is used to build a spatial domain alignment network, calculate the affine matrix of the SAR features at T2 after the hyperspectral features at T1 are aligned with the radiation, and adjust the spatial geometric information of the SAR features through grid construction and sampler interpolation;

[0025] The fourth construction module is used to construct a change reconstruction network, extract the change information between the hyperspectral features at time T1 and the SAR features after the radiation-spatial alignment at time T2, and inject the change information into the hyperspectral features to predict the hyperspectral image with low spatial resolution at time T2;

[0026] The fifth construction module is used to extract the parameters of the SAR features processed by Gaussian blur at time T2, share the radiation-spatial domain alignment parameters with the original SAR image at time T2, and build a spatial reconstruction network to fuse the hyperspectral image with low spatial resolution at time T2 and the original SAR image after parameter sharing, so as to obtain the hyperspectral image with high spatial resolution at time T2.

[0027] According to a third aspect, a computer storage medium is provided, wherein a computer program is stored in the computer storage medium; when the computer program is executed on a computer, the computer executes any method described in the first aspect.

[0028] In a fourth aspect, an electronic device is provided, characterized in that it includes:

[0029] Memory, used to store computer programs;

[0030] A processor is used to execute the computer program to implement any method as described in the first aspect.

[0031] The beneficial effects of the present invention are as follows: the present invention first samples the hyperspectral image and performs feature extraction, and performs Gaussian blur and feature extraction on the SAR image. In order to solve the problem of radiation differences between different images, a radiation domain alignment network is constructed, and the hyperspectral image features are used as style images. The features of the SAR image that is Gaussian blurred are used as content images. In the convolution operation, the radiation distribution of the SAR features is continuously approached to the radiation distribution characteristics of the hyperspectral image. In order to weaken the spatial geometric differences between features, a spatial domain alignment network is constructed, and convolution operations are used to calculate the affine matrix of the hyperspectral features and the SAR features after radiation alignment. The spatial geometric information of the SAR features is adjusted through grid construction and sampler interpolation. After that, a change reconstruction network is constructed to extract the change information between image features through convolution operations, and information injection is used to predict the hyperspectral image with low spatial resolution at time T2. In order to improve The spatial resolution of the SAR image is increased by 10%, and the feature extraction parameters and radiation-spatial domain alignment parameters of the Gaussian blurred SAR are shared with the original SAR image, and a spatial reconstruction network fusion is constructed. The original SAR image after alignment is used to obtain the final hyperspectral image with high spatial resolution at time T2. Finally, a low-rank loss function, radiation alignment and spatial alignment loss functions, and root mean square error loss function are constructed to continuously optimize the network parameters. In short, the method proposed in the present invention greatly improves the spatiotemporal fusion accuracy of hyperspectral and SAR images, and provides reliable support for subsequent applications. Therefore, the method proposed in the present invention has important practical application significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The overall flow chart of the dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion method;

[0033] Figure 2 It is a structural diagram of the feature extraction network;

[0034] Figure 3 This is a schematic diagram of the structure of the radiation domain alignment network;

[0035] Figure 4 This is a schematic diagram of the structure of the spatial domain alignment network;

[0036] Figure 5 A schematic diagram of the structure of the network reconstructed for the changes;

[0037] Figure 6 A schematic diagram of the structure of the spatial reconstruction network;

[0038] Figure 7 Schematic diagram of spatiotemporal fusion results. DETAILED DESCRIPTION

[0039] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0040] Embodiment 1:

[0041] In order to solve the problems of the prior art, Example 1 of the present application provides a dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion method, which weakens the grayscale difference between images by designing a radiation domain alignment style transfer network, and eliminates the spatial position difference between images by designing a spatial domain alignment network based on convolution operation, and uses a multi-scale gating mechanism to construct a change information extraction network to extract change information, and constructs an attention fusion network to integrate the spatial difference information features between images. This method achieves high-quality fusion of unpaired hyperspectral and SAR images, and is highly practical.

[0042] Specifically, Figure 1As shown, the method includes:

[0043] S1, obtaining a hyperspectral image at time T1, and performing spatial upsampling processing on the hyperspectral image; obtaining a SAR image at time T2, and performing Gaussian blur processing on the SAR image.

[0044] S2, such as Figure 2 As shown, a feature extraction network is constructed to extract features from the spatially upsampled hyperspectral image at time T1 and the Gaussian blurred SAR image at time T2 to obtain the hyperspectral features at time T1 and the SAR features at time T2.

[0045] The formula for feature extraction in S2 is:

[0046] HS f =FEM(φ(HS))

[0047] SAR f = FEM(ψ(SAR))

[0048] Among them, HS is the hyperspectral image acquired at time T1, SAR is the SAR image acquired at time T2, φ is the upsampling operation, ψ is the Gaussian blur operation, and HS f and SAR f are the features of the upsampled hyperspectral image and the Gaussian blurred SAR image, respectively. FEM is the feature extraction network, which can be specifically expressed as:

[0049]

[0050] Where I = [φ(HS), ψ(SAR)] represents the input data, C3, C5, C7 represent convolution operations with 3×3, 5×5, and 7×7 convolution kernels respectively, M represents Mamba operation, is the output feature. At this time, these features are fused at multiple scales, and after the features are stacked, a 1×1 convolution operation is performed to output the final image features:

[0051]

[0052] Where C1 represents a convolution operation with a 1×1 convolution kernel, The final image feature.

[0053] S3, such as Figure 3 As shown in the figure, a radiation domain alignment network is constructed, the hyperspectral features at time T1 are used as the style image, and the SAR features at time T2 are used as the content image. In the convolution operation, the radiation distribution of the SAR features is continuously approached to the radiation distribution of the hyperspectral features.

[0054] Due to different imaging mechanisms, hyperspectral and SAR images have obvious radiation differences, which seriously affects the extraction of difference features between images. Considering that the style transfer algorithm can make the grayscale distribution of the content image close to the style image while keeping the details of the content image unchanged, S3 introduces the style transfer idea to design a radiation domain alignment network to weaken the radiation difference between hyperspectral and SAR images. Specifically, HS f is used as the style image, and the style vector is obtained after convolution operation, average pooling operation and linear operation, which can be expressed as:

[0055]

[0056] Where i represents the i-th layer of convolution operation, ap represents the average pooling operation, l represents the linear operation, and w represents the style vector. At this time, the SAR image is used as the content image, and the AdaIN algorithm is used to achieve style control after the convolution operation:

[0057]

[0058] Where μ and σ represent the mean and standard deviation respectively. It is worth noting that the radiation domain alignment network sets 9 layers of convolution operations when calculating w. f A total of 6 layers of convolution operations are set for style transfer, and the AdaIN algorithm is used for style control after each layer of convolution operation.

[0059] S4, such as Figure 4 As shown in the figure, a spatial domain alignment network is constructed, and a convolution operation is used to calculate the affine matrix of the SAR features at time T2 after the hyperspectral features at time T1 are aligned with the radiation. The spatial geometric information of the SAR features is adjusted through grid construction and sampler interpolation.

[0060] SAR and hyperspectral sensors have different imaging platforms and angles, which causes obvious spatial misalignment between images. In order to eliminate the spatial deviation between images, a spatial domain alignment network is constructed. The spatial domain alignment network mainly consists of three submodules: position network, grid generator, and sampler. For the position network, the main purpose is to predict the parameters of the spatial transformation. The position network can output a set of parameters that define a spatial transformation, including affine transformation parameters such as translation, rotation, and scaling. The spatial domain alignment network first converts the HS f After alignment with the irradiated domain The stacking is performed in the channel dimension, and 6 layers of convolution operations are performed on the stacked features. A Max Pool module is embedded in the middle of every 3 layers of convolution operations. After the fully connected layer, a reshape operation is performed to obtain the final affine matrix. The formula can be expressed as:

[0061]

[0062] in represents an affine matrix, Represents the SAR after radiation domain alignment f The features of v are fully connected layer operations. is the reshape operation, and mp represents the Max Pool operation. After that, a grid generator is defined. The grid generator is mainly used to receive the transformation parameters predicted by the positioning network and generate a coordinate grid. The grid represents the new position of each pixel in the input image mapped to the output image. The formula can be defined as:

[0063]

[0064] in represents the pixel coordinates after transformation, Represents the pixel coordinates before transformation, G represents the grid, and i represents the i-th pixel. Finally, the sampler is constructed using the bilinear interpolation method to sample pixel values ​​from the input image according to the coordinate grid provided by the grid generator to generate the transformed output image.

[0065] S5, such as Figure 5 As shown in the figure, a change reconstruction network is constructed to extract the change information between the hyperspectral features at time T1 and the SAR features after radiation-spatial alignment at time T2, and the change information is injected into the hyperspectral features to predict the hyperspectral image with low spatial resolution at time T2.

[0066] S6, the SAR feature extraction parameters after Gaussian blur processing at time T2, the radiation-spatial domain alignment parameters are shared with the original SAR image at time T2, and Figure 6 As shown, a spatial reconstruction network is constructed to fuse the hyperspectral image with low spatial resolution at time T2 and the original SAR image after parameter sharing to obtain a hyperspectral image with high spatial resolution at time T2.

[0067] Embodiment 2:

[0068] Based on Example 1, Example 2 of the present application provides a more specific dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal spectral fusion method, including:

[0069] S1, obtaining a hyperspectral image at time T1, and performing spatial upsampling processing on the hyperspectral image; obtaining a SAR image at time T2, and performing Gaussian blur processing on the SAR image.

[0070] S2. Construct a feature extraction network to extract features from the spatially upsampled hyperspectral image at time T1 and the Gaussian blurred SAR image at time T2 to obtain the hyperspectral features at time T1 and the SAR features at time T2.

[0071] S3. Construct a radiation domain alignment network, use the hyperspectral features at time T1 as the style image, and the SAR features at time T2 as the content image, and continuously approach the radiation distribution of the SAR features to the radiation distribution of the hyperspectral features in the convolution operation.

[0072] S4. Construct a spatial domain alignment network, calculate the affine matrix of the SAR features at time T2 after the hyperspectral features at time T1 are aligned with the radiation, and adjust the spatial geometric information of the SAR features through grid construction and sampler interpolation.

[0073] S5. Construct a change reconstruction network, use gated operations to extract the change information between the hyperspectral features at time T1 and the SAR features after radiation-spatial alignment at time T2, and inject the change information into the hyperspectral features to predict the hyperspectral image with low spatial resolution at time T2.

[0074] The variation reconstruction network first designs a difference information extraction network and With HS f As input data to obtain the initial change information. After that, a sigmoid activation function is used to analyze the distribution of the change information. In order to optimize the initial change information, eliminate and HS f The change information in the image is extracted again through the distribution law of the change information, and the residual change information is added to the initial change information. Finally, the change information is stacked, a 1×1 convolution operation is used to obtain the final change information, and the extracted change information is injected into the HS f To obtain The specific formula can be expressed as:

[0075]

[0076]

[0077] in represents the initial change information extraction network, E in is the initial change information, and Indicates the change information after optimization. It is mainly a two-stage operation consisting of two gating operations, and the gating operation is specifically constructed by the Mamba structure. In the first stage, the features of an image are first used as input data for gating operations. After the hidden state and cell are obtained, the initialized hidden state and cell are updated, and then the updated hidden state and cell replace the initialized hidden state and cell for network iterative optimization. After the iteration is completed, another image and the hidden state and cell obtained in the previous stage are used as input data for iterative optimization in the second stage. Each stage can be specifically expressed as follows:

[0078]

[0079] H t-1 =O t ⊙tanh(C t-1 )

[0080] Where w is the weight, b is the bias, F is the input feature, H is the hidden output, C is the unit output, f is the forget gate, O is the output gate, δ is the sigmoid function, tanh is the tanh function, and t represents the tth iteration. is the convolution operation, ⊙ is the multiplication operation, and ssm is the state space model.

[0081] S6. The SAR feature extraction parameters processed by Gaussian blur at time T2 and the radiation-spatial domain alignment parameters are shared with the original SAR image at time T2, and a spatial reconstruction network is constructed to fuse the hyperspectral image with low spatial resolution at time T2 and the original SAR image after parameter sharing to obtain a hyperspectral image with high spatial resolution at time T2.

[0082] Specifically, the feature extraction parameters, radiation domain alignment parameters and spatial domain alignment parameters of ψ(SAR) are first shared with SAR to obtain the corrected After that, a spatial reconstruction network is constructed and directly and As input data, we can obtain a high-spatial-resolution hyperspectral image at time T2. The spatial reconstruction network mainly consists of two branches. One branch is composed of a spatial spectrum convolution block and a Mamba block. For feature extraction, one branch is composed of a texture feature convolution block and a Mamba block. To extract features. In addition, in order to better fuse the features, there is a feature interaction branch between the spatial spectrum convolution block and the texture feature convolution block. The specific formula can be expressed as:

[0083]

[0084]

[0085] in Represents a spatial spectral convolution block, which consists of 3 convolutional layers. Represents the texture feature convolution block, which consists of 3 center difference convolution layers. represents the empty spectrum feature, Indicates texture features. IN indicates input data. and After extracting features, these features are stacked to obtain And by combining convolution with Mamba to design a three-branch network Perform feature extraction:

[0086]

[0087] Where Q, K, and V represent the output features, and mlp is a multi-layer perceptron. After that, Q and K are dot-producted, and dot-producted with V to obtain the attention feature:

[0088] A=V⊙((Q⊙K))

[0089] Where A is the attention feature. Finally, the attention feature is combined with The convolution features are added to obtain the final

[0090]

[0091] S7. Construct the radiation domain alignment loss function, the spatial domain alignment loss function, and the root mean square error loss function to jointly optimize the parameters of the feature extraction network, the radiation domain alignment network, the spatial domain alignment network, the variation reconstruction network, and the spatial reconstruction network.

[0092] Specifically, the network parameters in S2-S6 are effectively trained to obtain high quality The following loss function is designed for network parameter optimization:

[0093]

[0094] in Indicates the overall loss. Considering that SAR images contain a lot of noise signals, It is designed for noise signal removal of SAR images. represents the radiometric alignment loss function. is the spatial alignment loss function, is the root mean square error loss between the fusion result and the ground truth. can be expressed as follows:

[0095]

[0096] The first term in the formula represents the reconstruction loss, and the second term is the low-rank loss. n represents the number of pixels, Indicates that it has been rebuilt A, B, C, D means The factor matrix obtained after tensor decomposition, W is the weight vector, ‖·‖ F is the Frobenius norm. is defined as follows:

[0097]

[0098] Where K represents the number of bands, Indicates HS f The covariance matrix of express The covariance matrix of . In addition, and It can be expressed as follows:

[0099]

[0100] In order to reduce the spatial geometric error between images, a spatial domain alignment loss function is designed. The specific expression is as follows:

[0101]

[0102] in and Respectively represent HS f and Finally, the root mean square error loss function is designed to constrain the fusion result:

[0103]

[0104] This method is the first one for unpaired hyperspectral and SAR spatiotemporal fusion. The method is quantitatively verified by using a simulated data set. The peak signal-to-noise ratio PSNR of this method is 25.36, the spectral angle SAM is 9.18, and the root mean square error RMSE is 0.137. Figure 7As shown in the figure, the comparison method uses the typical CNN, GAN and Transformer networks in the image translation method. Among them, the PSNR of the CNN translation method is 9.24, SAM is 168.18, and RMSE is 0.997; the PSNR of the GAN translation method is 12.67, SAM is 96.54, and RMSE is 0.989; the PSNR of the Transformer translation method is 13.24, SAM is 105.09, and RMSE is 0.991. The PSNR of this method is significantly higher than that of all the comparison methods, and the SAM and RMSE are significantly lower than those of all the comparison methods, which is reliable and practical.

[0105] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be described in detail in this application.

[0106] Embodiment 3:

[0107] Based on Example 1, Example 3 of the present application provides a dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal spectral fusion system, including:

[0108] The acquisition module is used to acquire the hyperspectral image at time T1 and perform spatial upsampling processing on the hyperspectral image; acquire the SAR image at time T2 and perform Gaussian blur processing on the SAR image;

[0109] The first construction module is used to construct a feature extraction network to extract features from the spatially upsampled hyperspectral image at time T1 and the Gaussian blurred SAR image at time T2, so as to obtain the hyperspectral features at time T1 and the SAR features at time T2;

[0110] The second construction module is used to construct a radiation domain alignment network, which uses the hyperspectral features at time T1 as the style image and the SAR features at time T2 as the content image. In the convolution operation, the radiation distribution of the SAR features is continuously approached to the radiation distribution of the hyperspectral features.

[0111] The third building module is used to build a spatial domain alignment network, calculate the affine matrix of the SAR features at T2 after the hyperspectral features at T1 are aligned with the radiation, and adjust the spatial geometric information of the SAR features through grid construction and sampler interpolation;

[0112] The fourth construction module is used to construct a change reconstruction network, extract the change information between the hyperspectral features at time T1 and the SAR features after the radiation-spatial alignment at time T2, and inject the change information into the hyperspectral features to predict the hyperspectral image with low spatial resolution at time T2;

[0113] The fifth construction module is used to extract the parameters of the SAR features processed by Gaussian blur at time T2, share the radiation-spatial domain alignment parameters with the original SAR image at time T2, and build a spatial reconstruction network to fuse the hyperspectral image with low spatial resolution at time T2 and the original SAR image after parameter sharing, so as to obtain the hyperspectral image with high spatial resolution at time T2.

[0114] Specifically, the system provided in this embodiment is a system corresponding to the method provided in Embodiment 1. Therefore, the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be repeated in this application.

Claims

1. A dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion method, characterized in that: include: S1, obtaining a hyperspectral image at time T1, and performing spatial upsampling processing on the hyperspectral image; Acquire a SAR image at time T2, and perform Gaussian blur processing on the SAR image; S2, construct a feature extraction network, extract features from the spatially upsampled hyperspectral image at time T1 and the Gaussian blurred SAR image at time T2, and obtain the hyperspectral features at time T1 and the SAR features at time T2; S3. Construct a radiation domain alignment network, use the hyperspectral features at time T1 as the style image, and the SAR features at time T2 as the content image, and continuously approach the radiation distribution of the SAR features to the radiation distribution of the hyperspectral features in the convolution operation; S4, constructing a spatial domain alignment network, calculating the affine matrix of the SAR features at T2 after the hyperspectral features at T1 are aligned with the radiation, and adjusting the spatial geometric information of the SAR features through grid construction and sampler interpolation; S5, construct a change reconstruction network, extract the change information between the hyperspectral features at time T1 and the SAR features after the radiation-spatial alignment at time T2, inject the change information into the hyperspectral features to predict the hyperspectral image with low spatial resolution at time T2; S6. The SAR feature extraction parameters processed by Gaussian blur at time T2 and the radiation-spatial domain alignment parameters are shared with the original SAR image at time T2, and a spatial reconstruction network is constructed to fuse the hyperspectral image with low spatial resolution at time T2 and the original SAR image after parameter sharing to obtain a hyperspectral image with high spatial resolution at time T2.

2. The dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion method according to claim 1, characterized in that: Also includes: S7. Construct the radiation domain alignment loss function, the spatial domain alignment loss function, and the root mean square error loss function to jointly optimize the parameters of the feature extraction network, the radiation domain alignment network, the spatial domain alignment network, the variation reconstruction network, and the spatial reconstruction network.

3. The dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion method according to claim 2, characterized in that: In S4, the spatial domain alignment network includes a position network, a grid generator, and a sampler; The position network is used to output parameters defining spatial transformation; the grid generator is used to receive the transformation parameters predicted by the positioning network and generate a coordinate grid; the sampler is used to sample pixel values ​​from the input image according to the coordinate grid provided by the grid generator to generate a transformed output image.

4. The dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion method according to claim 3, characterized in that: In S4, a convolution operation is used to calculate the affine matrix of the SAR features at time T2 after the hyperspectral features at time T1 are aligned with the radiation.

5. The dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion method according to claim 4, characterized in that: In S5, a gating operation is used to extract the change information between the hyperspectral image features at time T1 and the SAR features at time T2 after the radiation-space alignment; the gating operation is constructed by the Mamba structure.

6. The dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion method according to claim 5, characterized in that: In S6, the spatial reconstruction network consists of a first branch and a second branch, the first branch is a combination of a spatial spectrum convolution block and a Mamba block, and the second branch is a combination of a texture feature convolution block and a Mamba block; there is a feature interaction branch between the spatial spectrum convolution block and the texture feature convolution block.

7. Dual-domain aligned unpaired hyperspectral and SAR image spatiotemporal fusion system, characterized by: Used to perform the method according to any one of claims 1 to 6, comprising: The acquisition module is used to acquire the hyperspectral image at time T1 and perform spatial upsampling processing on the hyperspectral image; acquire the SAR image at time T2 and perform Gaussian blur processing on the SAR image; The first construction module is used to construct a feature extraction network to extract features from the spatially upsampled hyperspectral image at time T1 and the Gaussian blurred SAR image at time T2, so as to obtain the hyperspectral features at time T1 and the SAR features at time T2; The second construction module is used to construct a radiation domain alignment network, which uses the hyperspectral features at time T1 as the style image and the SAR features at time T2 as the content image. In the convolution operation, the radiation distribution of the SAR features is continuously approached to the radiation distribution of the hyperspectral features. The third building module is used to build a spatial domain alignment network, calculate the affine matrix of the SAR features at T2 after the hyperspectral features at T1 are aligned with the radiation, and adjust the spatial geometric information of the SAR features through grid construction and sampler interpolation; The fourth construction module is used to construct a change reconstruction network, extract the change information between the hyperspectral features at time T1 and the SAR features after the radiation-spatial alignment at time T2, and inject the change information into the hyperspectral features to predict the hyperspectral image with low spatial resolution at time T2; The fifth construction module is used to extract the parameters of the SAR features processed by Gaussian blur at time T2, share the radiation-spatial domain alignment parameters with the original SAR image at time T2, and build a spatial reconstruction network to fuse the hyperspectral image with low spatial resolution at time T2 and the original SAR image after parameter sharing, so as to obtain the hyperspectral image with high spatial resolution at time T2.

8. A computer storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed on a computer, the computer executes any one of the methods described in claims 1 to 6.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 6.

Images