Hyperspectral super-resolution method and system based on cross-attention deep tensor network

Abstract

The present invention discloses a hyperspectral super-resolution method and system based on a cross-attention deep tensor network, the method comprising: using the concept of Tucker decomposition to regard an image as a combination of a core tensor and a factor matrix, and realizing image reconstruction through a constructed coupling factor matrix; establishing a feature tensor fusion network based on a cross-attention mechanism, and using the introduced cross-attention mechanism and a U-shaped structure feature tensor fusion network to fuse the original hyperspectral and multispectral images into a shared core tensor; establishing a simulated degradation structure of a hyperspectral image to simulate the degradation of spatial and spectral dimensions to assist model training and parameter iteration. The present invention is applicable to unsupervised hyperspectral and multispectral image fusion coupled tensor networks, and provides an innovative solution for hyperspectral image super-resolution processing.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing, and in particular relates to a hyperspectral super-resolution method and system based on a cross-attention deep tensor network. Background Art

[0002] With the continuous development of remote sensing technology, hyperspectral imaging technology has become one of the important technical means in the field of remote sensing. By capturing the reflection or radiation characteristics of the target object in multiple continuous bands, hyperspectral imaging technology can provide rich spectral information, which is of great significance to smart agriculture, mineral exploration, anomaly detection and other fields. However, although hyperspectral images have rich spectral information, due to the hardware constraints of imaging sensors, their own spatial resolution is often low, which brings challenges to the refined analysis and recognition in certain application scenarios. Therefore, improving the spatial resolution of hyperspectral images has become one of the important directions of current research. In order to solve the problem of low spatial resolution of hyperspectral images, a common method is to fuse hyperspectral images with images with high spatial resolution to obtain images with high spatial resolution and high spectral resolution.

[0003] For the goal of hyperspectral image super-resolution, the current common method is to use the fusion of multispectral images with panchromatic images or high-resolution images to improve the spatial resolution of the image. Among the traditional image fusion methods, a common method is to use a decomposition mechanism to achieve image fusion. More specifically, the method based on the decomposition mechanism usually extracts image features by matrix decomposition and spectral unmixing. However, this will involve the conversion of three-dimensional image cubes to two-dimensional matrices, thereby losing some high-dimensional structural information of spatial-spectral correlation. In recent years, with the deepening of the understanding of the high-dimensional data representation characteristics of tensors, more and more researchers have begun to try to use tensors to reconstruct input images in order to better preserve the relevant structures and features in the image. In addition, deep learning technology can learn effective feature representations from large-scale data, so it is widely used in the field of image processing. For the field of hyperspectral image fusion super-resolution, through the back propagation of neural networks, deep learning models can effectively deal with optimization problems and achieve excellent super-resolution effects.

[0004] However, the hyperspectral image fusion super-resolution method based on deep learning also faces many challenges. First, traditional supervised methods usually obtain the point spread function and spectral response function for image degradation from simulated imaging devices. However, these simulated parameters are often difficult to fully and accurately reflect the working conditions of the actual imaging device, resulting in some deviations in the fusion results. Secondly, the loss function design of the unsupervised hyperspectral image super-resolution model is also a major difficulty. Because it is impossible to use real high-spatial-resolution hyperspectral images for model training, the loss function can only be constructed indirectly by simulating degradation structures. Finally, since the spatial resolution of hyperspectral images is low and the spectral resolution of high-spatial-resolution images is low, information loss or distortion often occurs during the fusion process. This mismatch between spatial resolution and spectral resolution brings additional difficulties to the training and optimization of super-resolution models. Summary of the invention

[0005] The object of the present invention is to provide a hyperspectral super-resolution method and system based on a cross-attention deep tensor network.

[0006] The technical solution to achieve the purpose of the present invention is: a hyperspectral super-resolution method based on a cross-attention deep tensor network, comprising the following steps:

[0007] Step 1: Establish a feature tensor fusion network based on the cross-attention mechanism, receive high spatial resolution multispectral images and low spatial resolution hyperspectral images as input, and generate a shared core tensor through two modules: the cross-attention encoder and the U-shaped structure feature tensor fusion network;

[0008] Step 2: Based on the idea of ​​Tucker decomposition, the shared core tensor obtained in step 1 is used to construct three factor matrices for the reconstruction of high spatial resolution multispectral images, and three factor matrices for the reconstruction of low spatial resolution hyperspectral images. The coupling information in the two sets of factor matrices is then combined to reconstruct and predict the high spatial resolution hyperspectral images.

[0009] Step 3, through the constructed degradation structure, the input high spatial resolution multispectral image is spatially degraded, the input low spatial resolution hyperspectral image is spectrally degraded, and the high spatial resolution hyperspectral image reconstructed and predicted in step 2 is spatially degraded and spectrally degraded;

[0010] Step 4: Use the reconstructed image obtained in step 2 and the degraded image obtained in step 3 to establish a joint loss function, and obtain the final high spatial resolution hyperspectral image by continuously training the iterative model.

[0011] A hyperspectral super-resolution system based on a cross-attention deep tensor network, comprising:

[0012] In the first module, a feature tensor fusion network based on the cross-attention mechanism is established, which receives high spatial resolution multispectral images and low spatial resolution hyperspectral images as input, and generates a shared core tensor through two modules: the cross-attention encoder and the U-shaped structure feature tensor fusion network;

[0013] In the second module, based on the idea of ​​Tucker decomposition, the shared core tensor is used to construct three factor matrices for the reconstruction of high spatial resolution multispectral images and three factor matrices for the reconstruction of low spatial resolution hyperspectral images. The coupling information in the two sets of factor matrices is then combined to reconstruct and predict the high spatial resolution hyperspectral images.

[0014] The third module, through the constructed degradation structure, performs spatial degradation on the input high spatial resolution multispectral image, performs spectral degradation on the input low spatial resolution hyperspectral image, and performs spatial degradation and spectral degradation on the high spatial resolution hyperspectral image obtained by reconstruction prediction;

[0015] In the fourth module, a joint loss function is established using the reconstructed image and the degraded image, and the final high spatial resolution hyperspectral image is obtained by continuously training the iterative model.

[0016] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the hyperspectral super-resolution method based on a cross-attention deep tensor network is implemented.

[0017] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned hyperspectral super-resolution method based on a cross-attention deep tensor network.

[0018] Compared with the prior art, the present invention has the following significant advantages: (1) The present invention uses a coupled tensor network suitable for unsupervised hyperspectral and multispectral image fusion to simulate and realize spatial degradation structure and spectral degradation structure when the point spread function and the spectral response function are unknown; (2) The present invention constructs a feature tensor fusion network based on the cross-attention mechanism, uses a cross-attention encoder for multispectral and hyperspectral images to extract deep features of the image, and uses a U-shaped structure feature tensor fusion network to fuse the feature tensors to obtain a shared feature tensor; (3) The present invention constructs an unsupervised hyperspectral and multispectral image fusion coupled tensor network based on the Tucker decomposition idea, regards the image tensor as the product of a core tensor and three factor matrices, and realizes image reconstruction through a shared feature tensor and a coupled shared factor matrix structure.

[0019] The hyperspectral super-resolution method based on cross-attention deep tensor network provided by the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a framework diagram of the hyperspectral super-resolution method based on the cross-attention deep tensor network of the present invention.

[0021] Figure 2 This is a flow chart of the hyperspectral super-resolution method based on the cross-attention deep tensor network of the present invention.

[0022] Figure 3 It is the skip connection fusion block in the U-shaped structure feature tensor fusion network of the present invention.

[0023] Figure 4 It is the spatial attention extraction mechanism used by the present invention.

[0024] Figure 5 It is the spectral attention extraction mechanism used by the present invention.

[0025] Figure 6 It is the cross attention residual block in the U-shaped structure feature tensor fusion network of the present invention.

[0026] Figure 7 It is the Tucker decomposition joint reconstruction structure used in the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] like Figure 1 As shown in FIG. 1 , a specific framework structure of a hyperspectral super-resolution method based on a cross-attention deep tensor network of the present invention is shown. Figure 2 As shown, the present invention comprises the following steps:

[0029] (1) Establish a feature tensor fusion network based on the cross-attention mechanism:

[0030] Step 1.1: Convert low spatial resolution high spectral image and high spatial resolution multispectral images Input into the cross-attention encoder, where the relationship between the image height, width and number of channels satisfies the conditions w<<W, h<<H and s<<S. The encoder is designed to extract features and transform the dimension of the input image. First, the image is input into a module composed of several convolutional layers and activation functions. Among them, the low spatial resolution hyperspectral image branch uses convolution kernels of 5×5 and 3×3 sizes, and the high spatial resolution multispectral image branch uses convolution kernels of 1×1 size. The activation function uses the LeakyReLU function with a negative slope coefficient of 0.2; then the spatial attention is extracted from the high spatial resolution multispectral image branch, and the spectral attention is extracted from the low spatial resolution hyperspectral image branch. Among them, the spatial attention uses a convolution kernel of the same size as the spatial scale multiple scale = W / w = H / h of the hyperspectral image and the multispectral image, thereby adjusting the spatial attention to the same spatial scale as the low spatial resolution hyperspectral image, that is, Spectral attention uses a 1×1 convolution kernel to obtain linear spectral attention, that is, Where s' represents the number of multispectral feature channels after multi-layer convolution operations. Next, the original feature is multiplied with the attention from another branch, and the result of the multiplication is concatenated with the original feature using the concat function. Finally, a 1×1 convolution layer is used to change the dimension of the feature to R to reduce the amount of feature data and obtain the hyperspectral feature tensor. and the multispectral feature tensor Since hyperspectral images have high redundancy, the value of R satisfies R≤S, where S is the number of channels of the hyperspectral image. The model can adjust the super-resolution effect of the model by adjusting the hyperparameter R.

[0031] In the present invention, high spatial resolution and low spatial resolution are relative concepts. For example, for the same shooting scene, the resolution of a high spatial resolution multispectral image may be 512*512, while the resolution of a low spatial resolution hyperspectral image may be 64*64. The spatial resolution difference between the two is often several times or even dozens of times. In order to better represent this feature of the image, high spatial resolution and low spatial resolution are used for representation.

[0032] Step 1.2: The feature tensor obtained by the cross-attention encoder is used as the input of the U-shaped structure feature tensor fusion network. This network aims to merge the hyperspectral feature tensor X and the multispectral feature tensor Y to obtain a shared core tensor Since X and Y have different spatial scales, the network continuously downsamples the multispectral feature tensor Y, and connects and reconstructs it when it reaches the same size as the hyperspectral feature tensor X to achieve feature tensor fusion. Finally, the fused feature tensor is continuously upsampled to reach the same spatial scale as the original multispectral feature tensor, and the spatial scale of the feature tensor is changed to obtain the core tensor Among them, in the fusion feature tensor upsampling stage, use Figure 3 The skip connection fusion block that combines spatial attention and spectral attention as shown further supplements the feature information. For the features of the downsampling stage, two sets of "convolution layer + activation function" structures are adopted, in which the size of the convolution kernel is 1×1; for the features of the upsampling stage, a convolution kernel of size 3×3 is adopted, and an upsampling module is added to the convolution layer so that the two types of features have the same spatial scale. At the same time, better feature fusion is achieved by introducing the attention mechanism. Specifically, the spatial attention is first extracted from the features of the downsampling stage and multiplied with the features of the upsampling stage, and then the spatial attention is extracted from the features of the upsampling stage and multiplied with the features of the downsampling stage to better align the spatial features. In addition, in order to further strengthen the spectral features, the spectral attention is extracted from the features of the upsampling stage and multiplied with itself. Among them, the spatial attention extraction mechanism is as follows Figure 4 As shown in Figure 2, the spectral attention extraction mechanism is as follows: Figure 5 As shown in , both use average pooling and maximum pooling. The spatial attention extraction mechanism connects the results of average pooling and maximum pooling, and uses a 3×3 convolution kernel and a Sigmoid function to obtain the final spatial attention; the spectral attention inputs the results of average pooling and maximum pooling into two 1×1 convolution kernel convolution layers and an activation function layer for further feature extraction, and then adds the two together and uses the Sigmoid function to obtain the final spectral attention. In the downsampling stage of the feature tensor X, use Figure 6 The cross-attention residual block shown in the figure combines the cross-attention mechanism and the residual connection structure to align the feature tensors X and Y. First, the spatial attention is extracted from the multispectral feature tensor, and the spectral attention is extracted from the hyperspectral feature tensor. Then, the feature information in the feature tensor is further extracted through convolution operations and activation functions. A 1×1 convolution kernel is used to extract deep spectral features for the hyperspectral feature tensor; a 3×3 convolution kernel is used to extract deep spatial features for the multispectral feature tensor. The spatial attention is multiplied with the hyperspectral feature tensor, and the spectral attention is multiplied with the multispectral feature tensor to obtain the attention-weighted multispectral features and hyperspectral features. Finally, the attention-weighted multispectral feature tensor and hyperspectral feature tensor are added to the original feature tensor to form a cross-attention residual structure and obtain the fused feature representation.

[0033] (2) Reconstruct and predict the image based on Tucker decomposition idea:

[0034] Step 2.1: Use the shared core tensor constructed in step 1 Construct Tucker decomposition and reconstruction module to realize image reconstruction. Figure 7 As shown in Figure 2, the model uses three factor matrices for the reconstruction of high spatial resolution multispectral images and three factor matrices for the reconstruction of low spatial resolution hyperspectral images. and multispectral images The Tucker decomposition and reconstruction module can be expressed as:

[0035]

[0036]

[0037] The core tensor satisfies The hyperspectral image reconstruction factor matrix satisfies and The multispectral image reconstruction factor matrix satisfies and and Represents the intermediate product that is not constrained by the clamp. and They represent the reconstructed low spatial resolution hyperspectral image and high spatial resolution hyperspectral image respectively. The clamp constraint is a nonlinear constraint that ensures the output is [0,1]. The specific formula is:

[0038]

[0039] Step 2.2: By using low spatial resolution hyperspectral images and high spatial resolution multispectral images Reconstruction of , and obtain two sets of factor matrix parameters. In the coupled tensor network, these factor matrices can be regarded as shared parameters, and together constitute the shared factor matrix. The reconstruction prediction can use the W, H and S factor matrices in the shared factor matrix, and the reconstruction process is described as:

[0040]

[0041] The core tensor satisfies The shared factor matrix satisfies and Represents the intermediate product without the clamp constraint. The clamp constraint is a nonlinear constraint that ensures the output is [0,1].

[0042] (3) Constructing the degradation structure of hyperspectral images

[0043] When high spatial resolution multispectral images The spatial scale is low spatial resolution hyperspectral image When the image is scale times larger, a convolution kernel of size scale×1 is used to simulate the spatial degradation structure. The degradation process of image width and height is expressed as f PSF1 =(x;D1) and f PSF2 =(x;D2). is the image width degradation matrix satisfying w = D1W, is the image height degradation matrix satisfying h = D2H. Therefore, the spatial degradation structure is recorded as f PSF =f PSF1 (f PSF2 (x; D2); D1). Similarly, a 1×1 convolution kernel is used to simulate the spectral degradation structure, and the degradation process is represented by f SRF (x; D3). Among them, It is the image spectrum degradation matrix satisfying s=D2S.

[0044] Through the spatial degradation structure and spectral degradation structure, the input high spatial resolution multispectral image is spatially degraded and represented as The spectral degradation of the input low spatial resolution hyperspectral image is expressed as The spatial degradation and spectral degradation of the high spatial resolution hyperspectral image reconstructed and predicted in step 2 are expressed as and

[0045] (4) Establish a joint loss function for model training

[0046] In step 4, the reconstructed image obtained in step 2 and the degraded image obtained in step 3 are used to establish a joint loss function, and the specific contents are as follows:

[0047] In order to describe the reconstruction loss of the reconstructed image in step 2, the reconstruction loss function is constructed:

[0048]

[0049] Taking the reconstruction of low spatial resolution hyperspectral images as an example, for the factor matrices w, h and S used, the model is trained in an end-to-end manner, unlike the traditional Tucker decomposition, and is learned together with other parameters in the network. rec, use gradient back propagation to train the factor matrix. rec Take the gradient of the factor matrix w as an example, assuming the element coordinates are (i1,i2):

[0050]

[0051] in Elements representing the coordinates (i1, j1, k1) of the original low spatial resolution hyperspectral image; The element representing the position of the reconstructed low spatial resolution hyperspectral image coordinate (i1, j1, k1); Represents the element of the coordinate position (i1, j1, k1) of the reconstructed low spatial resolution hyperspectral image that does not pass the clamp constraint; Represents loss pair The derivatives of each element in; c′(·) represents the derivative of the clamp constraint; Represents the element of the factor matrix w at the coordinate (i1,i2); Represents the element of the factor matrix h at the coordinate (j1, j2); Represents the element of the factor matrix S at the coordinate (k1, k2). Similarly, we can obtain the derivatives of the factor matrices h and S at the element coordinates (j1, j2) and (k1, k2), respectively, as well as the training process of the factor matrices W, H and s for high spatial resolution multispectral image reconstruction.

[0052] In order to describe the degradation loss of the high spatial resolution hyperspectral image predicted in step 3, an image degradation loss function is constructed:

[0053]

[0054] At the same time, using and Two construct difference constraints:

[0055]

[0056] Finally, the loss function based on the degenerate structure and the total loss function are obtained:

[0057] L PSF-SRF =L degraded +βL LR-MSI

[0058] L sum =L rec +αL PSF-SRF

[0059] Where α and β represent weight coefficients. In the model, we set α to 0.1 and β to 100.

[0060] The application of the present invention requires a certain amount of data set support, including low spatial resolution hyperspectral images and high spatial resolution multispectral images, which are used to learn a hyperspectral super-resolution method based on a cross-attention deep tensor network; the unsupervised hyperspectral super-resolution method constructed on this basis has higher fusion accuracy and a wider range of application scenarios than many supervised hyperspectral super-resolution methods.

[0061] Based on the same inventive concept, the present invention also provides a hyperspectral super-resolution system based on a cross-attention deep tensor network, comprising:

[0062] In the first module, a feature tensor fusion network based on the cross-attention mechanism is established, which receives high spatial resolution multispectral images and low spatial resolution hyperspectral images as input, and generates a shared core tensor through two modules: the cross-attention encoder and the U-shaped structure feature tensor fusion network;

[0063] In the second module, based on the idea of ​​Tucker decomposition, the shared core tensor is used to construct three factor matrices for the reconstruction of high spatial resolution multispectral images and three factor matrices for the reconstruction of low spatial resolution hyperspectral images. The coupling information in the two sets of factor matrices is then combined to reconstruct and predict the high spatial resolution hyperspectral images.

[0064] The third module, through the constructed degradation structure, performs spatial degradation on the input high spatial resolution multispectral image, performs spectral degradation on the input low spatial resolution hyperspectral image, and performs spatial degradation and spectral degradation on the high spatial resolution hyperspectral image obtained by reconstruction prediction;

[0065] In the fourth module, a joint loss function is established using the reconstructed image and the degraded image, and the final high spatial resolution hyperspectral image is obtained by continuously training the iterative model.

[0066] The specific implementation method of the first to fourth modules mentioned above is the same as the aforementioned hyperspectral super-resolution method based on cross-attention deep tensor network, which will not be repeated here.

[0067] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hyperspectral super-resolution method based on a cross-attention deep tensor network, characterized in that: The following steps are involved: Step 1: Establish a feature tensor fusion network based on the cross-attention mechanism, receive high spatial resolution multispectral images and low spatial resolution hyperspectral images as input, and generate a shared core tensor through two modules: the cross-attention encoder and the U-shaped structure feature tensor fusion network; The specific steps of the feature tensor fusion network based on the cross attention mechanism are as follows: Step 1.1: Convert low spatial resolution high spectral image and high spatial resolution multispectral images Input into the cross-attention encoder; the encoder is designed to extract features and transform dimensions of the input image. First, the image is input into a module composed of several convolutional layers and activation functions. Then, the spatial attention is extracted from the high spatial resolution multispectral image branch, and the spectral attention is extracted from the low spatial resolution hyperspectral image branch. Next, the original feature is multiplied with the attention from another branch, and the concat function is used to connect the product with the original feature. Finally, the dimension of the feature is changed to R to reduce the amount of feature data and obtain the hyperspectral feature tensor. and the multispectral feature tensor Step 1.2: The feature tensor obtained by the cross-attention encoder is used as the input of the U-shaped structure feature tensor fusion network; this network aims to merge the hyperspectral feature tensor X and the multispectral feature tensor Y to obtain a shared core tensor Since X and Y have different spatial scales, the network continuously downsamples the multispectral feature tensor Y, connects and reconstructs it when it reaches the same size as the hyperspectral feature tensor X, and realizes the fusion of feature tensors; finally, the fused feature tensor is continuously upsampled to reach the same spatial scale as the original multispectral feature tensor, and the spatial scale of the feature tensor is changed to obtain the core tensor Among them, in the downsampling stage of the multispectral feature tensor Y, a cross-attention residual block combining the cross-attention mechanism and the residual connection structure is used to align the features of the hyperspectral feature tensor X and the multispectral feature tensor Y; in the upsampling stage of the fused feature tensor, a skip connection fusion block combining spatial attention and spectral attention is used to supplement the feature information; Step 2: Based on the idea of ​​Tucker decomposition, the shared core tensor obtained in step 1 is used to construct three factor matrices for the reconstruction of high spatial resolution multispectral images, and three factor matrices for the reconstruction of low spatial resolution hyperspectral images. The coupling information in the two sets of factor matrices is then combined to reconstruct and predict the high spatial resolution hyperspectral images. The image is reconstructed and predicted according to the Tucker decomposition idea. The specific steps are as follows: Step 2.1: Use the shared core tensor constructed in step 1 Construct Tucker decomposition and reconstruction module to realize image reconstruction; that is, three factor matrices are used for the reconstruction of high spatial resolution multispectral images, and three factor matrices are used for the reconstruction of low spatial resolution hyperspectral images; for hyperspectral images and multispectral images Tucker decomposition and reconstruction module is expressed as: The core tensor satisfies The hyperspectral image reconstruction factor matrix satisfies and The multispectral image reconstruction factor matrix satisfies and and Represents the intermediate product without the clamp constraint. The clamp constraint is a nonlinear constraint that ensures the output is [0,1]. The specific formula is: Step 2.2: By using low spatial resolution hyperspectral images and high spatial resolution multispectral images Reconstruction of two sets of factor matrix parameters is obtained; in the coupled tensor network, these factor matrices are regarded as shared parameters and together constitute a shared factor matrix; if high spatial resolution hyperspectral images are to be realized The reconstruction prediction can use the W, H and S factor matrices in the shared factor matrix, and the reconstruction process is described as: The core tensor satisfies The shared factor matrix satisfies and Represents the intermediate product without the clamp constraint. The clamp constraint is a nonlinear constraint that ensures the output is [0,1]; Step 3, through the constructed degradation structure, the input high spatial resolution multispectral image is spatially degraded, the input low spatial resolution hyperspectral image is spectrally degraded, and the high spatial resolution hyperspectral image reconstructed and predicted in step 2 is spatially degraded and spectrally degraded; Step 4: Use the reconstructed image obtained in step 2 and the degraded image obtained in step 3 to establish a joint loss function, and obtain the final high spatial resolution hyperspectral image by continuously training the iterative model.

2. The hyperspectral super-resolution method based on cross-attention deep tensor network according to claim 1 is characterized in that: The degradation structure described in step 3 is simulated by a convolutional layer; when the high spatial resolution multispectral image The spatial scale is low spatial resolution hyperspectral image When the image is scale times larger, a convolution kernel of size scale×1 is used to simulate the spatial degradation structure. The degradation process of image width and height is expressed as f PSF1 =(x;D1) and f PSF2 =(x; D2); where is the image width degradation matrix satisfying w = D1W, is the image highly degraded matrix satisfying h = D2H; hence, the spatially degraded structure is denoted by f PSF =f PSF1 (f PSF2 (x; D2); D1); Similarly, a 1×1 convolution kernel is used to simulate the spectral degradation structure, and the degradation process is represented by f SRF (x; D3); where is the image spectrum degradation matrix satisfying s = D2S; Through the spatial degradation structure and spectral degradation structure, the input high spatial resolution multispectral image is spatially degraded and represented as The spectral degradation of the input low spatial resolution hyperspectral image is expressed as The spatial degradation and spectral degradation of the high spatial resolution hyperspectral image reconstructed and predicted in step 2 are expressed as and 3. The hyperspectral super-resolution method based on cross-attention deep tensor network according to claim 2 is characterized in that: In step 4, the reconstructed image obtained in step 2 and the degraded image obtained in step 3 are used to establish a joint loss function, and the specific contents are as follows: In order to describe the reconstruction loss of the reconstructed image in step 2, the reconstruction loss function is constructed: In order to describe the degradation loss of the high spatial resolution hyperspectral image predicted in step 3, an image degradation loss function is constructed: At the same time, using and Two construct difference constraints: Finally, the loss function based on the degenerate structure and the total loss function are obtained: L PSF-SRF =L degraded +βL LR-MSI L sum =L rec +αL PSF-SRF Where α and β represent weight coefficients.

4. A hyperspectral super-resolution system based on a cross-attention deep tensor network, characterized in that: include: The first module establishes a feature tensor fusion network based on the cross attention mechanism, receives high spatial resolution multispectral images and low spatial resolution hyperspectral images as input, and generates a shared core tensor through two modules: a cross attention encoder and a U-shaped structure feature tensor fusion network; the feature tensor fusion network based on the cross attention mechanism is as follows: Convert low spatial resolution high spectral images and high spatial resolution multispectral images Input into the cross-attention encoder; the encoder is designed to extract features and transform dimensions of the input image. First, the image is input into a module composed of several convolutional layers and activation functions. Then, the spatial attention is extracted from the high spatial resolution multispectral image branch, and the spectral attention is extracted from the low spatial resolution hyperspectral image branch. Next, the original feature is multiplied with the attention from another branch, and the concat function is used to connect the product with the original feature. Finally, the dimension of the feature is changed to R to reduce the amount of feature data and obtain the hyperspectral feature tensor. and the multispectral feature tensor The feature tensor obtained by the cross-attention encoder is used as the input of the U-shaped structure feature tensor fusion network; the network aims to merge the hyperspectral feature tensor X and the multispectral feature tensor Y to obtain a shared core tensor Since X and Y have different spatial scales, the network continuously downsamples the multispectral feature tensor Y, connects and reconstructs it when it reaches the same size as the hyperspectral feature tensor X, and realizes the fusion of feature tensors; finally, the fused feature tensor is continuously upsampled to reach the same spatial scale as the original multispectral feature tensor, and then the spatial scale of the feature tensor is changed to obtain the core tensor c; in the downsampling stage of the multispectral feature tensor Y, a cross-attention residual block designed by combining the cross-attention mechanism and the residual connection structure is used to align the features of the hyperspectral feature tensor X and the multispectral feature tensor Y; in the upsampling stage of the fused feature tensor, a skip connection fusion block combining spatial attention and spectral attention is used to supplement the feature information; The image is reconstructed and predicted according to the Tucker decomposition idea, as follows; Shared core tensors built using the first module Construct Tucker decomposition and reconstruction module to realize image reconstruction; that is, three factor matrices are used for the reconstruction of high spatial resolution multispectral images, and three factor matrices are used for the reconstruction of low spatial resolution hyperspectral images; for hyperspectral images and multispectral images Tucker decomposition and reconstruction module is expressed as: The core tensor satisfies The hyperspectral image reconstruction factor matrix satisfies and The multispectral image reconstruction factor matrix satisfies and and Represents the intermediate product without the clamp constraint. The clamp constraint is a nonlinear constraint that ensures the output is [0,1]. The specific formula is: By using low spatial resolution hyperspectral images and high spatial resolution multispectral images Reconstruction of two sets of factor matrix parameters is obtained; in the coupled tensor network, these factor matrices are regarded as shared parameters and together constitute a shared factor matrix; if high spatial resolution hyperspectral images are to be realized The reconstruction prediction can use the W, H and S factor matrices in the shared factor matrix, and the reconstruction process is described as: The core tensor satisfies The shared factor matrix satisfies and Represents the intermediate product without the clamp constraint. The clamp constraint is a nonlinear constraint that ensures the output is [0,1]; In the second module, based on the idea of ​​Tucker decomposition, the shared core tensor is used to construct three factor matrices for the reconstruction of high spatial resolution multispectral images and three factor matrices for the reconstruction of low spatial resolution hyperspectral images. The coupling information in the two sets of factor matrices is then combined to reconstruct and predict the high spatial resolution hyperspectral images. The third module, through the constructed degradation structure, performs spatial degradation on the input high spatial resolution multispectral image, performs spectral degradation on the input low spatial resolution hyperspectral image, and performs spatial degradation and spectral degradation on the high spatial resolution hyperspectral image obtained by reconstruction prediction; In the fourth module, a joint loss function is established using the reconstructed image and the degraded image, and the final high spatial resolution hyperspectral image is obtained by continuously training the iterative model.

5. The hyperspectral super-resolution system based on cross-attention deep tensor network according to claim 4, characterized in that: The degradation structure is simulated by convolutional layers; when high spatial resolution multispectral images The spatial scale is low spatial resolution hyperspectral image When the image is scale times larger, a convolution kernel of size scale×1 is used to simulate the spatial degradation structure. The degradation process of image width and height is expressed as f PSF1 =(x;D1) and f PSF2 =(x; D2); where is the image width degradation matrix satisfying w = D1W, is the image highly degraded matrix satisfying h = D2H; hence, the spatially degraded structure is denoted by f PSF =f PSF1 (f PSF2 (x; D2); D1); Similarly, a 1×1 convolution kernel is used to simulate the spectral degradation structure, and the degradation process is represented by f SRF (x; D3); where is the image spectrum degradation matrix satisfying s = D2S; Through the spatial degradation structure and spectral degradation structure, the input high spatial resolution multispectral image is spatially degraded and represented as The spectral degradation of the input low spatial resolution hyperspectral image is expressed as The spatial degradation and spectral degradation of the high spatial resolution hyperspectral image reconstructed and predicted in the second module are expressed as and The reconstructed image obtained in the second module and the degraded image obtained in the third module are used to establish a joint loss function. The specific contents are as follows: In order to describe the reconstruction loss of the reconstructed image in the second module, the reconstruction loss function is constructed: In order to describe the degradation loss of the high spatial resolution hyperspectral image predicted in the third module, an image degradation loss function is constructed: At the same time, using and Two construct difference constraints: Finally, the loss function based on the degenerate structure and the total loss function are obtained: L PSF-SRF =L degraded +βL LR-MSI L sum =L rec +αL PSF-SRF Where α and β represent weight coefficients.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the hyperspectral super-resolution method based on the cross-attention deep tensor network as described in any one of claims 1-3 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the hyperspectral super-resolution method based on a cross-attention deep tensor network as described in any one of claims 1 to 3 are implemented.

Images