Multi-spectral image spectral super-resolution method based on adaptive learning and application

Abstract

The invention provides a multispectral image spectral super-resolution method based on adaptive learning and application, and belongs to the technical field of remote sensing image processing. The method solves the problems that in an existing spectrum super-resolution method, manual setting of prior information and complex parameter selection are carried out, in a deep learning method, pixel dependence between space long distances and self-similarity between spectrums are ignored, and deep network parameters cannot be dynamically adjusted in real time. Constructing a pseudo label based on the multispectral data set; constructing a spectrum super-resolution network, and performing supervised training on the spectrum super-resolution network based on a pseudo tag to generate a preliminary reconstructed image; inputting the preliminary reconstructed image into an adaptive learning module, learning the residual error of the preliminary reconstructed image and the real image, and performing residual error correction; constructing an adaptive learning network, and updating parameters of an adaptive learning module; and carrying out residual connection on the adaptive learning network after parameter updating and a to-be-processed image, and outputting a target high-spatial-resolution hyperspectral image.

Description

Technical Field

[0001] The invention relates to a multi-spectral image spectral super-resolution method based on adaptive learning and an application thereof, and belongs to the technical field of remote sensing image processing. Background Art

[0002] In practical applications, remote sensing images with wide width, high revisit, high temporal and spectral resolution have extremely important application value in many fields such as fine measurement and precision agriculture. However, restricted by the imaging capability of sensors and the law of conservation of energy, hyperspectral images have high spectral resolution, but low temporal resolution, small width, and usually pixel mixing due to low spatial resolution; multispectral images have high spatial and temporal resolution and large width, but low spectral resolution. In order to take into account the advantages of multi- and hyperspectral images and obtain the above ideal remote sensing images, the collaborative representation of the advantages of multi-hyperspectral information has become the main research direction at this stage. Therefore, research on the collaborative use of multi-hyperspectral images has been carried out, with the main purpose of obtaining high spatial resolution hyperspectral images through image fusion or spectral super-resolution.

[0003] Image fusion requires that the multispectral and hyperspectral images used have the same observation scene. Since the original hyperspectral image has a small width, this obviously limits the width of the obtained high spatial resolution hyperspectral image. Spectral super-resolution is the process of obtaining a hyperspectral image from the corresponding multispectral image, which is not limited by the imaging width. Dictionary learning and deep learning are two effective methods to achieve spectral super-resolution. Dictionary learning usually manually sets the prior to standardize the reconstruction, so the quality of the reconstructed image is often limited by prior knowledge and computing resources. In addition, it is mainly based on linear models, and its representation ability and model generalization ability are relatively limited; deep learning learns the nonlinear mapping between multi-hyperspectral through a data-driven strategy, and optimizes the network by minimizing the difference between the reconstructed image and the actual hyperspectral, and has more and more extensive applications in the field of spectral super-resolution. However, due to the adaptive parameters and network structure of different data, the fixed parameters in the traditional network are difficult to extract hidden features in the input data; and the traditional convolutional neural network ignores non-local self-similarity and long-range correlation in training, which leads to the low accuracy of the reconstructed image at this stage.

[0004] As a method to accurately capture and utilize the key features of input images, the attention mechanism has attracted widespread attention in the field of deep learning. It is worth noting that the traditional super-resolution model designed based on convolutional neural networks ignores the long-distance spatial pixel dependency and spectral self-similarity during training, which leads to information loss in the reconstruction process and affects the spectral reconstruction effect. The attention mechanism can effectively capture the key spatial-spectral features of the input image and further process the non-local self-similarity and long-range correlation of the image. Therefore, the attention mechanism has been widely used in the field of spectral super-resolution. However, in the face of the characteristics of different data with their corresponding adaptive network structures and parameters, the current spectral super-resolution method based on the attention mechanism is difficult to extract hidden features in the input data due to fixed parameters. Summary of the invention

[0005] In order to solve the problems of manual setting of prior information and complex parameter selection in existing spectral super-resolution methods, ignoring the long-distance spatial pixel dependence and self-similarity between spectra in deep learning methods, and the inability to dynamically adjust deep network parameters in real time, the present invention proposes a multispectral image spectral super-resolution method and application based on adaptive learning.

[0006] The technical solution adopted by the present invention to solve the above-mentioned problem is: the steps of the present invention include:

[0007] Step 1: Build a multispectral dataset and construct pseudo labels based on the multispectral dataset;

[0008] Step 2: Build a spectral super-resolution network, perform supervised training on the spectral super-resolution network based on pseudo labels, and generate a preliminary reconstructed image based on the trained spectral super-resolution network;

[0009] Step 3: Input the preliminary reconstructed image into the adaptive learning module to learn the residual between the preliminary reconstructed image and the real image and perform residual correction;

[0010] Step 4: Construct an adaptive learning network and update the parameters of the adaptive learning module by combining the learned preliminary reconstructed image and the residual of the real image;

[0011] Step 5: Perform a residual connection between the adaptive learning network with updated parameters and the image to be processed to output a hyperspectral image with high spatial resolution.

[0012] Optionally, constructing pseudo labels based on the multispectral dataset in step 1 specifically includes:

[0013] Input the images in the multispectral dataset into the image fusion module, use the multi-hyperspectral fusion method based on the generalized Laplace pyramid to fuse the images in an unsupervised manner, obtain a fused hyperspectral image, and use the fused hyperspectral image as a pseudo label, wherein the multispectral dataset includes low-resolution hyperspectral images and high-resolution multispectral image data pairs covering the same area at the same time;

[0014] The expression of image fusion is:

[0015] HSI ik =HSU ik +a ik (MS-MS i )(1);

[0016] In formula (1), HSI ik is the fused hyperspectral image, HSU ik is the upsampled hyperspectral image, MS is the multispectral image, and MS i is the i-th filtered multispectral band, a ik is the gain coefficient of the kth hyperspectral band in the i-th group of data.

[0017] Optionally, the step of generating a preliminary reconstructed image in step 2 includes:

[0018] Step 2.1: Map each image in the multispectral dataset into a unit and map the input image Reassemble the image

[0019] Step 2.2: Image Perform three feature mappings to obtain Q, K and V;

[0020] Step 2.3: Obtain based on the calculated Q and K Combined with the learnable parameter σ j Calculate self-attention A j ;

[0021] Step 2.4: Through the learnable parameters σ j right Reweight the loss calculation to adapt to the self-attention A j ;

[0022] Step 2.5: Self-attention A based on loss calculation j And the parameter V corresponding to the jth head in the parameter V obtained after feature mapping j Calculate the head output features;

[0023] Step 2.6: Aggregate the output features of the N heads and perform linear mapping, and embed the position information of the spectral features to obtain the output preliminary reconstructed image

[0024] The expressions for Q, K and V are:

[0025] Q=XW Q ,K=XW K ,V=XW V (2);

[0026] In formula (2), W Q , W K , All are learnable parameters;

[0027] Self-Attention j The calculation formulas for the output features of the and head are:

[0028]

[0029] In formula (3), K T The transpose of the matrix, σ j are learnable parameters;

[0030] The calculation formula for the preliminary reconstructed image is:

[0031]

[0032] In formula (4), is a learnable parameter, f p () is the function for generating position information, f p ()The specific structure includes two depth 3×3 convolutions, a GELU nonlinear activation function and a reorganization operation.

[0033] Optionally, in step 3, learning the residual between the preliminary reconstructed image and the real image and performing residual correction specifically includes:

[0034] The preliminary reconstructed image is mapped to the original hyperspectral image and multispectral image respectively through the spatial degradation network and the given spectral response function, the residual is adjusted pixel by pixel and the details of the image are refined to complete the unsupervised adaptive learning;

[0035] The expression of unsupervised adaptive learning of adaptive learning module is:

[0036]

[0037] In formula (5), Y m is the original multispectral image, Y sis the original hyperspectral image, Z' is the preliminary reconstructed image generated by the spectral super-resolution network, Z is the reconstructed image finally output by the adaptive learning module, G(·; θ) is the feature mapping process of the adaptive learning module, is the process of image space degradation, θ is the learnable parameter in the adaptive learning module, are the learnable parameters in the spatially degenerate network.

[0038] Optionally, the step of updating the parameters of the adaptive learning module in step 4 includes:

[0039] Step 4.1: Introduce the residual architecture into the adaptive learning network. The adaptive learning network includes a feature mapping module, a self-guided module, and an output convolutional layer. The feature mapping module contains three layers of 3×3 convolutional layers and ReLU activation functions with unchanged input and output feature dimensions.

[0040] Step 4.2: The spectral features of the image to be processed are convolved through the feature mapping module and mapped to the self-guided module;

[0041] Step 4.3: The self-guided module guides the input features through branches A and B and maps them into the output convolutional layer;

[0042] Step 4.4: The output convolution layer performs image spatial degradation, and the spatial downsampling and spatial blur in the image spatial degradation process are used as the image to be convolved with the specified kernel. The noise destruction is used as additive zero-mean random noise to simplify the number of parameters, and the residual between the image initially reconstructed by the spectral super-resolution network and the ground truth image is restored and the parameters of the adaptive learning module are updated;

[0043] The expression of the self-bootstrap module mapping process is:

[0044] A=P 1 (F;υ 1 ); B = P 2 (F;υ 2 )(6);

[0045] F′=A·F+B(7);

[0046] In formulas (6) and (7), F is the feature input to the self-guided module, F′ is the output feature, and P 1 () is Conv1 and Sigmoid activation function, P 2 () is the Conv2 convolution operation, A and B are the outputs of the middle layer, υ 1 and 2 The weight parameters of the bootstrap module through two branches;

[0047] The expression of the spatial degradation process is:

[0048] ZH=(Z*k)↓ s +N(8);

[0049] In formula (8), k is a two-dimensional spatial convolution kernel, * is the convolution of each band of the image with the same convolution kernel k, s is the scale factor of spatial downsampling, and N is the additive noise.

[0050] Applications of multispectral image spectral super-resolution methods based on adaptive learning include:

[0051] The output high spatial resolution hyperspectral image of the target is used to perform subsequent reconstruction accuracy verification and image object classification tasks.

[0052] The beneficial effects of the present invention are:

[0053] 1. The present invention effectively captures the non-local self-similarity and long-range correlation of the input multi-hyperspectral images and realizes the automatic adjustment of network parameters, further improving the authenticity of the reconstructed images and the accuracy of subsequent land object classification tasks.

[0054] 2. The present invention can effectively solve the problems of manual adjustment of model parameters, neglect of long-distance spatial pixel dependence and self-similarity between spectra, and inability to dynamically adjust network parameters in real time in existing multispectral image spectral super-resolution methods, thereby significantly improving the authenticity of spectral super-resolution.

[0055] 3. The present invention can effectively solve the problem of small amount and limited width of existing hyperspectral data, provide a large amount of data source for multiple downstream tasks in the remote sensing field, and can significantly improve the authenticity of the reconstructed image. Compared with the existing contrast learning method, it also has extremely high classification accuracy in the task of ground object classification and has good practical application value.

[0056] 4. The present invention designs a multispectral image spectral super-resolution model based on adaptive learning and multi-hyperspectral collaborative representation, which is composed of an image fusion module, a spectral super-resolution network and an adaptive learning network, and belongs to the field of multispectral remote sensing image spectral super-resolution. In order to further extract the spatial spectrum hidden features in the input data and improve the accuracy of the reconstructed image, the present invention innovatively designs an adaptive learning module and a self-attention module, and cleverly combines them with the image fusion and spectral resolution modules to enhance the generalization ability of the model, significantly improve the quality and reconstruction accuracy of the model, and the image fusion module uses the spatial consistency and spectral consistency of the training data to provide pseudo labels for spectral super-resolution training; a self-attention mechanism is designed in the spectral super-resolution network to effectively capture long-range dependencies and spectral self-similarity; the adaptive learning network introduces spectral response priors through unsupervised learning, and adjusts the output of the super-resolution network to retain the spectral information in the reconstructed data. The present invention can further supplement the spatial and spectral information for the multispectral image spectral super-resolution network, thereby generating more accurate and realistic super-resolution results. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of the multispectral image spectral super-resolution method based on adaptive learning provided by the present invention;

[0058] Figure 2 The overall block diagram of the multi-spectral image spectral super-resolution model provided by the present invention;

[0059] Figure 3 A schematic diagram of the spectral super-resolution network and its internal structure provided by the present invention;

[0060] Figure 3 In the figure, (a) represents the overall structure of the spectral super-resolution network; (b) represents the U-shaped structure of the single-stage spectral Transformer module (SST); (c) represents the internal structure of a spectral attention module (SAB) in an SST, and (d) represents the internal structure of a spectral multi-head self-attention module (S-MSA).

[0061] Figure 4 A schematic diagram of the adaptive learning network and its internal structure provided by the present invention. DETAILED DESCRIPTION

[0062] Combination Figure 1-4 This embodiment is described as follows. Figure 1 and Figure 2 As shown, the steps of the multispectral image spectral super-resolution method based on adaptive learning described in this embodiment include:

[0063] S1: Construct a spectral super-resolution model for multispectral images;

[0064] The multispectral image spectral super-resolution model includes an image fusion module, a spectral super-resolution network and an adaptive learning network.

[0065] S2: Construct a multispectral dataset and input the images of the multispectral dataset into the image fusion module to generate pseudo labels;

[0066] The multispectral dataset includes a low-resolution hyperspectral image and a high-resolution multispectral image data pair covering the same area at the same time. The image of the multispectral dataset is input into the image fusion module, and a multi-hyperspectral fusion method based on a generalized Laplace pyramid is used to perform image fusion in an unsupervised manner to obtain a fused hyperspectral image, and the fused hyperspectral image is used as a pseudo label;

[0067] The expression of image fusion is:

[0068] HSI ik =HSU ik +a ik (MS-MS i )(1);

[0069] In formula (1), HSI ik is the fused hyperspectral image, HSU ik is the upsampled hyperspectral image, MS is the multispectral image, and MS i is the i-th filtered multispectral band, a ik is the gain coefficient of the kth hyperspectral band in the i-th group of data.

[0070] The purpose of using the image fusion module is to generate pseudo labels, so as to facilitate the supervised training of the spectral super-resolution network based on the pseudo labels to learn the general image prior information of the mapping from multispectral images to hyperspectral images.

[0071] S3: Based on the constructed pseudo-labels, supervised training is performed on the spectral super-resolution network, and loss calculation is performed to guide the update of network parameters to generate preliminary reconstructed images;

[0072] The supervised training network consists of two feature mapping modules and a multi-level spectral Transformer module. The multi-level spectral Transformer module is cascaded by multiple single-level spectral Transformer modules. Each module is U-connected by upsampling, downsampling and spectral attention module (SAB), as shown in Figure 3 As shown, the steps for the spectral super-resolution network to generate a preliminary reconstructed image include:

[0073] S301: First, the first step of feature mapping is performed on the input multispectral image, wherein the mapping module of Conv1 includes two 3×3 convolution operations of progressive feature dimension increase, and the mapping module of Conv2 is a 3×3 convolution operation that keeps the input feature dimension unchanged, and the output of Conv2 is residually connected with the output of Conv1;

[0074] S302: The multi-level spectral Transformer module consists of upsampling, downsampling and spectral attention modules (SAB) connected to each other to ensure that the network can generate features at different levels. At the same time, jump connections are designed between the corresponding levels of the encoder and decoder to complete feature aggregation.

[0075] S303: The spectral attention module is composed of a normalization layer, a feedforward network, and a spectral multi-head self-attention module connected in series. The spectral multi-head self-attention module regards each spectral feature map as a unit and calculates the autocorrelation along the spectral dimension.

[0076] To capture non-local dependencies:

[0077] S30301: Input feature map Recombining Then three linear feature maps are performed to obtain Q, K and V respectively:

[0078] The expressions for Q, K and V are:

[0079] Q=XW Q ,K=XW K ,V=XW V (2);

[0080] In formula (2), W Q , W K , All are learnable parameters;

[0081] S30302: Calculate self-attention A based on the obtained K and Q j ;

[0082] S30303: Through σ j Come to Reweight to adapt self-attention A j , based on the self-attention A after loss calculation j And the parameter V corresponding to the jth head in the parameter V obtained after feature mapping j Calculate the head output features;

[0083] Self-Attention j The calculation formulas for the output features of the and head are:

[0084]

[0085] In formula (3), K T The transpose of the matrix, σ j are learnable parameters;

[0086] S30304: Aggregate the output features of the N heads and perform linear mapping, and finally embed the position information of the spectral features to obtain the output preliminary reconstructed image

[0087] The calculation formula for the preliminary reconstructed image is:

[0088]

[0089] In formula (4), is a learnable parameter, f p () is the function for generating position information, f p () The specific structure includes two depth 3×3 convolutions, a GELU nonlinear activation function and a reorganization operation, f p () The purpose is to ensure that the position information of different spectral units can be saved in advance through the network during the module's operation on the input.

[0090] S4: input the preliminary reconstructed image into the adaptive learning module, learn the residual between the preliminary reconstructed image and the ground truth image and perform residual correction;

[0091] This implementation introduces a spectral response function prior in the adaptive learning module and learns the specific spatial degradation function of the image. The network designs an adaptive learning module for the preliminary reconstructed image. The preliminary reconstructed image is mapped to the original hyperspectral image and multispectral image respectively through the spatial degradation network and the given spectral response function. The residual is adjusted pixel by pixel and the specific details of the image are refined to correct the influence of pseudo-labels on the reconstruction result, thereby completing unsupervised adaptive learning.

[0092] The expression of unsupervised adaptive learning of adaptive learning module is:

[0093]

[0094] In formula (5), Y m is the original multispectral image, Y s is the original hyperspectral image, Z' is the preliminary reconstructed image generated by the spectral super-resolution network, Z is the reconstructed image finally output by the adaptive learning module, G(·; θ) is the feature mapping process of the adaptive learning module, is the process of image space degradation, θ and are the learnable parameters in the adaptive learning module and the spatial degradation network, respectively.

[0095] S5: combining the learned preliminary reconstructed image and the residual of the real image to update the parameters of the adaptive learning network to obtain a pre-trained adaptive learning network;

[0096] In order to restore specific details of the image, this embodiment introduces a residual architecture into the adaptive learning network. The main structure includes feature mapping, self-guided module and output convolution layer to ensure that the main body of the module can restore the residual between the image initially reconstructed by the spectral super-resolution network and the ground truth image. The internal structure of the adaptive learning network is as follows: Figure 4 As shown, the specific process includes:

[0097] S501: introducing a residual architecture into the adaptive learning network, wherein the adaptive learning network includes a feature mapping module, a self-guided module, and an output convolution layer, wherein the feature mapping module includes three layers of 3×3 convolution layers with unchanged input and output feature dimensions and a ReLU activation function;

[0098] S502: performing a convolution operation on the spectral features of the image to be processed through a feature mapping module and mapping the convolution operation to a self-guiding module;

[0099] S503: The self-guided module guides the input features through branch A and branch B and maps them to the output convolution layer;

[0100] The expression of the self-bootstrap module mapping process is:

[0101] A=P 1 (F;υ 1 ); B = P 2 (F;υ 2 )(6);

[0102] F′=A·F+B(7);

[0103] In formulas (6) and (7), F is the feature input to the self-guided module, F′ is the output feature, and P 1 () is Conv1 and Sigmoid activation function, P 2 () is the Conv2 convolution operation, A and B are the outputs of the middle layer, υ 1 and 2 The weight parameters of the bootstrap module through two branches;

[0104] S504: Considering the limited amount of multispectral and hyperspectral data pairs, in order to reduce the number of parameters in the learning process, the adaptive learning network adopts a simple structure to implement the spatial degradation network to greatly reduce the number of parameters in the learning process. In the actual image spatial degradation process, spatial downsampling, spatial blurring and noise destruction are involved. The process of spatial downsampling and spatial blurring can be expressed as convolution of the image with a specific kernel, and the noise destruction is closer to additive zero-mean random noise.

[0105] The process of spatial degradation can be expressed as:

[0106] ZH=(Z*k)↓ s +N(8);

[0107] In formula (8), k is a two-dimensional spatial convolution kernel, * is the convolution of each band of the image with the same convolution kernel k, s is the scale factor of spatial downsampling, and N is the additive noise.

[0108] The present invention realizes a spatially degenerate network through a simple structure to greatly reduce the number of parameters in the learning process, which can effectively solve the problem of small data volume and limited width of existing hyperspectral data, and provide a large amount of data source for multiple downstream tasks in the remote sensing field. In addition, the present invention can significantly improve the authenticity of the reconstructed image. Compared with the existing contrast learning method, it also has extremely high classification accuracy in the task of ground object classification and has great practical application value.

[0109] like Figure 4 As shown in Figure 4, the adaptive learning network uses a single two-dimensional convolutional layer with a step size of s to achieve spatial downsampling and blurring. Its main learnable parameter is the convolution kernel k. At the same time, because the loss function designed in S4 has the function of absorbing additive noise, there is no need to further design a specific structure for the noise N.

[0110] S6: Perform residual connection between the adaptive learning network with updated parameters and the image to be processed to output a target hyperspectral image with high spatial resolution.

[0111] By using the output high spatial resolution hyperspectral image of the target, high precision and high authenticity spectral super-resolution of multispectral remote sensing images can be achieved.

[0112] In summary, the present invention utilizes the spatial consistency and spectral consistency of training data through an image fusion module to provide pseudo labels for spectral super-resolution training; designs a self-attention mechanism in the spectral super-resolution network to effectively capture long-range dependencies and spectral self-similarity; the adaptive learning network introduces spectral response priors through unsupervised learning, and adjusts the output of the super-resolution network to retain the spectral information in the reconstructed data. The present invention can further supplement the spatial and spectral information for the spectral super-resolution network of multispectral images, generate more accurate and realistic super-resolution results, and effectively solve the problems of manual adjustment of model parameters, neglect of spatial long-distance pixel dependence and spectral self-similarity, and inability to dynamically adjust network parameters in real time in existing multispectral image spectral super-resolution methods, thereby significantly improving the authenticity of spectral super-resolution.

[0113] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A multispectral image spectral super-resolution method based on adaptive learning, characterized in that: The specific steps include: Step 1: Build a multispectral dataset and construct pseudo labels based on the multispectral dataset; Step 2: Build a spectral super-resolution network, perform supervised training on the spectral super-resolution network based on pseudo labels, and generate a preliminary reconstructed image based on the trained spectral super-resolution network; Step 3: Input the preliminary reconstructed image into the adaptive learning module to learn the residual between the preliminary reconstructed image and the real image and perform residual correction; Step 4: Construct an adaptive learning network and update the parameters of the adaptive learning network by combining the learned preliminary reconstructed image and the residual of the real image; Step 5: Perform a residual connection between the adaptive learning network with updated parameters and the image to be processed to output a hyperspectral image with high spatial resolution.

2. The multispectral image spectral super-resolution method based on adaptive learning according to claim 1, characterized in that: In step 1, constructing pseudo labels based on multispectral datasets specifically includes: Input the images in the multispectral dataset into the image fusion module, use the multi-hyperspectral fusion method based on the generalized Laplace pyramid to fuse the images in an unsupervised manner, obtain a fused hyperspectral image, and use the fused hyperspectral image as a pseudo label, wherein the multispectral dataset includes low-resolution hyperspectral images and high-resolution multispectral image data pairs covering the same area at the same time; The expression of image fusion is: HSI ik =HSU ik +a ik (MS-MS i )(1); In formula (1), HSI ik is the fused hyperspectral image, HSU ik is the upsampled hyperspectral image, MS is the multispectral image, and MS i is the i-th filtered multispectral band, a ik is the gain coefficient of the kth hyperspectral band in the i-th group of data.

3. The multispectral image spectral super-resolution method based on adaptive learning according to claim 1, characterized in that: The steps of generating the preliminary reconstructed image in step 2 include: Step 2.1: Map each image in the multispectral dataset into a unit, and map the input image Reassemble the image Step 2.2: For the image Perform three feature mappings to obtain Q, K and V; Step 2.3: Obtain based on the calculated Q and K Combined with the learnable parameter σ j Calculate self-attention A j ; Step 2.4: Through the learnable parameters σ j right Reweight the loss calculation to adapt to the self-attention A j ; Step 2.5: Self-attention A based on loss calculation j And the parameter V corresponding to the jth head in the parameter V obtained after feature mapping j Calculate the head output features; Step 2.6: Aggregate the output features of the N heads and perform linear mapping, and embed the position information of the spectral features to obtain the output preliminary reconstructed image The expressions for Q, K and V are: Q=XW Q ,K=XW K ,V=XW V (2); In formula (2), All are learnable parameters; Self-Attention j The calculation formulas for the output features of the and head are: In formula (3), K T The transpose of the matrix, σ j are learnable parameters; The calculation formula for the preliminary reconstructed image is: In formula (4), is a learnable parameter, f p () is the function for generating position information, f p ()The specific structure includes two depth 3×3 convolutions, a GELU nonlinear activation function and a reorganization operation.

4. The multispectral image spectral super-resolution method based on adaptive learning according to claim 1, characterized in that: In step 3, the residual of the preliminary reconstructed image and the real image is learned and the residual correction is performed, which specifically includes: The preliminary reconstructed image is mapped to the original hyperspectral image and multispectral image respectively through the spatial degradation network and the given spectral response function, the residual is adjusted pixel by pixel and the details of the image are refined to complete the unsupervised adaptive learning; The expression of unsupervised adaptive learning of adaptive learning module is: In formula (5), Y m is the original multispectral image, Y s is the original hyperspectral image, Z' is the preliminary reconstructed image generated by the spectral super-resolution network, Z is the reconstructed image finally output by the adaptive learning module, G(·; θ) is the feature mapping process of the adaptive learning module, is the process of image space degradation, θ is the adaptive learning module, are the learnable parameters in the spatially degenerate network.

5. The multispectral image spectral super-resolution method based on adaptive learning according to claim 1, characterized in that: The steps of updating the parameters of the adaptive learning module in step 4 include: Step 4.1: Introduce the residual architecture into the adaptive learning network. The adaptive learning network includes a feature mapping module, a self-guided module, and an output convolutional layer. The feature mapping module contains three layers of 3×3 convolutional layers and ReLU activation functions with unchanged input and output feature dimensions. Step 4.2: The spectral features of the image to be processed are convolved through the feature mapping module and mapped to the self-guided module; Step 4.3: The self-guided module guides the input features through branches A and B and maps them into the output convolutional layer; Step 4.4: The output convolution layer performs image spatial degradation, and the spatial downsampling and spatial blur in the image spatial degradation process are used as the image to be convolved with the specified kernel. The noise destruction is used as additive zero-mean random noise to simplify the number of parameters, and the residual between the image initially reconstructed by the spectral super-resolution network and the ground truth image is restored and the parameters of the adaptive learning module are updated; The expression of the self-bootstrap module mapping process is: A=P1(F; υ1); B=P2(F; υ2)(6); F′=A·F+B(7); In formulas (6) and (7), F is the feature input to the self-guided module, F′ is the output feature, P1() is the Conv1 and Sigmoid activation functions, P2() is the Conv2 convolution operation, A and B are the outputs of the intermediate layer, υ1 and υ2 are the weight parameters of the self-guided module through the two branches; The expression of the spatial degradation process is: ZH=(Z*k)↓ s +N(8); In formula (8), k is a two-dimensional spatial convolution kernel, * is the convolution of each band of the image with the same convolution kernel k, s is the scale factor of spatial downsampling, and N is the additive noise.

6. Application of the multispectral image spectral super-resolution method based on adaptive learning, applied to the multispectral image spectral super-resolution method based on adaptive learning according to any one of claims 1 to 5, characterized in that: The output high spatial resolution hyperspectral image of the target is used to perform subsequent reconstruction accuracy verification and image object classification tasks.

Images