A remote sensing image spectral super-resolution reconstruction method

Abstract

The application discloses a remote sensing image spectral super-resolution reconstruction method, comprising the following steps: acquiring a remote sensing RGB image dataset, and converting each remote sensing RGB image into a reference hyperspectral image; inputting the remote sensing RGB image dataset into a constructed layered semantic perception CNN-Transformer model to generate a corresponding reconstructed hyperspectral image; constructing a frequency spectrum and space loss function according to the reference hyperspectral image and the reconstructed hyperspectral image, training the model to obtain a trained layered semantic perception CNN-Transformer model; inputting a remote sensing RGB image to be reconstructed into the trained model to obtain a reconstructed hyperspectral image. The application can effectively prevent information loss, improve the stability of the model in different use scenarios, and more accurately reconstruct a hyperspectral image from a remote sensing RGB image.

Description

Technical Field

[0001] This invention relates to the technical field of remote sensing image processing, and more specifically, to a method for spectral super-resolution reconstruction of remote sensing images. Background Technology

[0002] Image spectral super-resolution reconstruction is a technique used to improve the spatial and spectral resolution of images. It combines the advantages of image super-resolution and spectral imaging techniques, aiming to reconstruct high-resolution images from low-resolution hyperspectral images, ultimately resulting in images with both high spectral and high spatial resolution. Currently, hyperspectral imaging faces the following challenges: the high cost of spectral imagers for acquiring hyperspectral images; the high requirements of spectral imagers for lighting and weather conditions; and the mutual constraints between spatial and spectral resolution, making it difficult to simultaneously obtain images with both high spectral and high spatial resolution. Therefore, spectral reconstruction technology can overcome the performance bottlenecks of spectral imagers, reduce costs, and further unleash the potential of hyperspectral images. Spectral reconstruction technology includes two strategies: image fusion and spectral super-resolution. Since image fusion relies on the availability of MSI-HSI (Multi-Spectral Image-Hyperspectral Image) image pairs, image spectral super-resolution reconstruction focuses on reconstructing low-spectral resolution images from the same sensor along the spectral dimension, refining the spectral feature information, and ultimately resulting in images with both high spectral and high spatial resolution.

[0003] Existing spectral super-resolution techniques can be categorized into dictionary learning methods, deep network methods, and deep prior methods. These techniques still suffer from the following drawbacks:

[0004] In terms of model architecture: the encoder-decoder (EnDeCoder) architecture and the continuous convolution (SeqConv) architecture with residual connections are common SSR (Spectral Super-Resolution) model architectures, but they have corresponding problems. The former is prone to losing spatial details, while the latter has inconsistent reconstruction performance in different scenarios.

[0005] Regarding prior embedding: In SSR networks, it is more common to obtain prior information through data preprocessing rather than through learning.

[0006] Regarding lightweight deployment: Current SSR methods focus more on reconstruction performance than model lightweighting, making it difficult to achieve final deployment.

[0007] Regarding application feasibility: Most SSR methods perform very well on standard datasets, but still have limitations on cross-sensor data. Specifically, although existing fusion methods can effectively obtain high spatial resolution HSI (Hyperspectral Image), their model performance depends on the availability of MSI-HSI image pairs, but in reality, corresponding MSI-HSI image pairs are difficult to obtain. Summary of the Invention

[0008] To overcome the shortcomings of existing model reconstruction methods, such as loss of technical information and low model reconstruction performance, this invention provides a method for spectral super-resolution reconstruction of remote sensing images.

[0009] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0010] This invention proposes a method for spectral super-resolution reconstruction of remote sensing images, comprising:

[0011] S1: Obtain the remote sensing RGB image dataset and convert each remote sensing RGB image into a corresponding reference hyperspectral image;

[0012] S2: Input the remote sensing RGB image dataset into the constructed hierarchical semantic perception CNN-Transformer model to generate the corresponding reconstructed hyperspectral image;

[0013] S3: Construct a total training loss function based on the reference hyperspectral image and the reconstructed hyperspectral image, and train the hierarchical semantic perception CNN-Transformer model to obtain the trained hierarchical semantic perception CNN-Transformer model;

[0014] S4: Obtain the remote sensing RGB image to be reconstructed, input it into the trained hierarchical semantic perception CNN-Transformer model, and obtain the reconstructed hyperspectral image.

[0015] Preferably, the hierarchical semantic-aware CNN-Transformer model includes an ISE module, a first deep convolutional layer, a Concat layer, a second deep convolutional layer, a first SACT unit, a third deep convolutional layer, a first concatenation layer, a first convolutional layer, a PixelUnshuffle module, a fourth deep convolutional layer, a second SACT unit, a fifth deep convolutional layer, and a PixelShuffle module.

[0016] The ISE module, the first deep convolutional layer, the Concat layer, the second deep convolutional layer, the first SACT unit, the third deep convolutional layer, the first splicing layer, and the first convolutional layer are connected in sequence;

[0017] The PixelUnshuffle module, the fourth deep convolutional layer, the second SACT unit, the fifth deep convolutional layer, and the PixelShuffle module are connected in sequence, and the output of the PixelShuffle module is connected to the input of the Concat layer.

[0018] The output of the ISE module is also connected to the input of the PixelUnshuffle module and the first stitching layer, respectively.

[0019] The first SACT unit includes N sequentially connected SACT modules, and the second SACT unit includes N-1 sequentially connected SACT modules.

[0020] Preferably, the ISE module includes an in-wavelength estimation submodule and an out-of-wavelength estimation submodule connected in sequence;

[0021] Let R be an RGB image. The original spectral channels of image R include S. r S g and S b S r S g and S b Let R represent the red, green, and blue spectral channels of image R, respectively. Image R is input into the wavelength estimation submodule for estimation, obtaining estimated spectral channel values, including red-green spectral channel estimates and green-blue spectral channel estimates.

[0022] S rg =(S r +S g ) / 2

[0023] S gb =(S g +S b ) / 2

[0024] Among them, S rg S represents the estimated values ​​of the red and green spectral channels of image R. gb S represents the estimated values ​​of the green and blue spectral channels of image R. r S g and S b These represent the red, green, and blue spectral channels of image R, respectively.

[0025] The original spectral channels and the estimated spectral channels are combined to form the refined spectral information of image R, which is then used as the output image of the in-wavelength estimation submodule.

[0026] S R ={S r ,S rg ,S g ,S gb ,S b}

[0027] Among them, S R Represents the fine spectral information of image R;

[0028] The output image from the in-wavelength estimation submodule is input into the out-of-wavelength estimation submodule to obtain the initial spectral estimation features. The specific process is as follows:

[0029]

[0030] Where i = {λ R ,λ G ,λ B} represents the i-th wavelength group, λ s and λ e These represent the left and right endpoints of the wavelength interval λ, respectively. R , λ G , λ B The three molecular regions, DWConv, are in sequence three equal intervals within the wavelength range λ. R (·), DWConv G (·) and DWConv B (·) represent the depth convolutional layers for the red, green, and blue channels, respectively.

[0031] Preferably, each of the SACT modules includes a first PSConv submodule, an SACNN submodule, a second splicing layer, a LeakyReLU activation layer, a second PSConv submodule, an SAFormer submodule, and a third splicing layer connected in sequence.

[0032] Both the first PSConv submodule and the second PSConv submodule are connected sequentially by N PSConv devices;

[0033] The input of the first PSConv submodule is also connected to the input of the second splicing layer, and the input of the LeakyReLU activation layer is also connected to the input of the third splicing layer.

[0034] Preferably, the PSConv unit includes a superpixel semantic unit, a fourth stitching layer, a SpeConv unit, a SpaConv unit, a SpeSpaConv unit, and a fusion unit;

[0035] The input and output of the superpixel semantic unit are both connected to the input of the fourth stitching layer. The output of the fourth stitching layer is connected to the input of the SpeConv, SpaConv, and SpeSpaConv units, respectively. The outputs of the SpeConv, SpaConv, and SpeSpaConv units are all connected to the input of the fusion unit.

[0036] Preferably, the superpixel semantic unit includes a sixth deep convolutional layer, a second convolutional layer, a probability processing layer, a softmax processing layer, and a region label processing layer connected in sequence.

[0037] Preferably, in step S3, the method for determining the total training loss function includes:

[0038] L total =L1+αL spe-spa

[0039] Among them, L total Represents the total training loss function. W, L, and C represent the width, height, and number of spectral channels of the hyperspectral image, respectively. For reference hyperspectral images, To reconstruct the hyperspectral image; L spe-spa Let α represent the spectral spatial loss function, and let α represent the equilibrium parameter.

[0040] Preferably, the method for determining the spectral spatial loss function includes:

[0041] l spe-spa =2(l spe +l spa )

[0042] Among them, L spe L represents the spectral loss function. spa This represents the space loss function.

[0043] Preferably, the method for determining the spectral loss function includes:

[0044]

[0045] in, and respectively composed of H and Expanding along the spectral dimension, we get: <·> represents the dot product of the spectral radiance of each pixel, and ∥·∥ represents the square root of the cumulative square of the spectral radiance of each pixel.

[0046] Preferably, the method for determining the spatial loss function includes:

[0047]

[0048] Among them, H spa and respectively composed of H and It is obtained through metric transformation.

[0049] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0050] This invention proposes a method for spectral super-resolution reconstruction of remote sensing images. First, a dataset of remote sensing RGB images is acquired, and each RGB image is converted into a corresponding reference hyperspectral image. The RGB image dataset is then input into a constructed hierarchical semantic-aware CNN-Transformer model to generate the corresponding reconstructed hyperspectral image. Based on the reference and reconstructed hyperspectral images, spectral and spatial loss functions are constructed, and the model is trained to obtain a trained hierarchical semantic-aware CNN-Transformer model. Finally, the RGB image to be reconstructed is input into the trained model to obtain the reconstructed hyperspectral image. This invention effectively prevents information loss, improves the model's stability under different usage scenarios, and more accurately reconstructs hyperspectral images from remote sensing RGB images. Attached Figure Description

[0051] Figure 1 This is a schematic flowchart of the remote sensing image spectral super-resolution reconstruction method described in Example 1;

[0052] Figure 2 This is a schematic diagram of the hierarchical semantic perception CNN-Transformer model described in Example 2;

[0053] Figure 3 This is a schematic diagram of the ISE module described in Example 2;

[0054] Figure 4 This is a schematic diagram of the SACT module described in Example 2;

[0055] Figure 5 This is a schematic diagram of the PSConv device described in Example 2;

[0056] Figure 6 This is a schematic diagram of the superpixel semantic unit described in Example 2;

[0057] Figure 7 The image shows the performance of the hierarchical semantic perception CNN-Transformer proposed in Example 3 for spectral super-resolution reconstruction of remote sensing images under different datasets and different numbers of superpixels. Detailed Implementation

[0058] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0059] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions.

[0060] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0061] First, let me explain some concepts that may be involved in this application:

[0062] Hierarchical Semantic-Aware CNN-Transformer (HSACT) is a model architecture that combines a convolutional neural network (CNN) and a transformer (Transformer). It aims to leverage the hierarchical feature extraction capabilities of CNNs and the global dependency modeling capabilities of Transformers to improve performance on certain tasks.

[0063] Semantic-Aware CNN-Transformer (SACT).

[0064] Initial Spectral Estimation (ISE).

[0065] Polymorphic superpixel convolution (PSConv) is a method that can process polymorphic or multimodal features in images to improve the performance and effectiveness of image processing algorithms.

[0066] Structure-Aware CNN (SACNN) is a type of convolutional neural network that can perceive the structural features of the input data.

[0067] The Spectrum-Aware Transformer (SAFormer) is a spectrum-aware Transformer network used for hyperspectral image change detection.

[0068] Multi-Spectral Image-Hyperspectral Image (MSI-HSI).

[0069] Hyperspectral Image (HSI).

[0070] Spectral Super-Resolution (SSR).

[0071] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0072] Example 1

[0073] This embodiment provides a method for spectral super-resolution reconstruction of remote sensing images, such as... Figure 1 As shown, it includes:

[0074] S1: Obtain the remote sensing RGB image dataset and convert each remote sensing RGB image into a corresponding reference hyperspectral image;

[0075] S2: Input the remote sensing RGB image dataset into the constructed hierarchical semantic perception CNN-Transformer model to generate the corresponding reconstructed hyperspectral image;

[0076] S3: Construct a total training loss function based on the reference hyperspectral image and the reconstructed hyperspectral image, and train the hierarchical semantic perception CNN-Transformer model to obtain the trained hierarchical semantic perception CNN-Transformer model;

[0077] S4: Obtain the remote sensing RGB image to be reconstructed, input it into the trained hierarchical semantic perception CNN-Transformer model, and obtain the reconstructed hyperspectral image.

[0078] In the specific implementation process, firstly, a remote sensing RGB image dataset is acquired, and each remote sensing RGB image is converted into a corresponding reference hyperspectral image, which is then input into the constructed hierarchical semantic perception CNN-Transformer model to generate the corresponding reconstructed hyperspectral image. Then, based on the reference hyperspectral image and the reconstructed hyperspectral image, a total training loss function is constructed to train the hierarchical semantic perception CNN-Transformer model, resulting in a trained hierarchical semantic perception CNN-Transformer model. Finally, the remote sensing RGB image to be reconstructed is input into the trained hierarchical semantic perception CNN-Transformer model to obtain the reconstructed hyperspectral image.

[0079] Example 2

[0080] This embodiment proposes a method for spectral super-resolution reconstruction of remote sensing images, including:

[0081] S1: Obtain the remote sensing RGB image dataset and convert each remote sensing RGB image into a corresponding reference hyperspectral image;

[0082] Suppose a remote sensing RGB image is represented as The corresponding hyperspectral image is represented as Where L and W refer to the height and width of R and H respectively, and c and C are the number of spectral channels of the two images respectively, with c = 3 << C.

[0083] From the perspective of spectral imaging, the value I of the RGB camera at spatial location (x,y) s(x,y) is defined as follows:

[0084] I s (x,y)=∫ Λ E(x,y,λ)A s (λ)dλ

[0085] Where s∈{S r ,S g ,S b} represents the color channels in an RGB image, E(x,y,λ) n A represents the spectral radiance within the wavelength range λ at position (x,y). s (λ) represents the spectral channel, s is the spectral response function (SRF) from hyperspectral image to RGB image, and Λ represents the wavelength interval.

[0086] In real-world scenarios, wavelength intervals are typically sampled discretely. s (x,y) is redefined as follows:

[0087]

[0088] Where n represents the number of spectral bands in the hyperspectral spectrum.

[0089] The conversion relationship between R and H is described as follows:

[0090] R = AH

[0091] in, This represents the spectral response of an RGB camera.

[0092] S2: Input the remote sensing RGB image dataset into the constructed hierarchical semantic perception CNN-Transformer model to generate the corresponding reconstructed hyperspectral image;

[0093] Deep learning-based SSR methods overcome the uncertainty problem in computing A by learning a mapping function f(R; θ) to reconstruct H from R.

[0094]

[0095] Where L(·) represents the loss function.

[0096] A hierarchical network architecture is used to design the mapping model f(·), instead of the traditional EnDeCoder and SeqConv design modules, and the R structure is embedded into f(·) in an end-to-end learning manner. Finally, the model is optimized using the spectral space loss function. The objective function of the proposed hierarchical semantic-aware CNN-Transformer method in this embodiment is:

[0097]

[0098] Among them, L spe and L spa These refer to the spectral loss function and the spatial loss function, respectively, where α represents the trade-off parameter between spectral and spatial information, and P... s (R) provides the structural prior embedding for the designed PSConv module.

[0099] like Figure 2 The diagram shown is a schematic of a hierarchical semantic perception CNN-Transformer model. The hierarchical semantic perception CNN-Transformer model includes an ISE module, a first deep convolutional layer, a Concat layer, a second deep convolutional layer, a first SACT unit, a third deep convolutional layer, a first concatenation layer, a first convolutional layer, a PixelUnshuffle module, a fourth deep convolutional layer, a second SACT unit, a fifth deep convolutional layer, and a PixelShuffle module.

[0100] The ISE module, the first deep convolutional layer, the Concat layer, the second deep convolutional layer, the first SACT unit, the third deep convolutional layer, the first splicing layer, and the first convolutional layer are connected in sequence;

[0101] The PixelUnshuffle module, the fourth deep convolutional layer, the second SACT unit, the fifth deep convolutional layer, and the PixelShuffle module are connected in sequence, and the output of the PixelShuffle module is connected to the input of the Concat layer.

[0102] The output of the ISE module is also connected to the input of the PixelUnshuffle module and the first stitching layer, respectively.

[0103] The first SACT unit includes N sequentially connected SACT modules, and the second SACT unit includes N-1 sequentially connected SACT modules.

[0104] like Figure 3 The diagram shown is a schematic of an ISE module, which includes an in-wavelength estimation submodule and an out-of-wavelength estimation submodule connected in sequence.

[0105] The raw spectral channels of image R include S r S g and S b S r S g and S b These represent the red, green, and blue spectral channels of image R, respectively. Image R is input into the wavelength estimation submodule for estimation, obtaining estimated spectral channel values, including estimates for the red-green and green-blue spectral channels:

[0106] S rg =(S r +S g ) / 2

[0107] S gb =(S g +S b ) / 2

[0108] Among them, S rg S represents the estimated values ​​of the red and green spectral channels of image R. gb S represents the estimated values ​​of the green and blue spectral channels of image R. r S g and S b These represent the red, green, and blue spectral channels of image R, respectively.

[0109] The original spectral channels and the estimated spectral channels are combined to form the refined spectral information of image R, which is then used as the output image of the in-wavelength estimation submodule.

[0110] S R ={S r ,S rg ,S g ,S gb ,S b}

[0111] Among them, S R Represents the fine spectral information of image R;

[0112] The output image from the in-wavelength estimation submodule is input into the out-of-wavelength estimation submodule to obtain the initial spectral estimation features. The specific process is as follows:

[0113]

[0114] Where i = {λ R ,λ G ,λ B} represents the i-th wavelength group, λ s and λ e These represent the left and right endpoints of the wavelength interval λ, respectively. R , λ G , λ B The three molecular regions, DWConv, are in sequence three equal intervals within the wavelength range λ. R (·), DWConv G (·) and DWConv B (·) represent the depth convolutional layers for the red, green, and blue channels, respectively, which include a 1×1 Conv layer, a specular fill layer, a 3×3 Group Conv layer, and a 1×1 Conv layer. As the initial spectral estimation features, they are gradually restored into a hyperspectral image through subsequent networks.

[0115] like Figure 4 The diagram shows a schematic of the SACT module. Each SACT module includes a first PSConv submodule, an SACNN submodule, a second concatenation layer, a LeakyReLU activation layer, a second PSConv submodule, an SAFormer submodule, and a third concatenation layer connected in sequence. The PSConv submodule is used to capture features, the SACNN submodule is used to reconstruct structural details, and the SAFormer submodule is a spectrum-aware transformer.

[0116] Both the first PSConv submodule and the second PSConv submodule are connected sequentially by N PSConv devices;

[0117] The input of the first PSConv submodule is also connected to the input of the second splicing layer, and the input of the LeakyReLU activation layer is also connected to the input of the third splicing layer.

[0118] like Figure 5 The diagram shown is a schematic of the PSConv device, which includes a superpixel semantic unit, a fourth stitching layer, a SpeConv device, a SpaConv device, a SpeSpaConv device, and a fusion unit.

[0119] The input and output of the superpixel semantic unit are both connected to the input of the fourth stitching layer. The output of the fourth stitching layer is connected to the input of the SpeConv, SpaConv, and SpeSpaConv units, respectively. The outputs of the SpeConv, SpaConv, and SpeSpaConv units are all connected to the input of the fusion unit.

[0120] like Figure 6 The diagram shows the structure of a superpixel semantic processor, which includes a sixth deep convolutional layer, a second convolutional layer, a probability processing layer, a softmax processing layer, and a region label processing layer connected in sequence. The processing expression of the superpixel semantic processor is as follows:

[0121]

[0122] Among them, F s Let X represent the output features after passing through the superpixel semanticizer, and let Conv represent the input features of the superpixel semanticizer. 1×1 (·) represents a 1×1 convolutional layer, DWConv(·) represents a depthwise convolutional layer, and N s This indicates the preset number of superpixels, and Softmax(·) represents the Softmax processing layer function.

[0123] This embodiment employs a learnable method to extract superpixel semantic information from images. This differs from traditional preprocessing superpixel segmentation and embedding. The SpeConv converter outputs spectral features F spe SpaConv output spatial characteristics F spa Spatial characteristics of the output spectrum of the SpeSpaConv device F spe-spa As shown below:

[0124] F spe =SpeConv(X+w1·F) s )

[0125] F spa =SpaConv(X+w2·F s )

[0126] F spe-spa =SpeSpaConv(X+w3·F s )

[0127] Among them, F spe F spa and F spe-spa Let w1, w2, and w3 represent the output features of the SpeConv, SpaConv, and SpeSpaConv processors, respectively. SpeConv(·) represents the processing function of the SpeConv processor, which is a 1×1 convolutional layer. SpaConv(·) represents the processing function of the SpaConv processor, which is a 3×3 depth convolutional layer. SpeSpaConv(·) represents the processing function of the SpeSpaConv processor, which is a 1×1 convolutional layer followed by a 3×3 depth convolutional layer. w1, w2, and w3 are the learnable adaptive weights of the SpeConv, SpaConv, and SpeSpaConv processors, respectively.

[0128] The purpose of the SpeConv processor is to learn deep semantic features to recover spectral information, while the SpaConv processor focuses on recovering spatial details by deeply aggregating contextual information, and the SpeSpaConv processor achieves both spectral and spatial recovery.

[0129] Furthermore, this multimodal aggregation operator significantly enhances feature extraction capabilities and enriches the feature learning space. Then, adaptive weights are added to the output features of the SpeConv, SpaConv, and SpeSpaConv operators respectively to obtain the output features of the PSConv operator. Its expression is as follows:

[0130] F = w1·F spe +w2·F spa +w3·F spe-spa

[0131] For the SACNN submodule, the input features of SACNN are represented by F1, and the output features are represented by... This indicates that the relationship between the output features and the input features is as follows:

[0132]

[0133] Among them, W str Represents structural attention. W str for:

[0134] W str =f s (Conv 1×1 C(MaxPool(F1),AvgPool(F1)))

[0135] Among them, f s (·) represents the Sigmoid function, Conv 1×1 (·) represents a 1×1 convolutional layer, C(·) represents a connection operation, and MaxPool(·) and AvgPool(·) represent max pooling and average pooling, respectively. Represents the matrix dot product.

[0136] For the SAFormer submodule, the designed SAFormer submodule is also applicable to the features obtained by the PSConv, further refining the spectral information. First, by inputting features... The spatial dimension expansion operation is used to obtain the corresponding query. key Sum The specific process is as follows:

[0137] W spe =Softmax(f SpeMax (QK T )-(QK T ))

[0138] In the formula, W spe f represents the spectral attention term. SpeMax (·) indicates taking the maximum value, K T This represents the transpose of K.

[0139] For W spe Perform vector broadcasting to obtain the output features of the SAFormer submodule.

[0140]

[0141] In the formula, β represents the adaptive adjustment coefficient for network learning.

[0142] This embodiment designs a hierarchical network architecture to connect the ISE module, the first SACT unit, and the second SACT unit, thereby obtaining the reconstructed hyperspectral image. The advantages of a layered network architecture can be summarized as follows: (1) The reuse of feature decoders promotes the lightweighting of the network; (2) It enhances the feature capture capability of the model without losing the main spatial details. The specific process is as follows:

[0143]

[0144] in, It is the lth i The input features of each SACT unit, l i ≥2, It is the lth i Output characteristics of each SACT unit Indicates the lth i The equivalent function of the depthwise convolutional layer before and after each SACT unit, f PSF (·) represents the PixelShuffle operation with lossless spatial downsampling information, f PSUF (·) represents f PSF The inverse operation of (·), SACT(·) represents the SACT module. Indicates the lth i The SACT parameters corresponding to each SACT unit.

[0145] Figure 2 The network has a two-layer architecture, including a first SACT unit and a second SACT unit. The reconstructed hyperspectral image is output by the first convolutional layer. for:

[0146]

[0147] Among them, F in This is the output image of the ISE module.

[0148] To further reduce network complexity, the SACT module employs a weight-sharing strategy at each layer, calculated as follows:

[0149]

[0150] S3: Construct a total training loss function based on the reference hyperspectral image and the reconstructed hyperspectral image, and train the hierarchical semantic perception CNN-Transformer model to obtain the trained hierarchical semantic perception CNN-Transformer model;

[0151] To ensure the hierarchical semantic perception CNN-Transformer method proposed in this embodiment achieves its intended purpose, a spectral spatial loss function is designed. Specifically, the spectral spatial loss function is a function developed for SSR tasks, with two main uses: (1) measuring the similarity of spectral radiance within a single pixel's wavelength range; and (2) limiting the spatial fidelity of the spectral image at each sampling wavelength. The final loss function is defined as follows:

[0152] L total =L1+αL spe-spa

[0153] Among them, L total Represents the total training loss function. W, L, and C represent the width, height, and number of spectral channels of the hyperspectral image, respectively. For reference hyperspectral images, To reconstruct the hyperspectral image; L spe-spa Let α represent the spectral spatial loss function, and let α represent the equilibrium parameter.

[0154] The spectral spatial loss function is:

[0155] L spe-spa =2(L spe +L spa )

[0156] Among them, L spe L represents the spectral loss function. spa This represents the space loss function.

[0157] The spectral loss function is:

[0158]

[0159] in, and respectively composed of H and Expanding along the spectral dimension, we get: <·> represents the dot product of the spectral radiance of each pixel, and ∥·∥ represents the square root of the cumulative square of the spectral radiance of each pixel.

[0160] The space loss function is:

[0161]

[0162] Among them, H spa and respectively composed of H and It is obtained through metric transformation.

[0163] S4: Obtain the remote sensing RGB image to be reconstructed, input it into the trained hierarchical semantic perception CNN-Transformer model, and obtain the reconstructed hyperspectral image.

[0164] Finally, for the hierarchical semantic-aware CNN-Transformer method proposed in this embodiment, during the model training phase, the sample batch size, learning rate, and number of training epochs are set to 128 and 1×10, respectively. -3 And 200. To keep the model lightweight, the number of layers in the hierarchical network is preset to l=2, and the optimal weight hyperparameters of the network are obtained by using the designed spectral space loss function and Adam optimizer.

[0165] Example 3

[0166] This embodiment, building upon Embodiment 2, utilizes datasets from the University of Pavia, Washington DC, and Wenxian to explore the impact of the number of superpixels in the superpixel semantician on the performance of the hierarchical semantic-aware CNN-Transformer method. For example... Figure 7 The figure shows the performance of the proposed hierarchical semantic-aware CNN-Transformer for spectral super-resolution reconstruction of remote sensing images under different datasets and different numbers of superpixels. From left to right, these are (a), (b), and (c), corresponding to the University of Pavia, Washington DC, and Wenxian datasets, with preset superpixel numbers of 64, 128, and 256, respectively. The performance of the proposed hierarchical semantic-aware CNN-Transformer method for spectral super-resolution reconstruction of remote sensing images is analyzed using three metrics: CC, SAM, and ERGAS. It can be clearly seen from the figure that as the number of superpixel blocks increases, the performance of the proposed hierarchical semantic-aware CNN-Transformer method for spectral super-resolution reconstruction of remote sensing images improves. Figure 7 As can be seen from (b) and (c) in this application, the performance of the hierarchical semantic perception CNN-Transformer method proposed in this application has been significantly improved, and it also reflects the two advantages of superpixel semantic embedding: (1) the more superpixel blocks there are, the smaller the local superpixel blocks are, and the stronger the correlation between the internal pixels, which enhances the ability of the hierarchical semantic perception CNN-Transformer method proposed in this application to represent spectral spatial features; (2) superpixel semantics can effectively guide the deep network to learn features from local viewpoints, thereby improving the spectral super-resolution reconstruction capability of the hierarchical semantic perception CNN-Transformer.

[0167] The same or similar labels correspond to the same or similar parts;

[0168] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0169] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for spectral super-resolution reconstruction of remote sensing images, characterized in that, include: S1: Obtain the remote sensing RGB image dataset and convert each remote sensing RGB image into a corresponding reference hyperspectral image; S2: Input the remote sensing RGB image dataset into the constructed hierarchical semantic perception CNN-Transformer model to generate the corresponding reconstructed hyperspectral image; S3: Construct a total training loss function based on the reference hyperspectral image and the reconstructed hyperspectral image, and train the hierarchical semantic perception CNN-Transformer model to obtain the trained hierarchical semantic perception CNN-Transformer model; S4: Obtain the remote sensing RGB image to be reconstructed, input it into the trained hierarchical semantic perception CNN-Transformer model, and obtain the reconstructed hyperspectral image; The hierarchical semantic-aware CNN-Transformer model includes an initial spectrum estimation module, a first deep convolutional layer, a Concat layer, a second deep convolutional layer, a first semantic-aware convolutional neural network-transformer unit, a third deep convolutional layer, a first concatenation layer, a first convolutional layer, a PixelUnshuffle module, a fourth deep convolutional layer, a second semantic-aware convolutional neural network-transformer unit, a fifth deep convolutional layer, and a PixelShuffle module. The initial spectrum estimation module, the first deep convolutional layer, the Concat layer, the second deep convolutional layer, the first semantic-aware convolutional neural network-transformer unit, the third deep convolutional layer, the first concatenation layer, and the first convolutional layer are connected in sequence; The PixelUnshuffle module, the fourth deep convolutional layer, the second semantic-aware convolutional neural network-transformer unit, the fifth deep convolutional layer, and the PixelShuffle module are connected in sequence, and the output of the PixelShuffle module is connected to the input of the Concat layer. The output of the initial spectrum estimation module is also connected to the PixelUnshuffle module. The input connection for the first splicing layer is shown; The first semantic-aware convolutional neural network-converter unit includes N sequentially connected semantic-aware convolutional neural network-converter modules, and the second semantic-aware convolutional neural network-converter unit includes N-1 sequentially connected semantic-aware convolutional neural network-converter modules. Each of the semantic-aware convolutional neural network-transformer modules includes a first polymorphic superpixel convolutional submodule, a structure-aware convolutional neural network submodule, a second stitching layer, a LeakyReLU activation layer, a second polymorphic superpixel convolutional submodule, a spectrum-aware transformer submodule, and a third stitching layer connected in sequence. Both the first and second multi-state superpixel convolution submodules are composed of N multi-state superpixel convolutioners connected in sequence; The input of the first multi-state superpixel convolutional submodule is also connected to the input of the second stitching layer, and the input of the LeakyReLU activation layer is also connected to the input of the third stitching layer. The polymorphic superpixel convolutional unit includes a superpixel semantic unit, a fourth stitching layer, a restored spectral information unit, a restored spatial detail unit, a spectral spatial restorer, and a fusion unit. The input and output of the superpixel semantic unit are both connected to the input of the fourth stitching layer. The output of the fourth stitching layer is connected to the input of the restored spectrum information unit, the restored spatial detail unit, and the spectrum spatial restorer, respectively. The outputs of the restored spectrum information unit, the restored spatial detail unit, and the spectrum spatial restorer are all connected to the input of the fusion unit. The superpixel semantic unit comprises a sixth deep convolutional layer, a second convolutional layer, a probability processing layer, a softmax processing layer, and a region label processing layer connected in sequence.

2. The remote sensing image spectral super-resolution reconstruction method according to claim 1, characterized in that, The initial spectrum estimation module includes an in-wavelength estimation submodule and an out-of-wavelength estimation submodule connected in sequence; Remember RGB images ,image The original spectral channels include and ,in and Representing images respectively The red, green, and blue spectral channels of the image The input is fed into the wavelength estimation submodule for estimation, obtaining estimated values ​​for the spectral channels, including estimates for the red-green spectral channels and estimates for the green-blue spectral channels: in, Representing an image The estimated values ​​of the red and green spectral channels, Representing an image The estimated values ​​of the green and blue spectral channels, and Representing images respectively The red, green, and blue spectral channels; The original spectral channels and the estimated spectral channels are merged into an image. Fine spectral information is used as the output image of the wavelength estimation submodule: in, Representing an image Fine spectral information; The output image from the in-wavelength estimation submodule is input into the out-of-wavelength estimation submodule to obtain the initial spectral estimation features. The specific process is as follows: in, For the first Wavelength groups and respectively wavelength range The left and right endpoints, Wavelength ranges in sequence The three equal molecular regions, , and These represent the depth convolutional layers for the red, green, and blue channels, respectively.

3. The remote sensing image spectral super-resolution reconstruction method according to claim 1, characterized in that, In step S3, the method for determining the total training loss function includes: in, Denotes the total training loss function. , and These represent the width, height, and number of spectral channels of the hyperspectral image, respectively. For reference hyperspectral images, To reconstruct the hyperspectral image; Represents the spectral spatial loss function. This represents the equilibrium parameter.

4. The remote sensing image spectral super-resolution reconstruction method according to claim 3, characterized in that, The method for determining the spectral spatial loss function includes: in, Represents the spectral loss function. This represents the space loss function.

5. The remote sensing image spectral super-resolution reconstruction method according to claim 4, characterized in that, The method for determining the spectral loss function includes: in, and respectively by and We obtain it by expanding along the spectral dimension. This represents the dot product operation of the spectral radiance of each pixel. This represents the square root of the cumulative square of the spectral radiance of each pixel.

6. The remote sensing image spectral super-resolution reconstruction method according to claim 4, characterized in that, The method for determining the spatial loss function includes: in, and respectively by and It is obtained through metric transformation.

Images