Remote sensing image spectrum super-resolution reconstruction method

Abstract

The invention discloses a remote sensing image spectral super-resolution reconstruction method, which comprises the following steps: acquiring a remote sensing RGB image data set, and correspondingly converting each remote sensing RGB image into a reference hyperspectral image; the remote sensing RGB image data set is input into the constructed hierarchical semantic perception CNN-Transform model, and a corresponding reconstructed hyperspectral image is generated; constructing a frequency spectrum and space loss function according to the reference hyperspectral image and the reconstructed hyperspectral image, and training the model to obtain a trained hierarchical semantic perception CNN-Transform model; and inputting a remote sensing RGB image to be reconstructed into the trained model to obtain a reconstructed hyperspectral image. According to the method, information loss can be effectively prevented, the stability of the model in different use scenes is improved, and the hyperspectral image is more accurately reconstructed from the remote sensing RGB image.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and more specifically, to a remote sensing image spectral super-resolution reconstruction method. Background Art

[0002] Image spectral super-resolution reconstruction technology is a technology used to improve the spatial resolution and spectral resolution of images. It combines the advantages of image super-resolution and spectral imaging technology, and aims to reconstruct high-resolution images from low-resolution hyperspectral images, so that the processed images have both high-spectral and high-spatial resolution image data attributes. At present, hyperspectral imaging faces the following challenges: the spectral imager used to obtain hyperspectral images is expensive, resulting in increased costs; the spectral imager has high requirements for light and weather conditions; the spatial and spectral resolutions restrict each other, making it difficult to obtain images with high spectral resolution and high spatial resolution at the same time. Therefore, the use of spectral reconstruction technology can break through the performance bottleneck of spectral imagers, reduce costs, and further tap the potential of hyperspectral images. Spectral reconstruction technology includes two strategies: image fusion and spectral super-resolution. Since image fusion depends on the availability of MSI-HSI (Multi-Spectral Image-HyperspectralImage) image pairs, image spectral super-resolution reconstruction technology focuses on reconstructing low-spectral resolution images of the same sensor from the spectral dimension, so that the spectral feature information can be refined, and finally the processed images have both high-spectral and high-spatial resolution image data attributes.

[0003] Existing spectral super-resolution technologies can be divided into dictionary learning method, deep network method and deep prior method. These technologies still have the following disadvantages:

[0004] In terms of model architecture: the encoder-decoder (EnDeCoder) architecture and the continuous convolution (SeqConv) architecture with residual connection are common SSR (Spectral Super-Resolution) model architectures, but there are corresponding problems. The former is prone to lose spatial details, and the latter has inconsistent reconstruction performance in different scenarios.

[0005] Prior embedding: In SSR networks, it is more common to obtain prior information through data preprocessing rather than learning prior information.

[0006] Lightweight deployment: Current SSR methods focus more on reconstruction performance rather than model lightweight, making it difficult to achieve final deployment.

[0007] Application feasibility: Most SSR methods perform very well on standard datasets, but they still have limitations on cross-sensor data. Specifically, although existing fusion methods can effectively obtain high spatial resolution HSI (Hyperspectral Image), their model performance depends on the availability of MSI-HSI image pairs, but in fact, the corresponding MSI-HSI image pairs are difficult to obtain. Summary of the invention

[0008] In order to overcome the defects of technical information loss and low model reconstruction performance caused by the above-mentioned existing model reconstruction, the present invention provides a remote sensing image spectral super-resolution reconstruction method.

[0009] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0010] The present invention proposes a remote sensing image spectral super-resolution reconstruction method, comprising:

[0011] S1: Obtain a remote sensing RGB image dataset and convert each remote sensing RGB image into a reference hyperspectral image;

[0012] S2: Input the remote sensing RGB image dataset into the constructed hierarchical semantic perception CNN-Transformer model to generate the corresponding reconstructed hyperspectral image;

[0013] S3: constructing a total training loss function according to the reference hyperspectral image and the reconstructed hyperspectral image, training the hierarchical semantic perception CNN-Transformer model, and obtaining a trained hierarchical semantic perception CNN-Transformer model;

[0014] S4: Obtain the remote sensing RGB image to be reconstructed and input it into the trained hierarchical semantic perception CNN-Transformer model to obtain the reconstructed hyperspectral image.

[0015] Preferably, the hierarchical semantic perception CNN-Transformer model includes an ISE module, a first deep convolutional layer, a Concat layer, a second deep convolutional layer, a first SACT unit, a third deep convolutional layer, a first concatenation layer, a first convolutional layer, a PixelUnshuffle module, a fourth deep convolutional layer, a second SACT unit, a fifth deep convolutional layer and a PixelShuffle module;

[0016] The ISE module, the first deep convolutional layer, the Concat layer, the second deep convolutional layer, the first SACT unit, the third deep convolutional layer, the first concatenation layer, and the first convolutional layer are connected in sequence;

[0017] The PixelUnshuffle module, the fourth depth convolution layer, the second SACT unit, the fifth depth convolution layer and the PixelShuffle module are connected in sequence, and the output end of the PixelShuffle module is connected to the input end of the Concat layer;

[0018] The output of the ISE module is also connected to the input of the PixelUnshuffle module and the first splicing layer respectively;

[0019] The first SACT unit includes N SACT modules connected in sequence, and the second SACT unit includes N-1 SACT modules connected in sequence.

[0020] Preferably, the ISE module comprises an intra-wavelength estimation submodule and an extra-wavelength estimation submodule connected in sequence;

[0021] Remember the RGB image R, the original spectral channels of image R include S r , S g and S b , where S r , S g and S b Respectively represent the red, green and blue spectral channels of image R. Image R is input into the wavelength estimation submodule for estimation to obtain the estimated spectral channel estimation values, including the red and green spectral channel estimation values ​​and the green and blue spectral channel estimation values:

[0022] S rg =(S r +S g ) / 2

[0023] S gb =(S g +S b ) / 2

[0024] Among them, S rg Represents the estimated value of the red and green spectral channels of image R, S gb Represents the estimated value of the green and blue spectral channels of image R, S r , S g and S b denote the red, green and blue spectral channels of the image R respectively;

[0025] The original spectral channels and the estimated spectral channels are combined into the refined spectral information of image R as the output image of the wavelength estimation submodule:

[0026] S R ={S r ,S rg ,S g ,S gb ,S b}

[0027] Among them, S R Represents the fine spectral information of image R;

[0028] The output image of the intra-wavelength estimation submodule is input into the extra-wavelength estimation submodule to obtain the initial spectral estimation feature. The specific process is as follows:

[0029]

[0030] Where i = {λ R ,λ G ,λ B} is the i-th wavelength group, λ s and λ e are the left and right endpoints of the wavelength interval λ, respectively. R ,λ G ,λ B The three equal molecular intervals of wavelength interval λ are respectively, DWConv R (·), DWConv G (·) and DWConv B (·) denote the red, green and blue channel deep convolutional layers respectively.

[0031] Preferably, each of the SACT modules comprises a first PSConv submodule, a SACNN submodule, a second splicing layer, a LeakyReLU activation layer, a second PSConv submodule, a SAFormer submodule and a third splicing layer connected in sequence;

[0032] The first PSConv submodule and the second PSConv submodule are both connected in sequence by N PSConv devices;

[0033] The input of the first PSConv submodule is also connected to the input of the second concatenation layer, and the input of the LeakyReLU activation layer is also connected to the input of the third concatenation layer.

[0034] Preferably, the PSConv device includes a superpixel semantic device, a fourth splicing layer, a SpeConv device, a SpaConv device, a SpeSpaConv device and a fusion device;

[0035] The input and output ends of the superpixel semantic device are connected to the input end of the fourth splicing layer, the output end of the fourth splicing layer is connected to the input ends of the SpeConv device, the SpaConv device and the SpeSpaConv device respectively, and the output ends of the SpeConv device, the SpaConv device and the SpeSpaConv device are all connected to the input end of the aggregator.

[0036] Preferably, the superpixel semantic device includes a sixth deep convolutional layer, a second convolutional layer, a Probability processing layer, a Softmax processing layer and a Region Labels processing layer connected in sequence.

[0037] Preferably, in S3, the method for determining the total training loss function includes:

[0038] L total =L1+αL spe-spa

[0039] Among them, L total represents the total training loss function, W, L, and C are the width, height, and number of spectral channels of the hyperspectral image, respectively. is the reference hyperspectral image, To reconstruct the hyperspectral image; L spe-spa represents the spectral space loss function, and α represents the balance parameter.

[0040] Preferably, the method for determining the spectral spatial loss function comprises:

[0041] l spe-spa =2(l spe +l spa )

[0042] Among them, L spe represents the spectrum loss function, L spa represents the spatial loss function.

[0043] Preferably, the method for determining the spectrum loss function includes:

[0044]

[0045] in, and Respectively, H and Expanded along the spectral dimension, <·> represents the dot product operation of the spectral radiance of each pixel point, and ∥·∥ represents the square root of the cumulative square of the spectral radiance of each pixel point.

[0046] Preferably, the method for determining the spatial loss function includes:

[0047]

[0048] Among them, H spa and Respectively, H and Obtained by metric transformation.

[0049] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0050] The present invention proposes a remote sensing image spectral super-resolution reconstruction method, firstly, a remote sensing RGB image data set is obtained, and each remote sensing RGB image is converted into a reference hyperspectral image; the remote sensing RGB image data set is input into a constructed hierarchical semantic perception CNN-Transformer model to generate a corresponding reconstructed hyperspectral image; spectrum and spatial loss functions are constructed according to the reference hyperspectral image and the reconstructed hyperspectral image, and the model is trained to obtain a trained hierarchical semantic perception CNN-Transformer model; the remote sensing RGB image to be reconstructed is input into the trained model to obtain a reconstructed hyperspectral image. The present invention can effectively prevent information loss, improve the stability of the model in different usage scenarios, and more accurately reconstruct hyperspectral images from remote sensing RGB images. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 The figure is a flow chart of the remote sensing image spectral super-resolution reconstruction method described in Example 1;

[0052] Figure 2 This is a schematic diagram of the hierarchical semantic perception CNN-Transformer model described in Example 2;

[0053] Figure 3 This is a schematic diagram of the ISE module described in Example 2;

[0054] Figure 4 This is a schematic diagram of the SACT module described in Example 2;

[0055] Figure 5 Schematic diagram of the structure of the PSConv device described in Example 2;

[0056] Figure 6 This is a schematic diagram of the structure of the superpixel semantic device described in Example 2;

[0057] Figure 7 This is a graph showing the spectral super-resolution reconstruction performance of remote sensing images using the hierarchical semantic-aware CNN-Transformer proposed in Example 3 under different data sets and different numbers of superpixels. DETAILED DESCRIPTION

[0058] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0059] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0060] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0061] First, some concepts that may be involved in this application are explained:

[0062] Hierarchical Semantic-Aware CNN-Transformer (HSACT) is a model architecture that combines convolutional neural networks (CNNs) and transformers. It aims to simultaneously utilize the CNN hierarchical feature extraction capabilities and the Transformer global dependency modeling capabilities to improve performance in certain tasks.

[0063] Semantic-Aware CNN-Transformer (SACT).

[0064] Initial Spectral Estimation (ISE).

[0065] Polymorphic Superpixel Convolution (PSConv) is a method that can process polymorphic or multimodal features in images to improve the performance and effect of image processing algorithms.

[0066] Structure-Aware CNN (SACNN) is a convolutional neural network that can perceive the structural characteristics of input data.

[0067] Spectrum-Aware Transformer (SAFormer) is a spectrum-aware Transformer network for change detection in hyperspectral images.

[0068] Multi-Spectral Image-Hyperspectral Image (MSI-HSI).

[0069] Hyperspectral Image (HSI).

[0070] Spectral Super-Resolution (SSR).

[0071] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0072] Example 1

[0073] This embodiment provides a remote sensing image spectral super-resolution reconstruction method, such as Figure 1 As shown, including:

[0074] S1: Obtain a remote sensing RGB image dataset and convert each remote sensing RGB image into a reference hyperspectral image;

[0075] S2: Input the remote sensing RGB image dataset into the constructed hierarchical semantic perception CNN-Transformer model to generate the corresponding reconstructed hyperspectral image;

[0076] S3: constructing a total training loss function according to the reference hyperspectral image and the reconstructed hyperspectral image, training the hierarchical semantic perception CNN-Transformer model, and obtaining a trained hierarchical semantic perception CNN-Transformer model;

[0077] S4: Obtain the remote sensing RGB image to be reconstructed and input it into the trained hierarchical semantic perception CNN-Transformer model to obtain the reconstructed hyperspectral image.

[0078] In the specific implementation process, a remote sensing RGB image dataset is first obtained, and each remote sensing RGB image is converted into a reference hyperspectral image, which is input into the constructed hierarchical semantic perception CNN-Transformer model to generate a corresponding reconstructed hyperspectral image; then a total training loss function is constructed according to the reference hyperspectral image and the reconstructed hyperspectral image, and the hierarchical semantic perception CNN-Transformer model is trained to obtain a trained hierarchical semantic perception CNN-Transformer model; finally, the remote sensing RGB image to be reconstructed is input into the trained hierarchical semantic perception CNN-Transformer model to obtain a reconstructed hyperspectral image.

[0079] Example 2

[0080] This embodiment provides a remote sensing image spectral super-resolution reconstruction method, including:

[0081] S1: Obtain a remote sensing RGB image dataset and convert each remote sensing RGB image into a reference hyperspectral image;

[0082] Assume that a remote sensing RGB image is represented as The corresponding hyperspectral image is expressed as Wherein, L and W refer to the height and width of R and H respectively, c and C refer to the number of spectral channels of the two images respectively, and c=3<<C.

[0083] From the perspective of spectral imaging, the value I of the RGB camera at the spatial position (x, y) s(x,y) is defined as follows:

[0084] I s (x,y)=∫ Λ E(x,y,λ)A s (λ)dλ

[0085] Among them, s∈{S r ,S g ,S b} represents the color channel in the RGB image, E(x,y,λ n ) represents the spectral radiance within the wavelength range λ at the position (x, y), A s (λ) is the spectral channel, s is the spectral response function (SRF) from the hyperspectral image to the RGB image, and Λ represents the wavelength interval.

[0086] In practical scenarios, the wavelength interval is usually discretely sampled. s (x,y) is redefined as follows:

[0087]

[0088] Where n represents the number of spectral bands of the hyperspectral spectrum.

[0089] The conversion relationship between R and H is described as follows:

[0090] R=AH

[0091] in, is the spectral response of the RGB camera.

[0092] S2: Input the remote sensing RGB image dataset into the constructed hierarchical semantic perception CNN-Transformer model to generate the corresponding reconstructed hyperspectral image;

[0093] The deep learning-based SSR method overcomes the uncertainty problem of calculating A by learning a mapping function f(R; θ) to reconstruct H from R:

[0094]

[0095] Where L(·) represents the loss function.

[0096] A hierarchical network architecture is used to design the mapping model f(·), instead of the traditional EnDeCoder and SeqConv design modules, and the R structure is embedded in f(·) in an end-to-end learning manner. Finally, the model is optimized using the spectrum space loss function. The objective function of the hierarchical semantic perception CNN-Transformer method proposed in this embodiment is:

[0097]

[0098] Among them, L spe and L spa They refer to the spectral loss function and the spatial loss function respectively, α represents the trade-off parameter between spectral information and spatial information, and P s (R) Structural prior embedding provided for the designed PSConv module.

[0099] like Figure 2 , which is a schematic diagram of a hierarchical semantic perception CNN-Transformer model, wherein the hierarchical semantic perception CNN-Transformer model includes an ISE module, a first deep convolutional layer, a Concat layer, a second deep convolutional layer, a first SACT unit, a third deep convolutional layer, a first concatenation layer, a first convolutional layer, a PixelUnshuffle module, a fourth deep convolutional layer, a second SACT unit, a fifth deep convolutional layer and a PixelShuffle module;

[0100] The ISE module, the first deep convolutional layer, the Concat layer, the second deep convolutional layer, the first SACT unit, the third deep convolutional layer, the first concatenation layer, and the first convolutional layer are connected in sequence;

[0101] The PixelUnshuffle module, the fourth depth convolution layer, the second SACT unit, the fifth depth convolution layer and the PixelShuffle module are connected in sequence, and the output end of the PixelShuffle module is connected to the input end of the Concat layer;

[0102] The output of the ISE module is also connected to the input of the PixelUnshuffle module and the first splicing layer respectively;

[0103] The first SACT unit includes N SACT modules connected in sequence, and the second SACT unit includes N-1 SACT modules connected in sequence.

[0104] like Figure 3 As shown, it is a schematic diagram of an ISE module, and the ISE module includes an intra-wavelength estimation submodule and an extra-wavelength estimation submodule connected in sequence;

[0105] The original spectral channels of image R include S r , S g and S b , where S r , S g and S b Respectively represent the red, green and blue spectral channels of image R. Image R is input into the wavelength estimation submodule for estimation to obtain the estimated spectral channel estimation values, including the red and green spectral channel estimation values ​​and the green and blue spectral channel estimation values:

[0106] S rg =(S r +S g ) / 2

[0107] S gb =(S g +S b ) / 2

[0108] Among them, S rg Represents the estimated value of the red and green spectral channels of image R, S gb Represents the estimated value of the green and blue spectral channels of image R, S r , S g and S b denote the red, green and blue spectral channels of the image R respectively;

[0109] The original spectral channels and the estimated spectral channels are combined into the refined spectral information of image R as the output image of the wavelength estimation submodule:

[0110] S R ={S r ,S rg ,S g ,S gb ,S b}

[0111] Among them, S R Represents the fine spectral information of image R;

[0112] The output image of the intra-wavelength estimation submodule is input into the extra-wavelength estimation submodule to obtain the initial spectral estimation feature. The specific process is as follows:

[0113]

[0114] Where i = {λ R ,λ G ,λ B} is the i-th wavelength group, λ s and λ e are the left and right endpoints of the wavelength interval λ, respectively. R ,λ G ,λ B The three equal molecular intervals of wavelength interval λ are respectively, DWConv R (·), DWConv G (·) and DWConv B (·) denote the red, green, and blue channel deep convolutional layers, respectively, which include a 1×1Conv layer, a specular padding layer, a 3×3GroupConv layer, and a 1×1Conv layer in sequence. As the initial spectral estimation features, they are gradually restored to hyperspectral images through subsequent networks.

[0115] like Figure 4 As shown in FIG. 1 , it is a schematic diagram of a SACT module, each of which includes a first PSConv submodule, a SACNN submodule, a second splicing layer, a LeakyReLU activation layer, a second PSConv submodule, a SAFormer submodule, and a third splicing layer connected in sequence; the PSConv submodule is used to capture features, the SACNN submodule is used to reconstruct structural details, and the SAFormer submodule is a spectrum sensing transformer.

[0116] The first PSConv submodule and the second PSConv submodule are both connected in sequence by N PSConv devices;

[0117] The input of the first PSConv submodule is also connected to the input of the second concatenation layer, and the input of the LeakyReLU activation layer is also connected to the input of the third concatenation layer.

[0118] like Figure 5 , which is a schematic diagram of the structure of the PSConv device, wherein the PSConv device includes a superpixel semantic device, a fourth splicing layer, a SpeConv device, a SpaConv device, a SpeSpaConv device and a fusion device;

[0119] The input and output ends of the superpixel semantic device are connected to the input end of the fourth splicing layer, the output end of the fourth splicing layer is connected to the input ends of the SpeConv device, the SpaConv device and the SpeSpaConv device respectively, and the output ends of the SpeConv device, the SpaConv device and the SpeSpaConv device are all connected to the input end of the aggregator.

[0120] like Figure 6 The figure shows the structure of the superpixel semantic device, which includes the sixth deep convolution layer, the second convolution layer, the probability processing layer, the softmax processing layer and the region labels processing layer connected in sequence. The processing expression of the superpixel semantic device is:

[0121]

[0122] Among them, F s represents the output features after the superpixel semantics, X represents the input features of the superpixel semantics, Conv 1×1 (·) represents a 1×1 convolutional layer, DWConv(·) represents a deep convolutional layer, N s represents the preset number of superpixels, and Softmax(·) represents the Softmax processing layer function.

[0123] This embodiment uses a learnable method to extract superpixel semantic information from an image. This is different from the traditional pre-processing superpixel segmentation and embedding. The SpeConv outputs the spectral feature F spe , SpaConv outputs spatial features F spa , SpeSpaConv outputs the spectral spatial feature F spe-spa , as shown below:

[0124] F spe =SpeConv(X+w1·F s )

[0125] F spa = SpaConv(X+w2·F s )

[0126] F spe-spa =SpeSpaConv(X+w3·F s )

[0127] Among them, F spe 、F spa and F spe-spa denote the output features of SpeConv, SpaConv and SpeSpaConv respectively. SpeConv(·) denotes the processing function of SpeConv, which is a 1×1 convolutional layer. SpaConv(·) denotes the processing function of SpaConv, which is a 3×3 deep convolutional layer. SpeSpaConv(·) denotes the processing function of SpeSpaConv, which is a 1×1 convolutional layer and a 3×3 deep convolutional layer connected in sequence, i.e., a 3×3 deep convolutional layer after a 1×1 convolutional layer. w1, w2 and w3 are the learnable adaptive weights of SpeConv, SpaConv and SpeSpaConv respectively.

[0128] The SpeConv aims to learn deep semantic features to recover spectral information, the SpaConv focuses on recovering spatial details by deeply aggregating contextual information, and the SpeSpaConv comprehensively achieves spectral and spatial recovery.

[0129] In addition, this multimodal aggregation operator greatly enhances the feature extraction capability and enriches the feature learning space. Then, adaptive weights are added to the output features of SpeConv, SpaConv, and SpeSpaConv to obtain the output features of PSConv. Its expression is as follows:

[0130] F=w1·F spe +w2·F spa +w3·F spe-spa

[0131] For the SACNN submodule, the input features of SACNN are represented by F1 and the output features are represented by Indicates that the relationship between the output features and the input features is:

[0132]

[0133] Among them, W str represents structural attention, W str for:

[0134] W str =f s (Conv 1×1 C(MaxPool(F1),AvgPool(F1)))

[0135] Among them, f s (·) represents the Sigmoid function, Conv 1×1 (·) represents a 1×1 convolutional layer, C(·) represents a concatenation operation, MaxPool(·) and AvgPool(·) represent maximum pooling and average pooling, respectively. Represents matrix dot product.

[0136] For the SAFormer submodule, the designed SAFormer submodule is also applicable to the features obtained by the PSConv to further improve the spectral information. First, by inputting the features Expand the spatial dimension of key Sum The specific process is as follows:

[0137] W spe =Softmax(f SpeMax (QK T )-(QK T ))

[0138] Where W spe represents the spectral attention term, f SpeMax (·) means taking the maximum value, K T represents the transpose of K.

[0139] To W spe Perform vector broadcast operation to obtain the output features of the SAFormer submodule

[0140]

[0141] In the formula, β represents the network learning adaptive adjustment coefficient.

[0142] This embodiment designs a hierarchical network architecture to connect the above-mentioned ISE module, the first SACT unit, and the second SACT unit to obtain a reconstructed hyperspectral image. The advantages of the layered network architecture can be summarized as follows: (1) The reuse of feature decoders promotes the lightweight of the network; (2) The feature capture ability of the model is enhanced without losing the main spatial detail information. The specific process is as follows:

[0143]

[0144] in, It is the first i The input features of the SACT unit, l i ≥2, It is the first i The output characteristics of a SACT unit, Indicates the first i The equivalent function of the deep convolutional layer before and after the SACT unit, f PSF (·) is the PixelShuffle operation with lossless spatial downsampling information, f PSUF (·) indicates f PSF (·) is the reverse operation, SACT(·) represents the SACT module, Indicates the first i SACT parameters corresponding to each SACT unit.

[0145] Figure 2 It is a two-layer network architecture, including a first SACT unit and a second SACT unit. The first convolutional layer outputs a reconstructed hyperspectral image. for:

[0146]

[0147] Among them, F in It is the output image of the ISE module.

[0148] In order to further reduce the complexity of the network, the SACT module adopts a weight sharing strategy at each layer, and its calculation formula is as follows:

[0149]

[0150] S3: constructing a total training loss function according to the reference hyperspectral image and the reconstructed hyperspectral image, training the hierarchical semantic perception CNN-Transformer model, and obtaining a trained hierarchical semantic perception CNN-Transformer model;

[0151] In order to achieve the desired purpose of the hierarchical semantic perception CNN-Transformer method proposed in this embodiment, a spectral spatial loss function is designed in this embodiment. Specifically, the spectral spatial loss function is a function developed for the SSR task, which has two main uses: (1) measuring the similarity of spectral radiance within the wavelength range of a single pixel; (2) limiting the spatial fidelity of the spectral image at each sampling wavelength. The final loss function is defined as:

[0152] L total =L1+αL spe-spa

[0153] Among them, L total represents the total training loss function, W, L, and C are the width, height, and number of spectral channels of the hyperspectral image, respectively. is the reference hyperspectral image, To reconstruct the hyperspectral image; L spe-spa represents the spectral space loss function, and α represents the balance parameter.

[0154] The spectral spatial loss function is:

[0155] L spe-spa =2(L spe +L spa )

[0156] Among them, L spe represents the spectrum loss function, L spa represents the spatial loss function.

[0157] The spectrum loss function is:

[0158]

[0159] in, and Respectively, H and Expanded along the spectral dimension, <·> represents the dot product operation of the spectral radiance of each pixel point, and ∥·∥ represents the square root of the cumulative square of the spectral radiance of each pixel point.

[0160] The spatial loss function is:

[0161]

[0162] Among them, H spa and Respectively, H and Obtained by metric transformation.

[0163] S4: Obtain the remote sensing RGB image to be reconstructed and input it into the trained hierarchical semantic perception CNN-Transformer model to obtain the reconstructed hyperspectral image.

[0164] Finally, for the hierarchical semantic perception CNN-Transformer method proposed in this embodiment, in the model training stage, the sample batch, learning rate and number of training rounds are set to 128, 1×10 -3 and 200. In order to keep the model lightweight, the number of layers of the hierarchical network is preset to l = 2, and the designed spectrum space loss function and Adam optimizer are used to obtain the optimal weight hyperparameters of the network.

[0165] Example 3

[0166] Based on Example 2, this example uses the datasets of University of Pavia, Washington DC and Wenxian to explore the effect of the number of superpixels of the superpixel semanticizer on the performance of the hierarchical semantic perception CNN-Transformer method. Figure 7 As shown in the figure, the remote sensing image spectral super-resolution reconstruction performance of the hierarchical semantic perception CNN-Transformer proposed in this application under different data sets and different numbers of superpixels, from left to right are (a), (b), (c), corresponding to the University of Pavia, Washington DC and Wenxian data sets, the preset numbers of superpixels are 64, 128 and 256 respectively, and the CC, SAM and ERGAS three indicators are used to analyze the remote sensing image spectral super-resolution reconstruction performance of the hierarchical semantic perception CNN-Transformer method proposed in this application. It can be clearly seen from the figure that with the increase of the number of superpixel blocks, Figure 7 From (b) and (c), we can see that the performance of the hierarchical semantic-aware CNN-Transformer method proposed in this application has been significantly improved, which also reflects the two advantages of superpixel semantic embedding: (1) The more superpixel blocks there are, the smaller the local superpixel blocks are, and the stronger the correlation between internal pixels is, which enhances the ability of the hierarchical semantic-aware CNN-Transformer method proposed in this application to represent spectral spatial features; (2) Superpixel semantics can effectively guide the deep network to learn features from a local viewpoint, improving the spectral super-resolution reconstruction capability of the hierarchical semantic-aware CNN-Transformer.

[0167] The same or similar reference numerals correspond to the same or similar components;

[0168] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0169] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A remote sensing image spectral super-resolution reconstruction method, characterized in that: include: S1: Obtain a remote sensing RGB image dataset and convert each remote sensing RGB image into a reference hyperspectral image; S2: Input the remote sensing RGB image dataset into the constructed hierarchical semantic perception CNN-Transformer model to generate the corresponding reconstructed hyperspectral image; S3: constructing a total training loss function according to the reference hyperspectral image and the reconstructed hyperspectral image, training the hierarchical semantic perception CNN-Transformer model, and obtaining a trained hierarchical semantic perception CNN-Transformer model; S4: Obtain the remote sensing RGB image to be reconstructed and input it into the trained hierarchical semantic perception CNN-Transformer model to obtain the reconstructed hyperspectral image.

2. The remote sensing image spectral super-resolution reconstruction method according to claim 1, characterized in that: The hierarchical semantic perception CNN-Transformer model includes an ISE module, a first deep convolutional layer, a Concat layer, a second deep convolutional layer, a first SACT unit, a third deep convolutional layer, a first concatenation layer, a first convolutional layer, a PixelUnshuffle module, a fourth deep convolutional layer, a second SACT unit, a fifth deep convolutional layer and a PixelShuffle module; The ISE module, the first deep convolutional layer, the Concat layer, the second deep convolutional layer, the first SACT unit, the third deep convolutional layer, the first concatenation layer, and the first convolutional layer are connected in sequence; The PixelUnshuffle module, the fourth depth convolution layer, the second SACT unit, the fifth depth convolution layer and the PixelShuffle module are connected in sequence, and the output end of the PixelShuffle module is connected to the input end of the Concat layer; The output of the ISE module is also connected to the input of the PixelUnshuffle module and the first splicing layer respectively; The first SACT unit includes N SACT modules connected in sequence, and the second SACT unit includes N-1 SACT modules connected in sequence.

3. The remote sensing image spectral super-resolution reconstruction method according to claim 2, characterized in that: The ISE module comprises an intra-wavelength estimation submodule and an extra-wavelength estimation submodule connected in sequence; Remember the RGB image R, the original spectral channels of image R include S r , S g and S b , where S r , S g and S b Respectively represent the red, green and blue spectral channels of image R. Image R is input into the wavelength estimation submodule for estimation to obtain the estimated spectral channel estimation values, including the red and green spectral channel estimation values ​​and the green and blue spectral channel estimation values: S rg =(S r +S g ) / 2 S gb =(S g +S b ) / 2 Among them, S rg Represents the estimated value of the red and green spectral channels of image R, S gb Represents the estimated value of the green and blue spectral channels of image R, S r , S g and S b denote the red, green and blue spectral channels of the image R respectively; The original spectral channels and the estimated spectral channels are combined into the refined spectral information of image R as the output image of the wavelength estimation submodule: S R ={S r ,S rg ,S g ,S gb ,S b } Among them, S R Represents the fine spectral information of image R; The output image of the intra-wavelength estimation submodule is input into the extra-wavelength estimation submodule to obtain the initial spectral estimation feature. The specific process is as follows: Where i = {λ R ,λ G ,λ B } is the i-th wavelength group, λ s and λ e are the left and right endpoints of the wavelength interval λ, respectively. R ,λ G ,λ B The three equal molecular intervals of wavelength interval λ are respectively, DWConv R (·), DWConv G (·) and DWConv B (·) denote the red, green and blue channel deep convolutional layers respectively.

4. The remote sensing image spectral super-resolution reconstruction method according to claim 2, characterized in that: Each of the SACT modules includes a first PSConv submodule, a SACNN submodule, a second splicing layer, a LeakyReLU activation layer, a second PSConv submodule, a SAFormer submodule and a third splicing layer connected in sequence; The first PSConv submodule and the second PSConv submodule are both connected in sequence by N PSConv devices; The input of the first PSConv submodule is also connected to the input of the second concatenation layer, and the input of the LeakyReLU activation layer is also connected to the input of the third concatenation layer.

5. The remote sensing image spectral super-resolution reconstruction method according to claim 4, characterized in that: The PSConv device includes a superpixel semantic device, a fourth splicing layer, a SpeConv device, a SpaConv device, a SpeSpaConv device and a fusion device; The input and output ends of the superpixel semantic device are connected to the input end of the fourth splicing layer, the output end of the fourth splicing layer is connected to the input ends of the SpeConv device, the SpaConv device and the SpeSpaConv device respectively, and the output ends of the SpeConv device, the SpaConv device and the SpeSpaConv device are all connected to the input end of the aggregator.

6. The remote sensing image spectral super-resolution reconstruction method according to claim 5, characterized in that: The superpixel semantic device includes a sixth deep convolution layer, a second convolution layer, a Probability processing layer, a Softmax processing layer and a Region Labels processing layer which are connected in sequence.

7. The remote sensing image spectral super-resolution reconstruction method according to claim 4, characterized in that: In S3, the method for determining the total training loss function includes: L total =L1+αL spe-spa Among them, L total represents the total training loss function, W, L, and C are the width, height, and number of spectral channels of the hyperspectral image, respectively. is the reference hyperspectral image, To reconstruct the hyperspectral image; L spe-spa represents the spectral space loss function, and α represents the balance parameter.

8. The remote sensing image spectral super-resolution reconstruction method according to claim 7, characterized in that: The method for determining the spectral space loss function includes: l spe-spa ＝2(l spe +l spa ) Among them, L spe represents the spectrum loss function, L spa represents the spatial loss function.

9. The remote sensing image spectral super-resolution reconstruction method according to claim 8, characterized in that: The method for determining the spectrum loss function includes: in, and Respectively, H and Expanded along the spectral dimension, <·> represents the dot product operation of the spectral radiance of each pixel point, and ∥·∥ represents the square root of the cumulative square of the spectral radiance of each pixel point.

10. The remote sensing image spectral super-resolution reconstruction method according to claim 8, characterized in that: The method for determining the spatial loss function includes: Among them, H spa and Respectively, H and Obtained by metric transformation.

Images