Dual-stage diffusion remote sensing hyperspectral reconstruction method for disaster scene

Abstract

The invention discloses a disaster scene-oriented double-stage diffusion remote sensing hyperspectral reconstruction method. The method comprises the following steps of: acquiring a data set of a multispectral image and a hyperspectral image; generating a prior noise image; constructing conditional prior; generating an intermediate hyperspectral image through reverse diffusion iteration; and a more refined hyperspectral image is obtained through further optimization of the spectrum correction network. According to the method, a prior-guided diffusion method is adopted, accurate hyperspectral construction under a single sample is realized by utilizing spectral prior information, and hyperspectral data can be effectively generated in a scene lacking hyperspectral data; in the spectrum correction process, a progressive design is adopted to divide an input image into a plurality of hierarchies, the noise problem introduced in the diffusion process is gradually eliminated in the hierarchies with different resolutions, and high-quality generation of a hyperspectral image is completed. According to the method, a spectrum super-resolution generation task can be completed based on a single sample, on the other hand, the structural precision of a hyperspectral image generated by an algorithm can be improved, and the hyperspectral data generation effect is remarkably improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of spectral image reconstruction, and particularly relate to a two-stage diffusion remote sensing hyperspectral reconstruction method for disaster scenes. BACKGROUND

[0002] With the development of imaging sensors in recent years, such as microwave radar, multispectral sensors, hyperspectral sensors, etc., microwave images, multispectral images, hyperspectral images, etc. can be generated respectively.

[0003] Different kinds of satellite imaging can provide complementary all-round information for disaster sites, and thus provide reliable information for emergency disaster relief. In particular, hyperspectral images have up to hundreds of spectral bands, which can capture rich spectral information in the scene and assist in rapid judgment of the disaster scene in disaster relief tasks. However, the high imaging cost and low spatial resolution of hyperspectral images hinder their application in the field of emergency rescue. Relatively speaking, multispectral images represented by RGB images have low cost and high spatial resolution. In view of the complementary relationship between the imaging characteristics of the two, the spectral super-resolution method for generating hyperspectral images from RGB images has strong application value.

[0004] Spectral super-resolution is a complex inverse problem, and the solution depends not only on simple difference methods, but also on advanced methods such as machine learning and deep learning to achieve high-precision spectral super-resolution. Through the constructed model, the complex spectral information is learned, and then the spectral band information with higher resolution is recovered. According to the specific technical approach, different methods can be summarized as spectral super-resolution methods based on sparse representation and spectral super-resolution methods based on deep learning. The former uses the sparsity of hyperspectral data to establish a mapping relationship between hyperspectral data and RGB images, and generates hyperspectral data by using sparse mapping. This kind of method needs to use spectral response function, and the acquisition of this function often needs to obtain the parameters related to the imaging sensor, thereby limiting the application scene of this kind of method. With the rapid development of deep learning in recent years, through end-to-end training, the characteristic distribution of the target data can be better learned. The network architecture of this kind of method usually includes convolutional neural network, generative adversarial network and attention mechanism, etc. key components.

[0005] Although the above methods can effectively learn the representation of the data itself, a large amount of data is needed to drive the accurate construction of hyperspectral data, and there is a lack of specific network structure design for the structure of hyperspectral data. The above methods still have great limitations in the field of spectral super-resolution.

[0006] Therefore, it is necessary to improve one or more problems existing in the above related technical solutions.

[0007] It should be noted that this section aims to provide background or context to the technical solutions of the disclosure stated in the claims. The description herein is not admitted to be prior art merely because it is included in this section. SUMMARY

[0008] The purpose of the embodiments of the present disclosure is to provide a two-stage diffusion remote sensing hyperspectral reconstruction method for disaster scenarios, thereby at least partially overcoming one or more problems caused by the limitations and defects of the related art.

[0009] The embodiments of the present disclosure first provide a two-stage diffusion remote sensing hyperspectral reconstruction method for disaster scenarios, comprising: obtaining original multi-spectral-hyperspectral data of multiple bands, dividing the original multi-spectral-hyperspectral data into a test set, a training set and a validation set comprising multi-spectral images and hyperspectral images; performing band difference processing on the multi-spectral images in the test set to form a multi-band image, and connecting the multi-band image and a noise image in the spectral dimension to form a prior noise image; performing singular value decomposition on the spectral response function, and then constructing a conditional prior using the singular value pseudo-inverse matrix, spectral singular vectors and spatial singular vectors obtained by the decomposition; constraining the inverse diffusion iteration process using the conditional prior, generating an intermediate hyperspectral image by inverse diffusion iteration on the prior noise image, optimizing the intermediate hyperspectral image using a loss function composed of a root mean square error function and a total variation loss, and an optimizer, to obtain an optimized intermediate hyperspectral image; performing spectral correction on the optimized intermediate hyperspectral image, obtaining three aggregated feature maps of different spectral dimensions using downsampling in the spectral correction process, connecting the three aggregated feature maps in the spectral dimension to obtain a fused feature map, and performing convolution and activation processing on the fused feature map to obtain a reconstructed hyperspectral image.

[0010] In an embodiment of the present disclosure, the step of performing band difference processing on the multi-spectral images in the test set to form a multi-band image, and connecting the multi-band image and a noise image in the spectral dimension to form a prior noise image, comprises: obtaining a first difference image by subtracting the blue band from the green band of the multi-spectral images in the test set; obtaining a second difference image by subtracting the green band from the red band of the multi-spectral images in the test set; forming a five-band image in the order of the blue band, the first difference image, the green band, the second difference image, and the red band; connecting the five-band image and a noise image in the spectral dimension to form a prior noise image.

[0011] In an embodiment of the present disclosure, the step of performing singular value decomposition on the spectral response function, and then constructing a conditional priori by using the singular value pseudo-inverse matrix, the spectral singular vector and the spatial singular vector obtained from the decomposition, comprises: performing singular value decomposition on the spectral response function to obtain a spatial singular vector, a spectral singular vector and a singular value matrix, and obtaining a singular value pseudo-inverse matrix from the singular value matrix; constructing a conditional priori by using the priori noise image, the spatial singular vector and the singular value pseudo-inverse matrix; obtaining a conditional priori variance according to the conditional priori.

[0012] In an embodiment of the present disclosure, the step of constraining the inverse diffusion iterative process by using the conditional priori, and generating an intermediate hyperspectral image by iteratively diffusing the priori noise image, comprises: embedding the conditional priori into the inverse diffusion iterative process for constraint; taking the priori noise image as the initial input of inverse diffusion, and performing the following multiple rounds of inverse diffusion iteration: for each time step, inputting the current variable into a noise prediction network, wherein the noise prediction network is a U-shaped structure without skip connection, and contains a channel-level multi-head attention mechanism for focusing on spectral dimension features and estimating the noise component of the current time step; based on an inverse recursive formula, eliminating part of the noise in the current variable by using the conditional priori and the estimated noise component to obtain the variable of the next time step; in the iterative process, the spatial structure priori information of the multi-band image is preserved by the conditional priori, while the noise of the noise image is removed, so that the variable gradually changes from noise-dominant to hyperspectral feature-dominant; when the iteration reaches time step 0, the output variable is the intermediate hyperspectral image.

[0013] In an embodiment of the present disclosure, in the process of optimizing the intermediate hyperspectral image by using a loss function composed of a root mean square error function and a total variation loss and an optimizer, 1000 rounds of iterative optimization are performed, and the loss in the first 500 rounds does not participate in gradient back propagation.

[0014] In an embodiment of the present disclosure, the step of performing spectral correction on the optimized intermediate hyperspectral image, and using down-sampling to obtain three kinds of aggregated feature maps with different spectral dimensions in the spectral correction process, comprises: performing down-sampling operation on the intermediate hyperspectral image to obtain three first feature maps with different spectral dimensions and spatial dimensions; performing convolution processing on the three first feature maps respectively to obtain corresponding three second feature maps; perform multi-head attention processing on the first feature map to obtain a third feature map; sum the second feature map and the third feature map to obtain an aggregated feature map of three different spectral dimensions.

[0015] In an embodiment of the present disclosure, the reconstruction method further comprises: The root mean square error function is used as a loss function, and an Adam optimizer is used to carry out self-supervised learning to optimize the spectral correction process.

[0016] In an embodiment of the present disclosure, the root mean square error function is used as a loss function, and an Adam optimizer is used to carry out self-supervised learning to optimize the spectral correction process for 500 iterations.

[0017] The present disclosure further provides a two-stage diffusion remote sensing hyperspectral reconstruction system for disaster scenarios, comprising: A data set construction processing module is configured to obtain original multi-spectral-hyperspectral data of multiple bands, and divide the original multi-spectral-hyperspectral data into a test set, a training set and a validation set comprising multi-spectral images and hyperspectral images. An a priori noise image generation module is configured to perform band difference processing on the multi-spectral images in the test set to form multi-band images, and form a priori noise images by connecting the multi-band images and noise images in the spectral dimension. A conditional prior construction module is configured to perform singular value decomposition on a spectral response function, and then construct a conditional prior comprising a singular value pseudo-inverse matrix and a conditional prior variance. A prior diffusion module is configured to constrain an inverse diffusion iteration process by using the conditional prior, generate an intermediate hyperspectral image by performing inverse diffusion iteration on the a priori noise image, and optimize the intermediate hyperspectral image by using a loss function composed of a root mean square error function and a total variation loss and an optimizer to obtain an optimized intermediate hyperspectral image. A spectral correction module is configured to perform spectral correction on the optimized intermediate hyperspectral image, obtain aggregated feature maps of three different spectral dimensions by using down-sampling in the spectral correction process, connect the three aggregated feature maps in the spectral dimension to obtain a fusion feature map, and perform convolution and activation processing on the fusion feature map to obtain a reconstructed hyperspectral image.

[0018] The technical scheme provided by the embodiments of the present disclosure can have the following beneficial effects: The two-stage diffusion remote sensing hyperspectral reconstruction method for disaster scenarios in the embodiments of the present disclosure adopts a priori guided diffusion method, uses spectral prior information to achieve precise hyperspectral construction under single sample, and can effectively generate hyperspectral data in a scene with insufficient hyperspectral data; the spectral correction process adopts a progressive design to divide the input image into multiple levels, and gradually eliminates the noise problem introduced in the diffusion process in different resolution levels, thereby completing the high-quality generation of the hyperspectral image. On the one hand, the present application can complete the spectral super-resolution generation task based on a single sample, and on the other hand, it can improve the structural accuracy of the hyperspectral image generated by the algorithm, and significantly improve the hyperspectral data generation effect. BRIEF DESCRIPTION OF DRAWINGS

[0019] The drawings incorporated into the specification and forming part of the specification, show embodiments consistent with the present disclosure, and together with the specification, serve to explain the principles of the present disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can also be obtained from these drawings without creative labor for those skilled in the art.

[0020] Figure 1 A flowchart of the two-stage diffusion remote sensing hyperspectral reconstruction method for disaster scenarios in the exemplary embodiments of the present disclosure is shown; Figure 2 A flowchart of generating a prior noise image in the exemplary embodiments of the present disclosure is shown; Figure 3 A process schematic diagram of generating a prior noise image in the exemplary embodiments of the present disclosure is shown; Figure 4 A flowchart of singular value decomposition of the spectral response function and then constructing a conditional prior containing a singular value pseudo-inverse matrix and a conditional prior variance in the exemplary embodiments of the present disclosure is shown; Figure 5 A flowchart of generating an intermediate hyperspectral image in the exemplary embodiments of the present disclosure is shown; Figure 6 A model architecture diagram of the reconstruction system in the exemplary embodiments of the present disclosure is shown; Figure 7 A flowchart of generating an aggregated feature map in the exemplary embodiments of the present disclosure is shown; Figure 8 An image processing flowchart in the reconstruction method in the exemplary embodiments of the present disclosure is shown; Figure 9 A module structure diagram of the two-stage diffusion remote sensing hyperspectral reconstruction system for disaster scenarios in the exemplary embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0021] Example implementations are now described with reference to the drawings. Example implementations can, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example implementations to those skilled in the art. The described features, structures, or characteristics can be combined in one or more implementations.

[0022] In addition, the drawings are merely schematic and are not drawn to scale. Identical reference numerals in the figures designate identical or similar parts throughout the figures and text. Some of the blocks in the drawings are functional blocks that do not necessarily have a corresponding physical or logical entity in an implantation, and the connection between blocks can be a direct connection, or can be indirect through one or more intervening blocks.

[0023] In the present example implementation, a two-stage diffusion remote sensing hyperspectral reconstruction method for disaster scenarios is first provided, please refer to Figure 1 , which can include S101-S105. Specifically as follows: S101, obtaining original multi-spectral-hyperspectral data of multiple bands, dividing the original multi-spectral-hyperspectral data into a test set, a training set and a validation set including multi-spectral images and hyperspectral images. The training set is used for data training, and the validation set is used to verify the performance of the reconstruction method.

[0024] S102, performing band difference processing on the multi-spectral images in the test set to form a multi-band image, and connecting the multi-band image with a noise image in the spectral dimension to form a prior noise image.

[0025] S103, singular value decomposition is performed on the spectral response function, and then a pseudo-inverse matrix, spectral singular vectors and spatial singular vectors obtained by decomposition are used to construct a conditional prior.

[0026] S104, the inverse diffusion iteration process is constrained by using the conditional prior, and the intermediate hyperspectral image is generated by inverse diffusion iteration on the prior noise image. A loss function composed of a root mean square error function and a total variation loss and an optimizer are used to optimize the intermediate hyperspectral image, and an optimized intermediate hyperspectral image is obtained.

[0027] S105, performing spectral correction on the optimized intermediate hyperspectral image, obtaining three kinds of aggregation feature maps of different spectral dimensions by using down-sampling in the spectral correction process, connecting the three kinds of aggregation feature maps in the spectral dimension to obtain a fusion feature map, and performing convolution and activation processing on the fusion feature map to obtain a reconstructed hyperspectral image.

[0028] In this embodiment, the prior guided diffusion method is used to realize the construction of hyperspectral image with high precision under single sample by using spectral prior information, which can effectively generate hyperspectral data in the scene with insufficient hyperspectral data; the spectral correction process adopts progressive design to divide the input image into multiple levels, and gradually eliminates the noise problem introduced in the diffusion process in different resolution levels, thereby completing the generation of high-quality hyperspectral image. On the one hand, the present application can complete the spectral super-resolution generation task based on single sample, on the other hand, it can improve the structural accuracy of the hyperspectral image generated by the algorithm, and significantly improve the effect of hyperspectral data generation.

[0029] The specific process of each step in the above embodiment is described below.

[0030] In S101, the data set used by the present application is the Oribiter hyperspectral satellite data set launched by China, the spectral resolution in the data set is 10 nanometers, the distribution of the wave band is 400 nanometers to 1000 nanometers, and the spatial resolution of the image is 10 meters. In the actual training process, the original L1B level data is respectively subjected to atmospheric correction and geometric correction, and the original data is cropped, and finally 9025 pieces of hyperspectral data with a spatial resolution of 256 256 are obtained. The corresponding RGB image is generated by means of the wave band selection recommended by the official manual. The final multi-spectral-hyperspectral data set is composed of 9025 pairs of multi-spectral-hyperspectral data set, among which 7581 pairs are randomly selected for training set for comparative model training, 722 pairs are for verification set for model verification, and 722 pairs are for test set for model testing.

[0031] Please refer to Figure 2 , S102 includes S201~S204, which are as follows: S201, the blue wave band and the green wave band of the multi-spectral image in the test set are subtracted to obtain a first difference image B1.

[0032] S202, the green wave band and the red wave band of the multi-spectral image in the test set are subtracted to obtain a second difference image B2.

[0033] S203, a five-wave band image F1 is formed according to the wave band order of the blue wave band, the first difference image B1, the green wave band, the second difference image B2 and the red wave band.

[0034] S204, the five-wave band image F1 and the noise image F2 are connected in parallel in the spectral dimension to form a prior noise image F3.

[0035] In this embodiment, combined with Figure 3 , the initial input is 256 256 3, after difference processing and sorting, the five-band image F1 is 256 256 5. The noise image F2 is composed of pure Gaussian noise, and the specific image size is 256 256 26. Then the five-band image F1 and the noise image F2 are connected in the spectral dimension to form a 256 256 31 prior noise image F3. The prior noise image F3 is the input data of the subsequent diffusion processing.

[0036] Please refer to Figure 4 , S103 includes S301~S303, as follows: S301, singular value decomposition is performed on the spectral response function to obtain spatial singular vectors, spectral singular vectors and singular value matrix, and singular value pseudo-inverse matrix is obtained according to the singular value matrix; S302, constructing a conditional prior by the prior noise image, spatial singular vectors and singular value pseudo-inverse matrix; S303, obtaining a conditional prior variance according to the conditional prior.

[0037] In this embodiment, the spectral response function , is singular value decomposed (Singular Value Decomposition, SVD) to obtain spatial singular vectors , spectral singular vectors and singular value matrix . Feature construction of input data is realized by singular value decomposition. Then the singular value matrix is further inverted to obtain the singular value pseudo-inverse matrix , the conditional prior and the iteration intermediate variable are obtained according to the prior noise image F3, and , wherein is the prior noise image F3, is the noise image at time step t.

[0038] Next, please refer to Figure 5 , the generation process of the intermediate hyperspectral image in S104 includes S401~S406, as follows: S401, embedding the conditional prior into the inverse diffusion iteration process for constraint; S402, taking the prior noise image as the initial input of inverse diffusion, and performing the following multiple rounds of inverse diffusion iteration: S403, For each time step, the current variable is input into the noise prediction network. The noise prediction network is a U-shaped structure without skip connections and includes a channel-level multi-head attention mechanism to focus on spectral dimension features and estimate the noise component of the current time step. S404, Based on the inverse recursive formula, the prior conditions and estimated noise components are used to eliminate some noise in the current variable to obtain the variable for the next time step; S405, During the iteration process, the spatial structure prior information of the multi-band image is retained through the aforementioned conditional prior, while noise in the noisy image is removed, so that the variable gradually shifts from noise-dominated to hyperspectral feature-dominated. S406, when iterating to time step 0, the output variable is the intermediate hyperspectral image.

[0039] In this process, the prior noisy image F3 is the input data for the diffusion process. The inverse recursive formula for the diffusion process is as follows:

[0040] in, Let be the variance of the variable at time step t. Depend on Decide; For conditional prior variance; and All of these are hyperparameters for controlling the noise level in each step; Singular value matrix The i-th singular value in These are the intermediate variables obtained through the sampling network during the iteration. Here are the network parameters at time step t; It follows a normal distribution.

[0041] Please refer to Figure 6 In stage 1, the sampling network has a U-shaped structure, and its specific structural components are as follows:

[0042] The sampling network does not contain any skip connection modules, which maximizes the prediction and elimination of noise. In addition, the Multi-Head Attention used calculates channel-level attention, which can better reconstruct spectral information and reduce over-resolution error.

[0043] The iteration starting variable of the diffusion process can be obtained by using the reverse recursive formula of the above diffusion process. That is, processed data and The intermediate hyperspectral image H1, with a size of 256, was obtained through recursion. 256 31.

[0044] The present application adopts a two-stage diffusion network. Please refer to Figure 6 and Figure 8 . The first stage is that the prior noise image constructed by noise and multispectral is input into the prior diffusion module, and after multiple iterations, it is a reverse diffusion process belonging to the diffusion model. In this process, the prior information in the prior noise image is saved as useful information to impose spatial constraints on the generated hyperspectral image, while the noise image is removed in the iteration process. In each time step, the noise prediction network is used to estimate and eliminate the image noise, and the prior noise information transmits the structural information to the feature image generated at different time steps to ensure that the generated feature image can have similar spatial features to the input multispectral image. The second stage is that the feature image is input into the spectral correction module to complete the recovery of the spectral features and spatial features of the image at different granularities. This process is achieved through a hierarchical network architecture. In this architecture, convolution and attention mechanism are applied to feature images at different scales to extract local information and global information respectively, and then the progressive generation of hyperspectral images is completed at full scale.

[0045] In the diffusion process, the present application uses the combination of mean square error (MSE) and total variation loss (TV Loss) as the loss function of the model , and uses the Adam optimizer (for example, the Adam optimizer proposed by Diederik et al. in the literature D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” arXiv.org , Dec. 22, 2014.) to optimize the diffusion process. 1000 iterations of optimization are performed, and the loss in the first 500 iterations does not participate in gradient backpropagation.

[0046] Next, please refer to Figure 7 , in S105, the steps of obtaining the aggregated feature map through spectral correction are as follows: S501, down-sampling the intermediate hyperspectral image to obtain three first feature maps with different spectral dimensions and spatial dimensions; S502, convolving the three first feature maps respectively to obtain corresponding three second feature maps; S503, performing multi-head attention processing on the first feature map to obtain a third feature map; S504, sum the second feature map and the third feature map to obtain an aggregated feature map of three different spectral dimensions.

[0047] The spectral correction network used in the spectral correction process is a three-level network structure. First, the input intermediate hyperspectral image H1 is subjected to a downsampling operation to obtain a first feature map D1 of size 256 256 31, a first feature map D2 of size 128 128 124, and a first feature map D3 of size 64 64 496. In the first level, D1 is first input into 1 1 and 3 3 convolution modules, and then the feature map D1 is input into 3 3 convolution modules in the spectral dimension to obtain a second feature map D1'; at the same time, D1 is input into a multi-head attention module, and then D1 is input into a LayerNorm module to obtain a third feature map D1'', and finally D1' and D1'' are summed to obtain an aggregated feature map D11 of size 256 256 31 in the first level. Similarly, the second level outputs an aggregated feature map D21 of size 256 256 124, and the third level outputs an aggregated feature map D31 of size 256 256 496.

[0048] Finally, D11, D21 and D31 are connected in the spectral dimension to form a fused feature map D4 of size 256 256 651, which is subjected to 3 3 convolution and LeakyReLU activation functions to obtain a final output of size 256 256 31, i.e. the reconstructed hyperspectral image.

[0049] Spectral correction is performed to restore the fine-grained spectral information of the generated hyperspectral feature map after diffusion. Referring to Figure 6 Stage 2, the structure of the spectral correction module is composed of: feature map H1→PixelShuffle spatial size is reduced in turn, and the spectral dimension is increased in turn Feature map D1, D2 and D3→Conv11 and Conv33 in parallel→Concat→Conv3 3→LeakyReLU→feature map D11, D21, D31→D1, D2, D3 are input into Multi-Head Attention respectively→LayerNorm feature map→D12, D22, D32→feature map D11, D21, D31 and feature map D12, D22, D32 are added to obtain E1, E2, E3→Upsample→Concat→output image size is 256 256 31 final result F. The spectral correction module designed in the application can divide the original feature map into different granularities to extract spatial features and spectral features respectively, and the cross-dimension aggregation of the two can generate a high-fidelity fine-grained hyperspectral image.

[0050] The application also optimizes the spectral correction process, adopts Mean Square Error (MSE) as the loss function of the model , and adopts the Adam optimizer to carry out self-supervised learning, and iteratively optimizes the spectral correction process for 500 times.

[0051] To sum up, the application proposes a two-stage diffusion model for a spectral super-resolution task, which mainly consists of a priori guided diffusion module and a spectral correction module. The application proposes a priori diffusion module that does not need to be trained. The module uses prior noise information and can complete high-quality conversion from multispectral to hyperspectral under limited data volume. The spectral correction network proposed in the application can effectively reduce the impression of noise introduced by the diffusion module, and the designed multi-level structure can enhance spectral fidelity at different scales.

[0052] The reverse diffusion process of the application is realized by a single sample sampling network, which mainly consists of convolution operation, deconvolution operation, MultiHeadAttention, LayerNorm function, GELU activation function and LeakyReLU activation function. The input is a three-channel multispectral image and a noise image, and the output is a multi-channel hyperspectral image. The spectral correction network mainly consists of PixelShuffle, Conv3 3, Conv1 1, LeakyReLU activation function, Upsample function and Multi-Head Attention function, the input is the feature image output by the first stage, and the output is the reconstructed hyperspectral image.

[0053] The application also provides a two-stage diffusion remote sensing hyperspectral reconstruction system for disaster scenarios, please refer to Figure 9The system comprises a dataset construction processing module 101, a prior noise image generation module 102, a conditional prior construction module 103, a prior diffusion module 104, and a spectral correction module 105.

[0054] Specifically, the dataset construction processing module 101 is configured to acquire original multi-spectral-hyperspectral data of multiple bands, divide the original multi-spectral-hyperspectral data into a test set, a training set, and a validation set comprising multi-spectral images and hyperspectral images. The prior noise image generation module 102 is configured to perform band difference processing on the multi-spectral images in the test set to form multi-band images, and form prior noise images by connecting the multi-band images and noise images in the spectral dimension. The conditional prior construction module 103 is configured to perform singular value decomposition on a spectral response function, and then construct a conditional prior by using a singular value pseudo-inverse matrix, spectral singular vectors, and spatial singular vectors obtained by the decomposition. The prior diffusion module 104 is configured to constrain an inverse diffusion iteration process by using the conditional prior, generate an intermediate hyperspectral image by performing inverse diffusion iteration on the prior noise image, and obtain an optimized intermediate hyperspectral image by using a loss function composed of a root mean square error function and a total variation loss and an optimizer to optimize the intermediate hyperspectral image. The spectral correction module 105 is configured to perform spectral correction on the optimized intermediate hyperspectral image, obtain three kinds of aggregated feature maps of different spectral dimensions by downsampling in the spectral correction process, connect the three kinds of aggregated feature maps in the spectral dimension to obtain a fused feature map, and obtain a reconstructed hyperspectral image by performing convolution and activation processing on the fused feature map.

[0055] In this embodiment, the roles of the modules in the system and the image processing process are the same as those of the foregoing reconstruction method, and will not be described here again.

[0056] The beneficial effects of the two-stage diffusion remote sensing hyperspectral reconstruction method for disaster scenarios of the present application will be further described below through specific experimental examples.

[0057] 1. Experimental environment and settings The software framework of the experiment of the application is PyTorch1.6, and the hardware platform is RTX3090 graphics card. In order to verify the effectiveness of the application, the application carries out experiments on the Zhuhai No. 1 hyperspectral satellite data set proposed by He et al. in the literature J. He, J. Li, Q. Yuan, H. Shen, and L. Zhang, “Spectral Response Function-Guided Deep Optimization-Driven Network for Spectral Super-Resolution,” IEEE Trans. Neural Netw. Learning Syst., pp. 1-15, 2021, and the performance comparison results between the application and other reconstruction methods are shown in Table 1.

[0058] Table 1 Performance index experimental results of the method of the application and the existing spectral reconstruction method

[0059] 2, Analysis of experimental results From the comparison results in Table 1, by comparing the effects of the method proposed in the application with the existing methods based on sparse coding, convolutional neural network, GAN and Transformer-based spectral super-resolution, the method proposed in the application leads the existing spectral reconstruction methods in the three indicators of root mean square error, average relative absolute error and relative root mean square error. This is because the application can fully utilize the characteristics of sparse spectrum in remote sensing hyperspectral images, establish an effective spectral attention mechanism to effectively model the correlation between different spectral bands; at the same time, the application uses a multi-level feature extraction method to remove the degradation existing in the image, and has good effect, which improves the expression ability of the model and the robustness in complex situations.

[0060] From the above, the prior diffusion module proposed in the application can realize effective reconstruction and generation of multispectral images by means of the diffusion model, and then realize the construction work of high-quality hyperspectral images; at the same time, the prior diffusion module guided by the prior and the spectral correction module form a complementary coupling relationship, which can accurately aim at the defects of the diffusion model in the generation of fine-grained structure information, realize the reconstruction of hyperspectral details under different granularities, and finally efficiently and accurately realize the generation of hyperspectral images.

[0061] It should be noted that although several modules of the system for action execution are mentioned in the above detailed description, such division is not mandatory. In fact, according to embodiments of the present application, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be embodied. The components shown as modules can or can not be physical units, i.e. can be located in one place, or can also be distributed to multiple network units. Part or all of the modules can be selected to achieve the purpose of the present application scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0062] The embodiments of the present application are described above in conjunction with the drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative, not restrictive, and those skilled in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, which are all within the protection of the present application.

Claims

1. A two-stage diffusion remote sensing hyperspectral reconstruction method for disaster scenarios, characterized by, The method comprises the following steps: acquiring original multi-spectral-hyperspectral data of multiple bands, and dividing the original multi-spectral-hyperspectral data into a test set, a training set and a validation set comprising multi-spectral images and hyperspectral images; performing band difference processing on the multi-spectral images in the test set to form multi-band images, and connecting the multi-band images with noise images in the spectral dimension to form prior noise images; performing singular value decomposition on a spectral response function, and then constructing a conditional prior using a singular value pseudo-inverse matrix, a spectral singular vector and a spatial singular vector obtained by the decomposition; constraining an inverse diffusion iteration process using the conditional prior, and performing inverse diffusion iteration on the prior noise images to generate intermediate hyperspectral images, and optimizing the intermediate hyperspectral images using a loss function composed of a root mean square error function and a total variation loss and an optimizer to obtain optimized intermediate hyperspectral images; performing spectral correction on the optimized intermediate hyperspectral images, obtaining three kinds of aggregated feature maps of different spectral dimensions by downsampling in the spectral correction process, connecting the three kinds of aggregated feature maps in the spectral dimension to obtain a fusion feature map, and performing convolution and activation processing on the fusion feature map to obtain reconstructed hyperspectral images.

2. The method of claim 1, wherein, The step of performing band difference processing on the multi-spectral images in the test set to form multi-band images, and connecting the multi-band images with noise images in the spectral dimension to form prior noise images comprises the following steps: obtaining a first difference image by subtracting the blue band from the green band of the multi-spectral images in the test set; obtaining a second difference image by subtracting the green band from the red band of the multi-spectral images in the test set; forming a five-band image in the order of the blue band, the first difference image, the green band, the second difference image and the red band; connecting the five-band image with noise images in the spectral dimension to form prior noise images.

3. The method of claim 2, wherein, The step of performing singular value decomposition on a spectral response function, and then constructing a conditional prior comprising a singular value pseudo-inverse matrix and a conditional prior variance comprises the following steps: performing singular value decomposition on the spectral response function to obtain a spatial singular vector, a spectral singular vector and a singular value matrix, and obtaining a singular value pseudo-inverse matrix from the singular value matrix; constructing a conditional prior through the prior noise image, the spatial singular vector and the singular value pseudo-inverse matrix; obtaining a conditional prior variance according to the conditional prior.

4. The method of claim 1, wherein, The step of constraining an inverse diffusion iteration process using the conditional prior, and performing inverse diffusion iteration on the prior noise images to generate intermediate hyperspectral images comprises the following steps: embedding the conditional prior into the inverse diffusion iteration process for constraint; taking the prior noise image as the initial input of inverse diffusion, and performing the following multiple rounds of inverse diffusion iteration: at each time step, inputting a current variable into a noise prediction network, the noise prediction network being a U-shaped structure without a skip connection and comprising a channel-level multi-head attention mechanism for focusing on spectral dimension features and estimating a noise component at the current time step; based on an inverse recursive formula, eliminating part of the noise in the current variable using the conditional prior and the estimated noise component to obtain a variable at the next time step. In the iteration process, the conditional prior preserves the spatial structure prior information of the multi-band image and removes the noise of the noise image, so that the variable gradually changes from noise dominance to hyperspectral feature dominance; When the iteration reaches time step 0, the output variable is the intermediate hyperspectral image.

5. The method of claim 4, wherein, In the optimization process of the intermediate hyperspectral image by using the loss function composed of the root mean square error function and the total variation loss and the optimizer, 1000 iterations of optimization are performed, and the loss in the first 500 iterations does not participate in the gradient back propagation.

6. The method of claim 1, wherein, The step of performing spectral correction on the optimized intermediate hyperspectral image and obtaining three aggregated feature maps of different spectral dimensions by downsampling in the spectral correction process comprises: performing downsampling operation on the intermediate hyperspectral image to obtain three first feature maps with different spectral dimensions and spatial dimensions; performing convolution processing on the three first feature maps respectively to obtain corresponding three second feature maps; performing multi-head attention processing on the first feature map to obtain a third feature map; summing the second feature map and the third feature map to obtain three aggregated feature maps of different spectral dimensions.

7. The method of claim 6, wherein, The reconstruction method further comprises: using the root mean square error function as the loss function, using the Adam optimizer to carry out self-supervised learning, and optimizing the spectral correction process.

8. The method of claim 7, wherein, Using the root mean square error function as the loss function, using the Adam optimizer to carry out self-supervised learning, and optimizing the spectral correction process for 500 iterations.

9. A two-stage diffusion remote sensing hyperspectral reconstruction system for disaster scenarios, characterized by, The system comprises: a data set construction processing module configured to obtain original multi-spectral-hyperspectral data of multiple bands, divide the original multi-spectral-hyperspectral data into a test set, a training set and a validation set comprising multi-spectral images and hyperspectral images; a prior noise image generation module configured to perform band difference processing on the multi-spectral images in the test set to form multi-band images, and form prior noise images by connecting the multi-band images and noise images in the spectral dimension; a conditional prior construction module configured to perform singular value decomposition on a spectral response function, and then construct a conditional prior by using the singular value pseudo-inverse matrix, spectral singular vectors and spatial singular vectors obtained by the decomposition; a prior diffusion module configured to constrain the inverse diffusion iteration process by using the conditional prior, generate an intermediate hyperspectral image by iteratively diffusing the prior noise image, optimize the intermediate hyperspectral image by using a loss function composed of a root mean square error function and a total variation loss and an optimizer, and obtain an optimized intermediate hyperspectral image; a spectral correction module configured to perform spectral correction on the optimized intermediate hyperspectral image, obtain three aggregated feature maps of different spectral dimensions by downsampling in the spectral correction process, connect the three aggregated feature maps in the spectral dimension to obtain a fusion feature map, and perform convolution and activation processing on the fusion feature map to obtain a reconstructed hyperspectral image.

Images