Hyperspectral generation method and device based on spatial super-division enhancement

Abstract

The invention discloses a hyperspectral generation method and device based on spatial super-division enhancement, and the method comprises the steps: carrying out the up-sampling of a preprocessed hyperspectral image through employing an interpolation technology, cutting a plurality of training blocks from the hyperspectral image after the up-sampling through employing a cutting technology, and generating a training sample; extracting three wave bands from each training block, and combining the three wave bands into an RGB image; constructing a generative model; wherein the generation model comprises a DSRNet model and a rectangular attention guiding network; setting a generation condition for the ultrahigh-resolution hyperspectral image; obtaining training samples as a training set; performing iterative training on the generative model by using the training set to obtain a final generative model; wherein the final generation model is used for responding to various generation conditions to generate a hyperspectral image meeting the generation conditions. The problem of how to realize high-quality, high-diversity and controllable hyperspectral image generation is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for generating a hyperspectral spectrum based on spatial super-resolution enhancement. Background Art

[0002] As a cutting-edge field in modern remote sensing and image analysis, hyperspectral imaging technology provides unprecedented spectral feature information for material identification and analysis by capturing continuous spectral bands. Compared with traditional color imaging, hyperspectral imaging can deeply reveal the internal details of materials and is widely used in agriculture, environment, mineral exploration, medical diagnosis and other fields.

[0003] Currently, in order to overcome the high cost of hyperspectral image data acquisition, researchers have turned to deep learning technology, especially generative adversarial networks and variational autoencoders, to synthesize high-quality hyperspectral images. Although these methods have achieved certain results, they still face many challenges in practical applications, including the lack of precise control over the content of generated images, the difficulty in meeting specific needs, and the lack of image diversity and authenticity due to data scarcity.

[0004] The direct application of existing technologies in the field of hyperspectral images is still insufficient, and has not fully resolved the contradiction between the scarcity of hyperspectral images and the quality of their generation. Therefore, how to achieve high-quality, high-diversity and controllable hyperspectral image generation has become a problem to be solved. Summary of the invention

[0005] In the embodiments of the present application, a method for generating a hyperspectral image based on spatial super-resolution enhancement is provided to solve the problem of how to achieve high-quality, high-diversity and controllable hyperspectral image generation.

[0006] In a first aspect, an embodiment of the present application provides a hyperspectral generation method based on spatial super-resolution enhancement, the method comprising: collecting hyperspectral images from multiple sources and preprocessing the collected hyperspectral images; upsampling the preprocessed hyperspectral images using interpolation technology, cropping multiple training blocks from the upsampled hyperspectral images using cropping technology, and generating training samples; extracting three bands from each training block and combining them into an RGB image; constructing a generation model; wherein the generation model comprises a DSRNet model and a rectangular guided attention network; the DSRNet model takes the RGB image as the input image, combines the VAE encoder and the UNet structure, and uses the control network The invention relates to a method for iteratively denoising a super-high-resolution hyperspectral image by using a rectangular guided attention network and a denoising diffusion latent model to generate a high-resolution RGB image; the rectangular guided attention network uses a plurality of guided attention layers and a cross-attention calculation within a rectangular window to obtain an ultra-high-resolution hyperspectral image by using the high-resolution RGB image as auxiliary information; the generation conditions are set for the ultra-high-resolution hyperspectral image; the generation conditions set include edge green detection, image segmentation, sketch generation, line segment detection, content encoding and text generation; the training samples are obtained as the training set; the generation model is iteratively trained using the training set to obtain a final generation model; the final generation model is used to respond to a plurality of generation conditions to generate a hyperspectral image that meets the generation conditions.

[0007] In a possible implementation manner, the preprocessing includes wavelength alignment, denoising operation and standardization processing.

[0008] In a possible implementation, the DSRNet model takes an RGB image as an input image, combines a VAE encoder and a UNet structure, and uses a control network and a denoising diffusion latent model to iteratively denoise to generate a final high-resolution RGB image, including: using a VAE encoder to encode the input image, compressing the input image into a low-dimensional latent space, while retaining the key features of the input image; using a denoising diffusion latent model in the latent space to perform a forward diffusion process and a reverse diffusion process to minimize the difference between the predicted noise and the actual noise; the forward diffusion process is to gradually add noise to the latent space representation through a series of time steps until its distribution is close to a Gaussian distribution; the reverse diffusion process is to generate an initial high-resolution RGB image by gradually denoising based on the input image, starting from the noise distribution; the control network receives the input image, extracts key features, and passes them to the UNet structure; the UNet structure uses the key features passed from the control network to reconstruct the initial high-resolution RGB image in the reverse diffusion process to ensure that the generated final high-resolution RGB image is consistent with the input image in content.

[0009] In a possible implementation, the rectangular guided attention network uses a plurality of guided attention layers and a cross-attention calculation within a rectangular window to obtain an ultra-high-resolution hyperspectral image using a high-resolution RGB image as auxiliary information, including: each guided attention layer includes a self-attention layer, a cross-attention layer, a spectral attention layer, and a feedforward layer; a self-attention mechanism is applied in the self-attention layer to enhance the spatial dependency within the feature map; wherein the feature map is an image extracted from the high-resolution RGB image by a convolution operation; in the cross-attention layer, both the training block and the feature map are divided into vertical and horizontal rectangular windows, and a convolution operation is applied to each window. Multiplication operation is performed to generate query features, key features and value features; multi-head cross-attention calculations are performed in the vertical and horizontal directions respectively to fuse the detailed information of the feature map into the training block, and the multi-head cross-attention calculation results in different directions are aggregated and converted back to the original feature map size; the spectral attention mechanism is applied in the spectral attention layer to enhance the dependency of the feature map on the spectral dimension; the feedforward layer is used to perform linear transformation and nonlinear activation on the outputs of self-attention, cross-attention and spectral attention respectively; multiple guided attention layers are stacked in sequence, and the output of each layer is used as the input of the next layer; ultra-high resolution hyperspectral images are obtained through the processing of multiple guided attention layers.

[0010] In a possible implementation, setting generation conditions for an ultra-high resolution hyperspectral image includes: extracting edge boundaries of the ultra-high resolution hyperspectral image by edge detection; generating a segmentation mask of the ultra-high resolution hyperspectral image by using a pre-trained segmentation model; simplifying the ultra-high resolution hyperspectral image into lines and contours by using a sketch drawing model; applying a multi-scale line segment detection model to identify straight line paths in the ultra-high resolution hyperspectral image; and using an image encoder of a CLIP model to extract a global representation of the content of the ultra-high resolution hyperspectral image.

[0011] In one possible implementation, the generative model is iteratively trained using the training set to obtain the final generative model, including: using the mean square error as the loss function, iteratively updating the generative model parameters through the AdamW optimizer; using the pre-trained weights to initialize the UNet structure in the generative model to accelerate the training process and improve the performance of the generative model to obtain the final generative model.

[0012] In the second aspect, an embodiment of the present application provides a hyperspectral generation device based on spatial super-resolution enhancement, the device comprising: a preprocessing module, used to collect hyperspectral images from multiple sources, and preprocess the collected hyperspectral images; a cropping module, used to upsample the preprocessed hyperspectral images using interpolation technology, and use cropping technology to crop multiple training blocks from the upsampled hyperspectral images, and generate training samples; a composition module, used to construct a generation model; wherein the generation model includes a DSRNet model and a rectangular guided attention network; the DSRNet model takes an RGB image as an input image, combines a VAE encoder and a UNet structure, and uses a control network and a denoising diffusion latent model to iteratively de-noise. noise to generate a high-resolution RGB image; the rectangular guided attention network uses its multiple guided attention layers and cross-attention calculations within the rectangular window to use the high-resolution RGB image as auxiliary information to obtain an ultra-high-resolution hyperspectral image; a setting module is used to set generation conditions for the ultra-high-resolution hyperspectral image; wherein the set generation conditions include edge green detection, image segmentation, sketch generation, line segment detection, content encoding and text generation; an acquisition module is used to obtain training samples as a training set; an iterative training module is used to iteratively train the generation model using the training set to obtain a final generation model; wherein the final generation model is used to respond to a variety of generation conditions to generate a hyperspectral image that meets the generation conditions.

[0013] In a third aspect, an embodiment of the present application provides a hyperspectral generation server based on spatial super-resolution enhancement, comprising a memory and a processor; the memory is used to store computer-executable instructions; the processor is used to execute the computer-executable instructions to implement the method described in the first aspect or any possible implementation method of the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores executable instructions, and when a computer executes the executable instructions, the method described in the first aspect or any possible implementation method of the first aspect can be implemented.

[0015] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects:

[0016] The embodiment of the present application provides a hyperspectral generation method based on spatial super-resolution enhancement. By collecting hyperspectral images from multiple sources, the diversity and representativeness of the data are increased, providing a solid foundation for the training of subsequent models. The preprocessing step further improves the data quality and lays a good foundation for the subsequent processing flow. By using interpolation technology and cropping technology, training blocks can be effectively extracted from hyperspectral images. These training blocks significantly improve the resolution of the image while retaining the key features of the original image. Through the iterative denoising process of the DSRNet model, the quality of the generated image is further enhanced, making the generated high-resolution RGB image clearer and more delicate. The rectangular guided attention network can make full use of the auxiliary information of the high-resolution RGB image through its unique rectangular window cross-attention mechanism to generate ultra-high-resolution hyperspectral images. This mechanism not only improves the spatial resolution of the generated image, but also significantly enhances the accuracy and richness of the spectral information. The present application sets a variety of generation conditions for the generation of images, including edge detection, image segmentation, sketch generation, line segment detection, content encoding and text generation. The setting of these generation conditions enables the generation model to flexibly generate hyperspectral images that meet specific conditions according to actual needs, greatly expanding the application scenarios and practicality of the model. By iteratively training the generative model, we finally obtained a final generative model that can respond to a variety of generative conditions. The model is continuously optimized during the training process, which improves the quality and efficiency of the generated images. At the same time, due to the flexibility and scalability of the model design, it is easy to introduce new generation conditions or improve the model structure in the future to adapt to changing needs and technological developments. The problem of how to achieve high-quality, high-diversity and controllable hyperspectral image generation is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of a method for generating a hyperspectral spectrum based on spatial super-resolution enhancement provided in an embodiment of the present application;

[0019] Figure 2 The DSRNet model provided for the embodiment of the present application uses an RGB image as an input image, combines a VAE encoder and a UNet structure, uses a control network and a denoising diffusion latent model to iteratively denoise, and generates a specific flow chart of a high-resolution RGB image;

[0020] Figure 3A specific flow chart of a rectangular guided attention network provided for an embodiment of the present application, through its multiple guided attention layers, transferring detail information of an RGB image to a final high-resolution hyperspectral image by cross-attention calculation within a rectangular window to obtain an ultra-high-resolution hyperspectral image;

[0021] Figure 4 A specific flow chart for setting generation conditions for ultra-high resolution hyperspectral images provided in an embodiment of the present application;

[0022] Figure 5 Generate an image effect diagram for the generation model provided in the embodiment of the present application;

[0023] Figure 6 A schematic diagram of a hyperspectral generation device based on spatial super-resolution enhancement provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of a hyperspectral generation server based on spatial super-resolution enhancement provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] The following describes some of the techniques involved in the embodiments of the present application to facilitate understanding, and they should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted in the following description.

[0027] The present application embodiment provides a method for generating a hyperspectral spectrum based on spatial super-resolution enhancement, such as Figure 1 As shown, the method includes steps S101 to S109. Among them, Figure 1 This is only an execution order shown in the embodiment of the present application, and does not represent the only execution order of a hyperspectral generation method based on spatial super-resolution enhancement. In the case where the final result can be achieved, Figure 1 The steps shown may be performed in parallel or reversed.

[0028] S101: Collect hyperspectral images from multiple sources and preprocess the collected hyperspectral images. The preprocessing includes wavelength alignment, denoising and standardization.

[0029] Specifically, multiple sources include but are not limited to specific datasets such as Xiongan, Chikusei, DFC2013, DFC2018, and Heihe, each of which has unique characteristics, such as different sensor types, resolutions, and widely covered ground features (such as urban buildings, agricultural crops, natural vegetation, etc.). Such a diverse selection ensures that the subsequent generative model training can access rich and comprehensive data, thereby improving its generalization ability. The collected hyperspectral images are preprocessed to improve data quality and promote the efficiency of subsequent generative model training. Since data from different sources may be collected based on different spectral bands, these images need to be wavelength aligned first. This step ensures the consistency of all images in the spectral dimension. Hyperspectral images may be interfered by various noises during the acquisition process, such as instrument noise, environmental noise, etc. In order to reduce the impact of these noises on image quality, denoising operations are used to process the images to restore their true spectral and spatial information. In order to eliminate the deviations between different datasets due to acquisition conditions, instrument differences, etc., we further standardize the images. This step includes adjusting the brightness and contrast of the images, as well as spectral normalization to ensure that all data are compared and analyzed on the same scale.

[0030] S102: up-sample the pre-processed hyperspectral image using an interpolation technique, crop a plurality of training blocks from the up-sampled hyperspectral image using a cropping technique, and generate training samples.

[0031] Specifically, it is necessary to ensure that the cropped training blocks are diversified in spatial distribution, covering different regions, textures and spectral features in the image. Through cropping, a large number of training samples can be generated from limited hyperspectral images, which are not only expanded in quantity but also richer and more diverse in content.

[0032] S103: Extract three bands from each training block and combine them into an RGB image.

[0033] Specifically, in this step, the goal is to generate an RGB image using the rich spectral information of the hyperspectral image, especially those bands that are most relevant to the human visual system. For each training block obtained from the upsampled hyperspectral image by cropping, three specific bands are first selected from it, which usually correspond to the wavelength ranges of red (R), green (G), and blue (B). The selection of these bands is based on their close relationship with the human visual system's perception of color. According to the specific spectral range of the hyperspectral image and the characteristics of the dataset, we need to determine which bands can best represent the three color channels of R, G, and B. This usually involves analyzing the spectral features of the hyperspectral image and selecting the bands that are closest to R, G, and B in the standard RGB color space. Once the three bands of R, G, and B are determined, the data of these bands are combined to form an RGB image. This RGB image not only retains the spatial information of the original hyperspectral image at that location, but also highlights the color information that is most relevant to human vision through selective band extraction. Through this step, a corresponding RGB image is generated for each cropped training block. These RGB images will be used in the subsequent super-resolution processing as guiding information or prior knowledge to help obtain super-resolution hyperspectral images.

[0034] S104: constructing a generation model, wherein the generation model includes a DSRNet model and a rectangular guided attention network.

[0035] S105: The DSRNet model takes RGB images as input images, combines the VAE encoder and UNet structure, uses the control network and denoising diffusion latent model for iterative denoising, and generates high-resolution RGB images.

[0036] Specifically, the DSRNet model is a super-resolution model based on a diffusion model.

[0037] Figure 2 The DSRNet model provided in the embodiment of the present application takes the RGB image as the input image, combines the VAE encoder and the UNet structure, uses the control network and the denoising diffusion latent model to iteratively denoise, and generates a specific flow chart of a high-resolution RGB image, as shown in FIG. Figure 2 As shown, it includes steps S201 to S206.

[0038] S201: Encode the input image using a VAE encoder to compress the input image into a low-dimensional latent space while retaining key features of the input image.

[0039] Specifically, the VAE encoder (Variational Autoencoder) part in the DSRNet model plays a crucial role. It is responsible for compressing the input image, i.e., the RGB image (denoted as x, whose dimensions are H×W×C, where H and W are the height and width of the image, respectively, and C is the number of color channels) into a low-dimensional latent space (denoted as z, whose dimensions are h×w×c, where h and w are smaller than H and W, and c is the number of channels in the latent space). This conversion process not only achieves the compression of image data, but more importantly, through the design of the VAE encoder, it can capture and retain the key features in the input image. Specifically, the VAE encoder maps the high-dimensional pixel information of the input image to a relatively low-dimensional latent space representation through a series of nonlinear transformations (such as convolutional layers, activation functions, etc.). Although this latent space representation has a low dimensionality, since the VAE encoder is optimized during the training process to minimize the reconstruction error (i.e., the difference between the input image and the image reconstructed from the latent representation by the decoder), it can effectively retain the key information in the input image, such as edges, textures, color distribution, etc.

[0040] S202: Using the denoising diffusion latent model in the latent space to perform forward diffusion and reverse diffusion to minimize the difference between the predicted noise and the actual noise.

[0041] S203: The forward diffusion process is to gradually add noise to the latent space representation through a series of time steps until its distribution is close to a Gaussian distribution.

[0042] Specifically, during the forward diffusion process, the latent space representation z (i.e., the low-dimensional representation output by the VAE encoder) goes through a series of time steps (t=1, 2, ..., T), where T is the total number of time steps. At each time step, a certain amount of noise is added to z, making the distribution of z gradually blurred and eventually approaching a Gaussian distribution. Specifically, at time step t, the latent space representation z t The generation of depends on the latent space representation z at the previous time step t-1 This dependency can be expressed by the conditional probability distribution q(z t ∣z t-1 ), which is a Gaussian distribution whose mean and variance are represented by the latent space z at the previous time step. t-1 and the variance parameter α of the current time step t Jointly determined. The forward diffusion process can be expressed as: Among them, z t represents the noisy data at time step t, i.e., the noisy version of the potential representation at the current time step, is the mean of the Gaussian distribution, which indicates that z t The expected value of z t-1 Proportional, the proportionality coefficient is αt This means that as t increases, z t With z t-1 The similarity between them will gradually decrease because the influence of noise gradually increases. (1-α t )I is the covariance matrix of the Gaussian distribution, where I is the identity matrix, (1-α t ) is the variance parameter, which controls the amount of t-1 The amount of noise in. As t increases, (1-α t ) gradually increases, which means that the amount of noise added at each time step is also increasing.

[0043] S204: The reverse diffusion process is to generate an initial high-resolution RGB image by gradually denoising based on the input image, starting from the noise distribution.

[0044] Specifically, the inverse diffusion process is responsible for gradually removing noise from the initial noise distribution to restore the initial high-resolution RGB image corresponding to the input image. This process is conditioned on the input image to ensure that the generated initial high-resolution RGB image is consistent with the original input image.

[0045] To achieve this goal, DSRNet adopts the sampling method in DDIM (Denoising Diffusion Implicit Model). A significant advantage of this method is that it allows stride sampling during the sampling process, which significantly speeds up the process of generating samples. Stride sampling means that the model can jump directly from the current time step to the previous time step without going through all the intermediate time steps, which greatly reduces the time required for the initial high-resolution RGB image. In the reverse diffusion process, the model starts with a noise distribution and gradually removes noise to generate high-resolution samples. Specifically, for each time step t (starting from T and decreasing to 1), the model will be based on the noise data z of the current time step t and the noise predicted by the model And the variance parameter (It is the sum of all α from time step 1 to t s The product of To calculate the latent space representation z of the previous time step t-1 Among them, α s is a variance parameter associated with time step s, which controls the amount of noise added to the latent space representation z at each step during forward diffusion. Specifically, α s is a scalar value between 0 and 1 that determines the proportion of the original data (or latent space representation) in the noisy data at time step s. This process is given by the following equation: in, is what the model learns during training and is used to predict the value added to z at time step t t-1 The noise in and is the scaling factor. By iterating this process step by step, the model can gradually remove noise from the noise distribution, and finally the initial high-resolution RGB image corresponding to the input image. During the training process, the model is trained to minimize the predicted noise The mean squared error (MSE) between the noise added to the data and the noise actually added to the data. This training objective ensures that the model can accurately predict and remove the noise added during the diffusion process, thereby generating high-quality high-resolution images.

[0046] S205: The control network receives the input image, extracts key features, and passes them to the UNet structure.

[0047] S206: The UNet structure uses the key features passed from the control network to reconstruct the initial high-resolution RGB image in the reverse diffusion process to ensure that the generated final high-resolution RGB image is consistent with the input image in content.

[0048] Specifically, the control network (ControlNet) receives the input low-resolution image as input and processes it to extract key features. These key features capture the most important content information in the input image, such as edges, textures, and semantic structures. The extracted key features are then passed to the UNet structure for use in the subsequent reverse diffusion process. The UNet structure is a convolutional neural network architecture for image segmentation and generation tasks. In the DSRNet model, the UNet structure uses the key features passed from the control network to gradually remove noise during the reverse diffusion process to reconstruct the initial high-resolution RGB image. The skip connections and symmetric encoder-decoder structure of UNet help to gradually restore higher-resolution image content while preserving image details. At each step of reverse diffusion, UNet updates the potential representation based on the noise data at the current time step, the predicted noise, and the key features passed from the control network. These key features provide UNet with prior knowledge about the content of the input image, ensuring that the final high-resolution RGB image is consistent with the input image in content.

[0049] S106: The rectangular guided attention network uses the high-resolution RGB image as auxiliary information to obtain an ultra-high-resolution hyperspectral image through its multiple guided attention layers and cross-attention calculations within the rectangular window.

[0050] Figure 3The rectangular guided attention network provided in the embodiment of the present application transfers the detail information of the RGB image to the final high-resolution hyperspectral image through its multiple guided attention layers and cross-attention calculations within the rectangular window to obtain a specific flow chart of an ultra-high-resolution hyperspectral image, as shown in FIG. Figure 3 As shown, it includes steps S301 to S307.

[0051] Specifically, the rectangular guided attention network (RGAN) is a network architecture designed for generating high-resolution hyperspectral images (HSI) using high-resolution RGB images as auxiliary information. Through its unique structure of multiple guided attention layers (GALs), RGAN effectively fuses the information of feature maps and training blocks, and realizes the conversion from low-resolution hyperspectral images to ultra-high-resolution hyperspectral images. RGAN is composed of multiple guided attention layers (GALs) stacked together, each of which contains four core components: self-attention layer (SAL), criss-cross attention layer (CAL, using rectangular criss-cross attention RCA module), spectral attention layer (SpecAL) and feed-forward layer (FFD).

[0052] S301: Applying a self-attention mechanism in a self-attention layer to enhance spatial dependencies within a feature map, wherein the feature map is an image extracted from a high-resolution RGB image through a convolution operation.

[0053] Specifically, in the self-attention layer, the self-attention mechanism is applied to enhance the spatial dependency within the feature map. The feature map is first extracted from the high-resolution RGB image through a convolution operation, and then the self-attention mechanism is used in the self-attention layer, so that the model can consider the information of all other positions in the image when processing the features of each position, thereby enhancing the spatial correlation of the feature map.

[0054] S302: In the cross attention layer, the training blocks and feature maps are divided into vertical and horizontal rectangular windows, and a convolution operation is applied to each window to generate query features, key features, and value features.

[0055] S303: Perform multi-head cross-attention calculations in the vertical and horizontal directions respectively, fuse the detail information of the feature map into the training block, aggregate the multi-head cross-attention calculation results in different directions and convert them back to the original feature map size.

[0056] Specifically, the cross attention layer is responsible for transferring the detailed information of the feature map (the image extracted from the high-resolution RGB image through the convolution operation) to the training block. The RCA module achieves this function through the following steps: Feature segmentation and mapping: The feature map (Z1) and the training block (Z2) are passed through a two-dimensional convolutional layer (Conv2d) to generate query (Q), key (K) and value (V) features, which are expressed as: Q1, K1, V1 = Split (Conv2d (Z1)), Q2, K2, V2 = Split (Conv2d (Z2)). Where Conv2d represents a two-dimensional convolutional layer, Q1, K1, V1∈R H×W×C The query, key, and value features generated for the feature graph, Q2,K2,V2∈R H×W×C The query, key, and value features generated for the training block. The Q, K, and V features are split into vertical and horizontal parts in the spectral domain, that is, the training block and feature map are split into rectangular windows and the convolution operation is applied to each window. The expression is:

[0057] Among them, Q1 is split into two parts by the Split function, respectively. and Indicates that K1 is split into two parts by the Split function, respectively. and Indicates that V1 is split into two parts by the Split function, respectively. and The same applies to Q2, K2 and V2.

[0058] Furthermore, the multi-head cross attention mechanism in the horizontal and vertical directions is performed on the segmented feature windows respectively. The attention weights are obtained by calculating the dot product of Q and K and applying SoftMax normalization. These weights are then used to weight the values ​​in V to fuse the features of the feature map and the training block. The calculated feature windows are aggregated and converted back to the original feature map size, and the final feature map is obtained through the connection operation.

[0059] The calculation of multi-head cross attention is as follows:

[0060] Where d is a scalar that represents the feature dimension (i.e., the length of the feature vector at each position in the feature map). When calculating the dot product of the query and the key, divide by is a scaling operation designed to stabilize the gradient of the SoftMax function, especially when the dimension is high. P is a learnable position embedding that provides information about different positions in the feature map. The SoftMax function is used to normalize the attention scores into a probability distribution. and is the output of the attention mechanism, and Corresponding to the feature map, and Corresponding to the training block. The calculated feature windows are aggregated and converted back to the original feature map size. The calculation formula for the final feature map obtained by the connection operation is as follows: in, and is the final feature map, corresponding to feature map Z1 and training block Z2, respectively, representing the results obtained after multi-head cross attention mechanism and feature window aggregation. Concat is a connection operation. Concat will and Concatenate along the channel dimension to restore (or generate) a feature map of the same size as the original feature map Z1 Likewise, Concat will and Connect to generate

[0061] S304: Apply the spectral attention mechanism in the spectral attention layer to enhance the dependency of the feature map in the spectral dimension.

[0062] Specifically, the spectral attention mechanism aims to enhance the dependency of feature maps in the spectral dimension, i.e., considering the correlation between different spectral bands.

[0063] S305: Use the feedforward layer to perform linear transformation and nonlinear activation on the outputs of self-attention, cross-attention and spectral attention respectively.

[0064] Specifically, the feed-forward layer (also called fully connected layer or dense layer) consists of a normalization layer and two linear layers with ReLU activation.

[0065] S306: Stack multiple guided attention layers in sequence, with the output of each layer serving as the input of the next layer.

[0066] Specifically, the output of each layer serves as the input of the next layer, so that information can flow and pass between layers. By stacking multiple such layers, higher-level feature representations can be gradually extracted and integrated, while using the attention mechanism to enhance important features and suppress irrelevant or noisy information. This hierarchical structure enables the rectangular guided attention network to better understand complex image data.

[0067] S307: Obtain an ultra-high resolution hyperspectral image through processing of multiple guided attention layers.

[0068] S107: Setting generation conditions for the ultra-high resolution hyperspectral image, wherein the set generation conditions include edge green detection, image segmentation, sketch generation, line segment detection, content encoding, and text generation.

[0069] Figure 4 A specific flow chart for setting generation conditions for ultra-high resolution hyperspectral images provided in an embodiment of the present application is as follows: Figure 4 As shown, it includes steps S401 to S405.

[0070] S401: Extract edge boundaries of ultra-high resolution hyperspectral images through edge detection.

[0071] Specifically, the pre-trained Holistically-nested Edge Detection model is used to perform edge detection on ultra-high-resolution hyperspectral images to extract the edge boundaries of the image, which helps the generative model maintain clear object contours and fine details when synthesizing images.

[0072] S402: Generate a segmentation mask of an ultra-high resolution hyperspectral image using a pre-trained segmentation model.

[0073] Specifically, the pre-trained segmentation model is used to segment the ultra-high-resolution hyperspectral image and generate the corresponding segmentation masks. These segmentation masks annotate the semantic information of different regions in the image in detail, so that the generative model can deeply understand the image content and better maintain the boundaries and structures of the objects during the synthesis process.

[0074] S403: Simplify ultra-high resolution hyperspectral images into lines and contours by sketching the model.

[0075] Specifically, the sketching model further simplifies the ultra-high-resolution hyperspectral image into basic lines and contours. This simplified visual representation not only helps capture the main shapes and structures in the image, but also provides an intuitive reference for the generative model, making it easier to understand and reproduce the core features of the image.

[0076] S404: Apply a multi-scale line segment detection model to identify straight line paths in ultra-high resolution hyperspectral images.

[0077] Specifically, a multi-scale line segment detection model is applied to perform line segment detection on ultra-high resolution hyperspectral images to identify straight line paths in the images. These line segment information provides additional geometric constraints for the generation process, which helps the generation model maintain the straight line features and geometric shapes of objects when synthesizing images, further improving the accuracy and aesthetics of the generated images.

[0078] S405: Use the image encoder of the CLIP model to extract a global representation of the content of the ultra-high resolution hyperspectral image.

[0079] Specifically, the image encoder of the CLIP model is used to encode the content of the ultra-high-resolution hyperspectral image and extract the global representation of the image content. This encoding method enables the generative model to understand and respond to textual prompts of the image content, so that it can better align with the user's text description when generating images, and enhance the personalization and customization capabilities of the generated images.

[0080] S108: Obtain training samples as a training set.

[0081] Specifically, in order to train an efficient and generalizable generative model, several representative hyperspectral image datasets were carefully selected, including Xiongan, Chikusei, DFC2013, DFC2018, and Heihe. These datasets cover different geographical regions, types of objects, and spectral characteristics, providing a rich information basis for model training. Strict wavelength alignment and preprocessing steps were performed on all datasets. This process ensures that all datasets contain 48 uniform bands in the spectral range of 400-1000nm, so that the model can learn on a uniform spectral scale. At the same time, we use linear interpolation methods to maintain the continuity and integrity of spectral information and avoid introducing unnecessary errors or distortions during preprocessing. In order to further improve the diversity and quality of training samples, spatial super-resolution processing was performed on the Heihe and Chikusei datasets. Through advanced image super-resolution techniques, we successfully increased the image resolution of these datasets to a higher level, so that more details and texture information can be captured. About 40k training images are cropped from these high-resolution images, which not only have higher spatial resolution but also maintain the spectral characteristics of the original dataset.

[0082] All training images after wavelength alignment, preprocessing and spatial super-resolution processing are integrated into a unified training set. This training set not only contains rich spectral information, but also has high spatial resolution and diversity, which provides strong support for the training of the generative model. By using this training set, we are able to train a generative model that can generate high-quality hyperspectral images that meet multiple generation conditions.

[0083] S109: Iteratively train the generation model using the training set to obtain a final generation model, wherein the final generation model is used to respond to a plurality of generation conditions to generate a hyperspectral image that meets the generation conditions.

[0084] The generative model is iteratively trained using the training set to obtain the final generative model, including: using the mean square error as the loss function and iteratively updating the generative model parameters through the AdamW optimizer. The UNet structure in the generative model is initialized with the pre-trained weights to accelerate the training process and improve the performance of the generative model to obtain the final generative model.

[0085] Specifically, in order to effectively guide the training of the generative model, this application uses the mean square error (MSE) as the loss function. By minimizing the MSE between the predicted noise and the actual noise, the model can continuously optimize its parameters, thereby improving the ability to generate images from noise and ensuring the quality of the generated images. During the training process, this application selected the AdamW optimizer and set an appropriate learning rate (10 -5 ) and the number of iterations (100k times) to ensure that the generative model can converge stably and efficiently. At the same time, in order to accelerate the training process and improve model performance, the present application also uses pre-trained weights to initialize the UNet structure in the generative model. As an advanced deep learning architecture, UNet is well-known for its powerful feature extraction and reconstruction capabilities. Its application in the generative model further enhances the performance of the model in the hyperspectral image synthesis task. During the training process, the generative model not only learns how to generate images from noise, but also deeply learns how to fine-tune the image generation process according to a variety of pre-set generation conditions (such as edge detection, image segmentation, sketch generation, line segment detection, content encoding, and text generation, etc.). This conditional generation mechanism enables the final generated hyperspectral image, that is, the hyperspectral image that meets the generation conditions, to closely fit the user's expectations and needs, and has a high degree of customizability and practicality.

[0086] Figure 5 The generation model provided in the embodiment of the present application generates an image effect diagram, such as Figure 5As shown in the figure, there are three main tabs at the top of the picture - "Conditions", "Generated Images" and "Real Images". These three tabs correspond to the experimental condition settings, generated images and real images respectively. Under each tab, different image processing techniques or methods are listed, such as "HED+Seg", "HED+Sketch" and "MLSD+Sketch". These represent different combinations of techniques used in the experiment, each of which aims to improve or enhance the image in a specific way. The HED+Seg combination refers to first using a high-order edge detection (HED) algorithm to detect edges in the image, and then further processing these edges using segmentation techniques to extract or emphasize specific structures or areas in the image. HED+Sketch: Here, a sketch rendering technique is added on the basis of HED edge detection. This technique may be intended to simulate the effect of hand-drawn sketches by simplifying image content or emphasizing edges, thereby providing another visual representation of image structure. In the MLSD+Sketch combination, multi-layer sparse decomposition (MLSD) is a technique for image decomposition or feature extraction. Combined with sketch techniques, this combination may aim to enhance the details or structural features of an image by decomposing multiple layers of the image and combining it with sketch rendering.

[0087] Furthermore, the evaluation indicators of the present application include structural similarity index (SSIM), Frechette initial distance (FID), natural image quality evaluator (NIQE), blind / no reference image space quality evaluator (BRISQUE), CLIP image quality assessment (ClipIQA) and CLIP score (ClipScore), each of which conducts in-depth analysis of the generated image, that is, the hyperspectral image that meets the generation conditions from different dimensions. Structural Similarity Index (SSIM): SSIM is an important indicator for measuring the structural similarity of two images. In the context of the present application, the higher the value of SSIM, the higher the similarity between the generated image and the given conditions (such as edges, shapes, structures, etc.) in visual structure, reflecting the model's ability to accurately capture and reproduce conditional information. Frechette initial distance (FID): FID evaluates the similarity of the distribution of the generated image and the real image by calculating the distance between them in the feature space. The smaller the FID value, the closer the generated image is to the real image at the feature level, that is, the final generation model can better simulate the data distribution of the real image and generate a more realistic image. Natural Image Quality Evaluator (NIQE) and Blind / No-reference Image Spatial Quality Evaluator (BRISQUE): Both are no-reference image quality assessment methods that do not rely on original images or subjective ratings. The smaller the values ​​of NIQE and BRISQUE, the higher the perceived quality of the generated image, which is closer to the human eye's cognitive standard for high-quality images. They objectively evaluate the generated images from two dimensions: the naturalness and spatial quality of the image. CLIP Image Quality Assessment (ClipIQA) and CLIP Score (ClipScore): Using the powerful image-text matching capabilities of the CLIP model, ClipIQA and ClipScore combine image quality assessment with text description. The larger the ClipIQA, the higher the consistency between the generated image and the text description, that is, the image content is more consistent with the semantic information expected by the user. Similarly, ClipScore also reflects the degree of match between the image and the text description. The larger its value, the more the image can perceptually meet the user's text needs.

[0088] The present application embodiment also provides a hyperspectral generation device 600 based on spatial super-resolution enhancement, such as Figure 6 As shown, the device includes: a preprocessing module 601, a cropping module 602, a construction module 603, a setting module 604, an acquisition module 605 and an iterative training module 606.

[0089] The preprocessing module 601 is used to collect hyperspectral images from multiple sources and preprocess the collected hyperspectral images.

[0090] The cropping module 602 is used to upsample the preprocessed hyperspectral image using an interpolation technique, crop multiple training blocks from the upsampled hyperspectral image using a cropping technique, and generate training samples.

[0091] The composition module 603 is used to construct a generation model. The generation model includes a DSRNet model and a rectangular guided attention network. The DSRNet model takes the RGB image as the input image, combines the VAE encoder and the UNet structure, and uses the control network and the denoising diffusion latent model for iterative denoising to generate a high-resolution RGB image. The rectangular guided attention network uses its multiple guided attention layers and the cross-attention calculation within the rectangular window to use the high-resolution RGB image as auxiliary information to obtain an ultra-high-resolution hyperspectral image.

[0092] The setting module 604 is used to set generation conditions for the ultra-high resolution hyperspectral image, wherein the set generation conditions include edge green detection, image segmentation, sketch generation, line segment detection, content encoding and text generation.

[0093] The acquisition module 605 is used to acquire training samples as training sets.

[0094] The iterative training module 606 is used to iteratively train the generation model using the training set to obtain a final generation model, wherein the final generation model is used to respond to a variety of generation conditions to generate a hyperspectral image that meets the generation conditions.

[0095] Some modules in the apparatus described in the present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0096] The devices or modules described in the above application embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described in various modules according to their functions. When implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, the module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0097] The methods, devices or modules described in this application can be implemented in a computer-readable program code mode. The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro) processor, a logic gate, a switch, an application-specific integrated circuit (English: Application Specific Integrated Circuit, referred to as: ASIC), a programmable logic controller and an embedded microcontroller. Examples of controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a pure computer-readable program code mode, the controller can be completely implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included in it for implementing various functions can also be regarded as structures within the hardware component. Or even, the means for realizing various functions may be regarded as both a software module for realizing the method and a structure within a hardware component.

[0098] like Figure 7 As shown, an embodiment of the present application also provides a hyperspectral generation server based on spatial super-resolution enhancement, including a memory 701 and a processor 702; the memory 701 is used to store computer-executable instructions; the processor 702 is used to execute computer-executable instructions to implement a hyperspectral generation method based on spatial super-resolution enhancement described above in an embodiment of the present application.

[0099] The embodiment of the present application further provides a computer-readable storage medium, which stores executable instructions. When a computer executes the executable instructions, the method for generating a hyperspectral spectrum based on spatial super-resolution enhancement described above in the embodiment of the present application can be implemented.

[0100] It can be known from the description of the above implementation mode that a person skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, or it can be reflected in the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the method described in the embodiment of the present application.

[0101] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations.

[0102] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A method for generating a hyperspectral spectrum based on spatial super-resolution enhancement, characterized in that: include: Collecting hyperspectral images from multiple sources and preprocessing the collected hyperspectral images; The preprocessed hyperspectral image is upsampled using interpolation technology, and multiple training blocks are cropped from the upsampled hyperspectral image using cropping technology to generate training samples; Extract three bands from each training block and combine them into an RGB image; Constructing a generation model; wherein the generation model includes a DSRNet model and a rectangular guided attention network; The DSRNet model takes RGB images as input images, combines VAE encoders and UNet structures, uses control networks and denoising diffusion latent models for iterative denoising, and generates high-resolution RGB images. The rectangular guided attention network uses the high-resolution RGB image as auxiliary information to obtain ultra-high-resolution hyperspectral images through cross-attention calculations within rectangular windows through its multiple guided attention layers; Setting generation conditions for ultra-high resolution hyperspectral images; wherein the generation conditions set include edge green detection, image segmentation, sketch generation, line segment detection, content encoding and text generation; Obtain training samples as training sets; The generation model is iteratively trained using the training set to obtain a final generation model; wherein the final generation model is used to respond to a variety of generation conditions to generate a hyperspectral image that meets the generation conditions.

2. The method for generating a hyperspectral spectrum based on spatial super-resolution enhancement according to claim 1, characterized in that: The preprocessing includes wavelength alignment, denoising operation and normalization processing.

3. The method for generating a hyperspectral spectrum based on spatial super-resolution enhancement according to claim 1, characterized in that: The DSRNet model takes the RGB image as the input image, combines the VAE encoder and the UNet structure, uses the control network and the denoising diffusion latent model to iteratively denoise, and generates the final high-resolution RGB image, including: The VAE encoder is used to encode the input image, compressing the input image into a low-dimensional latent space while retaining the key features of the input image; In the latent space, the denoising diffusion latent model is used to perform forward diffusion and reverse diffusion to minimize the difference between the predicted noise and the actual noise; The forward diffusion process is to gradually add noise to the latent space representation through a series of time steps until its distribution approaches a Gaussian distribution; The reverse diffusion process is to generate an initial high-resolution RGB image by gradually denoising the input image, starting from the noise distribution; The control network receives the input image, extracts key features, and passes them to the UNet structure; The UNet structure utilizes the key features passed from the control network to reconstruct the initial high-resolution RGB image in the reverse diffusion process to ensure that the generated final high-resolution RGB image is consistent with the input image in content.

4. The method for generating a hyperspectral spectrum based on spatial super-resolution enhancement according to claim 1, characterized in that: The rectangular guided attention network uses the high-resolution RGB image as auxiliary information to obtain an ultra-high-resolution hyperspectral image through cross-attention calculation within a rectangular window through its multiple guided attention layers, including: Each guided attention layer includes a self-attention layer, a cross-attention layer, a spectral attention layer, and a feed-forward layer; Applying a self-attention mechanism in the self-attention layer to enhance spatial dependencies within a feature map, wherein the feature map is an image extracted from a high-resolution RGB image by a convolution operation; In the cross-attention layer, both the training block and the feature map are divided into vertical and horizontal rectangular windows, and a convolution operation is applied to each window to generate query features, key features, and value features; Perform multi-head cross-attention calculations in the vertical and horizontal directions respectively, fuse the detailed information of the feature map into the training block, aggregate the multi-head cross-attention calculation results in different directions and convert them back to the original feature map size; Apply the spectral attention mechanism in the spectral attention layer to enhance the dependency of feature maps on the spectral dimension; Use feed-forward layers to perform linear transformation and nonlinear activation on the outputs of self-attention, cross-attention, and spectral attention, respectively; Multiple guided attention layers are stacked in sequence, with the output of each layer serving as the input of the next layer; Through the processing of multiple guided attention layers, ultra-high-resolution hyperspectral images are obtained.

5. The method for generating a hyperspectral spectrum based on spatial super-resolution enhancement according to claim 1, characterized in that: The generation conditions for setting the ultra-high resolution hyperspectral image include: Extract edge boundaries of ultra-high resolution hyperspectral images through edge detection; Generate segmentation masks of ultra-high-resolution hyperspectral images using pre-trained segmentation models; The sketching model simplifies ultra-high-resolution hyperspectral images into lines and contours; Applying a multi-scale line segment detection model to identify straight line paths in ultra-high resolution hyperspectral images; An image encoder using the CLIP model is used to extract a global representation of the content of ultra-high-resolution hyperspectral images.

6. The method for generating a hyperspectral spectrum based on spatial super-resolution enhancement according to claim 1, characterized in that: The method of iteratively training the generative model using the training set to obtain the final generative model includes: Using mean square error as the loss function, the parameters of the generative model are iteratively updated through the AdamW optimizer; The UNet structure in the generative model is initialized using pre-trained weights to accelerate the training process and improve the performance of the generative model to obtain the final generative model.

7. A hyperspectral generation device based on spatial super-resolution enhancement, characterized in that: include: A preprocessing module, used for collecting hyperspectral images from multiple sources and preprocessing the collected hyperspectral images; A cropping module is used to upsample the preprocessed hyperspectral image using interpolation technology, crop multiple training blocks from the upsampled hyperspectral image using cropping technology, and generate training samples; A composition module is used to construct a generation model; wherein the generation model includes a DSRNet model and a rectangular guided attention network; the DSRNet model takes an RGB image as an input image, combines a VAE encoder and a UNet structure, uses a control network and a denoising diffusion latent model to iteratively denoise, and generates a high-resolution RGB image; the rectangular guided attention network uses its multiple guided attention layers and cross-attention calculations within a rectangular window to use the high-resolution RGB image as auxiliary information to obtain an ultra-high-resolution hyperspectral image; A setting module, used to set generation conditions for ultra-high resolution hyperspectral images; wherein the set generation conditions include edge green detection, image segmentation, sketch generation, line segment detection, content encoding and text generation; An acquisition module, used for acquiring training samples as training sets; The iterative training module is used to iteratively train the generation model using the training set to obtain a final generation model; wherein the final generation model is used to respond to multiple generation conditions to generate a hyperspectral image that meets the generation conditions.

8. A hyperspectral generation server based on spatial super-resolution enhancement, characterized in that: including memory and processor; The memory is used to store computer executable instructions; The processor is used to execute the computer executable instructions to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores executable instructions, and when a computer executes the executable instructions, the method according to any one of claims 1 to 6 can be implemented.

Images