Hyperspectral image simulation sample generation method and device and electronic equipment

Abstract

The invention discloses a hyperspectral image simulation sample generation method and apparatus, and an electronic device. The method comprises the steps of obtaining a source image; preprocessing the source image, and determining a first image patch; encoding the first image patch into a first feature vector based on a trained adversarial auto-encoder; the first feature vector is subjected to noise adding processing, forward diffusion is carried out to obtain a second feature vector, and the information content of the first feature vector is different from that of the second feature vector; determining a target feature vector from the second feature vectors based on a trained deep neural network model; and de-noising the target feature vector based on the acquired auxiliary vector, and reversely generating a third feature vector. According to the method, advanced technologies such as deep learning and a diffusion model are combined, and on the basis of effectively simulating a noise removal process, a simulated hyperspectral image with better physical authenticity and statistical characteristics can be gradually generated.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device and electronic device for generating a hyperspectral image simulation sample. Background Art

[0002] With the acceleration of global urbanization, simulation of urban thermal environment and low-carbon planning have become important topics in environmental monitoring and sustainable development research. Urban heat island effect, energy consumption and carbon emissions are closely related, and traditional monitoring methods are difficult to effectively capture the complex dynamic changes in the thermal environment. Hyperspectral imaging technology is widely used in the study of urban thermal environment because it can capture a wide range of spectral information. However, due to the high cost of hyperspectral image acquisition and the limitations of external factors such as weather and lighting conditions, it is difficult to obtain enough real data to support the simulation and planning of urban thermal environment.

[0003] In the existing technology, the generation of simulated hyperspectral images has been applied to remote sensing, geographic information systems and other fields, but these technologies face new challenges in the simulation of urban thermal environment and low-carbon planning application scenarios. Specifically, the temperature distribution, thermal radiation characteristics, and spectral characteristics of building materials in the urban thermal environment will show large changes under different time and environmental conditions, which makes it difficult for traditional simulation methods to accurately simulate the hyperspectral characteristics under different environmental and temperature conditions. At the same time, low-carbon planning requires a large amount of hyperspectral image data to optimize building energy efficiency, evaluate green space coverage and other aspects, further exacerbating the demand for efficient simulation technology.

[0004] Traditional deep learning techniques, such as autoencoders (AE) and generative adversarial networks (GANs), have been used to generate simulated samples. However, these methods face unique difficulties when generating hyperspectral images. First, the autoencoder (AE) learns a low-dimensional representation of the data through an encoder-decoder architecture, but the generated samples often lack diversity and are insufficient in capturing the spectral details of hyperspectral images. The generated images are easily distorted and difficult to accurately reflect the complex spectral characteristics of urban thermal environments. Second, although the generative adversarial network (GAN) has certain advantages in terms of the authenticity and detail fidelity of generated image samples, its training process is extremely unstable, especially when processing high-dimensional and high-complexity hyperspectral image data, it is prone to mode collapse, resulting in insufficient diversity and quality of generated samples.

[0005] In addition, a major shortcoming of these traditional generation methods is that they often ignore the continuity of spectral dimensions and the dependence between bands in hyperspectral images, which are crucial for accurately simulating thermal radiation distribution, material properties, and temperature changes in urban thermal environments. The lack of processing of spectral continuity often results in the simulation samples being unable to effectively reflect the physical and environmental characteristics of real hyperspectral images, which also limits their application value in urban thermal environment simulation and low-carbon planning. Summary of the invention

[0006] The purpose of the embodiment of the present application is to provide a method for generating a hyperspectral image simulation sample, comprising: Get the source image; preprocessing the source image to determine a first image patch; encoding the first image patch into a first feature vector based on the trained adversarial autoencoder; Noising the first eigenvector and forward diffusing it into a second eigenvector, wherein the first eigenvector and the second eigenvector have different information contents; Based on the trained deep neural network model, determining a target feature vector from the second feature vector; Based on the acquired auxiliary vector, denoising the target feature vector, and reversely generating a third feature vector, wherein the third feature vector is the same as the first feature vector; Based on the trained adversarial autoencoder, the third feature vector is decoded into a second image patch, wherein the second image patch is similar to the first image patch.

[0007] As an optional embodiment, the acquiring the source image includes: Acquire a source image based on a preset resolution and a preset wavelength; Based on the pixel correspondence, the ground object category of the source image is marked.

[0008] As an optional embodiment, the preprocessing the source image to determine the first image patch includes: Normalizing the source image; The source image after segmentation processing is determined to determine a first image patch.

[0009] As an optional embodiment, the method further includes: Get the minimized reconstruction loss, spectral smoothness constraint, and adversarial loss; An adversarial autoencoder is trained based on the minimization of the reconstruction loss, the spectral smoothness constraint, and the adversarial loss.

[0010] As an optional embodiment, the step of adding noise to the first eigenvector and forward diffusing it into a second eigenvector includes: Step by step, adding noise to the first feature vector, wherein the noise added in different steps is different; A second eigenvector is determined based on the first eigenvector after noise is added.

[0011] As an optional embodiment, the denoising process of the target feature vector based on the acquired auxiliary vector and reversely generating the third feature vector includes: Encoding the pre-acquired auxiliary vector to determine a target auxiliary vector, wherein the auxiliary vector includes ground feature category information, physical constraint information, and temperature information; Determine, based on a preset time step, the target auxiliary vector and the target feature vector, the noise corresponding to the preset time step during denoising; Based on the noise, the deep neural network model denoises the target feature vector and reversely generates a third feature vector.

[0012] As an optional embodiment, the method further includes: Based on the difference between the first image patch and the second image patch, the deep neural network model is updated.

[0013] The purpose of the embodiment of the present application is to provide a method for simulating sample output results based on a hyperspectral image, the method comprising: Get the source image; According to the aforementioned generation method, a simulation image corresponding to the source image is obtained; Mixing the source image and the simulated image to determine a training sample; Based on the training samples, a neural network classifier is trained.

[0014] The purpose of the embodiment of the present application is to provide a device for generating a hyperspectral image simulation sample, comprising: a data acquisition module configured to acquire a source image; A preprocessing module configured to preprocess the source image to determine a first image patch; a first encoding module configured to encode the first image patch into a first feature vector based on a trained adversarial autoencoder; A noise processing module configured to perform noise processing on the first feature vector and forward diffuse it into a second feature vector, wherein the first feature vector has different information contents from the second feature vector; A first determination module, configured to determine a target feature vector from the second feature vector based on a trained deep neural network model; A denoising processing module, configured to denoise the target feature vector based on the acquired auxiliary vector, and reversely generate a third feature vector, wherein the third feature vector is the same as the first feature vector; A second determination module is configured to decode the third feature vector into a second image patch based on the trained adversarial autoencoder, wherein the second image patch is similar to the first image patch.

[0015] An object of an embodiment of the present application is to provide an electronic device, including a processor and a memory, wherein an executable program is stored in the memory, and the processor executes the executable program to implement the steps of the aforementioned method.

[0016] The beneficial effects of the embodiments of the present application are: (1) The present invention effectively reduces the difficulty and cost of hyperspectral image data collection. Compared with the traditional method that requires a large number of diverse ground object categories for high-cost collection, the present invention generates high-quality hyperspectral image simulation samples through a diffusion model based on physical constraints, which greatly reduces the dependence on field-collected hyperspectral data. This not only reduces the cost of data collection, but also simplifies the data acquisition process, so that even under conditions of limited resources, rich and diverse hyperspectral image data can be obtained.

[0017] (2) The present invention gradually restores the hyperspectral image through a diffusion model based on physical constraints. The generated simulation samples are highly consistent with the real data in terms of spectral characteristics, and can especially accurately reflect the temperature changes, thermal radiation characteristics and spectral response of building materials in the urban thermal environment. At the same time, the samples generated by this method have higher diversity, support simulation under different climate and thermal environment conditions, and improve the wide applicability of the data.

[0018] (3) The present invention can customize and generate hyperspectral images of specific areas according to the temperature, building materials and environmental conditions of different areas in the city, and accurately simulate the spectral distribution under different thermal environments. This is of great value for building thermal performance analysis and energy management optimization in low-carbon planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The process of the generation method of the embodiment of the present application Figure 1 ; Figure 2 This is a flowchart of step S10 in the generation method of an embodiment of the present application; Figure 3 This is a flowchart of step S20 in the generation method of an embodiment of the present application; Figure 4 This is a flowchart of step S30 in the generation method of an embodiment of the present application; Figure 5 This is a flowchart of step S40 in the generation method of an embodiment of the present application; Figure 6 This is a flowchart of step S60 in the generation method of an embodiment of the present application; Figure 7 This is a flow chart of step S9 of the method of the embodiment of the present application; Figure 8 is a structural block diagram of a generating device according to an embodiment of the present application; Fig. 9 The process of the generation method of the embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0020] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0021] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but only as an example of an embodiment. Other modifications within the scope and spirit of the present application will occur to those skilled in the art.

[0022] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0023] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0024] It should also be understood that although the present application has been described with reference to some specific examples, those skilled in the art will be able to readily implement many other equivalent forms of the present application.

[0025] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.

[0026] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that obscure the present application. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.

[0027] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.

[0028] The present application provides a method for generating a hyperspectral image simulation sample, such as Figure 1 and Figure 2 As shown, including: S10: Obtain a source image.

[0029] In this embodiment, the source image refers to original, unprocessed image data, which is the starting point of the simulation sample generation method, for example, a hyperspectral image of a city thermal environment.

[0030] S20: Preprocess the source image to determine a first image patch.

[0031] In this embodiment, preprocessing refers to a series of operations performed on the source image to prepare for subsequent processing steps, such as denoising, contrast enhancement, etc. For example, the source image is cropped and scaled to meet the processing requirements of the subsequent algorithm.

[0032] The first image patch is a local area extracted from the source image after preprocessing and used in the subsequent feature extraction and generation process, such as cropping an area containing buildings and green space from an urban thermal environment image.

[0033] S30: Based on the trained adversarial autoencoder, encode the first image patch into a first feature vector.

[0034] In this embodiment, the adversarial autoencoder is a deep learning model consisting of an encoder and a decoder, which is used to learn low-dimensional representations of data and reconstruct data from these representations. A feature vector is a vector representing data features and is usually used as input for machine learning models.

[0035] For example, a trained adversarial autoencoder can encode an image patch into a feature vector and reconstruct the image patch from the feature vector. A feature vector contains multiple dimensions, each dimension represents a specific attribute of the image patch, such as color, texture, etc.

[0036] S40: add noise to the first eigenvector and forward diffuse it into a second eigenvector, wherein the first eigenvector and the second eigenvector have different information contents.

[0037] In this embodiment, the noise addition process is to add noise to the data to simulate the imperfection in the data acquisition process, for example, adding random noise to the feature vector of the image to simulate the sensor noise.

[0038] Forward diffusion is a technique that simulates the data generation process by gradually adding noise to generate new data samples. For example, different levels of noise are gradually added to the feature vector to generate a series of samples that gradually move away from the original data.

[0039] S50: Determine a target feature vector from the second feature vector based on the trained deep neural network model.

[0040] In this embodiment, the deep neural network model is a multi-layer neural network that can learn complex data representations. For example, a deep neural network model can extract useful information from a noisy feature vector.

[0041] S60: Based on the acquired auxiliary vector, denoise the target feature vector and reversely generate a third feature vector, wherein the third feature vector is the same as the first feature vector.

[0042] In this embodiment, the auxiliary vector is a vector that provides additional information during the generation process to help improve the generation result. For example, an auxiliary vector containing ambient temperature information is used to guide the generation process. Denoising is to remove noise from the data and restore the original features of the data. For example, a deep learning model is used to restore clear image features from a noisy feature vector. Reverse generation is the process of reversely deriving the original feature vector from the target feature vector. For example, the feature vector of the original image is reversely generated from the feature vector processed by the deep neural network model.

[0043] S70. Based on the trained adversarial autoencoder, decode the third feature vector into a second image patch, wherein the second image patch is similar to the first image patch.

[0044] In this embodiment, the second image patch is an image patch obtained by decoding the third eigenvector, which is similar to the first image patch, for example, an image patch obtained by decoding an adversarial autoencoder, which is visually similar to the original image patch.

[0045] In this embodiment, it is assumed that there is a hyperspectral image of an urban thermal environment as a source image, and firstly, a region including buildings and green space is cropped from the image as a first image patch, and the region is preprocessed, such as adjusting the size and contrast, to adapt to subsequent algorithms.

[0046] Next, a trained adversarial autoencoder is used to encode this region into a feature vector. In order to simulate the noise in the real world, this feature vector is denoised and the noise is gradually increased to simulate the forward diffusion process.

[0047] Then, a deep neural network model is used to extract the target feature vector from the noisy feature vector, and auxiliary vectors (such as ambient temperature information) are used for denoising.

[0048] Finally, a third eigenvector identical to the original eigenvector is generated in reverse, and the adversarial autoencoder is used to decode it into a second image patch similar to the original area. In this way, a simulated hyperspectral image sample is obtained, which can be used for urban thermal environment simulation and low-carbon planning.

[0049] This application combines deep learning with advanced technologies such as diffusion models to gradually generate simulated hyperspectral images with more physical realism and statistical characteristics based on the effective simulation of the noise removal process. This application is suitable for generating simulated hyperspectral images under different temperature and environmental conditions. The generated simulated hyperspectral images have accurate spectral characteristics, are realistic and diverse, and improve the quality and diversity of simulated hyperspectral images, supporting accurate simulation of urban thermal environment and low-carbon planning.

[0050] Specifically, Fig. 9 As shown in the figure, a suitable wavelength range and resolution are selected, and a hyperspectral imager is used to photograph the ground, and the ground object category is marked according to the correspondence between the actual area and the pixels in the hyperspectral image. The image data is normalized using the maximum-minimum method, and the entire image is segmented into image patches using the sliding window method, and the corresponding infrared image patches are obtained by the feature band extraction method.

[0051] An adversarial autoencoder network structure is constructed, combined with spectral image smoothness constraints, and all image patches are used to perform unsupervised pre-training on the network.

[0052] The image patches are encoded into feature vectors using a pre-trained adversarial autoencoder as the initial state, and these image features are gradually transformed into pure noise features by adding a series of controlled noises to them. The addition of noise follows a certain rule of increasing noise levels to ensure that the gradual transformation of image data is both smooth and controllable.

[0053] A deep neural network model is used to predict each step of the noise removal process, thereby gradually reducing the noise level until the original hyperspectral image patch features are restored.

[0054] Use the pre-trained adversarial autoencoder to decode the restored original hyperspectral image patch features and restore the original input image patch.

[0055] Based on the pre-trained adversarial encoder and deep neural network model, the constructed auxiliary vector is input into the trained deep neural network model according to given conditions. The deep neural network model randomly samples a feature vector (target feature vector) from the Gaussian noise feature (the second feature vector), and performs a reverse generation process for a specific number of steps on it. After that, it is decoded by the decoder to generate a simulated image (the second image patch).

[0056] The auxiliary vector used in the above-mentioned deep neural network model refers to the reverse generation condition encoded by the encoder and output by the gate switching module; the encoded feature vector is the feature vector after adding noise; the multi-head cross-attention module uses multiple attention heads to perform multi-angle modeling of the relationship between the decoder input and the encoder output to enhance the information fusion capability; the multi-head self-attention module uses multiple attention heads to model the correlation between the positions within the sequence, thereby capturing the long-term and short-term dependencies of the data; the point-by-point feedforward network improves the feature expression capability of the model by independently performing linear transformation and nonlinear activation on the features of each position; layer normalization improves the stability of the training process and accelerates model convergence by normalizing the output of each layer of the network.

[0057] This application improves the authenticity of the simulation samples. By combining deep learning and diffusion models, the generated hyperspectral image simulation samples are closer to the real data, which improves the accuracy of the simulation. This application enhances sample diversity. Through the process of forward diffusion and reverse generation, it can generate a variety of simulation samples to meet the simulation needs under different environmental conditions.

[0058] In one embodiment, if Figure 2 As shown, the obtaining of the source image includes: S101 . Acquire a source image based on a preset resolution and a preset wavelength.

[0059] In this embodiment, the preset resolution refers to the image clarity standard preset before acquiring the image, that is, the number of pixels contained in each unit length of the image. For example, when acquiring a hyperspectral image of an urban thermal environment, the preset resolution may be 300 pixels per inch (DPI) to ensure that the image clarity meets the analysis requirements.

[0060] The preset wavelength refers to the spectral range that is preset before acquiring the image, that is, the wavelength range of light waves that the image can capture. For example, when acquiring a hyperspectral image of vegetation health, the preset wavelength range may be 400 nanometers to 700 nanometers, because the wavelengths in this range are most sensitive to the reflective characteristics of vegetation.

[0061] The source image refers to the original image data that serves as the starting point for the simulation sample generation method. For example, an image of an urban area taken by a hyperspectral camera contains rich spectral information.

[0062] S102: Label the ground object category of the source image based on the pixel correspondence relationship.

[0063] In this embodiment, the pixel correspondence refers to the correspondence between each pixel in the image and the actual ground object it represents. For example, in a hyperspectral image, each pixel may correspond to an area of ​​a specific size on the ground, and this correspondence helps with subsequent object category labeling. Object category labeling refers to the process of identifying and marking different objects (such as buildings, vegetation, water bodies, etc.) on an image. For example, in a hyperspectral image of an urban thermal environment, different objects such as buildings, roads, and green spaces are marked with different colors or labels for easy analysis.

[0064] When acquiring the source image, first, the source image is acquired based on the preset resolution and wavelength to ensure that the image quality and spectral information meet the analysis requirements; secondly, based on the pixel correspondence, the ground object category in the source image is labeled to provide accurate ground object information for subsequent simulation sample generation.

[0065] This application obtains source images by presetting resolution and wavelength, which can ensure the quality of the simulated samples and the accuracy of the spectral information, thereby improving the accuracy of the simulated samples. In addition, by labeling the object categories through pixel correspondence, different objects in the image can be more accurately identified and distinguished, providing richer object information for the generation of simulated samples.

[0066] Specifically, according to the analysis requirements, the resolution and wavelength range of the image are preset. A hyperspectral camera or other equipment is used to obtain the source image according to the preset parameters. The correspondence between each pixel in the image and the actual object on the ground is determined. Different ground object categories are identified and marked on the image.

[0067] In this embodiment, the hyperspectral image is defined as ,in , and Represents the height, width and number of spectral channels of the image respectively. to The ground image is collected to obtain the original hyperspectral data. The labeling of the corresponding ground object category can be expressed as a label matrix , where the value of each pixel location corresponds to the identifier of the object category.

[0068] For example, it is necessary to generate a sample of a hyperspectral image of an urban thermal environment. First, according to the analysis requirements, the resolution is preset to 300 pixels per inch and the wavelength range is 800 nanometers to 1400 nanometers to capture key spectral information in the urban thermal environment.

[0069] Then, a hyperspectral camera is used to capture images of the urban area under these parameters. Next, the correspondence between each pixel in the image and the actual object on the ground is analyzed to determine the size of the ground area represented by each pixel. Finally, different objects such as buildings, roads, and green spaces are marked with different colors or labels on the image to provide accurate object information for simulation sample generation. Through this process, high-quality source images can be obtained, laying a solid foundation for subsequent simulation sample generation.

[0070] In one embodiment, if Figure 3 As shown, the preprocessing of the source image to determine the first image patch includes: S201: Normalize the source image.

[0071] In this embodiment, normalization refers to the process of scaling image data to a specific value range (usually 0 to 1 or -1 to 1) to eliminate differences between different images due to factors such as illumination and contrast. For example, the pixel value of an urban thermal environment image is normalized from 0-255 to 0-1 so that subsequent processing is not affected by the original pixel value range.

[0072] S202: Segment the source image after processing and determine a first image patch.

[0073] In this embodiment, segmentation processing refers to the process of segmenting an image into multiple regions or objects, which have similar features. For example, in an urban thermal environment image, a segmentation algorithm is used to separate different regions such as buildings, roads and green spaces.

[0074] The first image patch refers to a local area or object randomly extracted from the source image after the segmentation process, which is used for the subsequent feature extraction or generation process. For example, a building area is extracted from a segmented hyperspectral image as the first image patch. In addition, the first image patch can be any intercepted partial image and / or any intercepted infrared image.

[0075] When preprocessing the source image, first, the source image is normalized so that the image data remains numerically consistent; secondly, the normalized source image is segmented to determine the first image patch, that is, to extract a specific area or object in the image.

[0076] This application can reduce the differences between different images through normalization processing, improve the efficiency and accuracy of subsequent data processing. In addition, through segmentation processing, specific areas or objects in the image can be more accurately identified and extracted, facilitating subsequent feature extraction and analysis.

[0077] Specifically, the pixel values ​​of the source image are normalized to conform to a preset value range, the normalized image is segmented to identify different regions or objects, and a specific region or object is extracted from the segmented image as a first image patch.

[0078] For example, the image data is normalized using the maximum-minimum method, and the entire image is segmented into image patches using the sliding window method. , use the maximum-minimum normalization method to normalize each spectral channel, and the normalized pixel value The calculation is as follows:

[0079] in, and Represent the maximum and minimum values ​​in the image respectively. Then, the sliding window method is used to divide the entire image into image patches for subsequent processing and analysis.

[0080] Next, for the Select infrared related bands from hyperspectral image patches , weighted fusion generates infrared patches:

[0081] in, is a synthetic infrared image, is the kth band of the hyperspectral image patch, is the weight of the current band.

[0082] In one embodiment, if Figure 4 As shown, the method also includes: S301, obtaining the minimized reconstruction loss, spectral smoothness constraint and adversarial loss; S302: training an adversarial autoencoder based on the minimization of the reconstruction loss, the spectral smoothness constraint and the adversarial loss.

[0083] In this embodiment, minimizing the reconstruction loss means that when training the adversarial autoencoder, the reconstructed output is made as close to the original input as possible by optimizing the model parameters, thereby minimizing the difference (loss) between the two. For example, in image processing, the goal of the autoencoder is to encode the input image into a low-dimensional representation, and then reconstruct the image from this representation, minimizing the difference between the reconstructed image and the original image.

[0084] Spectral smoothness constraint refers to a constraint imposed on the model in order to maintain the smoothness of the spectral curve in hyperspectral image processing. For example, in hyperspectral image analysis, spectral smoothness constraint can prevent the model from generating unrealistic spectral curves, which may be discontinuous or abrupt due to noise or errors.

[0085] In an adversarial autoencoder, adversarial loss refers to the loss incurred when the discriminator classifies the latent features generated by the encoder, with the goal of making the latent features as close to the real data distribution as possible. For example, in the training of an adversarial autoencoder, adversarial loss penalizes those latent features that the discriminator can easily distinguish, thereby forcing the encoder to generate features that are closer to the real data distribution.

[0086] Adversarial autoencoders are a model that combines autoencoders and adversarial networks (GANs), which introduce a discriminator to guide the encoder to generate higher quality latent features. For example, adversarial autoencoders can be used to generate more realistic images, or when generating simulated samples of hyperspectral images, generate images that are more consistent with actual physical properties.

[0087] Specifically, the method also includes obtaining a minimized reconstruction loss, a spectral smoothness constraint, and an adversarial loss, and based on these losses, training the adversarial autoencoder. During the training process, the adversarial autoencoder needs to simultaneously consider the reconstruction loss (ensuring that the reconstructed data is as similar as possible to the original data), the spectral smoothness constraint (maintaining the physical rationality of the spectral curve), and the adversarial loss (making the generated potential features difficult to distinguish by the discriminator), and train the adversarial autoencoder through the joint optimization of these loss functions.

[0088] This application can generate higher quality hyperspectral image simulation samples by comprehensively considering reconstruction loss, spectral smoothing constraint and adversarial loss. In addition, the training method of adversarial autoencoder enhances the model's adaptability to different data distributions and improves the model's generalization ability.

[0089] In this embodiment, an adversarial autoencoder is used to generate a simulated sample of a hyperspectral image of an urban thermal environment. First, a reconstruction loss (e.g., mean square error) needs to be defined to ensure that the reconstructed image is as similar as possible to the original image. Next, a spectral smoothness constraint is introduced, such as using a regularization term to maintain the smoothness of the spectral curve. In addition, an adversarial loss needs to be defined, such as using a binary cross entropy loss, to train the discriminator to distinguish between real and generated potential features. During the training process, the encoder and the discriminator are alternately optimized so that the potential features generated by the encoder are increasingly difficult to distinguish by the discriminator, while maintaining the high quality of the reconstructed image and the physical rationality of the spectral curve. In this way, an adversarial autoencoder model capable of generating realistic hyperspectral image simulated samples can be trained.

[0090] Exemplarily, an adversarial autoencoder network structure is constructed, and all image patches are used to perform unsupervised pre-training on the network. Specifically, an adversarial autoencoder (AAE) model is constructed, which includes an encoder , Decoder and the discriminator, where is the feature vector in the latent space. The Adversarial Autoencoder (AAE) is trained by minimizing the reconstruction loss, the spectral smoothness constraint, and the adversarial loss.

[0091] The reconstruction loss is used to measure the difference between the reconstructed image output by the decoder and the original input image. The commonly used reconstruction loss function is the root mean square error (RMSE), which is defined as follows:

[0092] in, is the pixel value of the original image, is the corresponding pixel value of the reconstructed image, is the total number of pixels in the image. By minimizing the reconstruction loss, the model is able to learn an efficient representation of the input image.

[0093] Spectral smoothing loss is used to ensure that the spectral information of the reconstructed image remains smooth and continuous. This is especially important for hyperspectral images because hyperspectral data usually have strong correlations between adjacent bands. The definition of spectral smoothing loss is as follows:

[0094] in, is the reconstructed image at position Place The pixel value of each spectral channel, and are the height and width of the image, is the number of spectral channels. By minimizing the spectral smoothing loss, the model can generate images with smoother spectral characteristics.

[0095] Adversarial loss is used to train the discriminator and encoder so that the potential vector generated by the encoder is as close to the prior distribution as possible. The definition of adversarial loss is based on the probability of the discriminator output. The commonly used adversarial loss function is the binary cross entropy loss, which is defined as follows:

[0096] in, is the discriminator's analysis of the latent vector The probability of belonging to the true underlying distribution, is the distribution of latent vectors generated by the encoder, is a prior distribution (such as Gaussian distribution), mathematical expectation It means taking the average of the samples in the distribution. Express The distribution generated from the encoder After sampling, calculate the average of the function value; Express From the prior distribution After sampling, the average of the function value is calculated. By minimizing the adversarial loss, the discriminator can effectively distinguish between the real and generated latent vectors, while the encoder makes the generated latent vector closer to the real distribution through adversarial training.

[0097] By weighted combination of the above three losses,

[0098] in, is the reconstruction loss (RMSE), is the spectral smoothness constraint, It is against loss. Weight of the spectral smoothness constraint.

[0099] The reconstruction quality, spectral smoothness and the realism of the latent spatial distribution can be comprehensively considered to generate high-quality hyperspectral image simulation samples.

[0100] In this embodiment, the specific structure of the adversarial autoencoder is designed as follows: 1) Encoder: Input layer: receives data of size An image patch of, preferably, Assume that the hyperspectral image has spectral channels, so the dimension of the input layer is .

[0101] Convolutional layers: Multiple convolutional layers are used to extract spatial-spectral features of images. Each convolutional layer may include an activation function (such as ReLU) and a pooling layer to reduce the feature dimension and increase the abstraction ability of the model.

[0102] Fully connected layer: The convolutional layer is followed by several fully connected layers to further compress the feature representation and finally output the feature vector of the latent space. .

[0103] 2) Decoder: Input layer: receives the feature vector of the latent space .

[0104] Fully connected layer: First, the feature vector is expanded through a series of fully connected layers Dimensions, ready for upsampling.

[0105] Deconvolution / Upsampling Layers: Use deconvolution layers (or a combination of upsampling and convolution) to gradually increase the spatial dimensions of the feature map, restoring it to the size of the original image patch.

[0106] Output layer: The last deconvolution layer outputs a reconstructed image of the same size as the original image patch, i.e. .

[0107] 3) Discriminator: Input layer: receives a feature vector from the latent space or a noise vector sampled from a prior distribution.

[0108] Fully connected layers: A series of fully connected layers are used to progressively compress feature representations.

[0109] Output layer: Outputs a scalar representing the probability that the input vector comes from the true underlying distribution.

[0110] Training process: The encoder and decoder minimize the reconstruction loss and spectral smoothness constraints The training is performed by taking the weighted sum of to make the reconstructed image as similar as possible to the original image and with smooth spectrum.

[0111] The discriminator uses adversarial loss Trained to distinguish between latent vectors generated by the encoder and noise vectors sampled from a prior distribution.

[0112] The encoder is trained adversarially to make the generated latent vector as close as possible to the prior distribution.

[0113] In one embodiment, if Figure 5 As shown, the noise addition process of the first eigenvector and the forward diffusion process of the first eigenvector to obtain the second eigenvector includes: S401, gradually adding noise to the first feature vector, wherein the noise added in different steps is different; In this embodiment, noise processing refers to intentionally adding noise to data during data processing to simulate the imperfection and randomness in the real world or for data enhancement. For example, in image processing, Gaussian noise is added to a clear picture to cause random fluctuations in its pixel values.

[0114] In diffusion models, the process of simulating data from a clear state to being gradually covered by noise usually involves multiple steps of noise addition. For example, in image generation, Gaussian noise is gradually added to the original image until the image becomes completely noisy.

[0115] S402: Determine a second eigenvector based on the first eigenvector after adding noise.

[0116] In this embodiment, the original or un-noised feature representation obtained in the feature extraction process is, for example, a vector representing the spectral characteristics extracted from a hyperspectral image. The second feature vector is a feature vector processed by forward diffusion, that is, a feature representation after adding noise. For example, in the forward diffusion process, the feature vector of the image is obtained after multiple steps of noise addition.

[0117] The process of adding noise to the first eigenvector and forward diffusing it into the second eigenvector includes gradually adding different noises to the first eigenvector and determining the second eigenvector based on the eigenvectors after adding the noise. This means that in the process of generating simulated data, the original eigenvector (the first eigenvector) will undergo a process of being gradually disturbed by noise, and finally form a new eigenvector (the second eigenvector) containing noise.

[0118] Through noise processing and forward diffusion, the model can learn how to identify and process data in the presence of noise, thereby enhancing the robustness of the model in practical applications. The forward diffusion process can generate a series of data with different noise levels, increasing the diversity of the data set and helping the model learn more data changes.

[0119] In this embodiment, different types and levels of noise are added to the first feature vector according to predetermined steps and strategies, and based on the result after adding the noise, a second feature vector processed by forward diffusion is determined.

[0120] Specifically, the feature vector of a hyperspectral image needs to generate a series of feature vectors with different noise levels through forward diffusion. First, the first feature vector is extracted from the original hyperspectral image, and then other noises such as Gaussian noise or uniform noise are gradually added to this feature vector. The noise added in each step is different, simulating the diversity of noise in the real world. Finally, a series of feature vectors with different degrees of noise interference are obtained, which constitute the second feature vector set. In this way, the data characteristics under different noise conditions can be simulated, providing richer data samples for subsequent model training and analysis.

[0121] Exemplarily, a pre-trained adversarial autoencoder is used to encode a first image patch into a first feature vector as an initial state, and these image features are gradually converted into pure noise features (i.e., a second feature vector) by adding a series of controlled noises thereto.

[0122] The addition of noise follows a certain rule of increasing noise levels to ensure that the gradual transformation of image data is both smooth and controllable. Add noise gradually.

[0123] Assume that the steps of adding noise are Step 1 The noise addition of the step can be expressed as:

[0124] in, represents the original eigenvector (i.e., the eigenvector without added noise), is the noise drawn from a standard normal distribution, is a coefficient that controls the noise level, usually Follow In this embodiment, The initial value of is 0.9999 and decays linearly to 0.98 with t, which is set to 1000.

[0125] In one embodiment, if Figure 6 As shown, the target feature vector is denoised based on the acquired auxiliary vector, and the third feature vector is generated inversely, including: S601. Encode the pre-acquired auxiliary vector to determine a target auxiliary vector, wherein the auxiliary vector includes terrain feature category information, physical constraint information, and temperature information.

[0126] In this embodiment, in data processing, the auxiliary vector refers to an additional information vector provided in addition to the main data, which is used to enhance the performance of the model or provide additional context information. For example, in image processing, the auxiliary vector may contain the category information, physical properties or ambient temperature of the object in the image, etc., which is used to assist in image denoising or classification.

[0127] The target auxiliary vector is an auxiliary vector that has been encoded and preprocessed and is used for a specific processing target, such as denoising. For example, the auxiliary vector containing the feature category information, physical constraint information, and temperature information is encoded to determine the target auxiliary vector.

[0128] S602: Based on a preset time step, the target auxiliary vector and the target feature vector, determine the noise corresponding to the preset time step during denoising.

[0129] In this embodiment, denoising refers to the process of removing noise from data and restoring the original features of the data, for example, using a deep neural network model to process a noisy image to remove the noise and restore a clear image.

[0130] S603: Based on the noise, a deep neural network model denoises the target feature vector and reversely generates a third feature vector.

[0131] In this embodiment, the third eigenvector refers to a eigenvector generated after denoising, which is similar or identical to the original eigenvector (the first eigenvector), for example, a eigenvector similar to the original hyperspectral image eigenvector generated after denoising.

[0132] Specifically, an auxiliary vector containing feature category information, physical constraint information, and temperature information is encoded. Based on a preset time step and a target auxiliary vector, the noise to be removed during denoising is determined. A deep neural network model is used to denoise the target feature vector according to the determined noise. A denoised third feature vector is generated, which is similar to the original first feature vector.

[0133] By using auxiliary vectors, this application can more accurately identify and remove noise and improve the accuracy of denoising. The auxiliary vectors provide additional environmental and physical information, allowing the model to better adapt to different conditions and scenarios.

[0134] In this embodiment, a deep neural network model is trained to predict a given The noise that should be removed next The goal of the model is to learn the mapping To predict the noise , thereby gradually reducing the noise level and restoring the original feature vector .

[0135] The deep neural network model adopts a U-Net-like structure, and in order to improve its learning ability, it is designed to accept auxiliary vectors. Specifically, the encoder and gated switching module composed of a feedforward neural network encode the category information, physical constraints, and temperature information into auxiliary vectors. The neural network model receives the encoded auxiliary vector and the current noisy image patch feature (the second feature vector) as input, and predicts the noise component in the image patch feature through its deep network structure. After iterations, the original feature vector can be restored to be approximately noise-free. (Second image patch).

[0136] The neural network adopts a U-Net-like structure, in which the noisy image patch and time step As the main input, in order to enhance the expressiveness of the model, auxiliary vectors such as category information, physical constraints and temperature information are also integrated. Specifically, the neural network fuses the auxiliary vectors into a comprehensive vector through the gated switching module. :

[0137] in, is a gating parameter used to control the influence of class information, physical constraints, and temperature information. is injected into each layer of the U-Net-like network, and the output of the model is affected by adjusting the parameters of the regularization layer. Specifically, the network adopts a conditional regularization mechanism, combined with an auxiliary vector To generate scaling and bias parameters, adjust the intermediate features Distribution:

[0138] in, represents the normalized intermediate features, and and From the comprehensive auxiliary vector The generated scaling and bias parameters. The U-Net-like structure consists of an encoder and a decoder. The encoder path gradually extracts features of different scales while incorporating time steps. and auxiliary vector The decoder path gradually restores the spatial resolution of the image by upsampling, and uses skip connections to pass the encoder features to the decoder to enhance the expressiveness of the features. The predicted value of , expressed as:

[0139] in, is the mapping function of the entire neural network, receiving the noisy image patch , time step and the comprehensive auxiliary vector , and predict the noise component that should be removed in the next step.

[0140] The encoding of category information includes: Category Information Encode through an independently set feedforward neural network encoder to generate an auxiliary vector This auxiliary vector As one of the additional inputs of U-Net, it is used to guide the model to generate images that meet the characteristics of this category.

[0141]

[0142] The encoding of physical constraints includes: Physical constraints Represents the typical spectral characteristic curve of the corresponding category. This physical constraint is encoded by the encoder in the adversarial autoencoder (AAE) to generate an auxiliary vector .

[0143]

[0144] The encoding of temperature information includes: Temperature information Through a dedicated feed-forward neural network encoder Encode and generate auxiliary vector This encoder converts the temperature information from scalar or vector form into a feature representation for incorporation into the main network.

[0145]

[0146] Gated switch module: The gated switching module uses preset gating parameters To combine the various auxiliary vectors to generate a comprehensive auxiliary vector , guiding the denoising process of the neural network.

[0147]

[0148] in, Control class information auxiliary vector The impact of Controlling the Physical Constraint Auxiliary Vector The impact of Control temperature information auxiliary vector By adjusting the value of the parameters in the gated switching module, specific auxiliary information can be selectively included or excluded. When , an infrared image at a specific temperature can be generated; when When the temperature is not taken into account, a hyperspectral image can be generated.

[0149] The deep neural network model also includes a hybrid attention module and a gated switching module to determine the target auxiliary vector according to the needs. , target auxiliary vector Input the mixed attention module to make the deep neural network model have a certain bias to generate the corresponding image. Through the mixed attention module, the deep neural network model is more inclined when generating images.

[0150] Specifically, a hyperspectral image of an urban thermal environment containing noise is being processed. First, auxiliary information related to the image is collected, such as the type of objects (buildings, roads), physical constraints (such as the spectral reflectance characteristics of materials), and ambient temperature. Then, this auxiliary information is encoded into a target auxiliary vector. Next, according to a preset time step (for example, simulating a process in which noise gradually increases), the noise that needs to be removed at each step in the denoising process is determined. Finally, a trained deep neural network model is used to denoise the target feature vector containing noise based on the target auxiliary vector and the determined noise, and a clear third feature vector similar to the original image feature vector is generated in reverse. In this way, a denoised hyperspectral image can be obtained for subsequent analysis and simulation.

[0151] Exemplarily, a deep neural network model is used to predict each step of the noise removal process, thereby gradually reducing the noise level until the original image patch features are restored.

[0152] Specifically, a deep neural network model is trained to predict given The noise that should be removed next The goal of the model is to learn the mapping To predict the noise , thereby gradually reducing the noise level and restoring the original feature vector .in, They are the encoded category information, physical constraints and temperature information respectively.

[0153] Finally, the recovered image patch features (the third feature vector) are decoded using the pre-trained adversarial autoencoder to restore the original input image patch.

[0154] Specifically, using the decoder The restored feature vector Decode back the image patch to reconstruct the entire target image. The decoding process can be expressed as:

[0155] in, is the feature vector recovered through the reverse generation process.

[0156] In one embodiment, if Figure 1 As shown, the method also includes: S80. Update the deep neural network model based on the difference between the first image patch and the second image patch.

[0157] In this embodiment, the difference refers to the degree of difference between two image patches (the first image patch and the second image patch), which is usually calculated by a specific metric method (such as mean square error, structural similarity index, etc.). For example, the first image patch is the original image, and the second image patch is the image processed by noise, and the difference can quantify the visual and spectral differences between the two images.

[0158] A deep neural network model is a neural network that contains multiple nonlinear transformation layers and is capable of learning complex data representations and patterns. For example, a deep neural network model for image recognition contains multiple convolutional layers and fully connected layers. In machine learning, updating refers to adjusting the parameters of a model based on new data or feedback to improve the performance of the model. For example, using the gradient descent algorithm to update the weights of a neural network based on new training data.

[0159] Specifically, a suitable metric method is used to calculate the difference between the first image patch and the second image patch. According to the calculated difference, the parameters of the deep neural network model are adjusted to reduce the difference between the model output and the target.

[0160] This application can improve the adaptability and generalization ability of the model to new data by updating the model according to the differences between image patches. The updated model can capture the characteristics of the data more accurately, thereby improving the performance of the model on related tasks (such as image recognition and classification).

[0161] Exemplarily, a deep neural network model is used for hyperspectral image analysis of urban thermal environment. There are two image patches: the first image patch is the original hyperspectral image, and the second image patch is the same image after noise processing. First, the difference between the two image patches is calculated, for example, using mean square error (MSE) as a metric. Then, the parameters of the deep neural network model are updated according to this difference, possibly by adjusting the learning rate or optimizer settings, or directly fine-tuning the network weights. For example, the learning rate can be reduced so that the model makes more detailed adjustments on new data instead of updating the weights drastically, thereby maintaining the model's memory of old data.

[0162] In this way, the model is able to gradually adapt to new data while retaining its ability to recognize old data.

[0163] Specifically, the goal of the training process is to make the noise of the deep neural network model predictions (Second image patch) with real noise The difference between (first image patch) is minimized by the mean square error loss function To measure:

[0164] During the training process, by optimizing the loss function , the network gradually learns how to effectively remove noise and restore the feature vector of the original image. represents the mathematical expectation operator, and the parameters of the network are updated using common optimization methods (such as Adam), ultimately achieving the ability to generate noise-free, high-quality images.

[0165] The present application also provides a method for outputting results based on a hyperspectral image simulation sample, such as Figure 7 As shown, the method includes: S91. Obtain a source image.

[0166] In this embodiment, the source image refers to the original, unprocessed image data, which serves as the starting point of the simulation sample generation method, for example, an image of an urban area taken by a hyperspectral camera, which contains rich spectral information.

[0167] S92. According to the aforementioned generation method, a simulation image corresponding to the source image is obtained.

[0168] In this embodiment, the simulated image refers to an image obtained by a specific generation method and simulating the characteristics of the source image, such as a hyperspectral image of an urban area generated by a deep learning model, which simulates the spectral characteristics of a real image.

[0169] S93, mixing the source image and the simulation image to determine a training sample; In this embodiment, it refers to combining the source image and the simulated image in a certain ratio or manner to form a new training sample. For example, a source image and a corresponding simulated image are mixed in a ratio of 50%, that is, the training sample contains both real and simulated data.

[0170] Training samples are datasets used to train machine learning models, usually containing input data and corresponding labels or results. For example, a dataset containing multiple hyperspectral images and their corresponding category labels is used to train a classifier.

[0171] S94: Train a neural network classifier based on the training samples.

[0172] In this embodiment, the neural network classifier is a machine learning model that uses a neural network model to classify data, such as a convolutional neural network (CNN), which is used to classify images based on their spectral features.

[0173] Specifically, a source image is selected from a data set or obtained by shooting with a hyperspectral camera. Using the aforementioned generation method, a simulation image corresponding to the source image is generated. The source image and the simulation image are mixed in a certain proportion or manner to form a training sample. The mixed training sample is used to train a neural network classifier.

[0174] This application can improve the generalization ability of the model when facing real-world data by mixing real and simulated images as training samples. The introduction of simulated images increases the diversity of training samples and helps the model learn a wider range of feature representations.

[0175] Exemplarily, a hyperspectral image classification system for urban thermal environment monitoring is being developed. First, a series of hyperspectral source images of urban areas are acquired. Then, the simulated sample generation method described previously is used to generate simulated images corresponding to these source images. Next, each source image is mixed with its corresponding simulated image, for example, the real image and the simulated image can be arranged alternately, or they can be mixed in a certain proportion to form a new training sample set.

[0176] Finally, this mixed training sample set is used to train a neural network classifier, such as a convolutional neural network, which is able to classify different objects in urban areas based on the spectral characteristics of the image. In this way, the classifier can not only learn the characteristics of the real image, but also adapt to the possible variations in the simulated image, making it more robust and accurate in practical applications.

[0177] Specifically, simulated images are mixed with real images, a neural network classifier is trained, and the classification effect is evaluated. The goal of the classifier is to minimize the difference between the predicted category and the true category, which is usually achieved through the cross entropy loss function:

[0178] in, is the total number of categories, It is a sample Belongs to category The indicator, is the model prediction sample Belongs to category probability.

[0179] In this embodiment, two data sets, Pavia University (PU) and Salina (SA), are selected as examples. 1% of the labeled data of Pavia University (PU) and Salina (SA) are used for neural network classifier training, which is denoted as the Baseline method. The neural network classifier consists of a three-layer feedforward neural network.

[0180] Similarly, 5 simulated samples of each class are mixed into the training set and trained using the same classifier, which is denoted as Ours. The final classification results on the Pavia University (PU) and Salina (SA) datasets are shown in Table 1. From the results in Table 1, it can be found that adding simulated images for training can effectively improve the accuracy of hyperspectral image classification, which confirms the effectiveness of the sample generation method proposed in the present invention.

[0181] Table 1

[0182]

[0183] In summary, when a simulated image needs to be generated, a trained deep neural network model is used to randomly sample a feature vector (target feature efficiency) from the Gaussian noise feature (the second feature vector), and a reverse generation process is performed on it for a specific number of steps to generate a simulated image.

[0184] Specifically, initialize a noise feature vector from a Gaussian distribution ,in represents the final step of forward diffusion.

[0185] Next, execute The reverse generation process is repeated step by step to gradually reduce the noise and recover the features of the hyperspectral image patch. , the reverse process can be expressed as:

[0186] in, is a deep neural network model used to predict from Removed noise components. is the coefficient controlling the noise level, The random noise added is used to ensure the diversity of the generation process. Step by step reverse to , gradually reducing the noise in the feature vector and finally recovering the hyperspectral image patch features close to the original data distribution.

[0187] Then, use the pre-trained adversarial autoencoder to recover the feature vector To decode:

[0188] Here, Represents the generated simulated hyperspectral / infrared image patch. By repeating this process, a large number of simulated images can be generated starting from Gaussian noise. These images are not only visually similar to real hyperspectral / infrared images, but also can statistically simulate the distribution characteristics of real data. By adjusting the noise control parameters in the reverse diffusion step and deep neural network models The structure can further optimize the quality and diversity of simulation samples.

[0189] like Figure 8 As shown, the embodiment of the present application also provides a hyperspectral image simulation sample generation device, including: a data acquisition module configured to acquire a source image; A preprocessing module configured to preprocess the source image to determine a first image patch; a first encoding module configured to encode the first image patch into a first feature vector based on a trained adversarial autoencoder; A noise processing module configured to perform noise processing on the first feature vector and forward diffuse it into a second feature vector, wherein the first feature vector has different information contents from the second feature vector; A first determination module, configured to determine a target feature vector from the second feature vector based on a trained deep neural network model; A denoising processing module, configured to denoise the target feature vector based on the acquired auxiliary vector, and reversely generate a third feature vector, wherein the third feature vector is the same as the first feature vector; A second determination module is configured to decode the third feature vector into a second image patch based on the trained adversarial autoencoder, wherein the second image patch is similar to the first image patch.

[0190] In this embodiment, the data acquisition module is a component responsible for collecting raw data, and is used to obtain data such as hyperspectral images. For example, a hyperspectral camera system is used to capture hyperspectral images of the ground within a selected wavelength range. The hyperspectral camera system can obtain hyperspectral data of the surface according to a preset resolution and wavelength range. In addition, the hyperspectral camera system is also equipped with GPS and object classification and annotation tools for real-time object classification and geographic location tagging to ensure that the collected data accurately reflects the spectral characteristics and spatial distribution of the objects.

[0191] The preprocessing module is a component responsible for the preliminary processing of raw data in preparation for subsequent analysis or processing. The preprocessing module is mainly responsible for preprocessing the collected raw hyperspectral data, including data normalization, denoising, and cutting the image into small image patches using the sliding window method to generate infrared image patches. This step is to make the data suitable for the subsequent simulation sample generation process and improve the quality of the generated samples.

[0192] The first encoding module is a component that encodes image data into a feature vector using an adversarial autoencoder. For example, a deep learning model that takes a preprocessed image patch and converts it into a numerical feature vector. The noise processing module is a component responsible for adding noise to the feature vector to simulate the imperfections in the data acquisition process. For example, an adversarial autoencoder that gradually adds different levels of noise to the feature vector to generate multiple versions of the data. The first determination module is a component that uses a deep neural network model to extract useful features from noisy data. For example, a deep learning model that takes a noisy feature vector and identifies key features. The denoising module is a component responsible for recovering the original data features from noisy data. For example, a denoising algorithm that uses auxiliary information (such as physical constraints) to remove noise from the feature vector.

[0193] The second deterministic module is a component that decodes the feature vector back into image data using an adversarial autoencoder, such as a deep learning model that takes in the denoised feature vector and reconstructs the original image patch.

[0194] Among them, the first encoding module, the noise processing module, the first determination module, the denoising processing module and the second determination module constitute a deep learning processing unit.

[0195] Exemplarily, the deep learning processing module includes an adversarial autoencoder (AAE) and a diffusion model. The deep learning processing unit uses high-performance computing hardware (such as GPU or TPU) to support large-scale deep learning operations. The adversarial autoencoder is used to learn low-dimensional representations of hyperspectral image data, while the diffusion model is used to perform forward diffusion and reverse generation processes on these low-dimensional representations to generate high-quality simulation samples.

[0196] Specifically, a data acquisition module is used to acquire a hyperspectral image. A preprocessing module is used to perform operations such as cropping and scaling on the image. A first encoding module is used to encode the image patch into a feature vector. A noise processing module is used to add noise to the feature vector. A first determination module is used to extract useful features from the noise data. A denoising processing module is used to restore the original data features. A second determination module is used to decode the feature vector back into image data.

[0197] Suppose that it is necessary to generate simulated samples of hyperspectral images of urban thermal environments. First, the data acquisition module captures images of urban areas through a hyperspectral camera. Then, the preprocessing module crops and contrast-enhances the images to suit subsequent processing. The first encoding module encodes these image patches into feature vectors using the trained adversarial autoencoder. The denoising module adds different levels of noise to these feature vectors to simulate the noise in the real world. The first determination module uses a deep neural network model to extract key features from the noisy data. The denoising module uses auxiliary information (such as physical constraints) to remove noise from the feature vectors. Finally, the second determination module uses the adversarial autoencoder to decode the denoised feature vectors back to the image data to generate simulated image patches similar to the original image patches. In this way, high-quality simulated samples of hyperspectral images can be generated for training and testing deep learning models.

[0198] The present application greatly reduces the reliance on field-collected hyperspectral data, and generates a large number of simulation samples through the system proposed by the present invention, which can significantly improve the learning efficiency and prediction accuracy of the classification model.

[0199] It should be noted that the principle of solving the technical problem by the generating device provided in the embodiment of the present application is similar to the generating method provided in the embodiment of the present application. Therefore, the implementation of the generating device provided in the embodiment of the present application can refer to the implementation of the generating method provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0200] In another embodiment, the apparatus further comprises: The simulation sample generation and evaluation module is based on the data output by the deep learning processing module. The simulation sample generation and evaluation module performs the final simulation sample generation and quality evaluation. This module can not only generate hyperspectral image simulation samples, but also evaluate the generated samples according to a series of predefined quality indicators (such as spectral consistency, visual authenticity, and classification accuracy) to ensure the practicality and effectiveness of the simulation samples.

[0201] The user interface module provides a user-friendly interface that allows users to configure simulation parameters (such as wavelength range, resolution, noise level, etc.), start the simulation process, and view the simulation results. In addition, users can also access the evaluation report of the simulation sample through this interface, as well as further analyze and apply the generated samples.

[0202] The data storage module is responsible for storing the original hyperspectral data, processed data, generated simulation samples, and related evaluation reports. This module uses a high-speed, large-capacity storage solution to support the secure storage and fast access of large amounts of data.

[0203] An embodiment of the present application further provides an electronic device, comprising: a memory and a processor, wherein an executable program is stored in the memory, and the processor executes the executable program to implement the steps of any method provided in the embodiment of the present application.

[0204] The processor may be a general purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. The general purpose processor may be a microprocessor or any conventional processor.

[0205] Since the electronic device introduced in the embodiments of the present application is an electronic device provided with a memory for implementing the method disclosed in the embodiments of the present application, based on the method introduced in the embodiments of the present application, technical personnel in this field can understand the structure and deformation of the electronic device introduced in the embodiments of the present application, so it is not repeated here.

[0206] The embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any method of the task processing method provided in the embodiment of the present application are implemented.

[0207] The storage medium in this embodiment may be included in the electronic device, or may exist independently without being installed in the electronic device. The storage medium carries one or more computer programs, and when the one or more computer programs are executed, the steps of any method of the storage control method provided in the embodiment of the present application are implemented.

[0208] It should be understood that each scheme in this embodiment has the corresponding technical effects in the above method embodiment, which will not be repeated here.

[0209] According to an embodiment of the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. Optionally, the specific examples in this embodiment can refer to the examples described in any embodiment of the present application, and this embodiment is not repeated here. Obviously, it should be understood by those skilled in the art that the above-mentioned modules or steps of the present application can be implemented with a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented with a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from here, or they can be made into individual integrated circuit modules respectively, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the present application is not limited to any specific combination of hardware and software.

[0210] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A method for generating a hyperspectral image simulation sample, characterized in that: include: Get the source image; preprocessing the source image to determine a first image patch; encoding the first image patch into a first feature vector based on the trained adversarial autoencoder; Noising the first eigenvector and forward diffusing it into a second eigenvector, wherein the first eigenvector and the second eigenvector have different information contents; Based on the trained deep neural network model, determining a target feature vector from the second feature vector; Based on the acquired auxiliary vector, denoising the target feature vector, and reversely generating a third feature vector, wherein the third feature vector is the same as the first feature vector; Based on the trained adversarial autoencoder, the third feature vector is decoded into a second image patch, wherein the second image patch is similar to the first image patch.

2. The generation method according to claim 1, characterized in that: The obtaining of the source image comprises: Acquire a source image based on a preset resolution and a preset wavelength; Based on the pixel correspondence, the ground object category of the source image is marked.

3. The generation method according to claim 1, characterized in that: The preprocessing of the source image to determine the first image patch comprises: Normalizing the source image; The source image after segmentation processing is determined to determine a first image patch.

4. The generation method according to claim 1, characterized in that: The method further comprises: Get the minimized reconstruction loss, spectral smoothness constraint, and adversarial loss; An adversarial autoencoder is trained based on the minimization of the reconstruction loss, the spectral smoothness constraint, and the adversarial loss.

5. The generation method according to claim 1, characterized in that: The step of adding noise to process the first eigenvector and forward diffusing the first eigenvector into a second eigenvector includes: Step by step, adding noise to the first feature vector, wherein the noise added in different steps is different; A second eigenvector is determined based on the first eigenvector after noise is added.

6. The generation method according to claim 1, characterized in that: The step of denoising the target feature vector based on the acquired auxiliary vector and reversely generating a third feature vector comprises: Encoding the pre-acquired auxiliary vector to determine a target auxiliary vector, wherein the auxiliary vector includes ground feature category information, physical constraint information, and temperature information; Determine, based on a preset time step, the target auxiliary vector and the target feature vector, the noise corresponding to the preset time step during denoising; Based on the noise, the deep neural network model denoises the target feature vector and reversely generates a third feature vector.

7. The generation method according to claim 1, characterized in that: The method further comprises: Based on the difference between the first image patch and the second image patch, the deep neural network model is updated.

8. A method for simulating sample output results based on hyperspectral images, characterized in that: The method comprises: Get the source image; According to any one of the generation methods of claims 1 to 7, obtaining a simulation image corresponding to the source image; Mixing the source image and the simulated image to determine a training sample; Based on the training samples, a neural network classifier is trained.

9. A device for generating a hyperspectral image simulation sample, characterized in that: include: a data acquisition module configured to acquire a source image; A preprocessing module configured to preprocess the source image to determine a first image patch; a first encoding module configured to encode the first image patch into a first feature vector based on a trained adversarial autoencoder; A noise processing module configured to perform noise processing on the first feature vector and forward diffuse it into a second feature vector, wherein the first feature vector has different information contents from the second feature vector; A first determination module, configured to determine a target feature vector from the second feature vector based on a trained deep neural network model; A denoising processing module, configured to denoise the target feature vector based on the acquired auxiliary vector, and reversely generate a third feature vector, wherein the third feature vector is the same as the first feature vector; A second determination module is configured to decode the third feature vector into a second image patch based on the trained adversarial autoencoder, wherein the second image patch is similar to the first image patch.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein an executable program is stored in the memory, and the processor executes the executable program to implement the steps of the method according to any one of claims 1 to 7.

Images