Hyperspectral image generation method and device based on spectral library physical constraint, equipment and storage medium

Abstract

This invention provides a method, apparatus, device, and storage medium for generating hyperspectral images based on physical constraints of a spectral library, relating to the field of remote sensing image processing technology. The method includes: inputting a multispectral image into a pre-trained spectral generation network to obtain a hyperspectral image containing the target ground object; the spectral generation network is trained based on sample multispectral images, sample hyperspectral images, and a physical consistency loss function; the physical consistency loss function is determined based on a spectral angle loss function and a root mean square loss function; the spectral angle loss function aims to minimize the difference between the generated spectrum and the true spectrum in the standard ground object spectral library; the root mean square loss function aims to minimize the difference between the sample hyperspectral image and the reconstructed hyperspectral image. Through this approach, the physical reliability and visual quality of the hyperspectral image are improved, meeting the application requirements of subsequent downstream tasks.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a method, apparatus, device, and storage medium for generating hyperspectral images based on physical constraints of a spectral library. Background Technology

[0002] Hyperspectral images (HSI) typically contain dozens to hundreds of continuous and narrow spectral bands. Hyperspectral images containing specific ground features can precisely capture the unique spectral reflectance characteristics of those features, thus playing a crucial role in remote sensing ground feature identification, target detection, and other fields.

[0003] However, due to physical constraints such as optical diffraction limit and detector noise, the spatial resolution of acquired hyperspectral images is usually limited, and the acquisition cost is high. Therefore, the industry has begun to use spectral generation technology to expand the number of hyperspectral images. This technology can generate corresponding hyperspectral images from multispectral images (MSI) or RGB images that have high spatial resolution but few spectral bands.

[0004] In some related technologies, pre-trained deep learning models can be used to reconstruct the spectrum from multispectral images, generating hyperspectral images corresponding to the multispectral images. However, the training process of existing deep learning models only optimizes pixel-level numerical errors, lacking explicit constraints on the inherent physical characteristics of ground object spectra. This easily leads to physical distortions in the spectra of the generated hyperspectral images. Specifically, there are serious deviations between the spectra in the generated images and the true spectra in the standard ground object spectral library, and even abrupt changes that violate real physical laws. Since the generated hyperspectral images lack physical credibility, this will further affect subsequent downstream tasks such as high-precision target detection and ground object classification.

[0005] In summary, existing hyperspectral image generation methods struggle to produce physically plausible images, making it difficult to meet the application requirements of subsequent downstream tasks. Summary of the Invention

[0006] This invention provides a method, apparatus, device, and storage medium for generating hyperspectral images based on physical constraints of a spectral library, in order to address the shortcomings of existing hyperspectral image generation methods that struggle to generate physically reliable images and meet the application requirements of subsequent downstream tasks.

[0007] This invention provides a hyperspectral image generation method based on physical constraints of a spectral library, comprising: acquiring a multispectral image; the multispectral image is obtained by photographing a region containing a target feature; inputting the multispectral image into a pre-trained spectral generation network to obtain a hyperspectral image containing the target feature output by the spectral generation network; wherein, the spectral generation network is trained based on sample multispectral images, sample hyperspectral images, and a physical consistency loss function; the physical consistency loss function is determined based on a spectral angle loss function and a root mean square loss function; the spectral angle loss function aims to minimize the difference between the generated spectrum and the true spectrum in a standard ground feature spectral library; the generated spectrum is the spectrum of the reconstructed hyperspectral image, and the reconstructed hyperspectral image is the image learned and generated by the spectral generation network during pre-training based on the sample multispectral images and sample hyperspectral images; the root mean square loss function aims to minimize the difference between the sample hyperspectral image and the reconstructed hyperspectral image.

[0008] According to the present invention, a hyperspectral image generation method based on physical constraints of a spectral library is provided, wherein the sample multispectral image and the sample hyperspectral image are images containing sample ground features; the spectral generation network is trained based on the following steps: convolving the sample multispectral image to generate a shallow feature map; obtaining the ground feature mask image corresponding to the sample multispectral image, and generating a spatial attention weight map based on the ground feature mask image; the ground feature mask image includes the ground feature ground value mask of the sample ground features; performing weighted fusion of the shallow feature map and the spatial attention weight map to generate a weighted feature map; and training the spectral generation network based on the weighted feature map and the sample hyperspectral image.

[0009] According to the present invention, a hyperspectral image generation method based on physical constraints of a spectral library generates a spatial attention weight map based on the following steps: determining the feature centers of sample land features on a shallow feature map based on a land feature mask image; determining the land feature category probability corresponding to each pixel in the sample multispectral image based on the similarity between the shallow features and the feature centers of each pixel in the sample multispectral image; and determining the feature weight value corresponding to each pixel in the sample multispectral image based on the land feature category probability corresponding to each pixel in the sample multispectral image, thereby generating a spatial attention weight map.

[0010] According to the present invention, a hyperspectral image generation method based on physical constraints of a spectral library is provided. The spatial attention weight map is generated based on the following steps: determining the background region and the region where the sample land cover is located based on the land cover mask image; assigning a first feature weight value to the background region and assigning a second feature weight value to the region where the sample land cover is located to generate a spatial attention weight map; wherein, the first feature weight value is less than the second feature weight value.

[0011] According to the present invention, a hyperspectral image generation method based on physical constraints of a spectral library is provided. The spectral generation network includes a convolutional layer, multiple SGB modules, an embedding layer, and a mapping layer connected in sequence. The convolutional layer, multiple SGB modules, the embedding layer, and the mapping layer are used to extract features from the weighted feature map, obtain spectral features, and learn to generate a reconstructed hyperspectral image based on the spectral features and sample hyperspectral images.

[0012] According to the present invention, a hyperspectral image generation method based on physical constraints of a spectral library is provided. The SGB module includes an encoder, a bottleneck layer, and a decoder; wherein the encoder includes a first level and a second level, the first level including... Each SAB module, the second level includes... One SAB module, the encoder is used for downsampling and feature extraction of the input features; the bottleneck layer includes One SAB module; the decoder includes a third layer and a fourth layer, the third layer including Each SAB module, the fourth level includes... Each SAB module has a decoder used to upsample and extract features from the input; a skip connection is set between the first and third layers, and between the second and fourth layers; , , It is a positive integer.

[0013] According to the present invention, a hyperspectral image generation method based on physical constraints of a spectral library is provided. The SAB module includes an SSA module, a feedforward network, and two LayerNorm layers. The SSA module is used to calculate the self-attention of multiple head networks based on the input features, perform splicing and linear projection processing on the self-attention of multiple head networks to generate attention features, and perform reshaping processing on the attention features to generate output features.

[0014] This invention also provides a hyperspectral image generation device based on physical constraints of a spectral library, comprising: an acquisition module for acquiring multispectral images; the multispectral images are obtained by photographing a region containing a target feature; and an image generation module for inputting the multispectral images into a pre-trained spectral generation network to obtain a hyperspectral image containing the target feature output by the spectral generation network; wherein the spectral generation network is trained based on sample multispectral images, sample hyperspectral images, and a physical consistency loss function; the physical consistency loss function is determined based on a spectral angle loss function and a root mean square loss function; the spectral angle loss function aims to minimize the difference between the generated spectrum and the true spectrum in the standard ground feature spectral library; the generated spectrum is the spectrum of the reconstructed hyperspectral image, and the reconstructed hyperspectral image is an image learned and generated by the spectral generation network during pre-training based on the sample multispectral images and sample hyperspectral images; and the root mean square loss function aims to minimize the difference between the sample hyperspectral image and the reconstructed hyperspectral image.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-described methods for generating hyperspectral images based on physical constraints of a spectral library.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described hyperspectral image generation methods based on physical constraints of a spectral library.

[0017] The present invention provides a method, apparatus, device, and storage medium for generating hyperspectral images based on physical constraints of a spectral library. In the model training phase, the loss function used during pre-training is improved. The spectral generation network is trained using sample multispectral images, sample hyperspectral images, and a physical consistency loss function. The physical consistency loss function is determined based on the spectral angle loss function and the root mean square loss function. The spectral angle loss function aims to minimize the difference between the generated spectrum of the reconstructed hyperspectral image learned by the spectral generation network during pre-training and the true spectrum in the standard ground cover spectral library. This explicitly constrains the inherent physical characteristics of the generated spectrum at the physical level, enabling the spectral morphology features of the generated spectrum to converge to the true spectrum in the standard ground cover spectral library. This enables the spectral generation network to reconstruct spectra that conform to the physical laws of real ground objects' spectra. This is beneficial for the spectral generation network to generate hyperspectral images with physical credibility during the model inference stage. The root mean square loss function aims to minimize the difference between the sample hyperspectral image and the reconstructed hyperspectral image, which enables the spectral generation network to reconstruct images with high accuracy. This is beneficial for the spectral generation network to generate hyperspectral images with both high physical credibility and high visual quality during the model inference stage. During the model inference stage, only a multispectral image containing the target ground object needs to be input into the trained spectral generation network to generate a hyperspectral image containing the target ground object, which improves the physical credibility and visual quality of the hyperspectral image and can meet the application requirements of subsequent downstream tasks. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the hyperspectral image generation method based on physical constraints of a spectral library provided by the present invention.

[0020] Figure 2 This is one of the schematic diagrams of the training process of the spectrum generation network provided by the present invention.

[0021] Figure 3 This is the second schematic diagram of the training process of the spectral generation network provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the feature weighted modulation process provided by the present invention.

[0023] Figure 5 This is a schematic diagram of the SGB module provided by the present invention.

[0024] Figure 6 This is a schematic diagram of the structure of the hyperspectral image generation device based on physical constraints of a spectral library provided by the present invention.

[0025] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] Please see Figures 1 to 5 , Figure 1 This is a flowchart illustrating the hyperspectral image generation method based on physical constraints of a spectral library provided by the present invention. Figure 2 This is one of the schematic diagrams of the training process of the spectrum generation network provided by the present invention. Figure 3 This is the second schematic diagram of the training process of the spectrum generation network provided by the present invention. Figure 4 This is a schematic diagram of the feature-weighted modulation process provided by the present invention. Figure 5 This is a schematic diagram of the SGB module provided by the present invention.

[0028] like Figure 1 As shown, the hyperspectral image generation method based on physical constraints of the spectral library includes steps S110 to S120, and the specific steps are as follows: S110: Acquire multispectral images.

[0029] Multispectral images are obtained by photographing areas containing target features.

[0030] S120: Input the multispectral image into the pre-trained spectral generation network to obtain a hyperspectral image containing the target ground features output by the spectral generation network.

[0031] The spectral generation network is trained based on sample multispectral images, sample hyperspectral images, and a physical consistency loss function.

[0032] The physical consistency loss function is determined based on the spectral angle loss function and the root mean square loss function.

[0033] The spectral angle loss function aims to minimize the difference between the generated spectrum and the true spectrum in the standard ground cover spectral library.

[0034] The generated spectrum is the spectrum of the reconstructed hyperspectral image, which is the image learned by the spectral generation network during the pre-training process based on the sample multispectral image and the sample hyperspectral image.

[0035] The root mean square loss function aims to minimize the difference between the sample hyperspectral image and the reconstructed hyperspectral image.

[0036] Specifically, such as Figure 2 and Figure 3 As shown, during the model training phase, an end-to-end deep neural network is first constructed, and then sample multispectral images and sample hyperspectral images are acquired.

[0037] Among them, the sample multispectral image and the sample hyperspectral image are obtained by taking pictures of areas containing the same sample features, that is, both the sample multispectral image and the sample hyperspectral image are images containing sample features.

[0038] Furthermore, using sample multispectral images and sample hyperspectral images By pre-training an end-to-end deep neural network with a pre-built physical consistency loss function, a spectral generation network can be obtained.

[0039] Specifically, standard land cover spectral libraries typically contain real spectral data for hundreds of land cover categories. However, in specific remote sensing scenarios, downstream tasks usually only need to focus on a few typical land cover categories. For example, in special scenarios such as airports, farmland, and ports, only typical land cover categories such as aircraft, vegetation, water bodies, and buildings need to be considered. Therefore, directly using all spectral data from the standard land cover spectral library for pre-training may introduce interference from irrelevant information, reducing the training efficiency and constraint accuracy of deep neural networks. Based on this, this embodiment adopts a scenario-based prior screening strategy for various typical scenarios of downstream tasks. It selects the ground truth spectra of typical land cover categories that are strongly correlated with various typical scenarios from the standard land cover spectral library. These ground truth spectra should cover various major land cover categories. The set of these ground truth spectra is denoted as […]. , Represents a set The first in ( The set of true value spectra of (positive integers) ground features. The first in Each end element, , Represent real numbers, Indicates the number of spectral bands. It is essentially a one-dimensional vector. ( (where the integers are positive) is a set The number of endmembers in the text.

[0040] During pre-training, deep neural networks can learn from sample multispectral images and sample hyperspectral images. The differences between them are used to train the ability to reconstruct spectra, and attempts are made to reconstruct a hyperspectral image corresponding to the sample multispectral image. To comprehensively constrain the reconstruction of hyperspectral images by deep neural networks during pre-training. To ensure the physical plausibility of the generated spectrum, this embodiment constructs a physical consistency loss function that includes spectral angle (SAM) loss and root mean square loss (i.e., image generation loss) to collaboratively optimize the training process of the deep neural network.

[0041] Among them, the spectral angle is the core physical indicator for measuring the similarity between two spectra; the smaller the value, the closer the two spectra are.

[0042] During the training process for spectral reconstruction, explicit constraints need to be applied to the reconstruction of hyperspectral images. The spectra of pixels representing various land cover types in the sample are analyzed to make them as close as possible to the true spectra corresponding to their respective land cover categories. Based on this, this embodiment constructs a spectral angle loss function. To minimize the difference between the generated spectrum and the true spectrum in the standard ground cover spectral library, the spectral angle loss function... It can be expressed by the following formula: ; in, Represents the reconstruction of hyperspectral images The height also indicates the height of the sample's hyperspectral image. Height, reconstructing hyperspectral images and sample hyperspectral images The heights are the same; Represents the reconstruction of hyperspectral images The width also represents the sample hyperspectral image. Width, reconstructing hyperspectral images and sample hyperspectral images The widths are the same; , representing a set The true spectrum in; This represents a function that takes the minimum value. Represents the inverse cosine function; Represents the reconstruction of hyperspectral images Spatial coordinates of the medium sample features The generated spectrum of the pixel at that location; This represents the L2 norm (i.e., the Euclidean norm). This indicates transpose.

[0043] The physical meaning of the above formula lies in the fact that, for reconstructing hyperspectral images... Spatial coordinates of each type of ground feature in the sample Calculate the generated spectrum of the pixel at that location. With sets The minimum spectral angle among all true spectra is taken as the spectral angle loss for that pixel. This spectral angle loss can forcefully constrain the generated spectrum. It closely matches the spectral morphology of real ground features, thereby avoiding spectral distortion.

[0044] Meanwhile, to ensure the reconstruction of hyperspectral images With sample hyperspectral images To ensure consistency between them, this embodiment also constructs a root mean square loss function. To minimize the sample hyperspectral image With reconstructed hyperspectral images The difference between them, root mean square loss function It can be expressed by the following formula: ; in, Indicates the number of spectral bands; Represents the reconstruction of hyperspectral images Spatial coordinates of the medium sample features The first The pixel values ​​corresponding to each spectral band; Represents the hyperspectral image of the sample Spatial coordinates of the medium sample features The first The pixel values ​​corresponding to each spectral band.

[0045] Physical consistency loss function This includes the spectral angle loss function. and root mean square loss function Physical consistency loss function It can be expressed by the following formula: ; in, For weight hyperparameters.

[0046] By introducing a physical consistency loss function The explicit constraints can overcome the limitations of traditional spectral generation schemes that rely solely on data-driven approaches. Furthermore, by organically integrating the true spectra of standard ground cover spectral libraries as physical priors with spectral generation techniques, the trained spectral generation network can possess excellent spectral reconstruction capabilities.

[0047] Furthermore, during the model inference stage, the region containing the target object is photographed to obtain a multispectral image. This multispectral image is then input into the trained spectral generation network to obtain a hyperspectral image containing the target object. This hyperspectral image has both high visual quality and high physical credibility, and can provide high-quality samples for subsequent downstream tasks, realizing cross-domain knowledge transfer from multispectral images to hyperspectral images.

[0048] The hyperspectral image generation method based on physical constraints of a spectral library provided in this embodiment improves the loss function used in the pre-training stage. It trains the spectral generation network using sample multispectral images, sample hyperspectral images, and a physical consistency loss function. The physical consistency loss function is determined based on the spectral angle loss function and the root mean square loss function. The spectral angle loss function aims to minimize the difference between the generated spectrum of the reconstructed hyperspectral image learned by the spectral generation network during pre-training and the true spectrum in the standard ground cover spectral library. This explicitly constrains the inherent physical properties of the generated spectrum at the physical level, enabling the spectral morphology of the generated spectrum to converge to the true spectrum in the standard ground cover spectral library, thereby improving the light... The spectral generation network has the ability to reconstruct spectra that conform to the physical laws of real ground objects' spectra. This is beneficial for the spectral generation network to generate hyperspectral images with physical credibility during the model inference stage. The root mean square loss function aims to minimize the difference between the sample hyperspectral image and the reconstructed hyperspectral image, which enables the spectral generation network to reconstruct images with high accuracy. This is beneficial for the spectral generation network to generate hyperspectral images with both high physical credibility and high visual quality during the model inference stage. During the model inference stage, only a multispectral image containing the target ground object needs to be input into the trained spectral generation network to generate a hyperspectral image containing the target ground object. This improves the physical credibility and visual quality of the hyperspectral image and can meet the application requirements of subsequent downstream tasks.

[0049] In some embodiments, the sample multispectral image and the sample hyperspectral image are images containing sample ground features; the spectral generation network is trained based on the following steps: convolving the sample multispectral image to generate a shallow feature map; obtaining the ground feature mask image corresponding to the sample multispectral image, and generating a spatial attention weight map based on the ground feature mask image; the ground feature mask image includes the ground feature ground value mask of the sample ground features; performing weighted fusion on the shallow feature map and the spatial attention weight map to generate a weighted feature map; and training the spectral generation network based on the weighted feature map and the sample hyperspectral image.

[0050] In existing technologies, models typically need to process all regions of a sample image equally. This training method is prone to interference from irrelevant background regions, thereby reducing the model's image generation accuracy. Therefore, this embodiment introduces a feature weighted modulation mechanism guided by a feature-of-interest mask. This mechanism explicitly enhances the feature representation of the region of interest during feature learning and assigns higher weights to pixels in the physical consistency loss function, thereby guiding the model to focus on key targets (i.e., sample features).

[0051] like Figure 4 As shown, this embodiment proposes a feature weighted modulation mechanism guided by ground-value masks of typical ground features. This mechanism aims to guide deep neural networks to focus on and enhance the feature representation of regions of interest during the feature extraction stage by utilizing ground-value masks of typical ground features.

[0052] Specifically, in acquiring multispectral images of samples ( The heights of both the multispectral and hyperspectral images of the samples are The width of both the multispectral and hyperspectral images of the samples is , After obtaining the total number of spectral bands in the sample multispectral image, the sample multispectral image can also be acquired. Corresponding pixel-level ground feature mask image Among them, ground feature mask images Includes ground truth masks for sample ground features.

[0053] Optionally, in the ground feature mask image In the middle, the spatial coordinates of each type of sample land cover The specific value of a pixel can represent the type of land cover to which that pixel belongs. For example, the number 0 represents the image background, the number 1 represents an airplane, the number 2 represents vegetation, and the number 3 represents water.

[0054] Optionally, multispectral images of the samples Corresponding pixel-level ground feature mask image It can be annotated from a public dataset or generated using weakly supervised or semi-supervised segmentation methods.

[0055] Furthermore, in order to mask the ground features in the image Prior information about land cover categories is injected into a deep neural network, which can then be used to analyze land cover mask images. Generate spatial attention weight map .

[0056] Meanwhile, multispectral images of the samples Perform 3×3 convolution processing (3×3 convolution processing refers to performing convolution operations using 3×3 convolution kernels) to generate shallow feature maps. .

[0057] Furthermore, for shallow feature maps Spatial attention weight map Perform weighted fusion to generate a weighted feature map. , thus completing feature-weighted modulation.

[0058] Optionally, weighted feature map It can be expressed by the following formula: ; Among them, the symbol " " indicates element-wise multiplication.

[0059] Furthermore, weighted feature map and sample hyperspectral images The data will be fed into a deep neural network for cross-band dependency modeling to train a spectral generation network.

[0060] The hyperspectral image generation method based on physical constraints of the spectral library provided in this embodiment generates a weighted feature map through a feature weighted modulation mechanism. This allows the deep neural network to be forced to allocate more computing resources to the high-weight regions of interest (i.e., the regions where the sample land features are located) in subsequent cross-band dependency modeling. As a result, the deep neural network can learn more accurate and discriminative spectral features of land features. This mechanism effectively solves the problem of uniformly processing the entire map and ignoring the spatial distribution of land features in traditional methods, and realizes differentiated feature learning guidance.

[0061] In some embodiments, the spatial attention weight map is generated based on the following steps: determining the feature center of the sample land cover on the shallow feature map based on the land cover mask image; determining the land cover category probability corresponding to each pixel in the sample multispectral image based on the similarity between the shallow features and the feature center of each pixel in the sample multispectral image; and determining the feature weight value corresponding to each pixel in the sample multispectral image based on the land cover category probability corresponding to each pixel in the sample multispectral image, thereby generating the spatial attention weight map.

[0062] In this embodiment, a spatial attention weight map can be generated using the category center-guided method. .

[0063] Specifically, for ground feature mask images For each type of ground cover in the sample, the multispectral image of that ground cover can be calculated. Shallow feature map Feature center of the corresponding region Feature center It can be expressed by the following formula: ; ; in, Represents ground feature mask images Above, the land feature category is Spatial coordinates of the sample features The set of corresponding pixels; Represents ground feature mask images Upper Earth Category Spatial coordinates of the sample features In shallow feature maps The corresponding position on; Represents ground feature mask images Above, the land feature category is Spatial coordinates of the sample features The corresponding pixels.

[0064] Understandably, due to the ground feature mask image There may be multiple ground features in the sample. For each ground feature, the shallow feature map of that ground feature can be calculated. The corresponding feature center is located in the upper region, therefore there may be multiple feature centers.

[0065] Furthermore, for sample multispectral images For each pixel in the dataset, the similarity between the shallow features of that pixel and all feature centers is calculated. Based on the similarity between the shallow features of that pixel and all feature centers, the probability value of that pixel belonging to each land cover category is determined. The highest probability value or the probability value of the target land cover category is selected from these probability values ​​as the land cover category probability corresponding to that pixel.

[0066] Optionally, the similarity between the shallow features of a pixel and all feature centers can be cosine similarity.

[0067] Furthermore, for sample multispectral images For each pixel in the dataset, the probability of the land cover category corresponding to that pixel is used as the feature weight value for that pixel: ; in, Representing spatial coordinates The feature weight value corresponding to the pixel at that location; Represents the set of all land cover categories; This represents the function that takes the maximum value. This represents the cosine function.

[0068] After calculating the feature weight values ​​for all pixels, a complete spatial attention weight map can be constructed. .

[0069] The hyperspectral image generation method based on physical constraints of the spectral library provided in this embodiment generates a spatial attention weight map through the category center-guided method. This weighting strategy allows the feature weight values ​​to change smoothly in space and can adaptively highlight the region most similar to the target land cover category.

[0070] In some embodiments, the spatial attention weight map is generated based on the following steps: determining the background region and the region where the sample land cover is located in the multispectral image of the sample based on the land cover mask image; assigning a first feature weight value to the background region and assigning a second feature weight value to the region where the sample land cover is located to generate the spatial attention weight map; wherein the first feature weight value is less than the second feature weight value.

[0071] In this embodiment, a spatial attention weight map can be generated using a binary mask-guided method. .

[0072] Specifically, based on binary ground feature mask images For sample multispectral images Perform region segmentation and determine the multispectral image of the sample. The background area and the area where the sample features are located.

[0073] Furthermore, a first feature weight value is assigned to the background region, and a second feature weight value is assigned to the region where the sample features are located, thereby constructing a spatial attention weight map. .

[0074] Among them, the weight value of the first feature is less than the weight value of the second feature.

[0075] For example, a higher second feature weight value (e.g., 1.0) is assigned to the pixel region of the sample land feature, and a lower first feature weight value (e.g., 0.1) is assigned to the background region.

[0076] Compared to the category center-guided method, the binary mask-guided method requires less computation and can make the training of deep neural networks more stable.

[0077] In some embodiments, the spectral generation network includes a convolutional layer, multiple SGB modules, an embedding layer, and a mapping layer connected in sequence; the convolutional layer, multiple SGB modules, the embedding layer, and the mapping layer are used to extract features from the weighted feature map, obtain spectral features, and learn to generate a reconstructed hyperspectral image based on the spectral features and the sample hyperspectral image.

[0078] In obtaining the weighted feature map Subsequently, deep neural networks can utilize cross-domain spectral self-attention mechanisms to model sample multispectral images. and sample hyperspectral images Long-range dependence between different spectral bands.

[0079] Specifically, for deep neural networks, sample multispectral images Given a source domain, the sample hyperspectral image For the target domain, the core training objective of the deep neural network is to learn a nonlinear mapping function under the physical constraints of a standard ground cover spectral library. This enables it to learn and generate reconstructed hyperspectral images. satisfy And reconstruct hyperspectral images It is necessary to maintain the multispectral image of the sample. The image consistency must also have physical consistency with the standard ground cover spectral library.

[0080] Optionally, nonlinear mapping function It can also be expressed by the following formula: .

[0081] To learn nonlinear mapping functions This embodiment designs the specific structure of the deep neural network.

[0082] In this embodiment, the deep neural network includes a convolutional layer, multiple SGB (Spectral Generation Block) modules, an embedding layer, and a mapping layer connected in sequence.

[0083] Specifically, the deep neural network first uses a 3×3 convolutional layer to weight the feature map. Feature extraction was performed to extract multispectral images of the samples. Shallow spectral characteristics ( The number of channels representing shallow spectral characteristics.

[0084] Furthermore, shallow spectral features The input is fed into multiple cascaded SGB modules for deep feature extraction to obtain deep spectral features. The embedding layer and mapping layer then use 3×3 convolution operations to process and learn the deep spectral features, enabling the deep neural network to perform deep feature extraction based on spectral features and sample hyperspectral images. Reconstructed into hyperspectral images The ability.

[0085] In some embodiments, the SGB module includes an encoder, a bottleneck layer, and a decoder; wherein the encoder includes a first layer and a second layer, the first layer including Each SAB module, the second level includes... One SAB module, the encoder is used for downsampling and feature extraction of the input features; the bottleneck layer includes One SAB module; the decoder includes a third layer and a fourth layer, the third layer including Each SAB module, the fourth level includes... Each SAB module has a decoder used to upsample and extract features from the input; a skip connection is set between the first and third layers, and between the second and fourth layers; , , It is a positive integer.

[0086] In this embodiment, the SGB module is the core feature extraction unit of the deep neural network, and multiple cascaded SGB modules can achieve spectral feature optimization from coarse to fine.

[0087] Specifically, such as Figure 5 As shown, each SGB module adopts an "encoder-bottleneck layer-decoder" structure. Its core design idea is to compress the feature space dimension and enhance the expression of spectral features by downsampling, and then restore the spatial resolution by upsampling, while retaining shallow detail information by skip connections.

[0088] The encoder comprises two levels: a first level and a second level. The first level includes... The second level includes one SAB module (Spectral Attention Block). One SAB module, the encoder can be used first. Each SAB module performs feature extraction and downsampling operations, and then uses... Each SAB module continues to perform feature extraction and downsampling operations.

[0089] The encoder's output becomes the input to the bottleneck layer, which includes... A series of SAB modules.

[0090] The output of the bottleneck layer becomes the input of the decoder. The decoder adopts a structure symmetrical to the encoder, and consists of two layers: the third layer and the fourth layer. The third layer includes... Each SAB module, the fourth level includes... Each SAB module, the decoder, is used to perform upsampling and feature extraction on the input features.

[0091] The decoder achieves upsampling through transposed convolution. The transposed convolution kernel has a size of 3×3, a stride of 2, and padding of 1, ensuring that the size of the upsampled feature map matches the size of the feature map in the encoder's downsampling stage. Figure 1 To.

[0092] Meanwhile, to compensate for the loss of detail during downsampling, skip connections are set between corresponding layers of the decoder and encoder, that is, skip connections are set between the first and third layers, and between the second and fourth layers. After adjusting the channel dimension with a 1×1 convolution kernel, feature stitching can be performed.

[0093] in, , , It is a positive integer, and Figure 5 The diagram only shows a simplified structure of the encoder, bottleneck layer, and decoder, and does not show the total number of SAB modules. In actual application scenarios, the number of SAB modules in the encoder, bottleneck layer, and decoder can be adjusted as needed.

[0094] In some embodiments, the SAB module includes an SSA module, a feedforward network, and two LayerNorm layers; wherein, the SSA module is used to compute the self-attention of multiple head networks based on the input features, concatenate and linearly project the self-attention of multiple head networks to generate attention features, and reshape the attention features to generate output features.

[0095] In this embodiment, the SAB module is the basic unit for spectral feature extraction in deep neural networks. It includes the SSA module (Spectral Self-Attention), a feedforward network, and two LayerNorm layers. The SAB module adopts a pre-normalization design, that is, normalization is performed before attention calculation and feedforward operation, which can effectively alleviate the gradient vanishing problem during training.

[0096] The SSA module can process the input features. ( express The channel number is flattened along the spatial dimension, transforming it into a two-dimensional word element. Subsequently, It will be linearly projected into a query vector Key vector Sum value vector : ; ; ; in, , , These are learnable parameters.

[0097] Furthermore, the SSA module can route the query vector along the spectral dimension. Key vector Sum value vector Divide into N (N is a positive integer) parallel branches respectively: ;

[0098] ; in, Represents the query vector The split ( (a positive integer) branches; Represents the key vector The split ( (a positive integer) branches; Value vector The split ( (a positive integer) branches; Dimension .

[0099] Furthermore, the SSA module treats each spectral representation as a lexical unit and computes self-attention for multiple head networks.

[0100] Among them, the Head network Self-attention It can be expressed by the following formula: ; ; in, Indicates the activation function; These are learnable parameters used to adjust the weights, through reweighting the head network. Matrix multiplication within To adjust self-attention Self-attention This is used to dynamically adjust the attention contribution of different spectral bands to address the problem of spectral density varying with wavelength, thereby better modeling the mapping relationship across bands.

[0101] After concatenating the outputs of all the head networks, the SSA module can obtain the final attention features through linear projection.

[0102] Among them, the attention features generated by the SSA module It can be expressed by the following formula: ; in, Indicates splicing; Represents the projection matrix; This is a bias term.

[0103] Furthermore, the SSA module can handle attention features. The reshaping process is performed to generate output features.

[0104] The hyperspectral image generation method based on physical constraints of the spectral library provided in this embodiment, through cross-band long-range dependency modeling and dynamic attention weighting of the spectral dimension, enables the SSA module to effectively model the cross-domain dependency relationship between sample multispectral images and sample hyperspectral images, accurately capture global features of the spectral dimension, which is beneficial for subsequent consistent alignment of cross-domain spectral features and provides a feature foundation for high-reliability spectrum generation. In addition, existing methods have limited mapping capabilities from multispectral images to hyperspectral images and are prone to mapping deviations between spectral bands. However, the spectral self-attention mechanism of this embodiment, through learnable band attention bias, dynamically adjusts the contribution weights of different spectral intervals, which can more accurately capture global dependencies of the spectral dimension, thereby reducing mapping deviations between spectral bands during the inference stage.

[0105] This invention also provides a hyperspectral image generation apparatus based on physical constraints of a spectral library. Please refer to [link / reference]. Figure 6 , Figure 6 This is a schematic diagram of the hyperspectral image generation device based on physical constraints of a spectral library provided by the present invention. In this embodiment, the hyperspectral image generation device based on physical constraints of a spectral library includes an acquisition module 610 and an image generation module 620.

[0106] The acquisition module 610 is used to acquire multispectral images.

[0107] Multispectral images are obtained by photographing areas containing target features.

[0108] The image generation module 620 is used to input multispectral images into a pre-trained spectral generation network to obtain a hyperspectral image containing the target ground features output by the spectral generation network.

[0109] The spectral generation network is trained based on sample multispectral images, sample hyperspectral images, and a physical consistency loss function.

[0110] The physical consistency loss function is determined based on the spectral angle loss function and the root mean square loss function.

[0111] The spectral angle loss function aims to minimize the difference between the generated spectrum and the true spectrum in the standard ground cover spectral library.

[0112] The generated spectrum is the spectrum of the reconstructed hyperspectral image, which is the image learned by the spectral generation network during the pre-training process based on the sample multispectral image and the sample hyperspectral image.

[0113] The root mean square loss function aims to minimize the difference between the sample hyperspectral image and the reconstructed hyperspectral image.

[0114] In some embodiments, the sample multispectral image and the sample hyperspectral image are images containing sample ground features; the spectral generation network is trained based on the following steps: convolving the sample multispectral image to generate a shallow feature map; obtaining the ground feature mask image corresponding to the sample multispectral image, and generating a spatial attention weight map based on the ground feature mask image; the ground feature mask image includes the ground feature ground value mask of the sample ground features; performing weighted fusion on the shallow feature map and the spatial attention weight map to generate a weighted feature map; and training the spectral generation network based on the weighted feature map and the sample hyperspectral image.

[0115] In some embodiments, the spatial attention weight map is generated based on the following steps: determining the feature center of the sample land cover on the shallow feature map based on the land cover mask image; determining the land cover category probability corresponding to each pixel in the sample multispectral image based on the similarity between the shallow features and the feature center of each pixel in the sample multispectral image; and determining the feature weight value corresponding to each pixel in the sample multispectral image based on the land cover category probability corresponding to each pixel in the sample multispectral image, thereby generating the spatial attention weight map.

[0116] In some embodiments, the spatial attention weight map is generated based on the following steps: determining the background region and the region where the sample land cover is located in the multispectral image of the sample based on the land cover mask image; assigning a first feature weight value to the background region and assigning a second feature weight value to the region where the sample land cover is located to generate the spatial attention weight map; wherein the first feature weight value is less than the second feature weight value.

[0117] In some embodiments, the spectral generation network includes a convolutional layer, multiple SGB modules, an embedding layer, and a mapping layer connected in sequence; the convolutional layer, multiple SGB modules, the embedding layer, and the mapping layer are used to extract features from the weighted feature map, obtain spectral features, and learn to generate a reconstructed hyperspectral image based on the spectral features and the sample hyperspectral image.

[0118] In some embodiments, the SGB module includes an encoder, a bottleneck layer, and a decoder; wherein the encoder includes a first layer and a second layer, the first layer including Each SAB module, the second level includes... One SAB module, the encoder is used for downsampling and feature extraction of the input features; the bottleneck layer includes One SAB module; the decoder includes a third layer and a fourth layer, the third layer including Each SAB module, the fourth level includes... Each SAB module has a decoder used to upsample and extract features from the input; a skip connection is set between the first and third layers, and between the second and fourth layers; , , It is a positive integer.

[0119] In some embodiments, the SAB module includes an SSA module, a feedforward network, and two LayerNorm layers; wherein, the SSA module is used to compute the self-attention of multiple head networks based on the input features, concatenate and linearly project the self-attention of multiple head networks to generate attention features, and reshape the attention features to generate output features.

[0120] The present invention also provides an electronic device. Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 7 As shown, the electronic device may include a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a hyperspectral image generation method based on physical constraints of a spectral library.

[0121] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0122] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hyperspectral image generation method based on physical constraints of the spectral library provided by the above methods.

[0123] The present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the hyperspectral image generation method based on physical constraints of the spectral library provided by the above methods.

[0124] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating hyperspectral images based on physical constraints of a spectral library, characterized in that, include: Acquire multispectral images; The multispectral image is obtained by photographing an area containing the target feature; The multispectral image is input into a pre-trained spectral generation network to obtain a hyperspectral image containing the target ground object output by the spectral generation network; The spectral generation network is trained based on sample multispectral images, sample hyperspectral images, and a physical consistency loss function. The physical consistency loss function is determined based on the spectral angle loss function and the root mean square loss function; The spectral angle loss function aims to minimize the difference between the generated spectrum and the true spectrum in the standard ground cover spectral library; the generated spectrum is the spectrum of the reconstructed hyperspectral image, and the reconstructed hyperspectral image is an image learned by the spectral generation network during the pre-training process based on the sample multispectral image and the sample hyperspectral image; The root mean square loss function aims to minimize the difference between the sample hyperspectral image and the reconstructed hyperspectral image.

2. The hyperspectral image generation method based on physical constraints of a spectral library according to claim 1, characterized in that, The multispectral image and the hyperspectral image of the sample are images containing ground features of the sample. The spectrum generation network was trained based on the following steps: The multispectral images of the samples are convolved to generate shallow feature maps; Obtain the land cover mask image corresponding to the multispectral image of the sample, and generate a spatial attention weight map based on the land cover mask image; the land cover mask image includes the ground truth mask of the land cover in the sample; The shallow feature map and the spatial attention weight map are weighted and fused to generate a weighted feature map; The spectral generation network is trained based on the weighted feature map and the sample hyperspectral image.

3. The hyperspectral image generation method based on physical constraints of a spectral library according to claim 2, characterized in that, The spatial attention weight map is generated based on the following steps: Based on the land cover mask image, determine the feature center of the sample land cover on the shallow feature map; Based on the similarity between the shallow features of each pixel in the sample multispectral image and the feature center, the probability of the land cover category corresponding to each pixel in the sample multispectral image is determined. Based on the probability of the land cover category corresponding to each pixel in the sample multispectral image, the feature weight value corresponding to each pixel in the sample multispectral image is determined, and the spatial attention weight map is generated.

4. The hyperspectral image generation method based on physical constraints of a spectral library according to claim 2, characterized in that, The spatial attention weight map is generated based on the following steps: Based on the land cover mask image, determine the background region and the region where the land cover is located in the multispectral image of the sample; Assign a first feature weight value to the background region and a second feature weight value to the region where the sample features are located, thereby generating the spatial attention weight map; Wherein, the first feature weight value is less than the second feature weight value.

5. The hyperspectral image generation method based on physical constraints of a spectral library according to claim 2, characterized in that, The spectrum generation network includes a convolutional layer, multiple SGB modules, an embedding layer, and a mapping layer connected in sequence. The convolutional layer, the multiple SGB modules, the embedding layer, and the mapping layer are used to extract features from the weighted feature map, obtain spectral features, and learn to generate the reconstructed hyperspectral image based on the spectral features and the sample hyperspectral image.

6. The hyperspectral image generation method based on physical constraints of a spectral library according to claim 5, characterized in that, The SGB module includes an encoder, a bottleneck layer, and a decoder; The encoder includes a first level and a second level, wherein the first level includes... A SAB module, the second level including The SAB module, wherein the encoder is used to perform downsampling and feature extraction on the input features; The bottleneck layer includes The SAB module described above; The decoder includes a third layer and a fourth layer, the third layer including... The SAB module, the fourth level includes The SAB module is described above, and the decoder is used to perform upsampling and feature extraction on the input features; A skip connection is set between the first level and the third level, and a skip connection is set between the second level and the fourth level; , , It is a positive integer.

7. The hyperspectral image generation method based on physical constraints of a spectral library according to claim 6, characterized in that, The SAB module includes an SSA module, a feedforward network, and two LayerNorm layers; The SSA module is used to calculate the self-attention of multiple head networks based on the input features, concatenate and linearly project the self-attention of the multiple head networks to generate attention features, and reshape the attention features to generate output features.

8. A hyperspectral image generation device based on physical constraints of a spectral library, characterized in that, include: The acquisition module is used to acquire multispectral images; The multispectral image is obtained by photographing an area containing the target feature; An image generation module is used to input the multispectral image into a pre-trained spectral generation network to obtain a hyperspectral image containing the target land cover output by the spectral generation network. The spectral generation network is trained based on sample multispectral images, sample hyperspectral images, and a physical consistency loss function. The physical consistency loss function is determined based on the spectral angle loss function and the root mean square loss function; The spectral angle loss function aims to minimize the difference between the generated spectrum and the true spectrum in the standard ground cover spectral library; the generated spectrum is the spectrum of the reconstructed hyperspectral image, and the reconstructed hyperspectral image is an image learned by the spectral generation network during the pre-training process based on the sample multispectral image and the sample hyperspectral image; The root mean square loss function aims to minimize the difference between the sample hyperspectral image and the reconstructed hyperspectral image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the hyperspectral image generation method based on physical constraints of a spectral library as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the hyperspectral image generation method based on physical constraints of a spectral library as described in any one of claims 1 to 7.

Images