A method and device for generating hyperspectral images

Abstract

The present application discloses a method and device for generating a hyperspectral image. The method comprises the following steps: obtaining a multispectral image to be super-resolved; inputting the multispectral image to be super-resolved into a trained spectral super-resolution model to obtain a super-resolved hyperspectral image; training the trained spectral super-resolution model based on multiple sample multispectral images; clustering the sample multispectral image blocks and sample hyperspectral image blocks; and obtaining the sample multispectral image blocks by clustering analysis; wherein the sample multispectral image blocks and the sample hyperspectral image blocks are located in the same spatial region. The spectral super-resolution model is then trained after dynamically assigning different weights to different ground object types in the image blocks. This avoids the problem of low super-resolution accuracy of ground object types with a small proportion being suppressed by ground object types with a large proportion during training due to insufficient prior knowledge provided, thereby improving super-resolution accuracy.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for generating a hyperspectral image. Background Art

[0002] While obtaining the spatial location information of ground objects, hyperspectral images can also obtain a continuous and complete spectral curve for each pixel. The spectral curve reflects the physical and chemical intrinsic characteristics of the ground object target, and is therefore widely used in many fields such as land resources, ecological environment, etc.

[0003] Currently, hyperspectral images are typically generated through methods such as dictionary learning, spectral unmixing, and linear regression. These methods utilize prior knowledge to build physical models to reconstruct hyperspectral images, or they can be combined with deep learning methods for spectral super-resolution to generate higher-quality hyperspectral images. However, existing deep learning methods are primarily designed for natural images, which typically capture close-ups, simple objects, and have low spectral resolution. In contrast, satellite remote sensing images are captured from high altitudes, contain complex ground objects, and have high spectral resolution, strong continuity, and strong correlation. Therefore, directly applying spectral super-resolution methods for natural images fails to fully exploit the complex characteristics of remote sensing images, thus affecting super-resolution accuracy.

[0004] Therefore, how to improve super-resolution accuracy has become an urgent problem to be solved in this field. Summary of the Invention

[0005] The present application provides a method and apparatus for generating a hyperspectral image, the purpose of which is to improve super-resolution accuracy.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] A method for generating a hyperspectral image, comprising:

[0008] Acquire the multispectral image to be super-resolved;

[0009] The multispectral image to be super-resolved is input into a trained spectral super-resolution model to obtain a super-resolved hyperspectral image; wherein the trained spectral super-resolution model is obtained by training the spectral super-resolution model based on multiple sample multispectral images; the sample multispectral images are obtained by clustering analysis of sample multispectral image blocks and sample hyperspectral image blocks; the sample multispectral image blocks and the sample hyperspectral image blocks are in the same spatial area; the trained spectral super-resolution model includes a spectral branch network and a spatial branch network.

[0010] Optionally, inputting the multispectral image to be super-resolved into a trained spectral super-resolution model to obtain a super-resolved hyperspectral image includes:

[0011] Inputting the multispectral image to be super-resolved into a spectral branch network to obtain spectral feature information of the hyperspectral image;

[0012] Inputting the multispectral image to be super-resolved into a spatial branch network to obtain spatial feature information of the hyperspectral image;

[0013] A super-resolved hyperspectral image is generated based on the spectral feature information of the hyperspectral image and the spatial feature information of the hyperspectral image.

[0014] Optionally, inputting the multispectral image to be super-resolved into a spectral branching network to obtain spectral information includes:

[0015] Segmenting the multispectral image to be super-resolved according to spectral dimensions to obtain multiple input vectors;

[0016] For each of the input vectors, linear mapping is performed on the input vector to obtain an output vector;

[0017] Decomposing the output vector according to the spectral dimension to obtain N heads;

[0018] Based on the output vector and the attention formula, the self-attention of each head is calculated;

[0019] The self-attention of each head is connected and linearly projected according to the spectral dimension to obtain a feature map; the feature map contains spectral information.

[0020] Optionally, the process of training the spectral super-resolution model based on the sample multispectral image to obtain the trained spectral super-resolution model includes:

[0021] Input the sample multispectral image into the spectral super-resolution model to obtain the super-resolved sample hyperspectral image;

[0022] Calculating the error between the super-resolved sample hyperspectral image and the preset hyperspectral image to obtain a loss function;

[0023] Determining whether the loss function converges;

[0024] If the loss function converges, the spectral super-resolution model is determined to be the trained spectral super-resolution model;

[0025] If the loss function does not converge, the model parameters in the spectral super-resolution model are adjusted, and the process returns to the step of inputting the sample multispectral image into the spectral super-resolution model to obtain the super-resolved sample hyperspectral image.

[0026] Optionally, the process of performing cluster analysis on the sample multispectral image blocks and the sample hyperspectral image blocks to obtain the sample multispectral image includes:

[0027] Normalizing the sample multispectral image blocks and the sample hyperspectral image blocks to obtain a plurality of normalized image blocks;

[0028] Select K normalized image blocks from all normalized image blocks and mark them as the center values ​​of K clusters;

[0029] For each normalized image block, calculating the Euclidean distance between the normalized image block and the center value of each cluster;

[0030] Assigning the normalized image blocks to the cluster corresponding to the minimum Euclidean distance;

[0031] For each of the clusters, calculating the average value of all points in the cluster, and determining the average value as a new center value;

[0032] Determine whether the new center value is consistent with the center value of the cluster;

[0033] If the new center value is inconsistent with the center value of the cluster, the center value of the cluster is updated to the new center value, and the process returns to the step of calculating the Euclidean distance between each normalized image block and the center value of each cluster.

[0034] If the new center value is consistent with the center value of the cluster, the weight of the cluster is calculated according to the number of normalized image blocks in the cluster and the total amount of data in the cluster;

[0035] The sample multispectral image is generated based on the normalized image blocks and the weights of the clusters.

[0036] A hyperspectral image generating device, comprising:

[0037] An acquisition unit, used for acquiring a multispectral image to be super-resolved;

[0038] The super-resolution unit is used to input the multispectral image to be super-resolved into a trained spectral super-resolution model to obtain a super-resolved hyperspectral image; wherein the trained spectral super-resolution model is obtained by training the spectral super-resolution model based on multiple sample multispectral images; the sample multispectral images are obtained by clustering analysis of sample multispectral image blocks and sample hyperspectral image blocks; the sample multispectral image blocks and the sample hyperspectral image blocks are in the same spatial area; the trained spectral super-resolution model includes a spectral branch network and a spatial branch network.

[0039] Optionally, the super-resolution unit is specifically configured to:

[0040] Inputting the multispectral image to be super-resolved into a spectral branch network to obtain spectral feature information of the hyperspectral image;

[0041] Inputting the multispectral image to be super-resolved into a spatial branch network to obtain spatial feature information of the hyperspectral image;

[0042] A super-resolved hyperspectral image is generated based on the spectral feature information of the hyperspectral image and the spatial feature information of the hyperspectral image.

[0043] Optionally, the super-resolution unit is specifically configured to:

[0044] Segmenting the multispectral image to be super-resolved according to spectral dimensions to obtain multiple input vectors;

[0045] For each of the input vectors, linear mapping is performed on the input vector to obtain an output vector;

[0046] Decomposing the output vector according to the spectral dimension to obtain N heads;

[0047] Based on the output vector and the attention formula, the self-attention of each head is calculated;

[0048] The self-attention of each head is connected and linearly projected according to the spectral dimension to obtain a feature map; the feature map contains spectral information.

[0049] Optionally, the super-resolution unit is specifically configured to:

[0050] Input the sample multispectral image into the spectral super-resolution model to obtain the super-resolved sample hyperspectral image;

[0051] Calculating the error between the super-resolved sample hyperspectral image and the preset hyperspectral image to obtain a loss function;

[0052] Determining whether the loss function converges;

[0053] If the loss function converges, the spectral super-resolution model is determined to be the trained spectral super-resolution model;

[0054] If the loss function does not converge, the model parameters in the spectral super-resolution model are adjusted, and the process returns to the step of inputting the sample multispectral image into the spectral super-resolution model to obtain the super-resolved sample hyperspectral image.

[0055] Optionally, the super-resolution unit is specifically configured to:

[0056] Normalizing the sample multispectral image blocks and the sample hyperspectral image blocks to obtain a plurality of normalized image blocks;

[0057] Select K normalized image blocks from all normalized image blocks and mark them as the center values ​​of K clusters;

[0058] For each normalized image block, calculating the Euclidean distance between the normalized image block and the center value of each cluster;

[0059] Assigning the normalized image blocks to the cluster corresponding to the minimum Euclidean distance;

[0060] For each of the clusters, calculating the average value of all points in the cluster, and determining the average value as a new center value;

[0061] Determine whether the new center value is consistent with the center value of the cluster;

[0062] If the new center value is inconsistent with the center value of the cluster, the center value of the cluster is updated to the new center value, and the process returns to the step of calculating the Euclidean distance between each normalized image block and the center value of each cluster.

[0063] If the new center value is consistent with the center value of the cluster, the weight of the cluster is calculated according to the number of normalized image blocks in the cluster and the total amount of data in the cluster;

[0064] The sample multispectral image is generated based on the normalized image blocks and the weights of the clusters.

[0065] The technical solution provided by the present application comprises the following steps: obtaining a multispectral image to be super-resolved; inputting the multispectral image to be super-resolved into a trained spectral super-resolution model to obtain a super-resolved hyperspectral image; training the trained spectral super-resolution model based on multiple sample multispectral images; clustering the sample multispectral image blocks and the sample hyperspectral image blocks; and the sample multispectral image blocks and the sample hyperspectral image blocks are located in the same spatial region. The spectral super-resolution model is trained after dynamically assigning different weights to different land object types in the image blocks. This avoids the problem of low super-resolution accuracy of land object types with a small proportion being suppressed by land object types with a large proportion during training due to insufficient prior knowledge provided, thereby improving the super-resolution accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0067] Figure 1 A flowchart of a method for generating a hyperspectral image provided in an embodiment of the present application;

[0068] Figure 2 A flowchart of a method for generating a hyperspectral image provided in an embodiment of the present application;

[0069] Figure 3 A flowchart of a method for determining spectral information provided in an embodiment of the present application;

[0070] Figure 4 A structural diagram of a spectrum branch provided in an embodiment of the present application;

[0071] Figure 5 A structural diagram of a spatial branch network provided in an embodiment of the present application;

[0072] Figure 6 A flowchart of a method for training a spectral super-resolution model provided in an embodiment of the present application;

[0073] Figure 7 A flowchart of an image clustering analysis method provided in an embodiment of the present application;

[0074] Figure 8 A schematic diagram of the architecture of a network model provided in an embodiment of the present application;

[0075] Figure 9 A schematic diagram of the architecture of a hyperspectral image generation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0077] In this application, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0078] like Figure 1FIG. 1 is a flow chart of a method for generating a hyperspectral image provided in an embodiment of the present application, comprising the following steps:

[0079] S101: Acquire a multispectral image to be super-resolved.

[0080] Multispectral images (e.g., super-resolved multispectral images) are images acquired across multiple spectral bands. Unlike color images, which typically have only three bands: red, green, and blue, multispectral images typically contain many more spectral bands.

[0081] S102: Input the multispectral image to be super-resolved into the trained spectral super-resolution model to obtain a super-resolved hyperspectral image.

[0082] Among them, the trained spectral super-resolution model is obtained by training the spectral super-resolution model based on multiple sample multispectral images; the sample multispectral images are obtained by clustering analysis of sample multispectral image blocks and sample hyperspectral image blocks; the sample multispectral image blocks and sample hyperspectral image blocks are in the same spatial area; the trained spectral super-resolution model includes a spectral branch network and a spatial branch network.

[0083] It can be understood that the spectral super-resolution model mainly uses the U-Net framework network to perform hierarchical feature learning on images. Since both spatial information and spectral features are helpful in learning the mapping from multispectral images to hyperspectral images, the multispectral image to be super-resolved is input into the trained spectral super-resolu-tion model, that is, the spectral super-resolu-tion model is used to perform spectral super-resolu-tion on the multispectral image to be super-resolved, and the super-resolved hyperspectral image is obtained.

[0084] Optionally, in another embodiment of the present application, the specific implementation of step S102 is as follows: Figure 2 As shown, the following steps are included:

[0085] S201: Input the multispectral image to be super-resolved into the spectral branch network to obtain spectral feature information of the hyperspectral image.

[0086] The spectral feature information generally refers to the reflectivity or radiation intensity of the image in different spectral bands.

[0087] It can be understood that the spectral feature information of the multispectral image is explored through the spectral branch network so as to subsequently generate a hyperspectral image according to the spectral feature information of the hyperspectral image.

[0088] Optionally, in another embodiment of the present application, the specific implementation of step S201 is as follows: Figure 3 As shown, the following steps are included:

[0089] S301: Segment the super-resolved multispectral image according to spectral dimensions to obtain multiple input vectors.

[0090] As you can see, by segmenting a multispectral image according to its spectral dimension, we can generate multiple single-band images (i.e., multiple input vectors). These images can be fed into the spectral branch network for super-resolution processing. After the super-resolution images for each band are processed, they are fused to generate the final high-resolution multispectral image.

[0091] S302: For each input vector, perform linear mapping on the input vector to obtain an output vector.

[0092] The output vector includes query, keyword, and value.

[0093] It is understandable that the input image is first marked as , where n is fixed. In the attention mechanism, n represents the number of tokens. In channel attention, a linear mapping is performed on the input image to obtain an output vector as the query , keywords Sum : , , .in, , , are learnable parameters represented by matrices.

[0094] S303: Decompose the output vector according to the spectral dimension to obtain N heads.

[0095] Specifically, the query, keyword, and value are decomposed into N heads according to the spectral dimension, where N is the number of spectral channels, and Q=[Q1,...,Q N ],K=[K1,...,K N ],V=[V1,...,V N ], the dimension of each head is .

[0096] S304: Based on the output vector and the attention formula, the self-attention of each head is calculated.

[0097] The specific expression of the attention formula is shown in formula (1).

[0098] (1)

[0099] In formula (1), A j is the self-attention of the head, V j is the jth head, K j The transposed matrix of is a learnable weight parameter used to adapt Matrix self-attention calculation.

[0100] It can be understood that spectral channel attention regards each spectral feature map as a token and calculates its attention along the spectral dimension.

[0101] S305: Connect the self-attention of each head according to the spectral dimension and perform linear projection to obtain a feature map.

[0102] Among them, the feature map contains spectral information.

[0103] In step S305, the self-attention of each head is connected and linearly projected according to the spectral dimension to obtain the specific representation of the feature map, as shown in formula (2).

[0104] (2)

[0105] In formula (2), X out is the feature map, , W is a learnable parameter, .

[0106] It should be noted that the spectral channel attention also considers each spectral feature as a tag. When calculating the self-attention A j When , the spectral channel attention will be global representation and A j The function is regarded as a global spatial position, so the receptive field of spectral channel attention is global and not limited to a window at a specific position.

[0107] In order to better Figure 3 For an explanation of the contents shown, see Figure 4 The structure of a spectral branch is shown. The multispectral image to be super-resolved is segmented along the spectral dimension for attention calculation. The attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, key, value, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is calculated using a compatibility function between the query and the corresponding key. The output is then layer-normalized and subjected to a series of convolutions to produce the final output of the spectral branch module, which represents the spectral information in the feature map.

[0108] S202: Input the multispectral image to be super-resolved into the spatial branch network to obtain spatial feature information of the hyperspectral image.

[0109] The structure diagram of the spatial branch network can be found in Figure 5The ResNet used in the spatial branching network consists of n basic modules connected in series. We denote H(x) as the underlying mapping for several stacked layers, where x represents the input to the first layer of these layers. Assuming that multiple nonlinear layers can asymptotically approximate complex functions is equivalent to assuming that they can asymptotically approximate the residual function, H(x)-x. Therefore, we explicitly approximate these layers to the residual function F(x)=H(x)-x.

[0110] It should be noted that before using the spatial branch network, due to the rich and complex scenes of remote sensing images, it is necessary to learn spatial information through the spatial branch network. Due to its ideal nonlinear modeling performance, ResNet has been proven to have the ability to implicitly learn priors. Through batch normalization and residual mapping, ResNet truly achieves good image restoration performance. At the same time, batch normalization will also increase computational memory overhead and destroy the independence of distribution between training examples within a batch. In order to replace batch normalization in ResNet, the embodiment of the present application uses ResNet without batch normalization in the spatial branch network to realize spatial feature learning of hyperspectral images.

[0111] Using residual learning for several stacked layers, formally, in this paper we define the building block as: , where x and y are the input vector and output vector of the considered layer, function Represents the residual mapping to be learned. For our network, there are two layers of 3×3 two-dimensional convolution between each residual connection, which can be expressed as ,in Denotes ReLU. To simplify notation, the bias is omitted. The formula F + x is implemented through a shortcut connection and element-wise addition. Similar n nonlinear mappings are then applied to the output of this module to complete the learning of spatial residual features.

[0112] The 3×3 two-dimensional convolution is computationally efficient and demonstrates excellent spatial feature extraction. The use of the ReLU activation function enhances the network's information transmission capabilities, preventing issues such as gradient explosion and vanishing that can disrupt information transmission. This improves the network's representational capabilities and prevents network degradation. Residual connections provide direct cross-layer connections, helping the model better learn features and further enhancing performance.

[0113] S203: Generate a super-resolved hyperspectral image based on the spectral feature information and the spatial feature information of the hyperspectral image.

[0114] Among them, the super-resolved hyperspectral image is based on spectral information and spatial information, and usually involves the use of deep learning and signal processing technology to improve the spatial resolution of the hyperspectral image while retaining its spectral information.

[0115] As you can understand, the goal of super-resolution is to recover a high-resolution image from a low-resolution hyperspectral image. This task typically involves two main pieces of information: spectral information and spatial information. Super-resolution restoration requires simultaneous utilization of both spectral and spatial information to generate a super-resolved hyperspectral image.

[0116] It should be noted that the hierarchical feature learning module of the network is to learn from the multi-spectral feature F M Obtaining hyperspectral features , where C h represents the number of channels in the hyperspectral feature. Because both spatial and spectral features contribute to learning the mapping M from multispectral data to hyperspectral data, the feature learning module is divided into a spectral branch and a spatial branch to simultaneously explore the spatial and spectral information of multispectral data. The spatial and spectral features are then aggregated into a spectral-spatial feature for use in subsequent operations.

[0117] Optionally, in another embodiment of the present application, a specific implementation method of obtaining a trained spectral super-resolution model by training the spectral super-resolution model based on a sample multispectral image is as follows: Figure 6 As shown, the following steps are included:

[0118] S601: Input the sample multispectral image into the spectral super-resolution model to obtain the super-resolved sample hyperspectral image.

[0119] Among them, the sample multispectral image indicates a remote sensing image.

[0120] S602: Calculate the error between the super-resolved sample hyperspectral image and the preset hyperspectral image to obtain a loss function.

[0121] Optionally, the mean square error between the super-resolved sample hyperspectral image and the preset hyperspectral image can be calculated, and the mean square error can be used as the loss function of the spectral super-resolution model.

[0122] S603: Determine whether the loss function converges.

[0123] If the loss function converges, step S604 is executed; if the loss function does not converge, step S605 is executed.

[0124] It can be understood that whether the loss function converges is judged. If the loss function converges, that is, the loss function gradually decreases with the increase of training steps, and eventually tends to be stable or no longer changes significantly, it means that the spectral super-resolution model training is completed at this time.

[0125] S604: Determine the spectral super-resolution model as the trained spectral super-resolution model.

[0126] S605: Adjust the model parameters in the spectral super-resolution model, and return to step S601.

[0127] It should be noted that if the loss function does not converge, it means that the loss function does not decrease with the increase of training steps. At this time, the model parameters in the spectral super-resolution model are adjusted, and the process returns to step S601 until the loss function converges.

[0128] Optionally, in another embodiment of the present application, a specific implementation method of performing cluster analysis on the sample multispectral image block and the sample hyperspectral image block to obtain the sample multispectral image is as follows: Figure 7 As shown, the following steps are included:

[0129] S701: performing normalization processing on the sample multispectral image blocks and the sample hyperspectral image blocks to obtain a plurality of normalized image blocks.

[0130] Understandably, hyperspectral and multispectral imagery often contains multiple spectral bands, each with a potentially different dynamic range or scale. If the values ​​of different bands vary significantly, some bands may have a greater impact on model training, while the contribution of other bands is suppressed. Normalization adjusts the values ​​of all spectral bands to the same scale or range (e.g., between 0 and 1, or with a mean of 0 and a variance of 1). This balances the impact of all spectral bands and prevents any particular band from unduly favoring the learning process.

[0131] S702: Select K normalized image blocks from all normalized image blocks and mark them as center values ​​of K clusters.

[0132] The value of K is a positive integer greater than 0.

[0133] S703: For each normalized image block, calculate the Euclidean distance between the normalized image block and the center value of each cluster.

[0134] A cluster refers to a group of image patches that are similar in spectral characteristics.

[0135] It can be understood that the specific form of calculating the Euclidean distance between the normalized image block and the cluster center in the cluster is as shown in formula (3).

[0136] (3)

[0137] In formula (3), d ij is the distance measurement value (i.e., Euclidean distance), c i is the cluster center of the first category, x j is the average spectral band of the j-th multispectral data block.

[0138] It should be noted that due to the normalization of the sample multispectral image blocks and the sample hyperspectral image blocks, the scale differences of different features or image blocks have been eliminated, so that the features of all input data (such as the pixel values ​​of the spectral bands) are in the same numerical range. Therefore, using the Euclidean metric as the distance metric does not affect the clustering results.

[0139] S704: Assign the normalized image blocks to the cluster corresponding to the minimum Euclidean distance.

[0140] It is understandable that the specific form of the minimum Euclidean distance is: , where K is the number of clusters and N is the total amount of data in the cluster.

[0141] S705: For each cluster, calculate the average value of all points in the cluster, and determine the average value as a new center value.

[0142] As you can understand, in each iteration, a data point is assigned to the nearest cluster center, which is then updated to the mean of all data points within that cluster. This process is repeated until the cluster center stops changing, indicating that the algorithm has converged and ultimately the optimal clustering result is obtained.

[0143] S706: Determine whether the new center value is consistent with the center value of the cluster.

[0144] If the new center value is inconsistent with the center value of the cluster, step S707 is executed; if the new center value is consistent with the center value of the cluster, step S708 is executed.

[0145] S707: Update the center value of the cluster to the new center value, and return to step S703.

[0146] S708: Calculate the weight of the cluster according to the number of normalized image blocks in the cluster and the total amount of data in the cluster.

[0147] According to the number of normalized image blocks in the cluster and the total amount of data in the cluster, the specific form of the cluster weight is calculated, as shown in formula (4).

[0148] (4)

[0149] In formula (4), a i is the weight of the cluster (i.e. the weight of the category), n i is the number of the i-th category (i.e., the number of normalized image blocks), and N is the total amount of data in the cluster.

[0150] S709: Generate a sample multispectral image based on the normalized image blocks and cluster weights.

[0151] It can be understood that weights are assigned to the objects in the normalized image blocks according to the cluster weights, thereby obtaining the weights of the objects, and a sample multispectral image is generated based on the weights of the objects and the normalized image blocks.

[0152] It should be noted that the surface cover classification product and the initial surface radiation product of the target area are obtained; among them, the initial surface radiation product is based on the reflectance product or emissivity product obtained by the MODIS sensor, and a large number of paired multispectral image blocks and hyperspectral image blocks are clustered using the k-means unsupervised clustering algorithm to cluster pixels with similar spectral correlations, and dynamically adjust the weights of different land object categories to balance the impact of uneven distribution of land objects.

[0153] from Figure 7 As shown, in large-scale, real-world scenarios, the distribution of different types of objects in a dataset is extremely uneven. A single remote sensing image can easily contain 90% bare land, 7% urban areas, and 3% water bodies. This significantly impacts the effectiveness of training for different objects. Therefore, before training, we propose using the k-means unsupervised clustering algorithm to perform cluster analysis on a large number of paired multispectral and hyperspectral image patches. This clustering allows for the clustering of pixels with similar spectral correlations. Once the distribution ratio of each object type is determined, each class is assigned a weight to mitigate the impact of uneven object distribution.

[0154] In order to better Figure 1 For an explanation of the contents shown, see Figure 8 The schematic diagram of the architecture of a network model shown is mainly composed of a cluster analysis module and a spectral super-resolution module. The purpose of the former is to learn the distribution ratio of various types of objects in the input image. When the distribution of various types of objects in the image is extremely uneven, this module is used to perform cluster analysis. Based on the clustering results, different weights are assigned to different objects, and the distribution of objects in the training set is adaptively adjusted. The spectral super-resolution module takes multispectral data as input and reconstructs its corresponding hyperspectral data. During the training phase, in order to explore the spatial information and spectral characteristics of local adjacent pixels, the multispectral data and hyperspectral data are first cropped into a large number of paired multispectral data blocks and hyperspectral data blocks. Then, paired multispectral data patches and hyperspectral data patches are used for training. During the testing phase, the test set is directly put into the spectral super-resolution module for testing to obtain the super-resolved hyperspectral data.

[0155] In the spectral super-resolution module, a 3×3 convolutional network is used for spectral channel mapping and feature filtering. Spectral channel mapping directly maps multispectral data with only a few spectral channels to hyperspectral data, but the hyperspectral data at this point lacks the expected spectral and spatial features. Feature filtering filters the features extracted by the hierarchical spectral-spatial feature learning module, retaining key spectral-spatial features and reducing redundancy. The hierarchical spectral-spatial feature learning module employs a U-net architecture, consisting primarily of an encoder, a bottleneck, and a decoder. The encoder generates hierarchical features of the image, while the decoder is symmetrical to the encoder. The bottleneck compresses and abstracts the high-level features extracted by the encoder and passes them to the decoder, achieving fine-grained segmentation of the input image. The downsampling operation is a 3×3 convolution to reduce the feature map and double the channels. The upsampling operation is a 3×3 deconv convolution layer to keep the feature map consistent with the upsampling. To avoid information loss during downsampling, skip connections are used between the encoder and decoder to concatenate the spectral channels. A 1×1 linear convolution layer is used to maintain the feature map dimensionality.

[0156] It should be noted that, based on the above process shown in S101-S102, this embodiment can achieve the following beneficial effects:

[0157] 1. Combining real satellite multispectral image data and hyperspectral image data, unified deep-level feature extraction is achieved through deep learning technology. This not only solves the problem that the spectral resolution of real remote sensing images is high and the spectral features are difficult to fully extract, but also solves the problem of mixed spectral pixels in the space of remote sensing images.

[0158] 2. By establishing an end-to-end mapping relationship between multispectral remote sensing images and hyperspectral remote sensing images, high-quality training data is provided for the spectral super-resolution model, so that high-precision hyperspectral images can be obtained through the corresponding multispectral data in surface areas where hyperspectral images are lacking.

[0159] In summary, by dynamically assigning different weights to different ground object types in the image block and then training the spectral super-resolution model, we can avoid the problem of low super-resolution accuracy of ground object types with a small proportion being suppressed by ground object types with a large proportion during training due to insufficient prior knowledge provided, thereby improving the super-resolution accuracy.

[0160] like Figure 9 , which is a schematic diagram of the architecture of a hyperspectral image generation device provided in an embodiment of the present application, the hyperspectral image generation device includes: an acquisition unit 100 and a super-resolution unit 200.

[0161] The acquisition unit 100 is used to acquire the multispectral image to be super-resolved.

[0162] The super-resolution unit 200 is used to input the multispectral image to be super-resolved into a trained spectral super-resolution model to obtain a super-resolved hyperspectral image; wherein the trained spectral super-resolution model is obtained by training the spectral super-resolution model based on multiple sample multispectral images; the sample multispectral images are obtained by clustering analysis of sample multispectral image blocks and sample hyperspectral image blocks; the sample multispectral image blocks and the sample hyperspectral image blocks are in the same spatial region; the trained spectral super-resolution model includes a spectral branch network and a spatial branch network.

[0163] The super-resolution unit 200 is specifically used to: input the multispectral image to be super-resolved into the spectral branch network to obtain spectral feature information of the hyperspectral image; input the multispectral image to be super-resolved into the spatial branch network to obtain spatial feature information of the hyperspectral image; and generate a super-resolved hyperspectral image based on the spectral feature information and spatial feature information of the hyperspectral image.

[0164] The super-resolution unit 200 is specifically used to: segment the multispectral image to be super-resolved according to the spectral dimension to obtain multiple input vectors; for each input vector, linearly map the input vector to obtain an output vector; decompose the output vector according to the spectral dimension to obtain N heads; calculate the self-attention of each head based on the output vector and the attention formula; connect the self-attention of each head according to the spectral dimension and perform linear projection to obtain a feature map; the feature map contains spectral information.

[0165] The super-resolution unit 200 is specifically used to: input the sample multispectral image into the spectral super-resolution model to obtain the super-resolution sample hyperspectral image; calculate the error between the super-resolution sample hyperspectral image and the preset hyperspectral image to obtain the loss function; determine whether the loss function converges; if the loss function converges, determine the spectral super-resolution model as the trained spectral super-resolution model; if the loss function does not converge, adjust the model parameters in the spectral super-resolution model, and return to execute the step of inputting the sample multispectral image into the spectral super-resolution model to obtain the super-resolution sample hyperspectral image.

[0166] The super-resolution unit 200 is specifically used to: perform normalization processing on the sample multispectral image blocks and the sample hyperspectral image blocks to obtain multiple normalized image blocks; select K normalized image blocks from all the normalized image blocks and mark them as the center values ​​of K clusters; for each normalized image block, calculate the Euclidean distance between the normalized image block and the center value of each cluster; assign the normalized image blocks to the cluster corresponding to the minimum Euclidean distance; for each cluster, calculate the average value of all points in the cluster, and determine the average value is the new center value; whether the new center value is consistent with the center value of the cluster is determined; if the new center value is inconsistent with the center value of the cluster, the center value of the cluster is updated to the new center value, and the process returns to execute the step of calculating the Euclidean distance between the normalized image block and the center value of each cluster for each normalized image block; if the new center value is consistent with the center value of the cluster, the weight of the cluster is calculated according to the number of normalized image blocks in the cluster and the total amount of data in the cluster; based on the normalized image blocks and the weight of the cluster, a sample multispectral image is generated.

[0167] In summary, by dynamically assigning different weights to different ground object types in the image block and then training the spectral super-resolution model, we can avoid the problem of low super-resolution accuracy of ground object types with a small proportion being suppressed by ground object types with a large proportion during training due to insufficient prior knowledge provided, thereby improving the super-resolution accuracy.

[0168] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Ordinary technicians in this field can understand and implement it without expending creative work.

[0169] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0170] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a hyperspectral image, characterized in that: include: Acquire the multispectral image to be super-resolved; The multispectral image to be super-resolved is input into a trained spectral super-resolution model to obtain a super-resolved hyperspectral image; wherein the trained spectral super-resolution model is obtained by training the spectral super-resolution model based on a plurality of sample multispectral images; the sample multispectral images are obtained by clustering analysis of sample multispectral image blocks and sample hyperspectral image blocks; the sample multispectral image blocks and the sample hyperspectral image blocks are in the same spatial region; the trained spectral super-resolution model includes a spectral branch network and a spatial branch network; The step of inputting the multispectral image to be super-resolved into a trained spectral super-resolution model to obtain a super-resolved hyperspectral image includes: Inputting the multispectral image to be super-resolved into a spectral branch network to obtain spectral feature information of the hyperspectral image; Inputting the multispectral image to be super-resolved into a spatial branch network to obtain spatial feature information of the hyperspectral image; generating a super-resolved hyperspectral image based on the spectral feature information of the hyperspectral image and the spatial feature information of the hyperspectral image; The step of inputting the multispectral image to be super-resolved into a spectral branching network to obtain spectral information includes: Segmenting the multispectral image to be super-resolved according to spectral dimensions to obtain multiple input vectors; For each of the input vectors, linear mapping is performed on the input vector to obtain an output vector; Decomposing the output vector according to the spectral dimension to obtain N heads; Based on the output vector and the attention formula, the self-attention of each head is calculated; Connecting the self-attention of each head according to the spectral dimension and performing linear projection to obtain a feature map; the feature map contains spectral information; The process of performing cluster analysis on the sample multispectral image blocks and the sample hyperspectral image blocks to obtain the sample multispectral image includes: Normalizing the sample multispectral image blocks and the sample hyperspectral image blocks to obtain a plurality of normalized image blocks; Select K normalized image blocks from all normalized image blocks and mark them as the center values ​​of K clusters; For each normalized image block, calculating the Euclidean distance between the normalized image block and the center value of each cluster; Assigning the normalized image blocks to the cluster corresponding to the minimum Euclidean distance; For each of the clusters, calculating the average value of all points in the cluster, and determining the average value as a new center value; Determine whether the new center value is consistent with the center value of the cluster; If the new center value is inconsistent with the center value of the cluster, the center value of the cluster is updated to the new center value, and the process returns to the step of calculating the Euclidean distance between each normalized image block and the center value of each cluster. If the new center value is consistent with the center value of the cluster, the weight of the cluster is calculated according to the number of normalized image blocks in the cluster and the total amount of data in the cluster; The sample multispectral image is generated based on the normalized image blocks and the weights of the clusters.

2. The method according to claim 1, characterized in that The process of training the spectral super-resolution model based on the sample multispectral image to obtain the trained spectral super-resolution model includes: Input the sample multispectral image into the spectral super-resolution model to obtain the super-resolved sample hyperspectral image; Calculating the error between the super-resolved sample hyperspectral image and the preset hyperspectral image to obtain a loss function; Determining whether the loss function converges; If the loss function converges, the spectral super-resolution model is determined to be the trained spectral super-resolution model; If the loss function does not converge, the model parameters in the spectral super-resolution model are adjusted, and the process returns to the step of inputting the sample multispectral image into the spectral super-resolution model to obtain the super-resolved sample hyperspectral image.

3. A hyperspectral image generation device, characterized in that: include: An acquisition unit, used for acquiring a multispectral image to be super-resolved; A super-resolution unit is configured to input the multispectral image to be super-resolved into a trained spectral super-resolution model to obtain a super-resolved hyperspectral image; wherein the trained spectral super-resolution model is obtained by training the spectral super-resolution model based on a plurality of sample multispectral images; the sample multispectral images are obtained by clustering analysis of sample multispectral image blocks and sample hyperspectral image blocks; the sample multispectral image blocks and the sample hyperspectral image blocks are in the same spatial region; and the trained spectral super-resolution model includes a spectral branch network and a spatial branch network; The super-resolution unit is specifically used for: Inputting the multispectral image to be super-resolved into a spectral branch network to obtain spectral feature information of the hyperspectral image; Inputting the multispectral image to be super-resolved into a spatial branch network to obtain spatial feature information of the hyperspectral image; generating a super-resolved hyperspectral image based on the spectral feature information of the hyperspectral image and the spatial feature information of the hyperspectral image; The super-resolution unit is specifically used for: Segmenting the multispectral image to be super-resolved according to spectral dimensions to obtain multiple input vectors; For each of the input vectors, linear mapping is performed on the input vector to obtain an output vector; Decomposing the output vector according to the spectral dimension to obtain N heads; Based on the output vector and the attention formula, the self-attention of each head is calculated; Connecting the self-attention of each head according to the spectral dimension and performing linear projection to obtain a feature map; the feature map contains spectral information; The super-resolution unit is specifically used for: Normalizing the sample multispectral image blocks and the sample hyperspectral image blocks to obtain a plurality of normalized image blocks; Select K normalized image blocks from all normalized image blocks and mark them as the center values ​​of K clusters; For each normalized image block, calculating the Euclidean distance between the normalized image block and the center value of each cluster; Assigning the normalized image blocks to the cluster corresponding to the minimum Euclidean distance; For each of the clusters, calculating the average value of all points in the cluster, and determining the average value as a new center value; Determine whether the new center value is consistent with the center value of the cluster; If the new center value is inconsistent with the center value of the cluster, the center value of the cluster is updated to the new center value, and the process returns to the step of calculating the Euclidean distance between each normalized image block and the center value of each cluster. If the new center value is consistent with the center value of the cluster, the weight of the cluster is calculated according to the number of normalized image blocks in the cluster and the total amount of data in the cluster; The sample multispectral image is generated based on the normalized image blocks and the weights of the clusters.

4. The device according to claim 3, characterized in that The super-resolution unit is specifically used for: Input the sample multispectral image into the spectral super-resolution model to obtain the super-resolved sample hyperspectral image; Calculating the error between the super-resolved sample hyperspectral image and the preset hyperspectral image to obtain a loss function; Determining whether the loss function converges; If the loss function converges, the spectral super-resolution model is determined to be the trained spectral super-resolution model; If the loss function does not converge, the model parameters in the spectral super-resolution model are adjusted, and the process returns to the step of inputting the sample multispectral image into the spectral super-resolution model to obtain the super-resolved sample hyperspectral image.

Images