Hyperspectral image generation method and device

Abstract

The invention discloses a hyperspectral image generation method and device. The method comprises the following steps: acquiring a to-be-hyper-divided multispectral image; inputting a multispectral image to be super-divided into the trained spectrum super-division model to obtain a super-divided hyperspectral image; the trained spectrum super-division model is obtained by training the spectrum super-division model based on a plurality of sample multispectral images; the sample multispectral image is obtained by performing clustering analysis on a sample multispectral image block and a sample hyperspectral image block; the sample multispectral image block and the sample hyperspectral image block are located in the same space area. According to the method, different weights are dynamically given to different ground feature types in the image blocks, and then the spectral super-division model is trained, so that the problem that the ground feature types with small proportions are pressed by the ground feature types with too large proportions during training due to insufficient priori knowledge, and the super-division precision of the ground feature types with small proportions is low is solved, and the super-division precision is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for generating a hyperspectral image. Background Art

[0002] While obtaining the spatial location information of ground objects, hyperspectral images can also obtain a continuous and complete spectral curve for each pixel. The spectral curve reflects the physical and chemical intrinsic characteristics of the ground object, and is therefore widely used in many fields such as land resources, ecological environment, etc.

[0003] At present, hyperspectral images are generally generated through dictionary learning, spectral unmixing, linear regression and other methods, and physical models are established using prior knowledge to reconstruct hyperspectral images, or spectral super-resolution is performed in combination with deep learning methods to generate higher quality hyperspectral images. However, existing deep learning methods are mainly designed for natural images, which usually capture close-ups, simple targets, and low spectral resolution. In contrast, satellite remote sensing images are taken from high altitudes, contain complex ground targets, and have high spectral resolution, strong continuity and correlation. Therefore, directly applying spectral super-resolution methods for natural images cannot fully exploit the complex characteristics of remote sensing images, thus affecting the super-resolution accuracy.

[0004] Therefore, how to improve super-resolution accuracy has become an urgent problem to be solved in this field. Summary of the invention

[0005] The present application provides a method and device for generating a hyperspectral image, the purpose of which is to improve super-resolution accuracy.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] A method for generating a hyperspectral image, comprising:

[0008] Acquire a multispectral image to be super-resolved;

[0009] The multispectral image to be super-resolved is input into a trained spectral super-resolution model to obtain a super-resolved hyperspectral image; wherein the trained spectral super-resolution model is obtained by training the spectral super-resolution model based on a plurality of sample multispectral images; the sample multispectral images are obtained by clustering analysis of sample multispectral image blocks and sample hyperspectral image blocks; the sample multispectral image blocks and the sample hyperspectral image blocks are in the same spatial region; the trained spectral super-resolution model includes a spectral branch network and a spatial branch network.

[0010] Optionally, inputting the multispectral image to be super-resolved into a trained spectral super-resolution model to obtain a super-resolved hyperspectral image includes:

[0011] Inputting the multispectral image to be super-resolved into a spectral branch network to obtain spectral feature information of the hyperspectral image;

[0012] Inputting the multispectral image to be super-resolved into a spatial branch network to obtain spatial feature information of the hyperspectral image;

[0013] Based on the spectral feature information of the hyperspectral image and the spatial feature information of the hyperspectral image, a super-resolved hyperspectral image is generated.

[0014] Optionally, inputting the multispectral image to be super-resolved into a spectral branching network to obtain spectral information includes:

[0015] Segmenting the multispectral image to be super-resolved according to spectral dimensions to obtain multiple input vectors;

[0016] For each of the input vectors, linear mapping is performed on the input vector to obtain an output vector;

[0017] Decomposing the output vector according to the spectral dimension to obtain N heads;

[0018] Based on the output vector and the attention formula, the self-attention of each head is calculated;

[0019] The self-attention of each head is connected and linearly projected according to the spectral dimension to obtain a feature map; the feature map contains spectral information.

[0020] Optionally, the process of training the spectral super-resolution model based on the sample multispectral image to obtain the trained spectral super-resolution model includes:

[0021] Input the sample multispectral image into the spectral super-resolution model to obtain the super-resolution sample hyperspectral image;

[0022] Calculating the error between the super-resolved sample hyperspectral image and the preset hyperspectral image to obtain a loss function;

[0023] Determining whether the loss function converges;

[0024] If the loss function converges, the spectral super-resolution model is determined to be the trained spectral super-resolution model;

[0025] If the loss function does not converge, the model parameters in the spectral super-resolution model are adjusted, and the step of inputting the sample multispectral image into the spectral super-resolution model to obtain the super-resolved sample hyperspectral image is returned to the step.

[0026] Optionally, the process of performing cluster analysis on the sample multispectral image block and the sample hyperspectral image block to obtain the sample multispectral image includes:

[0027] Normalizing the sample multispectral image blocks and the sample hyperspectral image blocks to obtain a plurality of normalized image blocks;

[0028] Select K normalized image blocks from all normalized image blocks and mark them as the center values ​​of K clusters;

[0029] For each normalized image block, calculating the Euclidean distance between the normalized image block and the center value of each cluster;

[0030] Assigning the normalized image blocks to the cluster corresponding to the smallest Euclidean distance;

[0031] For each of the clusters, calculating the average value of all points in the cluster, and determining the average value as a new center value;

[0032] Determine whether the new center value is consistent with the center value of the cluster;

[0033] If the new center value is inconsistent with the center value of the cluster, the center value of the cluster is updated to the new center value, and the step of calculating the Euclidean distance between the normalized image block and the center value of each cluster is returned to be executed for each normalized image block;

[0034] If the new center value is consistent with the center value of the cluster, the weight of the cluster is calculated according to the number of normalized image blocks in the cluster and the total amount of data in the cluster;

[0035] The sample multispectral image is generated based on the normalized image blocks and the weights of the clusters.

[0036] A hyperspectral image generating device, comprising:

[0037] An acquisition unit, used for acquiring a multispectral image to be super-resolved;

[0038] The super-resolution unit is used for inputting the multi-spectral image to be super-resolution into a trained spectral super-resolution model to obtain a super-resolution hyper-spectral image; wherein the trained spectral super-resolution model is obtained by training the spectral super-resolution model based on a plurality of sample multi-spectral images; the sample multi-spectral images are obtained by clustering analysis of sample multi-spectral image blocks and sample hyper-spectral image blocks; the sample multi-spectral image blocks and the sample hyper-spectral image blocks are in the same spatial region; the trained spectral super-resolution model includes a spectral branch network and a spatial branch network.

[0039] Optionally, the super-resolution unit is specifically used for:

[0040] Inputting the multispectral image to be super-resolved into a spectral branch network to obtain spectral feature information of the hyperspectral image;

[0041] Inputting the multispectral image to be super-resolved into a spatial branch network to obtain spatial feature information of the hyperspectral image;

[0042] Based on the spectral feature information of the hyperspectral image and the spatial feature information of the hyperspectral image, a super-resolved hyperspectral image is generated.

[0043] Optionally, the super-resolution unit is specifically used for:

[0044] Segmenting the multispectral image to be super-resolved according to spectral dimensions to obtain multiple input vectors;

[0045] For each of the input vectors, linear mapping is performed on the input vector to obtain an output vector;

[0046] Decomposing the output vector according to the spectral dimension to obtain N heads;

[0047] Based on the output vector and the attention formula, the self-attention of each head is calculated;

[0048] The self-attention of each head is connected and linearly projected according to the spectral dimension to obtain a feature map; the feature map contains spectral information.

[0049] Optionally, the super-resolution unit is specifically used for:

[0050] Input the sample multispectral image into the spectral super-resolution model to obtain the super-resolution sample hyperspectral image;

[0051] Calculating the error between the super-resolved sample hyperspectral image and the preset hyperspectral image to obtain a loss function;

[0052] Determining whether the loss function converges;

[0053] If the loss function converges, the spectral super-resolution model is determined to be the trained spectral super-resolution model;

[0054] If the loss function does not converge, the model parameters in the spectral super-resolution model are adjusted, and the step of inputting the sample multispectral image into the spectral super-resolution model to obtain the super-resolved sample hyperspectral image is returned to the step.

[0055] Optionally, the super-resolution unit is specifically used for:

[0056] Normalizing the sample multispectral image blocks and the sample hyperspectral image blocks to obtain a plurality of normalized image blocks;

[0057] Select K normalized image blocks from all normalized image blocks and mark them as the center values ​​of K clusters;

[0058] For each normalized image block, calculating the Euclidean distance between the normalized image block and the center value of each cluster;

[0059] Assigning the normalized image blocks to the cluster corresponding to the smallest Euclidean distance;

[0060] For each of the clusters, calculating the average value of all points in the cluster, and determining the average value as a new center value;

[0061] Determine whether the new center value is consistent with the center value of the cluster;

[0062] If the new center value is inconsistent with the center value of the cluster, the center value of the cluster is updated to the new center value, and the step of calculating the Euclidean distance between the normalized image block and the center value of each cluster is returned to be executed for each normalized image block;

[0063] If the new center value is consistent with the center value of the cluster, the weight of the cluster is calculated according to the number of normalized image blocks in the cluster and the total amount of data in the cluster;

[0064] The sample multispectral image is generated based on the normalized image blocks and the weights of the clusters.

[0065] The technical solution provided by the present application is to obtain a multispectral image to be super-resolved; input the multispectral image to be super-resolved into a trained spectral super-resolution model to obtain a super-resolved hyperspectral image; the trained spectral super-resolution model is obtained by training the spectral super-resolution model based on multiple sample multispectral images; the sample multispectral image is obtained by clustering the sample multispectral image block and the sample hyperspectral image block; the sample multispectral image block and the sample hyperspectral image block are in the same spatial region. After dynamically assigning different weights to different types of land objects in the image block, the spectral super-resolution model is trained, which avoids the problem of low super-resolution accuracy of land object types with a small proportion being suppressed by land object types with a large proportion during training due to insufficient prior knowledge provided, thereby improving the super-resolution accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 A flowchart of a method for generating a hyperspectral image provided in an embodiment of the present application;

[0068] Figure 2 A flowchart of a method for generating a hyperspectral image provided in an embodiment of the present application;

[0069] Figure 3 A flowchart of a method for determining spectral information provided in an embodiment of the present application;

[0070] Figure 4 A structural diagram of a spectrum branch provided in an embodiment of the present application;

[0071] Figure 5 A structural diagram of a spatial branch network provided in an embodiment of the present application;

[0072] Figure 6 A flowchart of a method for training a spectral super-resolution model provided in an embodiment of the present application;

[0073] Figure 7 A flowchart of an image clustering analysis method provided in an embodiment of the present application;

[0074] Figure 8 A schematic diagram of the architecture of a network model provided in an embodiment of the present application;

[0075] Fig. 9 A schematic diagram of the architecture of a hyperspectral image generating device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0077] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0078] like Figure 1FIG. 1 is a flowchart of a method for generating a hyperspectral image provided in an embodiment of the present application, comprising the following steps:

[0079] S101: Acquire a multispectral image to be super-resolved.

[0080] Among them, multispectral images (such as multispectral images to be super-resolved) refer to data images collected in multiple different spectral bands. Different from ordinary color images, which usually have only three bands of red, green, and blue, multispectral images usually contain more spectral bands.

[0081] S102: Input the multispectral image to be super-resolved into the trained spectral super-resolution model to obtain a super-resolved hyperspectral image.

[0082] Among them, the trained spectral super-resolution model is obtained by training the spectral super-resolution model based on multiple sample multispectral images; the sample multispectral images are obtained by clustering analysis of sample multispectral image blocks and sample hyperspectral image blocks; the sample multispectral image blocks and the sample hyperspectral image blocks are in the same spatial area; the trained spectral super-resolution model includes a spectral branch network and a spatial branch network.

[0083] It can be understood that the spectral super-resolution model mainly uses the U-Net framework network to perform hierarchical feature learning on images. Since both spatial information and spectral features are helpful to learn the mapping from multispectral images to hyperspectral images, the multispectral image to be super-resolved is input into the trained spectral super-resolution model, that is, the spectral super-resolution model is used to perform spectral super-resolution on the multispectral image to be super-resolved, and the super-resolved hyperspectral image is obtained.

[0084] Optionally, in another embodiment of the present application, the specific implementation of step S102 is as follows: Figure 2 As shown, the following steps are included:

[0085] S201: Input the multispectral image to be super-resolved into the spectral branch network to obtain spectral feature information of the hyperspectral image.

[0086] The spectral feature information generally refers to the reflectivity or radiation intensity of the image in different spectral bands.

[0087] It can be understood that the spectral feature information of the multispectral image is explored through the spectral branch network so as to subsequently generate a hyperspectral image according to the spectral feature information of the hyperspectral image.

[0088] Optionally, in another embodiment of the present application, the specific implementation of step S201 is as follows: Figure 3 As shown, the following steps are included:

[0089] S301: Segment the multispectral image to be super-resolved according to the spectral dimension to obtain multiple input vectors.

[0090] It can be understood that by segmenting the multispectral image according to the spectral dimension, multiple single-band images (i.e., multiple input vectors) can be obtained, which can be sent as input to the spectral branch network for super-resolution processing. After the super-resolution images of each band are processed, they are fused to generate the final high-resolution multispectral image.

[0091] S302: For each input vector, perform linear mapping on the input vector to obtain an output vector.

[0092] The output vector includes query, keyword, and value.

[0093] It is understandable that the input image is first marked as , where n is fixed. In the attention mechanism, n represents the number of tokens. In channel attention, the input image is linearly mapped to obtain an output vector as the query , keywords Sum : , , .in, , , are learnable parameters represented by matrices.

[0094] S303: Decompose the output vector according to the spectral dimension to obtain N heads.

[0095] Specifically, we decompose the query, keyword, and value into N heads according to the spectral dimension, where N is the number of spectral channels, and obtain Q=[Q1,...,Q N ],K=[K1,...,K N ],V=[V1,...,V N ], the dimension of each head is .

[0096] S304: Based on the output vector and the attention formula, the self-attention of each head is calculated.

[0097] The specific expression of the attention formula is shown in formula (1).

[0098] (1)

[0099] In formula (1), A j is the self-attention of the head, V j is the jth head, K j The transposed matrix of is a learnable weight parameter used to adapt Matrix self-attention calculation.

[0100] It can be understood that spectral channel attention regards each spectral feature map as a token and calculates its attention along the spectral dimension.

[0101] S305: Connect the self-attention of each head according to the spectral dimension and perform linear projection to obtain a feature map.

[0102] Among them, the feature map contains spectral information.

[0103] In step S305, the self-attention of each head is connected and linearly projected according to the spectral dimension to obtain the specific representation of the feature map, as shown in formula (2).

[0104] (2)

[0105] In formula (2), X out is the feature map, , W is a learnable parameter, .

[0106] It should be noted that the spectral channel attention also considers each spectral feature as a tag. j When , the spectral channel attention will be global representation and A j The function is considered as a global spatial position, so the receptive field of spectral channel attention is global and not limited to a window at a specific position.

[0107] In order to better Figure 3 For an explanation of the contents shown, see Figure 4 The structure diagram of a spectral branch is shown. The multispectral image to be super-resolved is segmented in the spectral dimension for attention calculation. The attention function can be described as mapping a query and a set of key-value pairs to the output, where the query, key, value, and output are all vectors. The output is calculated as a weighted sum of values, where the weight assigned to each value is calculated by the compatibility function of the query and the corresponding key. The output result is then layer-normalized and a series of convolution calculations are performed to obtain the output of the final spectral branch module, that is, the spectral information in the feature map.

[0108] S202: Input the multispectral image to be super-resolved into the spatial branch network to obtain spatial feature information of the hyperspectral image.

[0109] The structure diagram of the spatial branch network can be found in Figure 5, the ResNet used in the spatial branch network consists of n basic modules connected in series. We use H(x) as the underlying mapping of several stacked layers, and x represents the input of the first layer of these layers. If we assume that multiple nonlinear layers can asymptotically approximate complex functions, it is equivalent to assuming that they can asymptotically approximate the residual function, that is, H(x)-x. Therefore, we explicitly let these layers approximate the residual function F(x)=H(x)-x.

[0110] It should be noted that before using the spatial branch network, due to the rich and complex scenes of remote sensing images, it is necessary to learn spatial information through the spatial branch network. Due to its ideal nonlinear modeling performance, ResNet has been proven to have the ability to implicitly learn priors. Through batch normalization and residual mapping, ResNet truly achieves good image restoration performance. At the same time, batch normalization will also increase computational memory overhead and destroy the independence of distribution between training examples within a batch. In order to replace batch normalization in ResNet, the embodiment of the present application uses ResNet without batch normalization in the spatial branch network to realize spatial feature learning of hyperspectral images.

[0111] Using residual learning for several stacked layers, formally, in this paper our building block is defined as: , where x and y are the input vector and output vector of the considered layer, and the function Represents the residual mapping to be learned. For our network, there are two layers of 3×3 two-dimensional convolutions between each residual connection, which can be expressed as ,in ReLU represents ReLU. To simplify the notation, the bias is omitted. The formula F+x is executed through a shortcut connection and element addition. Then, similar n nonlinear mappings are used on the output of this module to complete the learning of spatial residual features.

[0112] Among them, the 3×3 two-dimensional convolution has high computational efficiency and shows excellent ability in spatial feature extraction. The use of the Relu activation function enhances the network's information transmission capability and prevents problems such as gradient explosion and gradient disappearance that lead to information transmission interruption. This improves the network's representation ability and prevents network degradation. The residual connection provides the effect of direct connection across layers, so it can help the model learn features better and further improve model performance.

[0113] S203: Generate a super-resolved hyperspectral image based on the spectral feature information of the hyperspectral image and the spatial feature information of the hyperspectral image.

[0114] Among them, the super-resolved hyperspectral image is based on spectral information and spatial information, and usually involves the use of deep learning and signal processing techniques to improve the spatial resolution of the hyperspectral image while retaining its spectral information.

[0115] It is understandable that the goal of super-resolution is to restore a high-resolution image from a low-resolution hyperspectral image. This task usually includes two main pieces of information: spectral information and spatial information. To use these two types of information simultaneously for super-resolution restoration, that is, to generate a super-resolved hyperspectral image based on spectral information and spatial information.

[0116] It should be noted that the hierarchical feature learning module of the network is to learn from the multi-spectral feature F M Obtaining hyperspectral features , where C h Represents the number of channels in the hyperspectral feature. Since both spatial information and spectral features contribute to learning the mapping M from multispectral data to hyperspectral data, the feature learning module is divided into a spectral branch and a spatial branch to simultaneously explore the spatial information and spectral information of multispectral data. The spatial features and spectral features are then aggregated into spectral-spatial features for use in subsequent operations.

[0117] Optionally, in another embodiment of the present application, a specific implementation method of obtaining a trained spectral super-resolution model by training the spectral super-resolution model based on a sample multispectral image is as follows: Figure 6 As shown, the following steps are included:

[0118] S601: Input the sample multispectral image into the spectral super-resolution model to obtain the super-resolved sample hyperspectral image.

[0119] Among them, the sample multispectral image indicates a remote sensing image.

[0120] S602: Calculate the error between the super-resolved sample hyperspectral image and the preset hyperspectral image to obtain a loss function.

[0121] Optionally, the mean square error between the super-resolved sample hyperspectral image and the preset hyperspectral image can be calculated, and the mean square error can be used as the loss function of the spectral super-resolution model.

[0122] S603: Determine whether the loss function converges.

[0123] If the loss function converges, step S604 is executed; if the loss function does not converge, step S605 is executed.

[0124] It can be understood that whether the loss function converges is judged. If the loss function converges, that is, the loss function gradually decreases with the increase of training steps, and finally tends to be stable or no longer changes significantly, it means that the spectral super-resolution model training is completed at this time.

[0125] S604: Determine the spectral super-resolution model as the trained spectral super-resolution model.

[0126] S605: Adjust the model parameters in the spectral super-resolution model, and return to execute step S601.

[0127] It should be noted that if the loss function does not converge, it means that the loss function does not decrease with the increase of training steps. At this time, the model parameters in the spectral super-resolution model are adjusted, and the process returns to step S601 until the loss function converges.

[0128] Optionally, in another embodiment of the present application, a specific implementation method of performing cluster analysis on the sample multispectral image block and the sample hyperspectral image block to obtain the sample multispectral image is as follows: Figure 7 As shown, the following steps are included:

[0129] S701: performing normalization processing on sample multispectral image blocks and sample hyperspectral image blocks to obtain a plurality of normalized image blocks.

[0130] It is understandable that hyperspectral and multispectral images usually contain multiple spectral bands, and the values ​​of each spectral band may have different dynamic ranges or scales. If the values ​​of different bands vary too much, some bands may have a greater impact on model training, while the contribution of other bands is suppressed. The normalization operation adjusts the values ​​of each spectral band to the same scale or range (such as between 0 and 1 or with a mean of 0 and a variance of 1), so that the influence of all spectral bands is more balanced and avoids unreasonable bias of some bands on the learning process.

[0131] S702: Select K normalized image blocks from all normalized image blocks and mark them as center values ​​of K clusters.

[0132] The value of K is a positive integer greater than 0.

[0133] S703: For each normalized image block, calculate the Euclidean distance between the normalized image block and the center value of each cluster.

[0134] A cluster refers to a group of image patches that are similar in spectral characteristics.

[0135] It can be understood that the specific form of calculating the Euclidean distance between the normalized image block and the cluster center in the cluster is as shown in formula (3).

[0136] (3)

[0137] In formula (3), d ij is the distance measurement (i.e., Euclidean distance), c i is the cluster center of the first category, x j is the average spectral band of the j-th multispectral data block.

[0138] It should be noted that, since the sample multispectral image blocks and sample hyperspectral image blocks are normalized, the scale differences of different features or image blocks have been eliminated, so that the features of all input data (such as the pixel values ​​of spectral bands) are in the same numerical range. Therefore, using the Euclidean metric as the distance metric does not affect the clustering results.

[0139] S704: Allocate the normalized image blocks to the cluster corresponding to the minimum Euclidean distance.

[0140] It is understandable that the specific form of the minimum Euclidean distance is: , where K is the number of clusters and N is the total amount of data in the clusters.

[0141] S705: For each cluster, calculate the average value of all points in the cluster, and determine the average value as a new center value.

[0142] It is understandable that in each iteration, the data point will be assigned to the nearest cluster center, and then the cluster center will be updated to the mean of all data points in the cluster. This process is repeated until the cluster center no longer changes, indicating that the algorithm converges and finally obtains the optimal clustering result.

[0143] S706: Determine whether the new center value is consistent with the center value of the cluster.

[0144] If the new center value is inconsistent with the center value of the cluster, step S707 is executed; if the new center value is consistent with the center value of the cluster, step S708 is executed.

[0145] S707: Update the center value of the cluster to a new center value, and return to execute step S703.

[0146] S708: Calculate the weight of the cluster according to the number of normalized image blocks in the cluster and the total amount of data in the cluster.

[0147] According to the number of normalized image blocks in the cluster and the total amount of data in the cluster, the specific table form of the cluster weight is calculated, as shown in formula (4).

[0148] (4)

[0149] In formula (4), a i is the weight of the cluster (i.e. the weight of the category), n i is the number of the i-th category (i.e., the number of normalized image blocks), and N is the total amount of data in the cluster.

[0150] S709: Generate a sample multispectral image based on the normalized image blocks and cluster weights.

[0151] It can be understood that weights are assigned to the objects in the normalized image blocks according to the weights of the clusters, thereby obtaining the weights of the objects, and based on the weights of the objects and the normalized image blocks, a sample multispectral image is generated.

[0152] It should be noted that the surface cover classification product and the initial surface radiation product of the target area are obtained; among which, the initial surface radiation product is based on the reflectance product or emissivity product obtained by the MODIS sensor, and a large number of paired multispectral image blocks and hyperspectral image blocks are clustered using the k-means unsupervised clustering algorithm to cluster pixels with similar spectral correlations, dynamically adjust the weights of different land object categories, and balance the impact of uneven distribution of land objects.

[0153] from Figure 7 It can be seen from the content shown that in large-scale real scenes, the distribution of different types of objects in the data set is extremely uneven. It is very likely that 90% of bare land, 7% of cities and 3% of water bodies will appear in a remote sensing image, which will greatly affect the training effect of different objects during the training process. Therefore, it is proposed to use the k-means unsupervised clustering algorithm to perform cluster analysis on a large number of paired multispectral image blocks and hyperspectral image blocks before training, so as to cluster pixels with similar spectral correlations. After obtaining the distribution ratio of each type of object, each type of object will be assigned a weight to balance the impact of uneven distribution of objects.

[0154] In order to better Figure 1 For an explanation of the contents shown, see Figure 8 The schematic diagram of the architecture of a network model shown in the figure mainly consists of a cluster analysis module and a spectral super-resolution module. The purpose of the former is to learn the distribution ratio of various types of objects in the input image. When the distribution of various types of objects in the image is extremely uneven, this module is used for cluster analysis, and different weights are assigned to different objects according to the clustering results, and the distribution of objects in the training set is adaptively adjusted. The spectral super-resolution module takes multispectral data as input and reconstructs its corresponding hyperspectral data. In the training stage, in order to explore the spatial information and spectral characteristics of local adjacent pixels, the multispectral data and hyperspectral data are first cropped into a large number of paired multispectral data blocks and hyperspectral data blocks. Then, paired multispectral data patches and hyperspectral data patches are used for training. In the testing stage, the test set is directly put into the spectral super-resolution module for testing to obtain the hyperspectral data after super-resolution.

[0155] In the spectral super-resolution module, a 3×3 convolutional network is used for spectral channel mapping and feature filtering. Spectral channel mapping directly maps multispectral data with only a few spectral channels to hyperspectral data, but the hyperspectral data at this time does not have the spectral and spatial features it should have. Feature filtering filters the features extracted by the hierarchical spectral spatial feature learning module, retains the main spectral spatial features, and reduces redundancy. The U-net structure is adopted in the hierarchical spectral spatial feature learning module, which is mainly composed of an encoder, a bottleneck, and a decoder. The encoder is to generate hierarchical features of the image, and the decoder is symmetrical with the encoder. The role of the bottleneck is to pass the high-level feature information extracted from the encoder to the decoder through compression and abstraction, thereby achieving fine segmentation of the input image. The downsampling operation is a conv3×3 convolution, which is used to reduce the feature map and double the channels. The upsampling operation is a deconv3×3 convolution layer to keep the feature map consistent with the upsampling. In order to avoid information loss in downsampling, jump connections are used between the encoder and the decoder to splice the spectral channels. And a 1×1 linear convolution layer is used to keep the feature map dimension consistent with the previous one.

[0156] It should be noted that, based on the above process shown in S101-S102, this embodiment can achieve the following beneficial effects:

[0157] 1. Combining real satellite multispectral image data and hyperspectral image data, unified deep-level feature extraction is achieved through deep learning technology, which not only solves the problem that the spectral resolution of real remote sensing images is high and the spectral features are difficult to fully extract, but also solves the problem that there are mixed spectral pixels in the pixels in the space of remote sensing images.

[0158] 2. By establishing an end-to-end mapping relationship between multispectral remote sensing images and hyperspectral remote sensing images, high-quality training data is provided for the spectral super-resolution model, so that high-precision hyperspectral images can be obtained through the corresponding multispectral data in surface areas where hyperspectral images are lacking.

[0159] In summary, by dynamically assigning different weights to different ground object types in the image block and then training the spectral super-resolution model, we can avoid the problem of low super-resolution accuracy of ground object types with a small proportion being suppressed by ground object types with a large proportion during training due to insufficient prior knowledge provided, thereby improving the super-resolution accuracy.

[0160] like Fig. 9 , which is a schematic diagram of the architecture of a hyperspectral image generation device provided in an embodiment of the present application, the hyperspectral image generation device includes: an acquisition unit 100 and a super-resolution unit 200.

[0161] The acquisition unit 100 is used to acquire the multispectral image to be super-resolved.

[0162] The super-resolution unit 200 is used to input the multispectral image to be super-resolution into the trained spectral super-resolution model to obtain a super-resolution hyperspectral image; wherein the trained spectral super-resolution model is obtained by training the spectral super-resolution model based on multiple sample multispectral images; the sample multispectral images are obtained by clustering the sample multispectral image blocks and the sample hyperspectral image blocks; the sample multispectral image blocks and the sample hyperspectral image blocks are in the same spatial area; the trained spectral super-resolution model includes a spectral branch network and a spatial branch network.

[0163] The super-resolution unit 200 is specifically used for: inputting the multispectral image to be super-resolution into the spectral branch network to obtain the spectral feature information of the hyperspectral image; inputting the multispectral image to be super-resolution into the spatial branch network to obtain the spatial feature information of the hyperspectral image; and generating a super-resolution hyperspectral image based on the spectral feature information of the hyperspectral image and the spatial feature information of the hyperspectral image.

[0164] The super-resolution unit 200 is specifically used for: segmenting the multi-spectral image to be super-resolved according to the spectral dimension to obtain multiple input vectors; for each input vector, linearly mapping the input vector to obtain an output vector; decomposing the output vector according to the spectral dimension to obtain N heads; based on the output vector and the attention formula, calculating the self-attention of each head; connecting the self-attention of each head according to the spectral dimension and performing linear projection to obtain a feature map; the feature map contains spectral information.

[0165] The super-resolution unit 200 is specifically used to: input the sample multispectral image into the spectral super-resolution model to obtain the super-resolution sample hyperspectral image; calculate the error between the super-resolution sample hyperspectral image and the preset hyperspectral image to obtain the loss function; determine whether the loss function converges; if the loss function converges, determine the spectral super-resolution model as the trained spectral super-resolution model; if the loss function does not converge, adjust the model parameters in the spectral super-resolution model, and return to execute the step of inputting the sample multispectral image into the spectral super-resolution model to obtain the super-resolution sample hyperspectral image.

[0166] The super-resolution unit 200 is specifically used for: normalizing the sample multispectral image blocks and the sample hyperspectral image blocks to obtain a plurality of normalized image blocks; selecting K normalized image blocks from all the normalized image blocks and marking them as the center values ​​of K clusters; for each normalized image block, calculating the Euclidean distance between the normalized image block and the center value of each cluster; assigning the normalized image blocks to the cluster corresponding to the minimum Euclidean distance; for each cluster, calculating the average value of all points in the cluster, and determining the average value is the new center value; whether the new center value is consistent with the center value of the cluster is determined; if the new center value is inconsistent with the center value of the cluster, the center value of the cluster is updated to the new center value, and the step of calculating the Euclidean distance between the normalized image block and the center value of each cluster for each normalized image block is returned to be executed; if the new center value is consistent with the center value of the cluster, the weight of the cluster is calculated according to the number of normalized image blocks in the cluster and the total amount of data in the cluster; based on the normalized image blocks and the weight of the cluster, a sample multispectral image is generated.

[0167] In summary, by dynamically assigning different weights to different ground object types in the image block and then training the spectral super-resolution model, we can avoid the problem of low super-resolution accuracy of ground object types with a small proportion being suppressed by ground object types with a large proportion during training due to insufficient prior knowledge provided, thereby improving the super-resolution accuracy.

[0168] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0169] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0170] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a hyperspectral image, characterized in that: include: Acquire a multispectral image to be super-resolved; The multispectral image to be super-resolved is input into a trained spectral super-resolution model to obtain a super-resolved hyperspectral image; wherein the trained spectral super-resolution model is obtained by training the spectral super-resolution model based on a plurality of sample multispectral images; the sample multispectral images are obtained by clustering analysis of sample multispectral image blocks and sample hyperspectral image blocks; the sample multispectral image blocks and the sample hyperspectral image blocks are in the same spatial region; the trained spectral super-resolution model includes a spectral branch network and a spatial branch network.

2. The method according to claim 1, characterized in that The step of inputting the multispectral image to be super-resolved into a trained spectral super-resolution model to obtain a super-resolved hyperspectral image comprises: Inputting the multispectral image to be super-resolved into a spectral branch network to obtain spectral feature information of the hyperspectral image; Inputting the multispectral image to be super-resolved into a spatial branch network to obtain spatial feature information of the hyperspectral image; Based on the spectral feature information of the hyperspectral image and the spatial feature information of the hyperspectral image, a super-resolved hyperspectral image is generated.

3. The method according to claim 1, characterized in that: The step of inputting the multispectral image to be super-resolved into a spectral branch network to obtain spectral information includes: Segmenting the multispectral image to be super-resolved according to spectral dimensions to obtain multiple input vectors; For each of the input vectors, linear mapping is performed on the input vector to obtain an output vector; Decomposing the output vector according to the spectral dimension to obtain N heads; Based on the output vector and the attention formula, the self-attention of each head is calculated; The self-attention of each head is connected and linearly projected according to the spectral dimension to obtain a feature map; the feature map contains spectral information.

4. The method according to claim 1, characterized in that: The process of training the spectral super-resolution model based on the sample multispectral image to obtain the trained spectral super-resolution model includes: Input the sample multispectral image into the spectral super-resolution model to obtain the super-resolution sample hyperspectral image; Calculating the error between the super-resolved sample hyperspectral image and the preset hyperspectral image to obtain a loss function; Determining whether the loss function converges; If the loss function converges, the spectral super-resolution model is determined to be the trained spectral super-resolution model; If the loss function does not converge, the model parameters in the spectral super-resolution model are adjusted, and the step of inputting the sample multispectral image into the spectral super-resolution model to obtain the super-resolved sample hyperspectral image is returned to the step.

5. The method according to claim 1, characterized in that The process of performing cluster analysis on the sample multispectral image block and the sample hyperspectral image block to obtain the sample multispectral image includes: Normalizing the sample multispectral image blocks and the sample hyperspectral image blocks to obtain a plurality of normalized image blocks; Select K normalized image blocks from all normalized image blocks and mark them as the center values ​​of K clusters; For each normalized image block, calculating the Euclidean distance between the normalized image block and the center value of each cluster; Assigning the normalized image blocks to the cluster corresponding to the smallest Euclidean distance; For each of the clusters, calculating the average value of all points in the cluster, and determining the average value as a new center value; Determine whether the new center value is consistent with the center value of the cluster; If the new center value is inconsistent with the center value of the cluster, the center value of the cluster is updated to the new center value, and the step of calculating the Euclidean distance between the normalized image block and the center value of each cluster is returned to be executed for each normalized image block; If the new center value is consistent with the center value of the cluster, the weight of the cluster is calculated according to the number of normalized image blocks in the cluster and the total amount of data in the cluster; The sample multispectral image is generated based on the normalized image blocks and the weights of the clusters.

6. A hyperspectral image generation device, characterized in that: include: An acquisition unit, used for acquiring a multispectral image to be super-resolved; The super-resolution unit is used for inputting the multi-spectral image to be super-resolution into a trained spectral super-resolution model to obtain a super-resolution hyper-spectral image; wherein the trained spectral super-resolution model is obtained by training the spectral super-resolution model based on a plurality of sample multi-spectral images; the sample multi-spectral images are obtained by clustering analysis of sample multi-spectral image blocks and sample hyper-spectral image blocks; the sample multi-spectral image blocks and the sample hyper-spectral image blocks are in the same spatial region; the trained spectral super-resolution model includes a spectral branch network and a spatial branch network.

7. The device according to claim 6, characterized in that The super-resolution unit is specifically used for: Inputting the multispectral image to be super-resolved into a spectral branch network to obtain spectral feature information of the hyperspectral image; Inputting the multispectral image to be super-resolved into a spatial branch network to obtain spatial feature information of the hyperspectral image; Based on the spectral feature information of the hyperspectral image and the spatial feature information of the hyperspectral image, a super-resolved hyperspectral image is generated.

8. The device according to claim 6, characterized in that The super-resolution unit is specifically used for: Segmenting the multispectral image to be super-resolved according to spectral dimensions to obtain multiple input vectors; For each of the input vectors, linear mapping is performed on the input vector to obtain an output vector; Decomposing the output vector according to the spectral dimension to obtain N heads; Based on the output vector and the attention formula, the self-attention of each head is calculated; The self-attention of each head is connected and linearly projected according to the spectral dimension to obtain a feature map; the feature map contains spectral information.

9. The device according to claim 6, characterized in that The super-resolution unit is specifically used for: Input the sample multispectral image into the spectral super-resolution model to obtain the super-resolution sample hyperspectral image; Calculating the error between the super-resolved sample hyperspectral image and the preset hyperspectral image to obtain a loss function; Determining whether the loss function converges; If the loss function converges, the spectral super-resolution model is determined to be the trained spectral super-resolution model; If the loss function does not converge, the model parameters in the spectral super-resolution model are adjusted, and the step of inputting the sample multispectral image into the spectral super-resolution model to obtain the super-resolved sample hyperspectral image is returned to the step.

10. The device according to claim 6, characterized in that The super-resolution unit is specifically used for: Normalizing the sample multispectral image blocks and the sample hyperspectral image blocks to obtain a plurality of normalized image blocks; Select K normalized image blocks from all normalized image blocks and mark them as the center values ​​of K clusters; For each normalized image block, calculating the Euclidean distance between the normalized image block and the center value of each cluster; Assigning the normalized image blocks to the cluster corresponding to the smallest Euclidean distance; For each of the clusters, calculating the average value of all points in the cluster, and determining the average value as a new center value; Determine whether the new center value is consistent with the center value of the cluster; If the new center value is inconsistent with the center value of the cluster, the center value of the cluster is updated to the new center value, and the step of calculating the Euclidean distance between the normalized image block and the center value of each cluster is returned to be executed for each normalized image block; If the new center value is consistent with the center value of the cluster, the weight of the cluster is calculated according to the number of normalized image blocks in the cluster and the total amount of data in the cluster; The sample multispectral image is generated based on the normalized image blocks and the weights of the clusters.

Images