Hyperspectral reconstruction method based on single RGB image

Abstract

The invention provides a hyperspectral reconstruction method based on a single RGB image, and the method is characterized in that a hyperspectral reconstruction model comprises a conical multi-scale feature extraction module which is used for carrying out the feature extraction of the feature mapping of the single RGB image after the slicing and partitioning processing, and obtaining the aggregation features of a shallow region; a plurality of multi-scale adaptive residual attention modules connected in sequence and used for processing the shallow region aggregation features to obtain deep features; the combination processing module is used for carrying out convolution and activation function processing on the deep features to obtain a hyperspectral image, and the multi-scale adaptive residual attention module comprises a conical multi-scale feature extraction module, an optimal non-local module, a pixel self-attention module, a LayerNorm module and a multi-layer perceptron module which are connected in sequence. In a word, the method can improve the precision of the reconstructed hyperspectral image.

Description

Technical Field

[0001] The invention relates to a hyperspectral reconstruction method, in particular to a hyperspectral reconstruction method based on a single RGB image. Background Art

[0002] Hyperspectral (HS) imaging technology records the reflectivity or transmittance of an object, combines imaging technology and spectral technology to detect the two-dimensional geometric space and one-dimensional spectral information of the target, and thus obtains a continuous narrow-band image with high spectral resolution. The hyperspectral image (HSI) obtained by this technology usually has multiple spectral bands from infrared spectrum to ultraviolet spectrum. Rich spectral features have been widely used in the field of remote sensing, such as geological exploration, marine environment monitoring, ship detection, military reconnaissance, ecological research and crop growth. However, the development of HS imaging has encountered bottlenecks in recent years, because the manufacturing of existing hyperspectral imaging equipment involves optics, machinery, computers and other fields, the integration complexity is high, and the equipment is large. At the same time, due to the limitations of imaging technology, it takes a long time to obtain such hyperspectral images with high spatiotemporal resolution containing rich spectral information, which inevitably limits the application scope of hyperspectral imaging technology. Especially in portable platforms and high-speed mobile scenes. One way to solve this problem is to generate such hyperspectral images by recovering the lost spectral information from a given RGB image. This method can be called hyperspectral reconstruction based on RGB images. It refers to the inverse process of hyperspectral imaging by finding an inverse response function. However, since the number of HSIs can be projected to any RGB input, there may be multiple reasonable HS image combinations for the same RGB image. How to reconstruct the best HS image has become the core and hot issue of research.

[0003] To solve this problem, a large number of spectral reconstruction (SR) methods have been proposed, which can be roughly divided into traditional methods, machine methods and deep learning methods.

[0004] Existing traditional methods include the use of interpolation technology, principal component analysis, pseudo-inverse method, Wiener method, etc., but these traditional methods have poor expression ability and limited generalization ability, which limits their good performance on images.

[0005] Existing machine learning-based spectral reconstruction methods mainly focus on establishing sparse coding or relatively shallow learning models from specific hyperspectral spectra. These methods still have problems such as being easily affected by external factors, requiring a large amount of training data, taking a long time to train, and the need to improve the quality of reconstructed images.

[0006] Existing deep learning methods have achieved good performance, but the accuracy of reconstructed hyperspectral images needs to be further improved. In addition, in order to obtain more advanced feature expressions, most methods are committed to designing deeper network structures, but also lack the exploration of information between local regions and long-range dependencies between pixels, which limits the network's ability to follow details and discriminate learning.

[0007] In conclusion, there is still room for further improvement in the quality of generated images using existing spectral reconstruction methods. Summary of the invention

[0008] The present invention is made to solve the above problems, and aims to provide a hyperspectral reconstruction method based on a single RGB image.

[0009] The invention provides a hyperspectral reconstruction method based on a single RGB image, which is used for inputting the single RGB image into a hyperspectral reconstruction model to obtain a corresponding hyperspectral image, and has the following characteristics: the hyperspectral reconstruction model comprises: a conical multi-scale feature extraction module, which is used for extracting features from a feature map of the single RGB image after being processed by slice partitioning, so as to obtain shallow region aggregation features; a plurality of multi-scale adaptive residual attention modules connected in sequence, which are used for processing the shallow region aggregation features to obtain deep features; a combination processing module, which is used for performing convolution and activation function processing on the deep features to obtain the hyperspectral image, wherein the multi-scale adaptive residual attention module comprises a conical multi-scale feature extraction module, an optimal non-local module, a pixel self-attention module, a LayerNorm module and a multi-layer perceptron module connected in sequence, and the conical multi-scale feature extraction module is used for taking the feature map of the single RGB image after being processed by slice partitioning or the input of the multi-scale adaptive residual attention module containing the conical multi-scale feature extraction module as an input feature F in , and input feature F in Perform preliminary extraction of multi-scale detail features and multi-layer splicing in sequence to obtain feature F Left , and then for the feature F Left The multi-scale detail feature is extracted alternately and multi-layered splicing is performed in sequence to obtain the shallow region aggregation feature or the input of the optimal non-local module connected to the cone multi-scale feature extraction module as the output feature F out , the optimal non-local module is used to transform the output feature F out The partition features are divided into four equal-sized regions, and the connection between the partition features is enhanced to obtain the feature sequence T i , the pixel self-attention module is used to i The long-range dependencies between pixels in are captured to obtain the fused representation T out , the LayerNorm module is used to eliminate the fusion representation T out and feature F LeftThe adverse effects caused by the singular sample data in the addition result are taken as the feature The multi-layer perceptron module is used to perform feature conversion and information reorganization on the output of the LayerNorm module to obtain feature In each multi-scale adaptive residual attention module, the input feature F in ,feature and Features The connection result is obtained by connecting as the input of the next multi-scale adaptive residual attention module connected in sequence with the multi-scale adaptive residual attention module, and the connection result of the last multi-scale adaptive residual attention module is the deep feature.

[0010] In the hyperspectral reconstruction method based on a single RGB image provided by the present invention, the following features may also be provided: wherein the cone multi-scale feature extraction module includes a Left submodule and a Right submodule connected in sequence, the Left submodule includes four layers of Left channel domains and corresponding four layers of convolutions of different scales, and the channels of the four layers of Left channel domains are all input features F of the cone multi-scale feature extraction module in One quarter of the channel, the Nth left The convolution kernel size corresponding to the Left channel domain of the layer is k1=2N left +1, input feature F in Input the Left submodule to get feature F Left The expression formula is: F Left =Relu(BN(F' out ), where ∑cat is a multi-layer concatenation operation, For N left The i-th group of feature maps in the Left channel domain of the layer, w k1,i is the i-th group of convolution kernels with k1 as the convolution kernel size, and the N-th left The number of channels in the Left channel domain of the layer is one-fourth of the original input feature F in As feature maps and corresponding convolution kernels are divided into m groups, BN is the batch normalization layer, Relu is the activation function, the Right submodule includes four layers of Right channel domain and corresponding four layers of convolutions of different scales. The channels of the four layers of Right channel domain are all input features F of the input cone multi-scale feature extraction module. in A quarter of the channel, the four-layer Right channel domain corresponds to the four-layer Left channel domain in sequence, the Nth right The convolution kernel size corresponding to the Right channel domain of the layer is k2=11-2N right , the feature F LeftInput the Right submodule to get the output feature F out The expression formula is: F out =Relu(BN(F”) out ), where ∑cat is a multi-layer concatenation operation, G i,j For N right The i-th group of feature maps in the layer channel domain, w k2,i is the i-th group of convolution kernels with k2 as the convolution kernel size, and the N-th right The number of channels in the layer channel domain is one-fourth of the original feature F Left The feature maps and the corresponding convolution kernels are divided into m groups, BN is the batch normalization layer, and Relu is the activation function.

[0011] The hyperspectral reconstruction method based on a single RGB image provided by the present invention may also have the following characteristics: wherein the output feature F out Input the optimal non-local module to obtain the feature sequence T i The expression formula is: T i =ONB(F out {lu,ld,ru,rd}), where ONB is the optimal non-local module, F out {lu,ld,ru,rd} is the feature F out Divide into four equal-sized areas: upper left, lower left, upper right, and lower right.

[0012] In the hyperspectral reconstruction method based on a single RGB image provided by the present invention, the following features may also be provided: wherein the pixel self-attention module performs a self-attention on the feature sequence T i Processing to obtain the fusion representation T out The specific steps are as follows: Step S1, for feature T i The convolution-activation function-convolution combination is used to obtain preliminary features, which are then divided into Q, K, and V of the attention mechanism; step S2, the weight of each pixel area is calculated based on Q and K; step S3, based on the feature sequence T i , weights and V are calculated to obtain the fusion representation T out .

[0013] In the hyperspectral reconstruction method based on a single RGB image provided by the present invention, the following features may also be provided: wherein, in step S1, the formulas of Q, K and V are as follows: Q = Conv(ρ(Conv(T i ))), K = Conv(ρ(Conv(T i ))), V = Conv(ρ(Conv(T i))), where ρ(·) is the activation function Prelu and Conv is the convolution operation.

[0014] The hyperspectral reconstruction method based on a single RGB image provided by the present invention may also have the following characteristics: wherein, in step S2, the weight formula is as follows: i =Corr(Q i ,Conv(K i )), μ i =softmax(U i ), where U i is the attention value of the i-th pixel area, Corr is the correlation matrix calculation, Q i is Q, K of the i-th pixel area i is the K of the i-th pixel area, softmax is the softmax function operation, μ i is the weight of the i-th pixel region.

[0015] The hyperspectral reconstruction method based on a single RGB image provided by the present invention may also have the following characteristics: wherein, in step S3, the fusion representation T out The formula is as follows: T o ={T o1 ,T o2 ,T o3 ,...T oi ,...T on}, T out =T i +δ(Conv(T o )), where T oi is the output of the i-th pixel area, is the transpose multiplication, V i is V of the i-th pixel area, T o is the set of outputs of all pixel regions, and δ is a learnable parameter.

[0016] Functions and Effects of the Invention

[0017] According to the hyperspectral reconstruction method based on a single RGB image involved in the present invention, on the one hand, the tapered multi-scale feature extraction module 10 interlacedly extracts detail features of different scales on four layers of different channels, thereby obtaining aggregated local information and relative global information containing edge space information, and finally achieving the purpose of extracting inter-regional contextual features at multiple scales; on the other hand, the pixel self-attention module 202 extracts the inter-pixel dependency of the local area while capturing the relationship between the pixel points in the area and all the pixel points in other farther areas, focusing on key information and reducing the attention paid to other irrelevant information, so that the relationship between different areas of the image can be better modeled. Therefore, the hyperspectral reconstruction method based on a single RGB image of the present invention can improve the accuracy of the reconstructed hyperspectral image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of a framework of a hyperspectral reconstruction model in an embodiment of the present invention;

[0019] Figure 2 Schematic diagram of the composition and working process of the cone multi-scale feature extraction module in an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of the composition and working process of a multi-scale adaptive residual attention module in an embodiment of the present invention;

[0021] Figure 4 Schematic diagram of the working principle of the pixel self-attention module in an embodiment of the present invention;

[0022] Figure 5 In the embodiment of the present invention, the fusion expression T out Schematic diagram of the process;

[0023] Figure 6 is a schematic diagram of typical example data ARAD_HS_0453 in an embodiment of the present invention;

[0024] Figure 7 is a schematic diagram comparing spectral response curves of various models at pixel point A in an embodiment of the present invention;

[0025] Figure 8 is a schematic diagram comparing spectral response curves of various models at pixel point B in an embodiment of the present invention;

[0026] Fig. 9 is a schematic diagram of typical example data ARAD_HS_0459 in an embodiment of the present invention;

[0027] Fig.10 is a schematic diagram comparing spectral response curves of various models at pixel point C in an embodiment of the present invention;

[0028] Fig.11 is a schematic diagram comparing spectral response curves of various models at pixel point D in an embodiment of the present invention;

[0029] Fig.12 is a schematic diagram of typical example data ARAD_HS_0456 in an embodiment of the present invention;

[0030] Fig.13 is a schematic diagram comparing spectral response curves of various models at pixel point E in an embodiment of the present invention;

[0031] Fig.14 is a schematic diagram comparing spectral response curves of various models at pixel point F in an embodiment of the present invention;

[0032] Fig.15 is a schematic diagram of typical example data ARAD_HS_0464 in an embodiment of the present invention;

[0033] Fig.16 is a schematic diagram comparing spectral response curves of various models at pixel point G in an embodiment of the present invention;

[0034] Fig.17 3 is a schematic diagram comparing spectral response curves of various models at pixel point H in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and accompanying drawings specifically illustrate the hyperspectral reconstruction method based on a single RGB image of the present invention.

[0036] In this embodiment, a single RGB image is input into a hyperspectral reconstruction model to obtain a corresponding hyperspectral image.

[0037] Figure 1 Schematic diagram of the framework of the hyperspectral reconstruction model in the embodiment of the present invention.

[0038] like Figure 1 As shown, the hyperspectral reconstruction model 100 includes a conical multi-scale feature extraction module 10, M sequentially connected multi-scale adaptive residual attention modules 20 and a combination processing module 30.

[0039] The cone multi-scale feature extraction module 10 is used to extract features from the feature map of a single RGB image after slice partitioning to obtain shallow region aggregation features. In this embodiment, the input feature F of the cone multi-scale feature extraction module 10 is in is the feature map, output feature F out Aggregation features for shallow areas.

[0040] Figure 2It is a schematic diagram of the composition and working process of the conical multi-scale feature extraction module in an embodiment of the present invention.

[0041] like Figure 2 As shown, the cone multi-scale feature extraction module 10 includes a Left submodule 101 and a Right submodule 102 connected in sequence. The cone multi-scale feature extraction module 10 extracts the input feature F in Processing to obtain output feature F out The specific working process is:

[0042] The Left submodule 101 includes four layers of Left channel domains and corresponding four layers of convolutions of different scales. left The convolution kernel size corresponding to the Left channel domain of the layer is k1=2N left +1, the input feature F of size H*W*C in Divide the channel into four equal parts as the input features of each Left channel domain, and take the Nth left The input features of the Left channel domain and the corresponding convolution kernel are divided into In each layer of the Left channel domain, the grouped input features of size H*W*(C / 4m) are processed using the corresponding convolution kernel of size k1*k1*(C / 4m), that is, the i-th group of convolution kernels only processes the i-th group of input features, thereby obtaining m convolution features, and then concatenating all the convolution features of the four layers of the Left channel domain to obtain a feature F' of size H*W*C out , and then for the feature F' out After batch normalization layer and activation function processing, the feature F with size H*W*C is obtained Left As the output of the Left submodule 101, the Left submodule 101 is based on the input feature F in Get feature F Left The expression formula is:

[0043]

[0044]

[0045] F Left =Relu(BN(F' out ),

[0046] Where ∑cat is a multi-layer concatenation operation, For N left The i-th group of feature maps in the Left channel domain of the layer, w k1,i is the i-th group of convolution kernels with k1 as the convolution kernel size, and the N-th lef The number of channels in the Left channel domain of layer t is one-fourth of the original input feature Fin The feature maps and the corresponding convolution kernels are divided into m groups, BN is the batch normalization layer, and Relu is the activation function.

[0047] The Right submodule 102 includes four layers of Right channel domains and corresponding four layers of convolutions of different scales. The four layers of Right channel domains correspond to the four layers of Left channel domains in sequence. right The convolution kernel size corresponding to the Right channel domain of the layer is k2=11-2N right , the input feature F Left Divide the channel into four equal parts as the input features of each Right channel domain, and take the Nth right The input features of the Left channel domain and the corresponding convolution kernel are divided into In each layer of the Right channel domain, the grouped input features of size H*W*(C / 4n) are processed using the corresponding convolution kernel of size k2*k2*(C / 4n), that is, the i-th group of convolution kernels only processes the i-th group of input features, thereby obtaining n convolution features, and then concatenating all the convolution features of the four layers of the Right channel domain to obtain a feature F of size H*W*C." out , and then for feature F" out After batch normalization layer and activation function processing, the output feature F with size H*W*C is obtained out As the output of the Right submodule 102, that is, the output of the cone multi-scale feature extraction module 10, the Right submodule 102 extracts the feature F according to the input Left Get the output feature F out The expression formula is:

[0048]

[0049]

[0050] F out =Relu(BN(F”) out ),

[0051] Where ∑cat is a multi-layer concatenation operation, G i,j For N right The i-th group of feature maps in the layer channel domain, w k2,i is the i-th group of convolution kernels with k2 as the convolution kernel size, and the N-th right The number of channels in the layer channel domain is one-fourth of the original feature F Left The feature maps and the corresponding convolution kernels are divided into m groups, BN is the batch normalization layer, and Relu is the activation function.

[0052] The input feature F of the cone multi-scale feature extraction module 10 in this embodiment is in And the output feature F out The size of the convolution kernels is the same as that of the previous one, because the sizes of the convolution kernels are inconsistent and are all odd numbers, so half padding is used: the padding size is set to [k / 2]k=k1,k2, and the compensation of the four layers of convolution is set to the same size.

[0053] In this embodiment, batch normalization layers and activation functions are used to ensure that the entire processing process of the cone multi-scale feature extraction module 10 adapts to nonlinear problems, accelerates convergence to obtain complex depth features, and improves the recognition performance of the network.

[0054] The conical multi-scale feature extraction module 10 of this embodiment, on the one hand, uses the four-layer channel domain with increasing convolution kernel sizes in the Left submodule 101 to extract input features F of different areas. in In the invention, the local details of different levels are captured, that is, the detail information of the local smaller area is extracted by a smaller convolution kernel size, and the detail information of the larger area including the context information is captured by a relatively larger convolution kernel size, and then the output features of the four channel domains are spliced, so as to achieve the purpose of merging local information and relative global information at the channel level; on the other hand, the Right submodule 102 is provided with four channel domains corresponding to the Left submodule 101 and the corresponding convolution kernels are successively reduced, so as to achieve the channel that uses a larger convolution kernel size in the Left submodule 101 to extract the information of the larger area, and the Right submodule 102 uses a smaller convolution kernel size to capture the detail information of the smaller area in the same channel, so as to achieve the interleaved extraction on the channel, and finally obtain the information complementarity at multiple scales. Through the Left submodule 101 and the Right submodule 102, the conical multi-scale feature extraction module 10 can obtain the aggregated local information and the relative global information including the edge space information, and finally achieve the purpose of extracting the inter-region context feature at multiple scales.

[0055] Figure 3 Schematic diagram of the composition and working process of the multi-scale adaptive residual attention module in an embodiment of the present invention.

[0056] like Figure 3 As shown, the multi-scale adaptive residual attention module 20 includes a cone multi-scale feature extraction module 10, an optimal non-local module 201, a pixel self-attention module 202, a LayerNorm module 203 and a multi-layer perceptron module 204 connected in sequence, and its working process is:

[0057] In the multi-scale adaptive residual attention module 20, the Left submodule 101 of the cone multi-scale feature extraction module 10 receives the input feature F inPerform preliminary extraction of multi-scale detail features and multi-layer splicing in sequence to obtain feature F Left , and then through the Right submodule 102 to the feature F Left The output feature F is obtained by sequentially performing multi-scale detail feature interleaving extraction and multi-layer splicing. out .

[0058] The optimal non-local module 201 will output the feature F out The partition features are divided into four equal-sized regions, and the connection between the partition features is enhanced to obtain the feature sequence T i .

[0059] Among them, the output feature F out Input the optimal non-local module 201 to obtain the feature sequence T i The expression formula is:

[0060] T i =ONB(F out {lu,ld,ru,rd}),

[0061] Where ONB is the optimal non-local module 201, F out {lu,ld,ru,rd} is the feature F out Divide into four equal-sized areas: upper left, lower left, upper right, and lower right.

[0062] The pixel self-attention module 202 performs the feature sequence T i The long-range dependencies between pixels in are captured to obtain the fused representation T out .

[0063] Figure 4 Schematic diagram of the working principle of the pixel self-attention module in an embodiment of the present invention.

[0064] like Figure 4 As shown, the pixel self-attention module 202 is input to the feature sequence T i Convolution, activation function and convolution processing are performed in sequence, and the processing results are divided into Q, K and V of the attention mechanism. Then K and V are convolved separately to decompose the feature information of each channel into information components on the convolution kernel. Each convolution kernel generates new channel information. At this time, the sequence is (K i ,V i) have different contributions to the key information and the amount of information they contain. Therefore, a weight coefficient is needed to indicate the importance of the relevance to the key information. Then, the correlation matrix is ​​calculated for the result after convolution of K and Q, and the result of the correlation matrix is ​​processed by softmax to obtain the attention value as the weight, that is, the weight coefficient of each channel and pixel area. The weight is transposed and multiplied by the result after convolution of V, and then after convolution, it is combined with the feature sequence T. i Add them together to get the fusion representation T out .

[0065] Figure 5 In the embodiment of the present invention, the fusion expression T out Schematic diagram of the process.

[0066] like Figure 5 As shown, the pixel self-attention module 202 performs an operation on the feature sequence T i Processing to obtain the fusion representation T out The specific steps are:

[0067] Step S1: for feature T i Use the combination of convolution-activation function-convolution to obtain preliminary features, and then divide the preliminary features into Q, K and V of the attention mechanism.

[0068] The formulas for Q, K, and V are as follows:

[0069] Q = Conv(ρ(Conv(T i ))),

[0070] K=Conv(ρ(Conv(T i ))),

[0071] V=Conv(ρ(Conv(T i ))),

[0072] Where ρ(·) is the activation function Prelu, and Conv is the convolution operation.

[0073] Step S2, calculating the weight of each pixel area according to Q and K.

[0074] The weight formula is as follows:

[0075] U i =Corr(Q i ,Conv(K i )),

[0076] μ i =softmax(U i ),

[0077] Where Ui is the attention value of the i-th pixel area, Corr is the correlation matrix calculation, Q i is Q, K of the i-th pixel area i is the K of the i-th pixel area, softmax is the softmax function operation, μ i is the weight of the i-th pixel region.

[0078] Step S3, according to the feature sequence T i , weights and V are calculated to obtain the fusion representation T out .

[0079] Among them, in step S3, the fusion representation T out The formula is as follows:

[0080]

[0081] T o ={T o1 ,T o2 ,T o3 ,...T oi ,...T on},

[0082] T out =T i +δ(Conv(T o )),

[0083] Where T oi is the output of the i-th pixel area, is the transpose multiplication, V i is V of the i-th pixel area, T o is the set of outputs of all pixel regions, δ is a learnable parameter. In this embodiment, the learnable parameter δ is initialized to 0 during the training process. The model network mainly relies on domain features in the initial stage of training, and then slowly increases the weight of relying on distant regions.

[0084] In this embodiment, the pixel self-attention module 202 captures the relationship between the pixels in the local area and all the pixels in other distant areas while extracting the dependencies between the pixels in the local area. In order to enhance the capture of the long-range dependencies between pixels, the self-attention mechanism focuses on the key information and reduces the attention of other irrelevant information. In addition, since the self-attention mechanism processes the same sequence, the similarity of any two elements in the sequence with the entire area as the observation range can be obtained after one matrix calculation, and there is no need to calculate the attention value multiple times as the window moves, thereby reducing the amount of calculation. In summary, the pixel self-attention module 202 can better model the relationship between different areas of the image and improve the accuracy of the generated hyperspectral image.

[0085] Then the fusion expression T out and feature F Left The addition result is used as the feature The data is input to the LayerNorm module 203, which eliminates the adverse effects caused by the singular sample data in the addition result.

[0086] The multi-layer perceptron module 204 performs feature conversion and information reorganization on the output of the LayerNorm module to obtain feature

[0087] The multi-scale adaptive residual attention module 20 finally inputs the feature F in ,feature and Features The connection result is obtained as the output of the multi-scale adaptive residual attention module 20, that is, the feature F i .

[0088] In this embodiment, the shallow region aggregation feature is used as the input feature F of the first multi-scale adaptive residual attention module 20 in , that is, the input of the conical multi-scale feature extraction module 10 in the first multi-scale adaptive residual attention module 20 is the shallow region aggregation feature.

[0089] In this embodiment, from the first multi-scale adaptive residual attention module 20 to the last second multi-scale adaptive residual attention module 20, the feature F output by each multi-scale adaptive residual attention module 20 is i As the input feature F of the next multi-scale adaptive residual attention module 20 connected to the multi-scale adaptive residual attention module 20 in .

[0090] In this embodiment, the feature F output by the last multi-scale adaptive residual attention module 20 is i As a deep feature.

[0091] The combined processing module 30 is used to perform convolution and activation function processing on the deep features to obtain a hyperspectral image. In this embodiment, the combined processing module includes a first convolution layer, an activation function, and a second convolution layer connected in sequence, thereby forming a linear-nonlinear combination to sort out the model network and enhance the fitting ability of the model network.

[0092] In this embodiment, the activation functions in the conical multi-scale feature extraction module 10, the multi-scale adaptive residual attention module 20 and the combination processing module 30 are all parameter-corrected linear units, namely Prelu.

[0093] In this embodiment, in order to test the performance of the hyperspectral reconstruction model 100 of the present invention, i.e., the model of the present invention, according to the existing three deep learning-based spectral reconstruction methods: HRNet method, AWAN method and DGCAMN method, corresponding models are constructed to obtain HRNet model, AWAN model and DGCAMN model respectively, and then these three models are retrained on the same hardware, programming environment and training set of the NTIRE 2020 hyperspectral dataset as the model of the present invention, and the RMSE index and MRAE index of the above four trained models are calculated by using the typical example data ARAD_HS_0453 in the validation set of the NTIRE 2020 hyperspectral dataset as shown in the following table:

[0094] Model Name RMSE metric MRAE indicator HRNet model 0.01964 0.07037 AWAN model 0.01662 0.04932 DGCAMN model 0.01474 0.04461 Model of the present invention 0.01299 0.03797

[0095] The first column in the table is the name of each model, the second column is the RMSE indicator calculation result of the corresponding model on the typical example data ARAD_HS_0453, and the third column is the MRAE indicator calculation result of the corresponding model on the typical example data ARAD_HS_0453. For example, the cell in the second column of the fifth row indicates that the RMSE indicator calculation result of the model of the present invention on the typical example data ARAD_HS_0453 is 0.01299. The RMSE indicator is used to calculate the square root of the average of the sum of squares of the differences between the predicted value and the true value. The smaller the calculation result of the RMSE indicator, the higher the accuracy of the model. The MRAE indicator is used to measure the relative error between the predicted value and the true value. The smaller the calculation result of the MRAE indicator, the higher the accuracy of the model. Therefore, on the NTIRE2020 hyperspectral dataset, the model of the present invention has better accuracy than the other three models, that is, it can reconstruct a hyperspectral image with higher accuracy.

[0096] Figure 6 Schematic diagram of typical example data ARAD_HS_0453 in an embodiment of the present invention.

[0097] like Figure 6 As shown, pixel A and pixel B are randomly selected on the typical example data ARAD_HS_0453, and then the spectral response curves of the above four models at these two pixel points are compared.

[0098] Figure 7 3 is a schematic diagram comparing spectral response curves of various models at pixel point A in an embodiment of the present invention.

[0099] like Figure 7As shown, (a) is the spectral response curve of the HRNet model at pixel point A, (b) is the spectral response curve of the AWAN model at pixel point A, (c) is the spectral response curve of the DGCAMN model at pixel point A, (d) is the spectral response curve of the model of the present invention at pixel point A, (e) is the ground truth spectral response curve at pixel point A, and (f) is a comparison diagram of the spectral response curves. The horizontal axes of (a), (b), (c), (d), (e) and (f) are all band numbers, i.e., Band, and the vertical axes are all normalized intensities of spectral response values, i.e., Normalized Intensity. In this embodiment, there are a total of 31 band channels, and the horizontal axis is 1, which represents the first band channel.

[0100] Figure 8 3 is a schematic diagram comparing spectral response curves of various models at pixel point B in an embodiment of the present invention.

[0101] like Figure 8 As shown, (a) is the spectral response curve of the HRNet model at pixel point B, (b) is the spectral response curve of the AWAN model at pixel point B, (c) is the spectral response curve of the DGCAMN model at pixel point B, (d) is the spectral response curve of the model of the present invention at pixel point B, (e) is the ground truth spectral response curve at pixel point B, and (f) is a comparison diagram of the spectral response curves. The horizontal axes of (a), (b), (c), (d), (e) and (f) are all band numbers, and the vertical axes are all normalized intensities of spectral response values.

[0102] according to Figure 7 and Figure 8 It can be obtained that at pixel point A, the spectral response curve of the model of the present invention is basically consistent with the true spectral response curve on the ground. At pixel point B, the spectral response curve of the model of the present invention is basically consistent with the true spectral response curve on the ground in the 1st-7th and 14-22nd band channels, namely 400nm-460nm and 530-610nm. In the remaining bands, it is also closer to the true spectral response curve on the ground than the other three models.

[0103] Fig. 9 Schematic diagram of typical example data ARAD_HS_0459 in an embodiment of the present invention.

[0104] like Fig. 9 As shown, pixel C and pixel D are randomly selected on the typical example data ARAD_HS_0459, and then the spectral response curves of the above four models at these two pixel points are compared.

[0105] Fig.10 3 is a schematic diagram comparing spectral response curves of various models at pixel point C in an embodiment of the present invention.

[0106] like Fig.10 As shown, (a) is the spectral response curve of the HRNet model at pixel point C, (b) is the spectral response curve of the AWAN model at pixel point C, (c) is the spectral response curve of the DGCAMN model at pixel point C, (d) is the spectral response curve of the model of the present invention at pixel point C, (e) is the ground truth spectral response curve at pixel point C, and (f) is a comparison diagram of the spectral response curves. The horizontal axes of (a), (b), (c), (d), (e) and (f) are all band numbers, and the vertical axes are all normalized intensities of spectral response values.

[0107] Fig.11 Schematic diagram of comparison of spectral response curves of various models at pixel point D in an embodiment of the present invention.

[0108] like Fig.11 As shown, (a) is the spectral response curve of the HRNet model at pixel point D, (b) is the spectral response curve of the AWAN model at pixel point D, (c) is the spectral response curve of the DGCAMN model at pixel point D, (d) is the spectral response curve of the model of the present invention at pixel point D, (e) is the ground truth spectral response curve at pixel point D, and (f) is a comparison diagram of the spectral response curves. The horizontal axes of (a), (b), (c), (d), (e) and (f) are all band numbers, and the vertical axes are all normalized intensities of spectral response values.

[0109] according to Fig.10 and Fig.11 It can be obtained that at pixel point C, the spectral response curve of the HRNet model in the 7th to 17th band, i.e., 460nm-560nm, is quite different from the ground truth spectral response curve. At pixel point D, the spectral response curves of the HRNet model, the AWAN model, and the DGCAMN model all fluctuate with the ground truth spectral response curve, while the spectral response curve of the model of the present invention still has the best fit with the ground truth spectral response curve.

[0110] In summary, the hyperspectral reconstruction method based on a single RGB image of the present invention can generate a higher quality hyperspectral image compared with the existing hyperspectral reconstruction method.

[0111] In order to verify the effectiveness of the pixel self-attention module 202 and the cone multi-scale feature extraction module 10 proposed in the hyperspectral reconstruction method based on a single RGB image of the present invention, the basic model for hyperspectral reconstruction is adjusted as follows: the pixel self-attention module 202 is adaptively adjusted and then added to the basic model to obtain the Ea model, the cone multi-scale feature extraction module 10 containing only a single component for obtaining multi-scale information complementarity is adaptively adjusted and then added to the basic model to obtain the Eb model, and the cone multi-scale feature extraction module 10 is adaptively adjusted and then added to the basic model to obtain the Ec model. Then, the three models are iterated 30 times respectively through the validation set data. In this process, the best RMSE index and MRAE index of each model are shown in the following table:

[0112] Model Name RMSE metric MRAE indicator Ea Model 0.01901 0.05940 Eb Model 0.01704 0.04725 Ec Model 0.01299 0.03797

[0113] The first column in the table is the name of each model, the second column is the RMSE indicator calculation result of the corresponding model, and the third column is the MRAE indicator calculation result of the corresponding model. For example, the cell in the second row and the second column indicates that the best RMSE indicator calculation result of the Ea model is 0.01901. It can be seen from the above table that compared with the basic model, the pixel self-attention module 202 added to the Ea model captures long-range dependencies by utilizing the features of all positions, and highlights the features of important areas, thereby improving the expression ability of spatial features. The Eb model changes the area of ​​each channel group to obtain input feature information through a single-component conical multi-scale feature extraction module 10 to capture detailed features at different regional levels, thereby achieving a strong learning ability of the network. The Ec model uses a complete conical multi-scale feature extraction module 10 to interleave information capture of regional information of different sizes on the channel, which can achieve the purpose of information complementarity, and improves the RMSE and MRAE indicators through aggregated local information and relative global information containing edge space information.

[0114] In order to more intuitively show the difference in the effects of the hyperspectral images generated by the above three models, two random pixels are selected from the typical example data ARAD_HS_0456 and ARAD_HS_0464 to compare the spectral response curves.

[0115] Fig.12 Schematic diagram of typical example data ARAD_HS_0456 in an embodiment of the present invention.

[0116] like Fig.12 As shown, pixel E and pixel F are selected in the typical example data ARAD_HS_0456 for comparison of the spectral response curves of each model.

[0117] Fig.13Schematic diagram of comparison of spectral response curves of various models at pixel point E in an embodiment of the present invention.

[0118] like Fig.13 As shown, (a) is the spectral response curve of the Ea model at the pixel point E, (b) is the spectral response curve of the Eb model at the pixel point E, (c) is the spectral response curve of the Ec model at the pixel point E, (d) is the ground truth spectral response curve at the pixel point E, and (e) is a comparison diagram of the spectral response curves. The horizontal axes of (a), (b), (c), (d) and (e) are all band numbers, and the vertical axes are all normalized intensities of the spectral response values.

[0119] Fig.14 3 is a schematic diagram comparing spectral response curves of various models at pixel point F in an embodiment of the present invention.

[0120] like Fig.14 As shown, (a) is the spectral response curve of the Ea model at the pixel point F, (b) is the spectral response curve of the Eb model at the pixel point F, (c) is the spectral response curve of the Ec model at the pixel point F, (d) is the ground truth spectral response curve at the pixel point F, and (e) is a comparison diagram of the spectral response curves. The horizontal axes of (a), (b), (c), (d) and (e) are all band numbers, and the vertical axes are all normalized intensities of the spectral response values.

[0121] according to Fig.13 and Fig.14 It can be obtained that at pixel E and pixel F, the spectral response curve corresponding to the Ea model is consistent with the ground-truth spectral response curve in direction, but there is a certain deviation in the value. In the 1-23 band, i.e. 400nm-620nm, the spectral response curves corresponding to the Eb model and the Ec model are basically consistent with the ground-truth spectral response curve. In the 24-31 band, i.e. 630nm-700nm, the spectral response curve corresponding to the Ec model is closer to the ground-truth spectral response curve, that is, the Ec model has the best hyperspectral reconstruction effect among the three models.

[0122] Fig.15 Schematic diagram of typical example data ARAD_HS_0464 in an embodiment of the present invention.

[0123] like Fig.15 As shown, pixel point G and pixel point H are selected in the typical example data ARAD_HS_0464 for comparison of the spectral response curves of each model.

[0124] Fig.16 Schematic diagram of comparison of spectral response curves of various models at pixel point G in an embodiment of the present invention.

[0125] like Fig.16As shown, (a) is the spectral response curve of the Ea model at the pixel point G, (b) is the spectral response curve of the Eb model at the pixel point G, (c) is the spectral response curve of the Ec model at the pixel point G, (d) is the ground truth spectral response curve at the pixel point G, and (e) is a comparison diagram of the spectral response curves. The horizontal axes of (a), (b), (c), (d) and (e) are all band numbers, and the vertical axes are all normalized intensities of the spectral response values.

[0126] Fig.17 3 is a schematic diagram comparing spectral response curves of various models at pixel point H in an embodiment of the present invention.

[0127] like Fig.17 As shown, (a) is the spectral response curve of the Ea model at the pixel point H, (b) is the spectral response curve of the Eb model at the pixel point H, (c) is the spectral response curve of the Ec model at the pixel point H, (d) is the ground truth spectral response curve at the pixel point H, and (e) is a comparison diagram of the spectral response curves. The horizontal axes of (a), (b), (c), (d) and (e) are all band numbers, and the vertical axes are all normalized intensities of spectral response values.

[0128] according to Fig.16 and Fig.17 It can be obtained that, at pixel point G, in the 1-25 band, i.e., 400nm-640nm, the spectral response curve corresponding to the Ec model is completely consistent with the ground truth spectral response curve. In the 26-31 band, i.e., 650nm-700nm, when the spectral response curves corresponding to the three models deviate from the ground truth spectral response curve, the spectral response curve corresponding to the Ec model is closer to the ground truth spectral response curve. At pixel point H, in the 24-31 band, i.e., 630nm-700nm, the spectral response curves corresponding to the Ea model and the Eb model deviate from the ground truth spectral response curve, but the spectral response curve corresponding to the Eb model is closer to the ground truth spectral response curve, and the curve corresponding to the Ec model is still basically consistent with the ground truth spectral response curve. Therefore, the Ec model has the best hyperspectral reconstruction effect among the three models.

[0129] In summary, the conical multi-scale feature extraction module 10 is more ideal for tracking lost information between local areas during the feature extraction process. The fusion of information from small areas and relatively large areas can help the model reconstruct a higher-precision hyperspectral image.

[0130] Functions and Effects of the Embodiments

[0131] According to the hyperspectral reconstruction method based on a single RGB image involved in this embodiment, on the one hand, the tapered multi-scale feature extraction module 10 is used to interlace the extraction of detail features of different scales on four different channels, thereby obtaining aggregated local information and relative global information containing edge space information, and finally achieving the purpose of extracting inter-regional contextual features at multiple scales; on the other hand, the pixel self-attention module 202 is used to extract the inter-pixel dependency of the local area while capturing the relationship between the pixel points in the area and all the pixel points in other distant areas, focusing on key information and reducing the attention paid to other irrelevant information, so as to better model the relationship between different areas of the image. In short, this method can improve the accuracy of the reconstructed hyperspectral image.

[0132] The above-mentioned embodiments are preferred examples of the present invention and are not intended to limit the protection scope of the present invention.

Claims

1. A hyperspectral reconstruction method based on a single RGB image, for inputting a single RGB image into a hyperspectral reconstruction model to obtain a corresponding hyperspectral image, characterized in that: The hyperspectral reconstruction model includes: A conical multi-scale feature extraction module is used to extract features from the feature map of the single RGB image after slice partitioning to obtain shallow region aggregation features; A plurality of sequentially connected multi-scale adaptive residual attention modules are used to process the shallow region aggregation features to obtain deep features; A combined processing module is used to perform convolution and activation function processing on the deep features to obtain the hyperspectral image. The multi-scale adaptive residual attention module includes a tapered multi-scale feature extraction module, an optimal non-local module, a pixel self-attention module, a LayerNorm module and a multi-layer perceptron module connected in sequence. The cone multi-scale feature extraction module is used to take the feature map of the single RGB image after slice partitioning or the input of the multi-scale adaptive residual attention module including the cone multi-scale feature extraction module as the input feature F in , and for the input feature F in Perform preliminary extraction of multi-scale detail features and multi-layer splicing in sequence to obtain feature F Left , and then for the feature F Left The multi-scale detail feature is extracted alternately and multi-layered splicing is performed in sequence to obtain the shallow region aggregation feature or the input of the optimal non-local module connected to the cone multi-scale feature extraction module as the output feature F out , The optimal non-local module is used to transform the output feature F out The partition features are divided into four equal-sized regions, and the connection between the partition features is strengthened to obtain the feature sequence T i , The pixel self-attention module is used to i The long-range dependencies between pixels in are captured to obtain the fused representation T out , The LayerNorm module is used to eliminate the fusion representation T out and the feature F Left The adverse effects caused by the singular sample data in the addition result are taken as the feature The multi-layer perceptron module is used to perform feature conversion and information reorganization on the output of the LayerNorm module to obtain feature In each of the multi-scale adaptive residual attention modules, the input feature F in , the characteristics and the characteristics The connection result is obtained as the input of the next multi-scale adaptive residual attention module connected in sequence with the multi-scale adaptive residual attention module, The connection result of the last multi-scale adaptive residual attention module is the deep feature.

2. The hyperspectral reconstruction method based on a single RGB image according to claim 1, characterized in that: in, The cone multi-scale feature extraction module includes a Left submodule and a Right submodule connected in sequence. The Left submodule includes four layers of Left channel domains and corresponding four layers of convolutions of different scales, and the channels of the four layers of Left channel domains are all the input features F of the cone multi-scale feature extraction module. in One quarter of the channel, the Nth left The convolution kernel size corresponding to the Left channel domain of the layer is k1=2N left +1, the input feature F in Input the Left submodule to obtain the feature F Left The expression formula is: F Left =Relu(BN(F' out ), Where ∑cat is a multi-layer concatenation operation, For N left The i-th group of feature maps in the Left channel domain of the layer, w k1,i is the i-th group of convolution kernels with k1 as the convolution kernel size, and the N-th left The number of channels in the Left channel domain of the layer is one-fourth of the original input feature F in As feature maps and corresponding convolution kernels are divided into m groups, BN is the batch normalization layer, Relu is the activation function, The Right submodule includes four layers of Right channel domains and corresponding four layers of convolutions of different scales, and the channels of the four layers of Right channel domains are all the input features F of the cone multi-scale feature extraction module. in The four layers of Right channel domains correspond to the four layers of Left channel domains in sequence. right The convolution kernel size corresponding to the Right channel domain of the layer is k2=11-2N right , the feature F Left Input the Right submodule to obtain the output feature F out The expression formula is: F out =Relu(BN(F″ out ), Where ∑cat is a multi-layer concatenation operation, G i,j For N right The i-th group of feature maps in the layer channel domain, w k2,i is the i-th group of convolution kernels with k2 as the convolution kernel size, and the N-th right The feature F in which the number of channels in the layer channel domain is one-fourth of the original Left The feature maps and the corresponding convolution kernels are divided into m groups, BN is the batch normalization layer, and Relu is the activation function.

3. The hyperspectral reconstruction method based on a single RGB image according to claim 1, characterized in that: in, The output feature F out Input the optimal non-local module to obtain the feature sequence T i The expression formula is: T i ONB(F out {lu,ld,ru,rd}), Where ONB is the optimal non-local module, F out {lu,ld,ru,rd} is the feature F out Divide into four equal-sized areas: upper left, lower left, upper right, and lower right.

4. The hyperspectral reconstruction method based on a single RGB image according to claim 1, Features: Among them, the pixel self-attention module is used to calculate the feature sequence T i Processing to obtain the fusion representation T out The specific steps are: Step S1, for the feature T i Use the combination of convolution-activation function-convolution to obtain preliminary features, and then divide the preliminary features into Q, K and V of the attention mechanism; Step S2, calculating the weight of each pixel area according to Q and K; Step S3: according to the characteristic sequence T i , the weight and V are calculated to obtain the fusion representation T out .

5. The hyperspectral reconstruction method based on a single RGB image according to claim 4, characterized in that: in, In step S1, the formulas of Q, K and V are as follows: Q=Conv(ρ(Conv(T i ))), K=Conv(ρ(Conv(T i ))), V=Conv(ρ(Conv(T i ))), Where ρ(·) is the activation function Prelu, and Conv is the convolution operation.

6. The hyperspectral reconstruction method based on a single RGB image according to claim 4, characterized in that: in, In step S2, the weight formula is as follows: U i =Corr(Q i ,Conv(K i )), μ i =softmax(U i ), Where U i is the attention value of the i-th pixel area, Corr is the correlation matrix calculation, Q i is Q, K of the i-th pixel area i is the K of the i-th pixel area, softmax is the softmax function operation, μ i is the weight of the i-th pixel area.

7. The hyperspectral reconstruction method based on a single RGB image according to claim 4, characterized in that: in, In step S3, the fusion representation T out The formula is as follows: T o ={T o1 ,T o2 ,T o3 ,...T oi ,...T on }, T out =T i +δ(Conv(T o )), Where T oi is the output of the i-th pixel area, is the transpose multiplication, V i is V, T of the ith pixel area o is the set of outputs of all pixel regions, and δ is a learnable parameter.

Images