A multi-scale hyperspectral image reconstruction algorithm and system from RGB images

Abstract

The present invention discloses a multi-scale hyperspectral image reconstruction algorithm and system from RGB images, which belongs to the field of computer vision hyperspectral images. The algorithm includes: processing the original RGB image through a multi-scale processing module, and outputting a feature map Y' i ; The feature map Y' i The feature image with multi-scale preprocessing information is added and stacked with the original RGB image itself in the spectral channel dimension to obtain a feature image with multi-scale preprocessing information; the feature image with multi-scale preprocessing information is passed through three spatial-spectral Transformer joint processing modules in sequence to jointly process the spatial-spectral dimensions to obtain a feature image with spatial-spectral feature information; the feature image with spatial-spectral feature information is processed through the spectral dimension Transformer to obtain a feature image with the spectral dimension feature information fully extracted by the Transformer; the feature image with the spectral dimension feature information fully extracted by the Transformer is passed through a 3×3 convolutional layer to adjust the channel dimension to the target 31-channel output, and finally a hyperspectral image is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision hyperspectral image reconstruction, and specifically relates to a multi-scale hyperspectral image reconstruction algorithm and system from RGB images. Background Art

[0002] Visual information is crucial for humans, with at least 80 percent of external information acquired through vision. When light strikes and reflects from an object, it carries with it the object's unique spectral information. Limited by the structure of the human eye, the naked eye can only read information within the visible light range. However, hyperspectral imagers can capture information across all wavelengths, revealing information that is difficult to detect with the naked eye. Hyperspectral imagers can fully record the spectral information of an image. Compared to conventional color cameras, they can simultaneously capture both spatial and spectral information about the object being examined. Due to this advantage, hyperspectral imagery has particularly broad applications in medical image processing, remote sensing, and object tracking.

[0003] With the development of related fields of hyperspectral image processing, the demand for hyperspectral images has become increasingly greater. However, the difficulty and high cost of obtaining traditional hyperspectral images have restricted the development of this field. Most traditional hyperspectral image acquisition methods usually use spectrometers with spatial or spectral scanning technology, such as push-broom scanners, whisk-broom scanners and band sequential scanners. However, hyperspectral imagers also have obvious disadvantages. Hyperspectral imagers are generally very large and very complicated to operate, which makes it difficult and costly to obtain a hyperspectral image. In recent years, with the development of convolutional neural network theory, a large number of convolutional neural network (CNN)-based methods have been applied to reconstruction work and have achieved relatively good results. Although the traditional CNN-based deep learning reconstruction algorithm has solved the high cost and difficulty of hardware problems, due to the shortcomings of the CNN model in obtaining spatial long-range correlation and self-similarity within the spectrum, its reconstruction quality still does not meet high standards. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a multi-scale hyperspectral image reconstruction algorithm and system from RGB images.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] A multi-scale algorithm for reconstructing a corresponding hyperspectral image from an RGB image includes the following steps:

[0007] S1, processes the original RGB image X through the multi-scale processing module and outputs the feature map Y' i ;

[0008] S2, the feature map Y' i Add and stack it with the original RGB image X in the spectral channel dimension to obtain a feature image F with multi-scale preprocessing information;

[0009] S3, the feature image F with multi-scale preprocessing information is passed through three spatial-spectral Transformer joint processing modules in sequence, and the spatial-spectral dimensions are jointly processed to obtain a feature image with spatial-spectral feature information;

[0010] S4, the feature image obtained by S3 is processed separately through a spectral dimension Transformer to obtain a feature image with the spectral dimension feature information fully extracted by the Transformer;

[0011] In S5, the feature image that has fully extracted the spectral dimension feature information through the Transformer is passed through a 3×3 convolutional layer to adjust the channel dimension to the target 31-channel output, and finally a hyperspectral image is obtained.

[0012] Furthermore, the feature map Y' is obtained by processing in S1 i The steps are:

[0013] S11, copy the original RGB image 3 times, and then downsample it at 8 times, 4 times, and 2 times the downsampling rate respectively;

[0014] S12, after downsampling, a convolution layer with a 3×3 convolution kernel is used to perform a convolution operation on the downsampled feature image. Except for the first layer, each layer is added and fused with the result of the previous layer immediately after the convolution operation, as shown in the following formula:

[0015] Y i =Conv(Down r (X))⊕Y i-1

[0016] Down r (·) represents a downsampling operation, r is the downsampling rate, Conv(·) represents a two-dimensional convolution operation with a convolution kernel size of 3×3, and the ⊕ symbol represents a channel stacking operation; all convolution operations in the multi-scale processing module use the LeakyReLU activation function; Represents the intermediate process output of the i-th layer, where i∈(1, 2, 3);

[0017] S13, convert S12 to get Y i Input into the spectral Transformer module to capture the contextual information between spectral channels and obtain Y' i ; as shown below:

[0018]

[0019] in, represents the intermediate process output of the i-th layer, Represents the final result of layer i, Up r (·) represents the upsampling operation, r is the upsampling rate, Conv(·) represents the two-dimensional convolution operation with a convolution kernel size of 1×1, Tran spe (·) represents the Transformer processing of the spectral dimension.

[0020] Furthermore, in S2, the steps of obtaining the feature image F with multi-scale preprocessing information are:

[0021] S21, use a 3×3 convolution layer to adjust the original RGB image to the feature map Y' i With the same spatial size (rh'3×rw'3), the output is

[0022] S22, the characteristic graph Y' in the spectral dimension i And the original RGB image is stacked, that is, c0+c1, where Finally, we get a feature image with multi-scale preprocessing information

[0023] Furthermore, in S3, the feature image is processed by the spatial transformer and the spectral transformer in each spatial-spectral transformer joint processing module.

[0024] Furthermore, the processing steps of the feature image in each spatial-spectral Transformer joint processing module are as follows:

[0025] S31, spatial Transformer processing:

[0026] 1) The input of this layer is set to the feature image F∈R c×h×w , the feature image F is evenly divided into small windows of size m, and the block operation obtains the feature image F i ∈R c×m×m Flatten and transpose to get Then, self-attention processing is performed on the obtained one-dimensional feature map; the specific formula is as follows:

[0027] F={F 1 ,F 2 ,…,F N}, N = hw / m 2

[0028] Ai =Attention(F i W Q ,F i W K ,F i W V ), i=1,…,N

[0029]

[0030] Where W Q , W K and W V ∈R c×c Respectively represent The projection matrix is ​​a learnable parameter. is the final output result for each window; the Attention(·) operation adds relative position encoding while implementing self-attention calculation, as shown in the following formula:

[0031]

[0032] Among them, B is the relative position bias, which is a shape of R (2m-1)×(2m-1) The learnable parameters of

[0033] 2) Then a simple multi-layer perceptron is used to integrate the results of the self-attention calculation And in the whole process, skip connections are used to reduce the difficulty of training. The calculation process can be as follows:

[0034]

[0035] F out =MLP(LN(F′))+F

[0036] where F' and F out are the processing results of Spa and MLP respectively. Meanwhile, F out represents the final result of spatial Transformer processing, and the LN(·) symbol represents layer normalization;

[0037] S32, the feature map result processed by the spatial transformer is then input into the spectral transformer processing block;

[0038] 1) Let the input feature map be H∈R c×h×w , first flatten the feature map H and transpose it to H∈R hw×c , then H∈R hw ×c Via W Q , W K and W V ∈Rc×c Linear projection to Q, K, V ∈ R hw×c ; The self-attention calculation process is as follows:

[0039] Attention(Q,K,V)=SoftMax(σK T Q)V

[0040] where σ∈R 1 is a learnable parameter;

[0041] 2) Then, the self-attention result Attention(Q,K,V) is linearly projected and the relative position encoding is added. The specific process is given by the following formula:

[0042] Spe(H)=Attention(Q,K,V)W+φ(V)

[0043] Where W∈R c×c is a learnable parameter, and the symbol φ(·) represents the relative position encoding, which consists of two 3×3 convolutional layers and a GELU activation function;

[0044] 3) Then a feedforward network is used to integrate the weight matrix obtained by the above formula, and skip connections are used to reduce the training difficulty in the whole process. spe The calculation process of (·) can be expressed as follows:

[0045] H′=Spe(LN(H))+H

[0046] H out =FFN(LN(H′))+H′

[0047] Where H' and H out are the processing results of Spe and FFN respectively. Meanwhile, H out represents the final result of spectral Transformer processing, and the LN(·) symbol represents layer normalization.

[0048] Furthermore, the feedforward network is composed of a 1×1 convolution layer, a GELU activation function, a 3×3 convolution layer, a GELU activation function and a 1×1 convolution layer in sequence.

[0049] Furthermore, in S5, the steps of obtaining a hyperspectral image are:

[0050] S51, set a convolution layer with an input of 32 and an output of 31, a convolution kernel size of 3×3, and a stride and padding size of 1;

[0051] In step S52, the feature image obtained in step S4, which has been fully extracted from the spectral dimension feature information by the Transformer, is input into the convolutional layer and activated by the LeakyRelu function to obtain a hyperspectral image with 31 channels and the same spatial resolution as the input RGB image.

[0052] A system for multi-scale reconstruction of a corresponding hyperspectral image from an RGB image, comprising: a multi-scale processing unit, a dimension superposition unit, a spatial-spectral Transformer joint processing unit, a spectral dimension Transformer processing unit, and a hyperspectral image output unit;

[0053] The multi-scale processing unit is used to process the original RGB image X through the multi-scale processing module and output the feature map Y' i ;

[0054] Dimensional overlay unit, used to transform the feature map Y' i Add and stack it with the original RGB image X in the spectral channel dimension to obtain a feature image F with multi-scale preprocessing information;

[0055] The spatial-spectral Transformer joint processing unit is used to pass the feature image F with multi-scale preprocessing information through three spatial-spectral Transformer joint processing modules in sequence, jointly process the spatial and spectral dimensions, and obtain a feature image with spatial-spectral feature information;

[0056] A spectral dimension transformer processing unit is used to process the feature image obtained by S3 through a spectral dimension transformer alone to obtain a feature image with fully extracted spectral dimension feature information through the transformer;

[0057] The hyperspectral image output unit is used to adjust the channel dimension of the feature image whose spectral dimension feature information has been fully extracted by the Transformer to the target 31-channel output through a 3×3 convolution layer, and finally obtain a hyperspectral image.

[0058] A computer storage medium stores a readable program, and when the program is run, the above algorithm is executed.

[0059] A device, characterized by comprising: one or more processors, and a memory for storing one or more programs;

[0060] When the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the above algorithm.

[0061] The beneficial effects of the present invention are as follows: Through a multi-scale Transformer-based spatial-spectral joint processing network architecture, it is possible to efficiently, cost-effectively, and accurately reconstruct corresponding hyperspectral images from RGB. Compared with traditional hyperspectral imagers and CNN-based deep learning algorithms, the present invention can not only significantly increase the speed and reduce the cost of acquiring hyperspectral images, but also overcome the limitations of CNN models in acquiring spatial long-range correlations and self-similarity within the spectrum, and adopt a multi-scale hierarchical feature extraction method, avoiding the disadvantage of information loss during the encoding and decoding process brought about by the U-Net network currently used in work. This method can significantly improve the accuracy of reconstructed hyperspectral images and has great practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0063] Figure 1 It is a framework diagram of the algorithm for reconstructing the corresponding hyperspectral image from the RGB image of the present invention;

[0064] Figure 2 It is a framework diagram of the multi-scale processing module of the present invention;

[0065] Figure 3 In the experiment of the present invention, the real hyperspectral image and the reconstructed hyperspectral image as well as the error map between the two;

[0066] Figure 4 This is a comparison chart of the true values ​​and generated values ​​of all bands at a certain coordinate point in the hyperspectral image in the experiment of the present invention. DETAILED DESCRIPTION

[0067] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0068] like Figure 1 As shown, a multi-scale hyperspectral image reconstruction algorithm and system from RGB images includes the following steps:

[0069] S1, the original RGB image X∈R 3×128×128After processing by the multi-scale processing module (MSP), the output feature map Y' i ;

[0070] Among them, the framework structure of the multi-scale processing module is as follows Figure 2 As shown, the specific steps of S1 are:

[0071] S11, downsample and reconstruct the image; the original RGB image of 3×128×128 is copied three times, and then downsampled to 192×16×16, 48×32×32 and 12×64×64 at downsampling rates of 8 times, 4 times and 2 times, respectively.

[0072] S12, after downsampling, passes through a convolution layer with a 3×3 convolution kernel to perform a convolution operation on the downsampled feature image; the convolution operation changes the number of channels but does not change the spatial resolution so that it can be fused with the result of the previous layer; except for the first layer, each layer is added and fused with the result of the previous layer immediately after the convolution operation, as shown in the following formula:

[0073] Y i =Conv(Down r (X))⊕Y i-1

[0074] Down r (·) represents a downsampling operation, r is the downsampling rate, Conv(·) represents a two-dimensional convolution operation with a convolution kernel size of 3×3, and the ⊕ symbol represents a channel stacking operation; all convolution operations in the multi-scale processing module use the LeakyReLU activation function; Represents the intermediate process output of the i-th layer, where i∈(1, 2, 3).

[0075] S13, the Y obtained in S12 i Input into the spectral transformer module (Spe) to capture the contextual information between spectral channels; because the spatial size of the first layer is too small after downsampling, a 1×1 convolution layer is passed at the end of the first layer to enable the network to adaptively adjust the weights; finally, an upsampling layer is used to adjust the spatial scale to the same size as the next layer to obtain Y' i ; The specific implementation process is given by the following formula:

[0076]

[0077] in, represents the intermediate process output of the i-th layer, Represents the final result of layer i, Up r (·) represents the upsampling operation, r is the upsampling rate, Conv(·) represents the two-dimensional convolution operation with a convolution kernel size of 1×1, Transpe (·) represents the Transformer processing of the spectral dimension.

[0078] S2, the final output feature map Y' in S1 i It is added and stacked with the original RGB image X itself in the spectral channel dimension to obtain a feature image with multi-scale preprocessing information, so that the backbone network can better extract the characteristics of the original information;

[0079] The specific superposition steps are:

[0080] S21, uses a 3×3 convolutional layer to transform the original input RGB image (X∈R 3×h×w ) is adjusted to the same spatial size (rh'3×rw'3) as the output result in S1, and the output result is

[0081] S22, two tensors (feature map Y' i and the original RGB image) are stacked, i.e. c0+c1, where Finally, we get a feature image with multi-scale preprocessing information

[0082] S3, the feature image F with multi-scale preprocessing information in S2 is sequentially passed through three spatial-spectral Transformer joint processing modules (SUB), which jointly processes the spatial and spectral dimensions to obtain a feature image with spatial-spectral feature information;

[0083] The feature image is processed by spatial transformer (Spa) and spectral transformer (Spe) in each spatial-spectral transformer joint processing module (SUB);

[0084] The specific steps of processing the feature image in the spatial-spectral Transformer joint processing module (SUB) are as follows:

[0085] S31, spatial Transformer processing;

[0086] The input of this layer is To simplify the characters in the introduction process, we assume that the feature image F∈R c×h×w The input of this module is the feature image F, which is evenly divided into small windows of size m, and the feature image F is obtained by block operation. i ∈R c×m×m Flatten and transpose to get Then, self-attention processing is performed on the obtained one-dimensional feature map; the specific implementation process is shown in the following formula:

[0087] F={F 1 ,F 2 ,…,F N}, N = hw / m 2

[0088] A i =Attention(F i W Q ,F i W K ,F i W V ), i=1,…,N

[0089]

[0090] Where W Q , W K and W V ∈R c×c Respectively represent The projection matrix is ​​a learnable parameter. is the final output result for each window; the Attention(·) operation adds relative position encoding while implementing self-attention calculation. The specific implementation details are given by the following formula:

[0091]

[0092] Where B is the relative position bias, which is a shape of R (2m-1)×(2m-1) The learnable parameters of

[0093] Then a simple multi-layer perceptron (MLP) is used to integrate the results of the self-attention calculation And through the whole process of skip connection to reduce the difficulty of training, in summary, Tran spa The calculation process of (·) can be summarized as follows:

[0094]

[0095] F out =MLP(LN(F′))+F′

[0096] where F' and F out are the processing results of Spa and MLP respectively. Meanwhile, F out represents the final result of spatial Transformer processing, and the LN(·) symbol represents layer normalization.

[0097] S32, spectral Transformer processing;

[0098] Feature map result F processed by spatial Transformer out Then it is input into the spectral Transformer processing block;

[0099] For the convenience of character description, we assume that the input is H∈R c×h×w , similar to the spatial processing, first flatten the feature image X and transpose it to H∈R hw×c , and then H∈R hw×c Via W Q , W K and W V ∈R c×c Linear projection to Q, K, V ∈ R hw×c ; Different from spatial processing, spectral processing treats all channels of a single pixel as a token. The specific process of self-attention (Spe) is given by the following formula:

[0100] Attention(Q,K,V)=SoftMax(σK T Q)V

[0101] where σ∈R 1 is a learnable parameter;

[0102] Then, the result of self-attention Attention(Q,K,V) is linearly projected and the relative position encoding is added. The specific process is given by the following formula:

[0103] Spe(H)=Attention(Q,K,V)W+φ(V)

[0104] Where W∈R c×c is a learnable parameter, and the symbol φ(·) represents the relative position encoding, which consists of two 3×3 convolutional layers and a GELU activation function;

[0105] Then, a feedforward network (FFN) is used to integrate the weight matrix Spe(H) obtained by the above formula, and skip connections are used to reduce the difficulty of training in the whole process. The feedforward network consists of a 1×1 convolution layer, a GELU activation function, a 3×3 convolution layer, a GELU activation function, and a 1×1 convolution layer in sequence; In summary, Tran spe The calculation process of (·) can be expressed as follows:

[0106] H′=Spe(LN(H))+H

[0107] H out =FFN(LN(H′))+H′

[0108] Where H' and Hout are the processing results of Spe and FFN respectively. Meanwhile, H out represents the final result of spectral Transformer processing, and the LN(·) symbol represents layer normalization.

[0109] S4: The feature image obtained by S3 is processed by a spectral dimension transformer (Spe+FFN) to obtain a feature image with fully extracted spectral dimension feature information.

[0110] The specific processing steps are: repeat the S32 operation (since the ultimate goal is to reconstruct a 31-channel hyperspectral image from a 3-channel RGB image and achieve a mapping of the channel dimension from low to high, the most important thing is the characteristic information of the spectral dimension, so the spectral Transformer processing is performed again here)

[0111] S5: The feature image obtained in S4, which has been fully extracted from the spectral dimension feature information by the Transformer, is passed through a 3×3 convolutional layer to adjust the channel dimension to the target 31-channel output; finally, a hyperspectral image is obtained;

[0112] The specific steps are:

[0113] S51, set a convolution layer with an input of 32 and an output of 31, a convolution kernel size of 3×3, and a stride and padding size of 1 to ensure that the feature map size does not change;

[0114] In S52, the final result of S4 is input into the convolution layer and activated through the LeakyRelu function to obtain a hyperspectral image with 31 channels and the same spatial resolution as the input RGB image.

[0115] In this embodiment, the dataset provided by the NTIRE 2022 spectral reconstruction challenge is selected for training and evaluation. The dataset contains 1000 RGB data pairs and their corresponding real hyperspectral images; each hyperspectral image has 31 channels, and each channel stores light intensity information every 10nm from 400nm to 700nm, with a spatial resolution of 482×512; the RGB image has 3 RGB channels, and the spatial resolution is consistent with the corresponding hyperspectral image; 90% of the data in the dataset is randomly selected as the training set, and the remaining 10% of the data is used as the test set.

[0116] During the training process, the RGB and hyperspectral images in the original dataset are randomly cropped to a spatial size of 128×128, and the pixel intensity values ​​are normalized to the range of [0,1]. Simple data augmentation is performed on the training data, such as random rotation and flipping. The processed RGB and the corresponding real hyperspectral images are input into the network model. After being processed by a series of network modules proposed in this paper, the training model learns the mapping from RGB to real hyperspectral images. The training goal is to pass the loss function l MRAE Update the parameters θ to minimize and Y i Distance between:

[0117]

[0118] in is the light intensity of the generated hyperspectral image at the pixel point, and S i It is the light intensity of the real hyperspectral image at the pixel point, and N represents all the pixels in the image.

[0119] Experimental process:

[0120] During the test, the original RGB image in the test data set is input into the trained network to obtain the corresponding hyperspectral image. The absolute difference between the reconstructed hyperspectral image and the real hyperspectral image is calculated according to the value of each pixel in each spectral dimension to obtain the numerical error map, such as Figure 3 . And randomly select the same coordinate position in the reconstructed hyperspectral image and the real hyperspectral image to draw the light intensity curve of all bands of the pixel point, such as Figure 4 , in order to judge the accuracy of the hyperspectral images generated by the method of the present invention. In addition, the method of the present invention is compared with the advanced CNN architecture-based HSCNN+ and AWAN in terms of MRAE indicators. At the same time, ablation experiments are conducted on the MSP and SSU modules of the method of the present invention to prove its effectiveness.

[0121] from Figure 3 It can be seen that the hyperspectral image generated from the RGB image has a strong similarity to the real hyperspectral image on the left. Furthermore, from the error graph of the two, it can be seen that in the entire image, the errors of all pixels are within a very small range (the darker the difference, the smaller the difference, and the whiter the difference, the larger the difference). Figure 4The intensity curves also clearly show that the method of the present invention can generate intensity curves that are very close to real data in all bands. As can be clearly seen from Table 1, the method proposed by the present invention has a lower MRAE, which means that the hyperspectral images generated by the method of the present invention are very similar to the real hyperspectral images. And as can be seen from Table 2, both MSP and SSU have a positive improvement on the performance of the method of the present invention, which shows the independent effectiveness of the two modules.

[0122]

[0123]

[0124] Table 1 Comparison of indicators with traditional methods

[0125]

[0126] Table 2 Ablation experiment

[0127] Whether from the perspective of indicator quantification results or visual effect diagrams, the algorithm proposed in the present invention can better generate high-quality hyperspectral images. In addition, the network of the present invention is more lightweight and can be easily installed on other hardware, providing a more convenient and lower-cost hyperspectral image acquisition method for various hyperspectral application fields, thereby promoting further development in this field.

[0128] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiment of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal translation scripting language JavaScript, etc.

[0129] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0132] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0133] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A multi-scale algorithm for reconstructing a corresponding hyperspectral image from an RGB image, characterized in that: The following steps are involved: S1, processes the original RGB image X through the multi-scale processing module and outputs the feature map Y' i ; S2, the feature map Y' i Add and stack it with the original RGB image X in the spectral channel dimension to obtain a feature image F with multi-scale preprocessing information; S3, the feature image F with multi-scale preprocessing information is passed through three spatial-spectral Transformer joint processing modules in sequence, and the spatial-spectral dimensions are jointly processed to obtain a feature image with spatial-spectral feature information; S4, the feature image obtained by S3 is processed separately through a spectral dimension Transformer to obtain a feature image with the spectral dimension feature information fully extracted by the Transformer; In S5, the feature image that has fully extracted the spectral dimension feature information through the Transformer is passed through a 3×3 convolutional layer to adjust the channel dimension to the target 31-channel output, and finally a hyperspectral image is obtained.

2. The multi-scale hyperspectral image reconstruction algorithm according to claim 1 is characterized in that: The feature map Y' is obtained by processing in S1 i The steps are: S11, copy the original RGB image 3 times, and then downsample it at 8 times, 4 times, and 2 times the downsampling rate respectively; S12, after downsampling, a convolution layer with a 3×3 convolution kernel is used to perform a convolution operation on the downsampled feature image. Except for the first layer, each layer is added and fused with the result of the previous layer immediately after the convolution operation, as shown in the following formula: Down r (·) represents the downsampling operation, r is the downsampling rate, Conv(·) represents the two-dimensional convolution operation with a convolution kernel size of 3×3, The symbol represents the stacking operation of the number of channels; all convolution operations in the multi-scale processing module use the LeakyReLU activation function; Represents the intermediate process output of the i-th layer, where i∈(1, 2, 3); S13, convert S12 to get Y i Input into the spectral Transformer module to capture the contextual information between spectral channels and obtain Y' i ; as shown below: in, represents the intermediate process output of the i-th layer, Represents the final result of layer i, Up r (·) represents the upsampling operation, r is the upsampling rate, Conv(·) represents the two-dimensional convolution operation with a convolution kernel size of 1×1, Tran spe (·) represents the Transformer processing of the spectral dimension.

3. The multi-scale hyperspectral image reconstruction algorithm according to claim 1 is characterized in that: In S2, the steps to obtain the feature image F with multi-scale preprocessing information are: S21, use a 3×3 convolution layer to adjust the original RGB image to the feature map Y' i With the same spatial size (rh'3×rw'3), the output is S22, the characteristic graph Y' in the spectral dimension i And the original RGB image is stacked, that is, c0+c1, where Finally, we get a feature image with multi-scale preprocessing information 4. The multi-scale hyperspectral image reconstruction algorithm according to claim 1, characterized in that: In S3, the feature image is processed by the spatial transformer and the spectral transformer in each spatial-spectral transformer joint processing module.

5. The multi-scale hyperspectral image reconstruction algorithm according to claim 4 is characterized in that: The processing steps of the feature image in each spatial-spectral Transformer joint processing module are as follows: S31, spatial Transformer processing: 1) The input of this layer is set to the feature image F∈R c×h×w , the feature image F is evenly divided into small windows of size m, and the block operation obtains the feature image F i ∈R c×m×m Flatten and transpose to get Then, self-attention processing is performed on the obtained one-dimensional feature map; the specific formula is as follows: F={F 1 ,F 2 ,…,F N },N=hw / m 2 A i =Attention(F i W Q ,F i W K ,F i W V ),i=1,…,N Where W Q , W K and W V ∈R c×c Represent Q, K, The projection matrix is ​​a learnable parameter. is the final output result for each window; the Attention(·) operation adds relative position encoding while implementing self-attention calculation, as shown in the following formula: Among them, B is the relative position bias, which is a shape of R (2m-1)×(2m-1) The learnable parameters of 2) Then a simple multi-layer perceptron is used to integrate the results of the self-attention calculation And in the whole process, skip connections are used to reduce the difficulty of training. The calculation process can be as follows: F out =MLP(LN(F′))+F′ where F' and F out are the processing results of Spa and MLP respectively. Meanwhile, F out represents the final result of spatial Transformer processing, and the LN(·) symbol represents layer normalization; S32, the feature map result processed by the spatial transformer is then input into the spectral transformer processing block; 1) Let the input feature map be H∈R c×h×w , first flatten the feature map H and transpose it to H∈R hw×c , then H∈R hw×c Via W Q , W K and W V ∈R c×c Linear projection to Q, K, V ∈ R hw×c ; The self-attention calculation process is as follows: Attention(Q,K,V)=SoftMax(σK T Q)V where σ∈R 1 is a learnable parameter; 2) Then, the self-attention result Attention(Q,K,V) is linearly projected and the relative position encoding is added. The specific process is given by the following formula: Spe(H)=Attention(Q,K,V)W+φ(V) Where W∈R c×c is a learnable parameter, and the symbol φ(·) represents the relative position encoding, which consists of two 3×3 convolutional layers and a GELU activation function; 3) Then a feedforward network is used to integrate the weight matrix obtained by the above formula, and skip connections are used to reduce the training difficulty in the whole process. spe The calculation process of (·) can be expressed as follows: H′=Spe(LN(H))+H H out =FFN(LN(H′))+H′ Where H' and H out are the processing results of Spe and FFN respectively. Meanwhile, H out represents the final result of spectral Transformer processing, and the LN(·) symbol represents layer normalization.

6. The multi-scale hyperspectral image reconstruction algorithm according to claim 5 is characterized in that The feedforward network consists of a 1×1 convolutional layer, a GELU activation function, a 3×3 convolutional layer, a GELU activation function, and a 1×1 convolutional layer in sequence.

7. The multi-scale hyperspectral image reconstruction algorithm according to claim 1, characterized in that: In S5, the steps for obtaining hyperspectral images are: S51, set a convolution layer with an input of 32 and an output of 31, a convolution kernel size of 3×3, and a stride and padding size of 1; In step S52, the feature image obtained in step S4, which has been fully extracted from the spectral dimension feature information by the Transformer, is input into the convolutional layer and activated by the LeakyRelu function to obtain a hyperspectral image with 31 channels and the same spatial resolution as the input RGB image.

8. A system for reconstructing corresponding hyperspectral images from RGB images at multiple scales, Its characteristics include: Multi-scale processing unit, dimension superposition unit, spatial-spectral Transformer joint processing unit, spectral dimension Transformer processing unit and hyperspectral image output unit; The multi-scale processing unit is used to process the original RGB image X through the multi-scale processing module and output the feature map Y' i ; Dimensional overlay unit, used to transform the feature map Y' i Add and stack it with the original RGB image X in the spectral channel dimension to obtain a feature image F with multi-scale preprocessing information; The spatial-spectral Transformer joint processing unit is used to pass the feature image F with multi-scale preprocessing information through three spatial-spectral Transformer joint processing modules in sequence, jointly process the spatial and spectral dimensions, and obtain a feature image with spatial-spectral feature information; A spectral dimension transformer processing unit is used to process the feature image obtained by S3 through a spectral dimension transformer alone to obtain a feature image with fully extracted spectral dimension feature information through the transformer; The hyperspectral image output unit is used to adjust the channel dimension of the feature image whose spectral dimension feature information has been fully extracted by the Transformer to the target 31-channel output through a 3×3 convolution layer, and finally obtain a hyperspectral image.

9. A computer storage medium storing a readable program, characterized in that: When the program is run, the algorithm described in any one of claims 1 to 7 is executed.

10. A device, characterized in that include: one or more processors, a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the algorithm according to any one of claims 1 to 7.

Images