Hyperspectral snapshot image restoration method, device, equipment and medium

Abstract

The present invention discloses a hyperspectral snapshot image restoration method, device, equipment and medium. The method includes obtaining a spectral image data set to write a data interface required for a convolutional neural network; constructing a training set and a test set; using a random mask to superimpose the training set, setting channel pixels, and superimposing the image in the spectral dimension after tilt processing to obtain an aliased image as input, using the training set as the data to be fitted, and training the model end-to-end; constructing a restoration algorithm in the form of deep iterative expansion; using a neural network based on a regional self-attention mechanism for modeling and learning, and constructing a restoration algorithm model; setting training parameters and loss function strategies, training and verifying the constructed restoration algorithm model; and restoring the compressed sensing image to a hyperspectral image through the restoration algorithm model. The present invention utilizes the spectral-spatial correlation unique to hyperspectral data to restore the original hyperspectral image information with high precision and high efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of spectral computational imaging, and in particular relates to a hyperspectral snapshot image restoration method, device, equipment and medium. Background Art

[0002] For hyperspectral images, two-dimensional spatial information and one-dimensional spectral information together constitute three-dimensional hyperspectral image information, so it is also called a data cube. The core component of ordinary camera sensors is a two-dimensional photosensitive array, which cannot directly capture three-dimensional information. Therefore, traditional hyperspectral imaging technology adopts various indirect implementation methods. One method is to use a multi-band spectral sensor to directly collect local spectral information and use a scanning method to collect spatial information. This method has high image accuracy, but because it requires motion scanning, the imaging time is slow and the stability of the motion platform is high. The second method is to use a staring sensor array and continuously replace the filter in front of the sensor to obtain multi-band spectral information. This method does not require scanning, but is limited by the frequency of filter replacement. Another method is to directly integrate multiple spectral band photosensitive devices on a two-dimensional array. The RGB camera used in daily life adopts this method, but the number of its spectral bands is limited by the integrated space size of the photosensitive array.

[0003] With the emergence and development of computational imaging technology, the combination of optical systems and image processing algorithms has provided new ideas for spectral imaging. In computational imaging systems, the optical system directly obtains a certain form of light information, which is then processed into a complete and usable image by a software algorithm. In this process, since the final result does not need to be obtained directly, the optical system can overcome some shortcomings. Computational spectral imaging technology has made great progress based on computational imaging technology. In computational spectral imaging technology based on compressed sensing technology, the system separates and modulates spectral information through a spectroscopic device, and the sensor receives the modulated spectral information and the original spatial information. The complete spectral image is restored by a recovery algorithm based on compressed sensing technology. The most representative of these is the coded aperture snapshot spectral imaging system (CASSI) proposed by Brady et al. in 2006. The system uses coded aperture to modulate the incident light information, and uses a prism for splitting. The detector receives the aliased image after splitting, and finally the spectral image is restored by the recovery algorithm. Due to the snapshot design, the system has obvious advantages in imaging cost and efficiency.

[0004] The traditional restoration algorithm of the CASSI system uses an iterative optimization algorithm to solve the inverse problem of compressed sensing. This type of algorithm takes a long time to calculate and has relatively low recovery accuracy. End-to-end solutions that directly use end-to-end neural networks for recovery lack interpretability and have poor generalization of data. Plug-and-play solutions that use denoising networks to replace denoisers in traditional iterative algorithms still require dozens of iterative optimization processes, have slow running speeds, and are difficult to achieve real-time processing of image data. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the prior art and provide a hyperspectral snapshot image restoration method, device, equipment and medium, which replaces global modeling with regional modeling, overcomes the disadvantage of unstable global features, and utilizes the spectral-spatial correlation unique to hyperspectral data to restore the original hyperspectral image information with high precision and high efficiency.

[0006] The object of the present invention is achieved through the following technical solutions:

[0007] A hyperspectral snapshot image restoration method, characterized in that the method comprises:

[0008] Acquire a spectral image data set, and write a data interface required by a convolutional neural network according to data features of the spectral image data set;

[0009] Splitting the spectral image data set into several parts to construct a training set and a test set;

[0010] Using a random mask to superimpose the training set, setting channel pixels, and performing tilt processing on the image and then superimposing it in the spectral dimension to obtain an aliased image, using the aliased image as input and the training set as data to be fitted, and training the model end-to-end;

[0011] The recovery algorithm is constructed in the form of deep iterative expansion, the algorithm includes multiple formally repeated sub-stages, each sub-stage optimizes the solution by alternating projection method, and gradually approaches the desired recovery solution;

[0012] Use a neural network based on regional self-attention mechanism for modeling and learning to build a recovery algorithm model;

[0013] Set training parameters and loss function strategies to train and verify the constructed recovery algorithm model;

[0014] The compressed sensing image is restored to a hyperspectral image through the restoration algorithm model.

[0015] Furthermore, the method also includes preprocessing the spectral image data set, specifically including:

[0016] During training, image patches with fixed spatial resolution are randomly cropped from the original image.

[0017] Performing data enhancement on the image block, wherein the data enhancement includes random flipping, random rotation, random block re-joining and / or adding random Gaussian noise;

[0018] A number of spectral channels are uniformly selected within a preset range length, and if the spectral channels of the data of the spectral image data set do not match the selected spectral channels, an interpolation method is used to obtain the spectral channels.

[0019] Furthermore, the spectral image dataset includes a public dataset and a simulated spectral image dataset, wherein the method for acquiring the simulated spectral image dataset specifically includes:

[0020] A plurality of RGB images are acquired, and R, G, and B channels of the RGB images are expanded into a hyperspectral spectrum.

[0021] Furthermore, the method of expanding the R, G, and B channels of the RGB image into a hyperspectral image includes:

[0022] Three curves are set, the horizontal coordinates of the curves are the spectrum wavelengths, the vertical coordinates are the coefficients, and each curve is generated by superimposing 1-4 Gaussian curves with different parameters according to random probability;

[0023] The curves are again added with random noise and normalized, and the three curves correspond to the three image channels of R, G, and B in sequence according to the peak order;

[0024] Evenly sample wavelengths equal to the number of spectral channels on the three curves to obtain a coefficient of 3 times the number of wavelengths;

[0025] The three coefficients of each channel are multiplied by the corresponding R, G, and B original image pixels and then added together, and the result is used as the simulated spectral image at the corresponding wavelength.

[0026] Furthermore, the specific form of optimizing the solution by the alternating projection method in each sub-stage is:

[0027] x (t) =θ (t-1) +A T (AA T ) -1 (y-Aθ t-1 ),

[0028]

[0029] Among them, x represents the restored image, θ is the projection auxiliary variable, A is the sensing matrix, y represents the initial image input to the algorithm, t represents the tth sub-stage, and n represents that the input image is divided into n parts based on the region.

[0030] Furthermore, the neural network based on the regional self-attention mechanism is used for modeling and learning, and the construction of the recovery algorithm model specifically includes the following steps:

[0031] Construct a shallow feature coding network module, which includes three convolution layers, three batch normalization layers, three activation function layers, and one downsampling layer. The convolution kernel sizes of the three convolution layers are 3×3, 3×3, and 1×1, respectively. The activation function layer uses a LeakyReLU layer, and the downsampling layer uses a maximum pooling layer with a downsampling factor of 2. When outputting, the shallow feature coding network module divides the image feature map into blocks according to a preset spatial size to obtain a block area, and the block area includes several spatial positions;

[0032] Construct a spatially guided spectral attention module. The data unit size processed by the spatially guided spectral attention module is the regional data divided by the shallow feature encoding network module. The spatially guided spectral attention module includes a self-attention layer and a feedforward neural network layer. The attention layer aggregates the spatial features of each block area as the feature representation of the spectral channel, and then calculates the attention between the spectral channels and reorganizes the features. The obtained results are sent to the feedforward neural network for processing;

[0033] Construct a spectrally guided spatial attention module. The input of the spectrally guided spatial attention module is the output of the spatially guided spectral attention module. The spectrally guided spatial attention module includes a self-attention layer and a feedforward neural network layer. The spatially guided spectral attention module aggregates all spectral features of each spatial position as the feature representation of the spatial position, calculates attention between spatial positions and reorganizes features, and the obtained results are sent to the feedforward neural network for processing;

[0034] Constructing a feature cross-region propagation module, which re-shifts and divides the block regions so that each new block region covers a plurality of old block regions;

[0035] Repeatedly construct the spatially guided spectral attention module and the spectrally guided spatial attention module and connect them after the feature cross-region propagation module to establish feature connections between regions;

[0036] Construct a feature decoding network module, which restores the region to the entire image and then processes the complete feature image. The feature decoding network module includes two convolutional layers, two batch normalization layers, two activation function layers and a deconvolution layer. The convolution kernel sizes of the convolution layer are 3×3 and 1×1 respectively, the activation function layer uses the LeakyReLU layer, the deconvolution kernel size is 2, and the step size is 2.

[0037] Furthermore, the setting of training parameters and loss function strategy, training and verifying the constructed recovery algorithm model specifically includes:

[0038] The network optimizer selected the Adam optimizer, with an initial learning rate of 0.0004, a cosine function to decay the learning rate, 1000 training epochs, and a batch size of 8. The RMSE loss function used for the restored image after the alternating projection stage is:

[0039]

[0040] Among them, x is the real image and x' is the predicted image.

[0041] On the other hand, the present invention also provides a hyperspectral snapshot image restoration device, the device comprising:

[0042] An image acquisition module is used to acquire a spectral image dataset and write a data interface required by a convolutional neural network according to the data features of the spectral image dataset;

[0043] A training and testing set construction module, used to divide the spectral image data set into several parts to construct a training set and a testing set;

[0044] An input construction module is used to use a random mask to superimpose the training set, set channel pixels, and tilt the image and then superimpose it in the spectral dimension to obtain an aliased image, use the aliased image as input, and the training set as data to be fitted to train the model end-to-end;

[0045] A recovery algorithm building module, for building a recovery algorithm in the form of deep iterative expansion, wherein the algorithm includes a plurality of formally repeated sub-stages, each of which optimizes a solution by an alternating projection method to gradually approach a desired recovery solution;

[0046] The model building module is used to use the neural network based on the regional self-attention mechanism for modeling and learning, and to build a recovery algorithm model;

[0047] The training and verification module is used to set the training parameters and loss function strategy, and to train and verify the constructed recovery algorithm model;

[0048] The image restoration module is used to restore the compressed sensing image into a hyperspectral image through the restoration algorithm model.

[0049] Optionally, the image acquisition module is also used to preprocess the spectral image data set, specifically including:

[0050] During training, image patches with fixed spatial resolution are randomly cropped from the original image.

[0051] Performing data enhancement on the image block, wherein the data enhancement includes random flipping, random rotation, random block re-joining and / or adding random Gaussian noise;

[0052] A number of spectral channels are uniformly selected within a preset range length, and if the spectral channels of the data of the spectral image data set do not match the selected spectral channels, an interpolation method is used to obtain the spectral channels.

[0053] Optionally, the spectral image dataset acquired by the image acquisition module includes a public dataset and a simulated spectral image dataset, wherein the acquisition method of the simulated spectral image dataset specifically includes:

[0054] A plurality of RGB images are acquired, and R, G, and B channels of the RGB images are expanded into a hyperspectral spectrum.

[0055] Optionally, the method in which the image acquisition module expands the R, G, and B channels of the RGB image into a hyperspectral spectrum includes:

[0056] Three curves are set, the horizontal coordinates of the curves are the spectrum wavelengths, the vertical coordinates are the coefficients, and each curve is generated by superimposing 1-4 Gaussian curves with different parameters according to random probability;

[0057] The curves are again added with random noise and normalized, and the three curves correspond to the three image channels of R, G, and B in sequence according to the peak order;

[0058] Evenly sample wavelengths equal to the number of spectral channels on the three curves to obtain a coefficient of 3 times the number of wavelengths;

[0059] The three coefficients of each channel are multiplied by the corresponding R, G, and B original image pixels and then added together, and the result is used as the simulated spectral image at the corresponding wavelength.

[0060] Optionally, the recovery algorithm building module optimizes the solution in each sub-stage by an alternating projection method. The specific form is:

[0061] x (t) =θ (t-1) +A T (AA T ) -1 (y-Aθ t-1 ),

[0062]

[0063] Among them, x represents the restored image, θ is the projection auxiliary variable, A is the sensing matrix, y represents the initial image input to the algorithm, t represents the tth sub-stage, and n represents that the input image is divided into n parts based on the region.

[0064] Optionally, the model building module uses a neural network based on a regional self-attention mechanism for modeling and learning, and building a recovery algorithm model specifically includes the following steps:

[0065] Construct a shallow feature coding network module, which includes three convolution layers, three batch normalization layers, three activation function layers, and one downsampling layer. The convolution kernel sizes of the three convolution layers are 3×3, 3×3, and 1×1, respectively. The activation function layer uses a LeakyReLU layer, and the downsampling layer uses a maximum pooling layer with a downsampling factor of 2. When outputting, the shallow feature coding network module divides the image feature map into blocks according to a preset spatial size to obtain a block area, and the block area includes several spatial positions;

[0066] Construct a spatially guided spectral attention module. The data unit size processed by the spatially guided spectral attention module is the regional data divided by the shallow feature encoding network module. The spatially guided spectral attention module includes a self-attention layer and a feedforward neural network layer. The attention layer aggregates the spatial features of each block area as the feature representation of the spectral channel, and then calculates the attention between the spectral channels and reorganizes the features. The obtained results are sent to the feedforward neural network for processing;

[0067] Construct a spectrally guided spatial attention module. The input of the spectrally guided spatial attention module is the output of the spatially guided spectral attention module. The spectrally guided spatial attention module includes a self-attention layer and a feedforward neural network layer. The spatially guided spectral attention module aggregates all spectral features of each spatial position as the feature representation of the spatial position, calculates attention between spatial positions and reorganizes features, and the obtained results are sent to the feedforward neural network for processing;

[0068] Constructing a feature cross-region propagation module, which re-shifts and divides the block regions so that each new block region covers a plurality of old block regions;

[0069] Repeatedly construct the spatially guided spectral attention module and the spectrally guided spatial attention module and connect them after the feature cross-region propagation module to establish feature connections between regions;

[0070] Construct a feature decoding network module, which restores the region to the entire image and then processes the complete feature image. The feature decoding network module includes two convolutional layers, two batch normalization layers, two activation function layers and a deconvolution layer. The convolution kernel sizes of the convolution layer are 3×3 and 1×1 respectively, the activation function layer uses the LeakyReLU layer, the deconvolution kernel size is 2, and the step size is 2.

[0071] Optionally, the training verification module sets training parameters and loss function strategies, and trains and verifies the constructed recovery algorithm model, specifically including:

[0072] The network optimizer selected the Adam optimizer, with an initial learning rate of 0.0004, a cosine function to decay the learning rate, 1000 training epochs, and a batch size of 8. The RMSE loss function used for the restored image after the alternating projection stage is:

[0073]

[0074] Among them, x is the real image and x' is the predicted image.

[0075] On the other hand, the present invention further provides a computer device, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement any one of the above-mentioned hyperspectral snapshot image restoration methods.

[0076] On the other hand, the present invention further provides a computer-readable storage medium, wherein a computer program is stored in the storage medium, and the computer program is loaded and executed by a processor to implement any one of the above-mentioned hyperspectral snapshot image restoration methods.

[0077] The beneficial effects of the present invention are:

[0078] (1) Based on the intrinsic characteristics of hyperspectral image data, the present invention constructs a region-based restoration algorithm, replacing global modeling with regional modeling, thus overcoming the disadvantage of unstable global features.

[0079] (2) The overall architecture of the algorithm of the present invention adopts a deep iterative expansion form. Each sub-stage optimizes the solution by the alternating projection method, gradually approaching the desired recovery solution, introducing the spectrally guided spatial attention module and the spatially guided spectral attention module to model the regional spectral-spatial correlation prior, and guiding the network to restore the original hyperspectral image information with high precision and high efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a schematic diagram of the process of a hyperspectral snapshot image restoration method provided by an embodiment of the present invention;

[0081] Figure 2 It is a schematic diagram of the process of using the restoration algorithm model constructed in the embodiment of the present invention for hyperspectral computational imaging;

[0082] Figure 3 It is a structural block diagram of a hyperspectral snapshot image restoration device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0083] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0084] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0085] Existing hyperspectral snapshot image restoration methods are mainly divided into traditional iterative algorithms and deep learning algorithms. Most of these methods are based on unstable global spectral characteristics and do not make good use of the spectral-spatial correlation unique to hyperspectral data.

[0086] The traditional restoration algorithm of the CASSI system uses an iterative optimization algorithm to solve the inverse problem of compressed sensing. This type of algorithm takes a long time to calculate and has relatively low recovery accuracy. End-to-end solutions that directly use end-to-end neural networks for recovery lack interpretability and have poor generalization of data. Plug-and-play solutions that use denoising networks to replace denoisers in traditional iterative algorithms still require dozens of iterative optimization processes, have slow running speeds, and are difficult to achieve real-time processing of image data.

[0087] In order to solve the above technical problems, the following embodiments of the hyperspectral snapshot image restoration method, device, equipment and medium of the present invention are proposed.

[0088] Example 1

[0089] The hyperspectral snapshot image restoration method provided in this embodiment is a region-based spectral-spatial mutual inductance restoration algorithm, which uses an interpretable deep network to implement some mathematical processes in the traditional iterative algorithm and expands the iterations. It has obvious advantages in running speed, restoration accuracy, network interpretability, etc.

[0090] This embodiment is based on coded aperture snapshot spectral imaging and hyperspectral computational imaging of self-attention neural network. The coded aperture snapshot spectral imaging system is used to image the target to be measured, and the modulated image received by the imaging detector is restored to a hyperspectral image using a neural network model based on the self-attention mechanism. The steps include:

[0091] A. The target is imaged by a coded aperture snapshot spectral imaging system, in which the original spatial intensity information and original spectral information of the target are modulated separately by the system. The imaging detector can receive a compressed sensing image containing rich spectral information after one imaging, which reduces the cost of the hyperspectral imaging system and improves the imaging efficiency.

[0092] B. The compressed sensing image is sent into the neural network model, and the characteristics of the neural network are used to restore the compressed sensing image, which was originally in an aliased form, into a hyperspectral image. The specific implementation method is to use the generalized alternating projection algorithm combined with the neural network structure to optimize the restored image stage by stage, where each stage contains a sub-network. The sub-network uses the self-attention mechanism to learn the spatial spectral correlation of the input data and adaptively model the spatial spectral prior, thereby effectively improving the restoration quality.

[0093] C. For the designed neural network model, the real hyperspectral image data and synthetic simulation data are used for training. The trained network is applied to the system to obtain a complete computational spectral imaging system. Usually, the acquisition cost of real hyperspectral data is high and time-consuming, so there is less real data available for network training, which limits the performance of the network. Therefore, synthetic simulation data is used to increase the amount of network training data. The specific method is to expand the values ​​of the three channels of R, G, and B into dozens of channels according to the assumed spectral response curve based on the easily available ordinary RGB images and imitate the spectral characteristics of natural objects, and introduce a certain degree of randomness to form pseudo-hyperspectral image data, thereby increasing the amount of data available for training and improving the performance of the network.

[0094] Reference Figure 1 ,like Figure 1 FIG. 2 is a flow chart of a hyperspectral snapshot image restoration method provided in this embodiment, and the method specifically includes:

[0095] Step S100: Acquire a spectral image dataset, and write a data interface required by a convolutional neural network according to the data features of the spectral image dataset.

[0096] Specifically, the public data sets are preprocessed, and the data interface required by the convolutional neural network is written based on the data features such as the number of spectral bands and image resolution of the public data sets CAVE and KAIST.

[0097] It should be noted that, due to the high resolution of the images in the public data set, direct use will bring a large memory overhead during the training process. Therefore, this embodiment first designs an image cropping processing method, that is, randomly cropping an image block with a spatial resolution of 256×256 from the original large-resolution image during training, and then performing data enhancement operations such as random flipping, random rotation, random block re-stitching and / or adding random Gaussian noise on the cropped image to improve the robustness of the trained network. For the hyperspectral channel, 28 spectral channels are uniformly selected between 450-650nm. If the data spectral channel of the public data set does not match these 28 channels, the interpolation method is used to obtain it. For the number of hyperspectral channels to be restored, the more the number of restorations, the greater the difficulty of restoration. The relevant application scenarios of hyperspectral imaging usually require more than 20 spectral channels. Therefore, considering the performance and recovery capability, 28 spectral wavelengths are uniformly selected between 450-650nm as the recovery wavelengths, corresponding to 28 spectral channels.

[0098] As an implementation method, for deep learning algorithms in this field, due to the small number of public hyperspectral data sets and the single image content, the design and training of the network are subject to great restrictions. For example, simply using public data sets for training will lead to poor generalization and robustness of the algorithm. Therefore, this embodiment additionally provides a set of hyperspectral data generation methods, which can expand RGB images into hyperspectral images for network training.

[0099] Methods for expanding the R, G, and B channels of an RGB image to a hyperspectral image include:

[0100] a. Set three curves, the horizontal axis of the curve is the spectrum wavelength, the vertical axis is the coefficient, and each curve is generated by superimposing 1-4 Gaussian curves with different parameters according to random probability;

[0101] b. The curves are normalized by adding random noise again. The three curves correspond to the three image channels of R, G, and B according to the peak order.

[0102] c. Evenly sample wavelengths equal to the number of spectral channels on the three curves to obtain a coefficient that is 3 times the number of wavelengths;

[0103] d. Multiply the three coefficients of each channel with the corresponding R, G, and B original image pixels and then add them together. The result is the simulated spectral image at the corresponding wavelength.

[0104] As an implementation method, this embodiment obtains more than 5,000 RGB images of various types through the Internet and other channels, and expands their R, G, and B channels into a hyperspectral spectrum:

[0105] Three curves are set, with the horizontal axis of the curve being the spectral wavelength and the vertical axis being the coefficient. Each curve is generated by superimposing 1-4 Gaussian curves with different parameters according to random probability. The generated curve will be added with random noise again and normalized. The three curves correspond to the three image channels of R, G, and B in sequence according to the peak order. 28 wavelengths are uniformly sampled on the three curves, corresponding to the 28 channels selected in the previous steps, to obtain 28×3 coefficients. The 3 coefficients of each channel are multiplied by the corresponding R, G, and B original image pixels and then added together. The result is used as the simulated spectral image at that wavelength.

[0106] The acquired simulated spectral images were also preprocessed according to the aforementioned method and mixed with the public dataset.

[0107] Step S200: Split the spectral image data set into several parts to construct a training set and a test set.

[0108] As an implementation method, this embodiment constructs a training set and a test set with the final complete data set (including a public data set and simulated spectral images), splits the data set, and divides the data set into ten equal parts to implement network training and network testing using the leave-one-out cross-validation method in the subsequent training process.

[0109] Step S300: Use a random mask to superimpose the training set, set the channel pixels, tilt the image, and then superimpose it in the spectral dimension to obtain an aliased image. Use the aliased image as input and the training set as the data to be fitted to train the model end-to-end.

[0110] This step is used to construct the input of the system. In this embodiment, the image data of 256×256×28 pixels is processed according to the process of the CASSI system (Coded Aperture Snapshot Spectral Imaging System), that is, a 256×256 random mask is first used to superimpose the image, and then the step size / channel is set to 2 pixels, and the image is tilted, representing the prism spectrometry step in the CASSI system. Finally, the data is superimposed in the spectral dimension to obtain a 310×256 pixel aliased image, which is used as the input of the recovery algorithm. The 256×256×28 pixel image before processing is used as the data to be fitted, and the algorithm model is trained end-to-end.

[0111] Step S400: constructing a recovery algorithm in the form of deep iterative expansion, the algorithm includes a plurality of sub-stages that are repeated in form, each sub-stage optimizes the solution by an alternating projection method, and gradually approaches the desired recovery solution.

[0112] As an implementation method, this embodiment constructs an algorithm in the form of deep iterative expansion. The algorithm includes multiple sub-stages that are repeated in form. In this embodiment, when the number of sub-stages is 4, the best effect is achieved. Each sub-stage optimizes the solution by the alternating projection method and gradually approaches the desired recovery solution. The alternating projection form of each sub-stage is

[0113] x (t) =θ (t-1) +A T (AA T ) -1 (y-Aθ t-1 ),

[0114]

[0115] Among them, x represents the restored image, θ is the projection auxiliary variable, A is the sensor matrix, y represents the detector measurement value, that is, the initial aliased image input to the depth expansion algorithm, t represents the tth sub-stage, and n represents that the input image is divided into n parts based on the region.

[0116] The input image is divided into n parts based on the region, in order to apply a more stable regional spectral-spatial prior f(·).

[0117] Step S500: Use a neural network based on a regional self-attention mechanism to perform modeling and learning to build a recovery algorithm model.

[0118] The specific form of the regional spectral-spatial prior f(·) is modeled and learned using a neural network based on the regional self-attention mechanism. The specific steps are as follows:

[0119] Step S501: Construct a shallow feature encoding network module. The module directly processes the entire image. The module includes three convolutional layers, three batch normalization layers, three activation function layers, and a downsampling layer. The convolution kernel sizes of the three convolutional layers are 3×3, 3×3, and 1×1 respectively. The activation function layer uses the LeakyReLU layer. The downsampling layer uses the maximum pooling layer with a downsampling factor of 2. The input data size is 256×256×28, and the output data space size is 128×128×180, where 180 is the number of channels. At the same time, when the module outputs, the image feature map is divided into blocks according to the spatial size of 8×8, and the data size becomes 256×8×8×180.

[0120] Step S502: Construct a spatially guided spectral attention module. The data unit size processed by the module is 8×8×180, that is, the regional data divided by the previous module. At the same time, the module processes all 256 regions of an image in parallel. This module adopts the Transformer network structure, including a self-attention layer and a feedforward neural network layer. The aggregated 8×8 spatial features of the attention layer are used as the feature representation of the spectral channel. After that, the attention is calculated and the features are reorganized between the spectral channels. The results are sent to the feedforward neural network composed of a multi-layer perceptron for processing. Due to the characteristics of the Transformer structure, the final output size is the same as the input.

[0121] Step S503: Construct a spectrally guided spatial attention module. This module also adopts a Transformer network structure similar to the previous module. The input of the attention layer is the output of the previous module. It aggregates all spectral features of each spatial position (each area includes 8×8 spatial positions) as the feature representation of the spatial position, and then calculates the attention between the spatial positions and reorganizes the features. The results are sent to a feedforward neural network composed of a multi-layer perceptron for processing. The final features obtained incorporate spectral-spatial correlation.

[0122] Step S504: construct a feature cross-region propagation module, which re-shifts and divides the 8×8 region to ensure that each new region covers multiple old regions.

[0123] Step S505: Repeat the construction of the two attention modules in step S502 and step S503, and connect them to the feature cross-region propagation module to further establish feature connections between regions.

[0124] Step S506: Construct a feature decoding network module, which corresponds to the shallow feature encoding module. First, the region is restored to the entire image, and then the complete feature image is processed. The module includes two convolutional layers, two batch normalization layers, two activation function layers, and a deconvolution layer. The convolution kernel size of the convolution layer is 3×3 and 1×1 respectively, the activation function layer uses the LeakyReLU layer, the deconvolution kernel size is 2, the step size is 2, that is, the upsampling coefficient is 2, and the final module output is 256*256*28, that is, the formula θ in .

[0125] Reference Figure 2 ,like Figure 2 Shown is a schematic diagram of the process of using the restoration algorithm model constructed in this embodiment for hyperspectral computational imaging.

[0126] Step S600: Setting training parameters and loss function strategy, training and verifying the constructed recovery algorithm model.

[0127] As an implementation method, this embodiment sets the training parameters and loss function strategy. The network optimizer selects the Adam optimizer, the initial learning rate is 0.0004, the cosine function is used to decay the learning rate, the training period is set to 1000, and the batch size is 8. The RMSE loss function used for the restored image obtained after 4 alternating projection stages is

[0128]

[0129] Where x is the real image and x' is the predicted image.

[0130] The algorithm network after training is completed and verified on the test set is the hyperspectral snapshot compression imaging restoration algorithm.

[0131] Step S700: restoring the compressed sensing image to a hyperspectral image through a restoration algorithm model.

[0132] It should be noted that, in this embodiment, the constructed hyperspectral snapshot compression imaging restoration algorithm is combined with the coded aperture snapshot spectral imaging system to restore the compressed sensing image into a hyperspectral image.

[0133] The hyperspectral snapshot image restoration method of this embodiment constructs a region-based restoration algorithm, replacing global modeling with region-based modeling, and overcomes the disadvantage of unstable global features. The overall architecture of the algorithm constructed by this method adopts a deep iterative expansion form. Each sub-stage optimizes the solution by alternating projection method, gradually approaches the desired restoration solution, introduces spectrally guided spatial attention module and spatially guided spectral attention module, models regional spectral-spatial correlation prior, and guides the network to restore the original hyperspectral image information with high precision and high efficiency.

[0134] Example 2

[0135] Reference Figure 3 ,like Figure 3 FIG. 1 is a block diagram of a hyperspectral snapshot image restoration device provided in this embodiment, and the device includes:

[0136] An image acquisition module 10 is used to acquire a spectral image data set and write a data interface required by a convolutional neural network according to the data features of the spectral image data set;

[0137] A training and testing set construction module 20 is used to divide the spectral image data set into several parts to construct a training set and a test set;

[0138] An input construction module 30 is used to superimpose the training set with a random mask, set channel pixels, and superimpose the image in the spectral dimension after tilt processing to obtain an aliased image, using the aliased image as input and the training set as data to be fitted, and an end-to-end training model;

[0139] A recovery algorithm construction module 40, for constructing a recovery algorithm in the form of deep iterative expansion, the algorithm comprising a plurality of formally repeated sub-stages, each of which optimizes the solution by an alternating projection method to gradually approach the desired recovery solution;

[0140] A model building module 50 is used to use a neural network based on a regional self-attention mechanism to perform modeling learning and build a recovery algorithm model;

[0141] The training and verification module 60 is used to set training parameters and loss function strategies, and train and verify the constructed recovery algorithm model;

[0142] The image restoration module 70 is used to restore the compressed sensing image into a hyperspectral image through a restoration algorithm model.

[0143] As an implementation manner, the image acquisition module 10 is also used to preprocess the spectral image data set, specifically including:

[0144] During training, image patches with fixed spatial resolution are randomly cropped from the original image.

[0145] Perform data augmentation on the image blocks, including random flipping, random rotation, random block re-stitching and / or adding random Gaussian noise;

[0146] A number of spectral channels are uniformly selected within a preset range length. If the spectral channels of the data of the spectral image dataset do not match the selected spectral channels, the spectral channels are obtained by interpolation.

[0147] As an implementation manner, the spectral image dataset acquired by the image acquisition module 10 includes a public dataset and a simulated spectral image dataset, wherein the acquisition method of the simulated spectral image dataset specifically includes:

[0148] Get several RGB images and expand the R, G, and B channels of the RGB images to a hyperspectral spectrum.

[0149] As an implementation manner, the method in which the image acquisition module 10 expands the R, G, and B channels of the RGB image into a hyperspectral image includes:

[0150] Three curves are set, the horizontal axis of the curve is the spectrum wavelength, the vertical axis is the coefficient, and each curve is generated by superimposing 1-4 Gaussian curves with different parameters according to random probability;

[0151] The curves are again added with random noise and normalized. The three curves correspond to the three image channels of R, G, and B according to the peak order;

[0152] Evenly sample wavelengths equal to the number of spectral channels on the three curves to obtain a coefficient of 3 times the number of wavelengths;

[0153] The three coefficients of each channel are multiplied by the corresponding R, G, and B original image pixels and then added together, and the result is used as the simulated spectral image at the corresponding wavelength.

[0154] As an implementation method, the restoration algorithm building module 40 optimizes the solution in each sub-stage by using the alternating projection method. The specific form is:

[0155] x (t) =θ (t-1) +A T (AA T ) -1 (y-Aθ t-1 ),

[0156]

[0157] Among them, x represents the restored image, θ is the projection auxiliary variable, A is the sensing matrix, y represents the initial image input to the algorithm, t represents the tth sub-stage, and n represents that the input image is divided into n parts based on the region.

[0158] As an implementation method, the model building module 50 uses a neural network based on a regional self-attention mechanism to perform modeling and learning, and constructing a recovery algorithm model specifically includes the following steps:

[0159] Construct a shallow feature coding network module, which includes three convolutional layers, three batch normalization layers, three activation function layers, and one downsampling layer. The convolution kernel sizes of the three convolutional layers are 3×3, 3×3, and 1×1 respectively. The activation function layer uses the LeakyReLU layer, and the downsampling layer uses the maximum pooling layer with a downsampling factor of 2. When outputting, the shallow feature coding network module divides the image feature map into blocks according to the preset spatial size to obtain block areas, which include several spatial positions.

[0160] Construct a spatially guided spectral attention module. The data unit size processed by the spatially guided spectral attention module is the regional data divided by the shallow feature encoding network module. The spatially guided spectral attention module includes a self-attention layer and a feedforward neural network layer. The attention layer aggregates the spatial features of each block area as the feature representation of the spectral channel, and then calculates the attention between the spectral channels and reorganizes the features. The obtained results are sent to the feedforward neural network for processing;

[0161] Construct a spectrally guided spatial attention module. The input of the spectrally guided spatial attention module is the output of the spatially guided spectral attention module. The spectrally guided spatial attention module includes a self-attention layer and a feedforward neural network layer. The spatially guided spectral attention module aggregates all spectral features of each spatial position as the feature representation of the spatial position, calculates attention between spatial positions and reorganizes features, and the obtained results are sent to the feedforward neural network for processing;

[0162] Construct a feature cross-region propagation module, which re-shifts and divides the block areas so that each new block area covers multiple old block areas;

[0163] Repeatedly construct the spatially guided spectral attention module and the spectrally guided spatial attention module and connect them after the feature cross-region propagation module to establish feature connections between regions;

[0164] Construct a feature decoding network module, which restores the region to the entire image and then processes the complete feature image. The feature decoding network module includes two convolutional layers, two batch normalization layers, two activation function layers and a deconvolution layer. The convolution kernel sizes of the convolution layer are 3×3 and 1×1 respectively, the activation function layer uses the LeakyReLU layer, the deconvolution kernel size is 2, and the step size is 2.

[0165] As an implementation method, the training verification module 60 sets training parameters and loss function strategies, and trains and verifies the constructed recovery algorithm model, specifically including:

[0166] The network optimizer selected the Adam optimizer, with an initial learning rate of 0.0004, a cosine function to decay the learning rate, 1000 training epochs, and a batch size of 8. The RMSE loss function used for the restored image after the alternating projection stage is:

[0167]

[0168] Among them, x is the real image and x' is the predicted image.

[0169] The hyperspectral snapshot image restoration device provided in this embodiment constructs a region-based restoration algorithm, replacing global modeling with region-based modeling, and overcomes the disadvantage of unstable global features. The overall architecture of the algorithm constructed by the device adopts a deep iterative expansion form, and each sub-stage optimizes the solution by alternating projection method, gradually approaches the desired restoration solution, introduces spectrally guided spatial attention module and spatially guided spectral attention module, models regional spectral-spatial correlation prior, and guides the network to restore the original hyperspectral image information with high precision and high efficiency.

[0170] Example 3

[0171] This preferred embodiment provides a computer device, which can implement the steps in any embodiment of the hyperspectral snapshot image restoration method provided in the embodiments of the present application. Therefore, the beneficial effects of the hyperspectral snapshot image restoration method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0172] Example 4

[0173] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium, which stores a plurality of instructions, and the instructions can be loaded by a processor to execute the steps of any embodiment of the hyperspectral snapshot image restoration method provided in the embodiment of the present invention.

[0174] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0175] Since the instructions stored in the storage medium can execute the steps in any hyperspectral snapshot image restoration method embodiment provided in the embodiments of the present invention, the beneficial effects that can be achieved by any hyperspectral snapshot image restoration method provided in the embodiments of the present invention can be achieved. For details, please refer to the previous embodiments and will not be repeated here.

[0176] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A hyperspectral snapshot image restoration method, characterized in that: The method comprises: Acquire a spectral image data set, and write a data interface required by a convolutional neural network according to data features of the spectral image data set; Splitting the spectral image data set into several parts to construct a training set and a test set; Using a random mask to superimpose the training set, setting channel pixels, and performing tilt processing on the image and then superimposing it in the spectral dimension to obtain an aliased image, using the aliased image as input and the training set as data to be fitted, and training the model end-to-end; The recovery algorithm is constructed in the form of deep iterative expansion, the algorithm includes multiple formally repeated sub-stages, each sub-stage optimizes the solution by alternating projection method, and gradually approaches the desired recovery solution; The specific form of optimizing the solution by the alternating projection method in each sub-stage is: x (t) =θ (t-1) +A T (AA T ) -1 (y-Aθ t-1 ), Where x represents the restored image, θ is the projection auxiliary variable, A is the sensing matrix, y represents the initial image of the input algorithm, t represents the tth sub-stage, and n represents that the input image is divided into n parts based on the region; Use a neural network based on regional self-attention mechanism for modeling and learning to build a recovery algorithm model; The neural network based on the regional self-attention mechanism is used for modeling and learning, and the construction of the recovery algorithm model specifically includes the following steps: Construct a shallow feature coding network module, which includes three convolution layers, three batch normalization layers, three activation function layers, and one downsampling layer. The convolution kernel sizes of the three convolution layers are 3×3, 3×3, and 1×1, respectively. The activation function layer uses a LeakyReLU layer, and the downsampling layer uses a maximum pooling layer with a downsampling factor of 2. When outputting, the shallow feature coding network module divides the image feature map into blocks according to a preset spatial size to obtain a block area, and the block area includes several spatial positions; Construct a spatially guided spectral attention module. The data unit size processed by the spatially guided spectral attention module is the regional data divided by the shallow feature encoding network module. The spatially guided spectral attention module includes a self-attention layer and a feedforward neural network layer. The attention layer aggregates the spatial features of each block area as the feature representation of the spectral channel, and then calculates the attention between the spectral channels and reorganizes the features. The obtained results are sent to the feedforward neural network for processing; Construct a spectrally guided spatial attention module. The input of the spectrally guided spatial attention module is the output of the spatially guided spectral attention module. The spectrally guided spatial attention module includes a self-attention layer and a feedforward neural network layer. The spatially guided spectral attention module aggregates all spectral features of each spatial position as the feature representation of the spatial position, calculates attention between spatial positions and reorganizes features, and the obtained results are sent to the feedforward neural network for processing; Constructing a feature cross-region propagation module, which re-shifts and divides the block regions so that each new block region covers a plurality of old block regions; Repeatedly construct the spatially guided spectral attention module and the spectrally guided spatial attention module and connect them after the feature cross-region propagation module to establish feature connections between regions; Construct a feature decoding network module, which restores the region to the entire image and then processes the complete feature image. The feature decoding network module includes two convolutional layers, two batch normalization layers, two activation function layers and a deconvolution layer. The convolution kernel sizes of the convolution layer are 3×3 and 1×1 respectively, the activation function layer uses the LeakyReLU layer, the deconvolution kernel size is 2, and the step size is 2; Set training parameters and loss function strategies to train and verify the constructed recovery algorithm model; The compressed sensing image is restored to a hyperspectral image through the restoration algorithm model.

2. The hyperspectral snapshot image restoration method according to claim 1, characterized in that: The method further comprises preprocessing the spectral image data set, specifically comprising: During training, image patches with fixed spatial resolution are randomly cropped from the original image. Performing data enhancement on the image block, wherein the data enhancement includes random flipping, random rotation, random block re-joining and / or adding random Gaussian noise; A number of spectral channels are uniformly selected within a preset range length, and if the spectral channels of the data of the spectral image data set do not match the selected spectral channels, an interpolation method is used to obtain the spectral channels.

3. The hyperspectral snapshot image restoration method according to claim 1, characterized in that: The spectral image dataset includes a public dataset and a simulated spectral image dataset, wherein the method for acquiring the simulated spectral image dataset specifically includes: A plurality of RGB images are acquired, and R, G, and B channels of the RGB images are expanded into a hyperspectral spectrum.

4. The hyperspectral snapshot image restoration method according to claim 3, characterized in that: The method for expanding the R, G, and B channels of the RGB image into a hyperspectral spectrum comprises: Three curves are set, the horizontal coordinates of the curves are the spectrum wavelengths, the vertical coordinates are the coefficients, and each curve is generated by superimposing 1-4 Gaussian curves with different parameters according to random probability; The curves are again added with random noise and normalized, and the three curves correspond to the three image channels of R, G, and B in sequence according to the peak order; Evenly sample wavelengths equal to the number of spectral channels on the three curves to obtain a coefficient of 3 times the number of wavelengths; The three coefficients of each channel are multiplied by the corresponding R, G, and B original image pixels and then added together, and the result is used as the simulated spectral image at the corresponding wavelength.

5. The hyperspectral snapshot image restoration method according to claim 1, characterized in that: The setting of training parameters and loss function strategy, training and verifying the constructed recovery algorithm model specifically includes: The network optimizer selected the Adam optimizer, with an initial learning rate of 0.0004, a cosine function to decay the learning rate, 1000 training epochs, and a batch size of 8. The RMSE loss function used for the restored image after the alternating projection stage is: Among them, x is the real image and x' is the predicted image.

6. A hyperspectral snapshot image restoration device, characterized in that: The device comprises: An image acquisition module is used to acquire a spectral image dataset and write a data interface required by a convolutional neural network according to the data features of the spectral image dataset; A training and testing set construction module, used to divide the spectral image data set into several parts to construct a training set and a testing set; An input construction module is used to use a random mask to superimpose the training set, set channel pixels, and tilt the image and then superimpose it in the spectral dimension to obtain an aliased image, use the aliased image as input, and the training set as data to be fitted to train the model end-to-end; A recovery algorithm building module, for building a recovery algorithm in the form of deep iterative expansion, wherein the algorithm includes a plurality of formally repeated sub-stages, each of which optimizes a solution by an alternating projection method to gradually approach a desired recovery solution; The specific form of optimizing the solution by the alternating projection method in each sub-stage is: x (t) =θ (t-1) +A T (AA T ) -1 (y-Aθ t-1 ), Where x represents the restored image, θ is the projection auxiliary variable, A is the sensing matrix, y represents the initial image of the input algorithm, t represents the tth sub-stage, and n represents that the input image is divided into n parts based on the region; The model building module is used to use the neural network based on the regional self-attention mechanism for modeling and learning, and to build a recovery algorithm model; The neural network based on the regional self-attention mechanism is used for modeling and learning, and the construction of the recovery algorithm model specifically includes the following steps: Construct a shallow feature coding network module, which includes three convolution layers, three batch normalization layers, three activation function layers, and one downsampling layer. The convolution kernel sizes of the three convolution layers are 3×3, 3×3, and 1×1, respectively. The activation function layer uses a LeakyReLU layer, and the downsampling layer uses a maximum pooling layer with a downsampling factor of 2. When outputting, the shallow feature coding network module divides the image feature map into blocks according to a preset spatial size to obtain a block area, and the block area includes several spatial positions; Construct a spatially guided spectral attention module. The data unit size processed by the spatially guided spectral attention module is the regional data divided by the shallow feature encoding network module. The spatially guided spectral attention module includes a self-attention layer and a feedforward neural network layer. The attention layer aggregates the spatial features of each block area as the feature representation of the spectral channel, and then calculates the attention between the spectral channels and reorganizes the features. The obtained results are sent to the feedforward neural network for processing; Construct a spectrally guided spatial attention module. The input of the spectrally guided spatial attention module is the output of the spatially guided spectral attention module. The spectrally guided spatial attention module includes a self-attention layer and a feedforward neural network layer. The spatially guided spectral attention module aggregates all spectral features of each spatial position as the feature representation of the spatial position, calculates attention between spatial positions and reorganizes features, and the obtained results are sent to the feedforward neural network for processing; Constructing a feature cross-region propagation module, which re-shifts and divides the block regions so that each new block region covers a plurality of old block regions; Repeatedly construct the spatially guided spectral attention module and the spectrally guided spatial attention module and connect them after the feature cross-region propagation module to establish feature connections between regions; Construct a feature decoding network module, which restores the region to the entire image and then processes the complete feature image. The feature decoding network module includes two convolutional layers, two batch normalization layers, two activation function layers and a deconvolution layer. The convolution kernel sizes of the convolution layer are 3×3 and 1×1 respectively, the activation function layer uses the LeakyReLU layer, the deconvolution kernel size is 2, and the step size is 2; The training and verification module is used to set the training parameters and loss function strategy, and to train and verify the constructed recovery algorithm model; The image restoration module is used to restore the compressed sensing image into a hyperspectral image through the restoration algorithm model.

7. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the hyperspectral snapshot image restoration method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is loaded and executed by a processor to implement the hyperspectral snapshot image restoration method according to any one of claims 1 to 5.

Images