Computational spectral imaging error correction method based on deep learning

Abstract

The application discloses a method for correcting errors of computational spectral imaging based on deep learning, which comprises the following steps: analyzing and modeling errors of a low-resolution spectral image acquired by an imaging system; building a neural network to correct the errors of the low-resolution spectral image; taking the corrected low-resolution spectral image and an RGB image captured by a color camera of the imaging system as inputs, building a fusion reconstruction network of the images by a deep learning method, calculating spectrum reconstruction to obtain a preliminary reconstructed target image; degrading the target image by the imaging system, and performing constraint on the degraded image by means of actual observation, realizing joint optimization of a chromatic aberration correction network and a spectral fusion network, and realizing error correction of spectral imaging. The application is based on differentiable modeling of actual errors of the imaging system, and establishes an end-to-end joint optimization framework of joint observation extraction, error correction and spectral reconstruction, so that a huge gap between actual imaging quality and theoretical simulation quality can be bridged.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of spectral imaging error correction, and particularly relates to a method for correcting errors in computational spectral imaging based on deep learning. BACKGROUND

[0002] Multispectral data represents the intensity information of electromagnetic radiation on the surface of an object, and has a very wide application in the fields of biological medicine, geological exploration, forestry and agricultural monitoring, weather prediction, disaster reduction and prevention, and military target detection. However, compared with the vigorous development of theory, the engineering process of the computational spectral imaging system is seriously lagging behind. The classical computational spectral imaging adopts a two-imaging system, which encodes and modulates the scene on the primary image plane by using an encoding template, then moves the image plane and disperses by using a relay mirror and a dispersive device, and finally images on the secondary image plane by using a sensor. At this time, the output result of the sensor cannot be directly used, and it is still necessary to establish the geometric correspondence between the two image planes through tedious calibration operations, to extract the observation values from the sensor response by means of affine, cropping, interpolation and other basic image processing methods, and to complete the calculation and reconstruction of the scene spectral information by means of a neural network or an optimizer. The existence of errors in the imaging system will seriously restrict the quality of the calculation and reconstruction of the scene spectral information.

[0003] The existing engineering mainly adopts a manual one-by-one scene and one-by-one pixel calibration of the position of the response on the encoding template and the CCD detector, the purpose being to establish the geometric correspondence between the two image planes through tedious calibration operations, to extract the observation values from the sensor response by means of affine, cropping, interpolation and other basic image processing methods, but this method is greatly limited in actual application, and the calibration of remote sensing spectral images cannot be realized in practical application scenarios such as satellite, airborne and even vehicle-mounted, and it is also time-consuming and laborious.

[0004] In terms of computational spectral reconstruction, the traditional method for calculation and reconstruction is superior to some deep learning methods in terms of the performance on the test set and the real scene, but the traditional method for reconstruction has low efficiency, needs a complex calculation process, has high computational complexity, needs a large amount of computing resources, and has general reconstruction accuracy. With the popularity of deep learning, many people choose to optimize reconstruction by using a deep neural network, then the deep learning method based on the network achieves good results on the simulation training set, but the reconstruction effect on the actual spectral observation image is even worse than that of the traditional algorithm, they build a model for the simulation data set, iteratively train, and may overfit, resulting in poor generalization ability of the model.

[0005] At present, some reconstruction algorithms consider errors during reconstruction, and simply include estimation of point PSF when performing spectral reconstruction through neural networks, etc. A team from Northwestern Polytechnical University proposed a reconstruction algorithm for unsupervised image adaptive learning, proposed a reconstruction framework including two stages of pre-fusion and adaptive adjustment, considered the actual sampled PSF function and spectral sampling model during the late adaptive adjustment, MIAE is an unsupervised deep learning-based method, which is established on the basis of the target image implicit automatic encoder network, and proposes blind estimation of SRF and PSF.

[0006] However, these at most predict the PSF function, and are all based on the simple assumption that the PSF of each point in the full image is consistent to establish a linear model for estimation, without considering local deformation. However, the estimation of PSF is a point-by-point different nonlinear transformation in engineering. Secondly, in the observation extraction process, these methods are based on the assumption that the two image planes are aligned, and the observation is extracted manually, without considering the error in the observation extraction process. Although the above methods can correct errors to some extent, they are far from the error model in actual engineering. Therefore, for the problem of calculating spectral imaging, it is urgent to establish a complete error model for error analysis and correction, and it is urgent to propose a spectral reconstruction algorithm considering complex nonlinear errors. SUMMARY

[0007] In order to overcome the problems existing in the prior art, the purpose of the present application is to provide a deep learning-based computational spectral imaging error correction method, which is based on the differentiable modeling of the actual error of the imaging system, establishes an end-to-end joint optimization framework of joint observation extraction, error correction, and spectral reconstruction, and bridges the gap between the actual system imaging quality and the theoretical simulation quality.

[0008] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0009] The deep learning-based computational spectral imaging error correction method comprises the following steps:

[0010] Step (1): analyzing and modeling the error of the low-resolution spectral image obtained by the imaging system;

[0011] Step (2): building a neural network to correct the error of the low-resolution spectral image, i.e. offline calibration;

[0012] Step (3): taking the corrected low-resolution spectral image and the RGB image captured by the color camera of the imaging system as input, building an image fusion reconstruction network through deep learning method, and calculating the spectral reconstruction to obtain the preliminary reconstructed target image;

[0013] Step (4): The preliminary reconstructed target image is degraded by the imaging system, and the joint optimization of the chromatic aberration correction network and the spectral fusion network is realized by means of actual observation and constraint on the degraded image, so as to realize the spectral imaging error correction.

[0014] The step (1) is specifically:

[0015] The imaging system error is modeled, the actual spectral imaging system observation and the theoretical design model are obtained, and the composition principle of the error is analyzed.

[0016] (1.1) Obtain the actual engineering imaging system observation and analyze it;

[0017] Firstly, the imaging system is used to take pictures of the scene, and the actual observation on the CCD detector is extracted and compared with the theoretical design model. The actual observation image has pixel position offset and non-rigid diffusion phenomenon for each waveband pixel point.

[0018] (1.2) Model the displacement offset;

[0019] Firstly, the spectral image I M*N is sampled and combined:

[0020]

[0021] The spectral image is sparsely sampled and combined, where k*k is the number of sampling points;

[0022]

[0023] J is the offset image, and A is the affine transformation, wherein the degree of freedom dF(A) of A is 2.

[0024] (1.3) Model the point diffusion phenomenon;

[0025] For the high-spectral image I λ (x,y) to be reconstructed, the observed RGB image J c (x,y) can be represented as:

[0026] J c (x,y)=∫∫∫Ω c (λ)I λ (μ,v)p λ (x-μ,y-v)dμdvdλ

[0027] Where Ω c (λ) is the spectral response curve, c∈{r,g,b}, p λ(x, y) is a point spread function related to position and wavelength, and Δσ is used to describe the variance of the Gaussian distribution of the point spread function; using a parameterized Gaussian to estimate the PSF can increase the constraints between points;

[0028] (1.4) Establishment of the overall model;

[0029] 1) Establishment of the offset model: including horizontal position offset Δx and vertical position offset Δy, 2) Establishment of the diffusion model: the diffusion of a single pixel point is represented using a SPF function conforming to a Gaussian function, and a Gaussian parameterized by Δσ is used to fit the point diffusion process, σ representing the variance, and the error model is decomposed into the solutions of Δx, Δy, and Δσ.

[0030] The step (2) is specifically:

[0031] (2.1) Making a simulation data set;

[0032] A public data set is selected, which contains a CAVE data set of indoor scene hyperspectral images, including spectral images (HR-HSI) and corresponding color images (HR-MSI);

[0033] (2.2) Color difference correction through a network. The step (2.1) is specifically:

[0034] (2.1.1) Obtaining a low-resolution spectral image;

[0035] Using the rule commonly used in the field of remote sensing to make a low-resolution spectral image, first filtering the HR-HSI using a Gaussian blur kernel, then obtaining pixels in the row and column directions of the hyperspectral image with a step size of the downsampling multiple, and then performing maximum and minimum normalization to obtain a low-resolution spectral image (LR-HSI);

[0036] (2.1.2) Obtaining a CCD image;

[0037] For the obtained low-resolution spectral image, first a sampling template is defined; then the corresponding position point coordinates are restored to the arrangement on the CCD to obtain a CCD image according to the preset sampling template; then the wavelength band progressive tilt and Gaussian filtering operation are performed on the pixel points on the CCD according to the established model; for different wavebands of the same pixel point, the higher the wavelength band tilt, the lower the diffusion filtering; through this operation, the error formation process of the imaging system can be simulated; the adjacent pixel points are progressively transformed, and the step size of each change is half of the maximum tilt angle / pixel width, and the Gaussian filter kernel size linearly changes from 3-10 according to the number of wavebands;

[0038] The step (2.2) is specifically:

[0039] (2.2.1) Perform patch operation on the above-mentioned CCD error data set;

[0040] (2.2.2) Send the patch into the network, pass through LeakyRelu() as the activation function, and then pass through the pooling layer to obtain the feature map;

[0041] (2.2.3) Then input the obtained feature map into the network, the input channel number of the network is the output channel number of the previous layer, the output channel number is twice the input channel number, then pass through the activation function and the pooling layer, the activation function is LeakyRelu(), and a total of 5 convolutional networks are set;

[0042] (2.2.4) Perform twice full connection on the feature map output by the last convolutional network;

[0043] (2.2.5) Train by setting the loss function as mean square error loss MSE_LOSS to obtain the preliminary result after error correction;

[0044] (2.2.6) Save the corrected parameters of the network, and obtain the correction result of the whole image through forward inference.

[0045] The offline calibration process is to train the network by using the data set, realize the correction and adjustment of the network to the error, and the main advantages of correcting the error by the method of deep learning are a) the network can replace the tedious manual calibration process, manual calibration is tedious and time-consuming, and it is not achievable in vehicle-mounted and satellite-mounted application scenarios, and the offline calibration process realized by the network can save a lot of manpower and time, and also increase the application range of the spectral imaging system; b) the network can realize online recalibration in real scenes, and adjusting the parameters of the error extraction network by using the degradation model in real scenes can realize online fine adjustment of parameters for different scenes. The traditional calculation spectral imaging technology does not consider the imaging system error when reconstructing the spectral information, which leads to that the reconstructed spectral image performs well on the simulation data set, but the effect is often not good for real scenes. Taking the neural network as a breakthrough, the inevitable errors of the imaging system are corrected, which makes up for the huge gap between the actual system imaging quality and the theoretical simulation quality.

[0046] The step (3) is specifically:

[0047] According to the corrected low spatial resolution spectral image and the high spatial resolution RGB image, a high-resolution spectral image is obtained by hierarchical progressive fusion, and scene information is reconstructed.

[0048] (3.1) Firstly, the CAVE hyperspectral image set is preprocessed, the CAVE dataset is provided by Columbia University, containing 32 scenes, each scene contains 31 spectral channels, the channel images of the scene are divided into a plurality of small patches, then the UP_HSI is obtained by interpolation to the same multiple as the HR_MSI for the LR_HSI, and the UP_HSI and the HR_MSI are respectively taken as the input of the network;

[0049] (3.2) The channels taken out from the UP-HSI for the first time are combined with the channels of the HR_MSI, and the output is obtained through the image fusion sub-network, and is taken as the input of the next layer;

[0050] (3.3) The channels taken out from the UP-HSI for the second time are combined with the channels of the HR_MSI, and the output is obtained through the image fusion sub-network, and is taken as the input of the next layer;

[0051] (3.4) The channels taken out from the next layer are combined with the channels of the HR_MSI and the input of the fusion sub-network of the previous layer, and the output result obtained through the fusion sub-network is added to the UP-HIS pixel by pixel to obtain the output;

[0052] (3.4) The output of the previous is input into the purification network to obtain the final output result, the purification network includes four convolutional networks, and the output is the reconstructed HR-HIS;

[0053] (3.5) Save the network parameters.

[0054] The step (4) is specifically:

[0055] (4.1) Imaging system detection;

[0056] Using the imaging system detection, a low-resolution chromatic spectral image (LR-HSI) and a high spatial resolution RGB image (HR-MSI) obtained by an RGB camera are obtained;

[0057] (4.2) Error correction technology;

[0058] After preprocessing the low-resolution chromatic spectral image obtained by the imaging system, the offline calibrated network parameters pre-trained in step (2) are loaded, and the network parameters are fixed, and forward inference is performed, and the error of the low-resolution spectral image is preliminarily corrected;

[0059] (4.3) Training of spectral fusion network;

[0060] For the spectrum image (LR_HSI) after preliminary error correction and the high spatial resolution spectrum image obtained by the RGB camera, the low resolution spectrum image is linearly interpolated, which is taken as the input of the network together with the RGB image, and the network parameters of the reconstruction network saved in step (3) are used to obtain the target spectrum image (HR-HSI) of the preliminary reconstructed scene;

[0061] (4.4) Establishing a degradation model between space and spectrum;

[0062] The spatial degradation model is simulated by a convolution network, and the inter-spectral degradation is simulated by multiplying the camera response curve, so as to obtain the low spatial resolution and low spectral resolution images;

[0063] (4.4) Establishing a degradation model of imaging error;

[0064] The modeling process of the error degradation is shown in step (1), and the depth separable convolution and the deformable convolution are used to simulate the point spread function and the displacement offset process, respectively;

[0065] (4.5) Correcting the reconstruction result and the model parameters by online observation;

[0066] The preliminary reconstructed image is subjected to the degradation model in steps (4.2), (4.3) and (4.4) to obtain a spectrum image 1, an RGB image 2 and an error CCD spectrum image 3, the spectrum image 1 and the RGB image 2 are constrained by the loss function together with the LR-HSI and the HR-MSI, and the error CCD spectrum image 3 is constrained by the CCD image, and the network back propagation is used to jointly optimize the spectrum reconstruction algorithm and the error correction network; the self-supervised learning strategy is used to finely adjust the reconstruction network parameters and the error model parameters online, and the joint optimization of the reconstruction, online re-calibration and error correction is realized according to the online prior in the real scene.

[0067] The step (4.4) is specifically:

[0068] Point spread function simulation: the depth separable convolution is used to realize different point spread functions for different spectral channels of the scene, and even for the same pixel point, the point spread function simulated by the depth separable convolution is different, which is more in line with the characteristics of the actual imaging system;

[0069] Displacement simulation: the deformable convolution is used to simulate the displacement, and the input and output channel numbers of the deformable convolution remain unchanged, and the displacement estimation is performed for each point.

[0070] The beneficial effects of the present application are:

[0071] 1.The present application establishes a differentiable model for calculating the error of a spectral imaging system, which is used to parameterize the degradation process between the theoretical observation model and the actual observation model, and the mapping relationship from the phase plane to the sensor no longer needs to meet the "one-to-one alignment and rigid mapping" assumption, so the "observation extraction process" previously independent of the calculation spectral reconstruction network can be combined with the calculation reconstruction for joint training.

[0072] 2.The present application reduces the impact of poor spectral image quality caused by system error on the later calculation of spectral imaging, and the calculation spectral imaging system has a complex coupled system error problem, in which the alignment problem between the encoding template and the extracted observation value cannot be met for all pixels, and the chromatic aberration and aberration problems inevitably exist in the imaging system, which seriously restricts the calculation spectral imaging process, and the neural network is used to correct these errors of the imaging system, which not only improves the quality of the obtained spectral image, but also is beneficial to further improve the quality of the calculation reconstructed spectral image.

[0073] 3.The present application solves the problem of repeated and tedious manual calibration, which is one of the main bottlenecks of the current calculation spectral imaging system engineering, and the system parameters will be affected by temperature changes, vibration and other physical environments during actual use, which makes it necessary to repeat the tedious offline recalibration operation before each actual use, which is impossible in actual application scenarios such as satellite, airborne and even vehicle, and since the "manual calibration observation extraction process" has been replaced by network learning, the network training method greatly improves the efficiency of the observation extraction process.

[0074] 4.The present application solves the problem that the reconstruction quality of the calculation reconstruction is greatly different between the simulation data set and the real scene, and adopts a self-supervised learning strategy to fine-tune the reconstruction network parameters and error model parameters online, and realizes the joint optimization of calculation reconstruction, online recalibration and error correction according to the online prior in the real scene. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 is the observation image of the actual imaging system of the present application.

[0076] Figure 2 is the schematic diagram of the establishment of the whole model of the present application.

[0077] Figure 3 is the schematic diagram of the double-channel spectral imaging system based on DMD of the present application.

[0078] Figure 4 is the schematic diagram of offline calibration error correction of the present application.

[0079] Figure 5 is the joint error correction spectral image calculation reconstruction diagram of the present application. DETAILED DESCRIPTION

[0080] The application will be described in further detail below with reference to the drawings.

[0081] The application provides a deep learning-based calculation spectral imaging error correction algorithm; the following steps:

[0082] (1) Calculate the spectral imaging system error differentiable modeling:

[0083] In order to better extract the system observation from the system observation, the imaging system error needs to be modeled. Therefore, the actual spectral imaging system observation and the theoretical design model need to be obtained, and the composition principle of the error needs to be analyzed.

[0084] (1.1) Obtain the actual engineering imaging system observation and analyze it;

[0085] First, the imaging system is built to take pictures of the scene, extract the actual observation on the CCD detector, and compare and analyze it with the theoretical model, such as Figure 1 As shown in the actual observation image, the actual observation is quite different from the ideal design model, and there is a pixel position offset and non-rigid diffusion phenomenon for each waveband pixel point, so the model is divided into the following two parts:

[0086] (1.2) Model the displacement offset;

[0087] First, the spectral image I M*N is sampled and combined:

[0088]

[0089] The spectral image is sparsely sampled and combined, where k*k is the number of sampling points;

[0090]

[0091] J is the offset image, and A is the affine transformation, where the degree of freedom dF(A) of A is 2;

[0092] (1.3) Model the point diffusion phenomenon:

[0093] For the high-resolution hyperspectral image I λ (x,y) to be reconstructed, the observed RGB image J c (x,y) can be represented as:

[0094] J c (x,y)=∫∫∫Ω c (λ)Iλ (x, y)p λ (x - μ, y - v)dμdvdλ

[0095] where Ω c (λ) is the spectral response curve, c e {r, g, b}. p λ (x, y) is the point spread function related to position and wavelength, through the actual observation model, it is found that the point spread function is a non-rigid Gaussian distribution, so Δσ is used to describe the variance of the Gaussian distribution of the point spread function; using parameterized Gaussian to estimate the PSF can increase the constraint between points, which is also not achieved by the traditional calibration method.

[0096] (1.4) Establishment of overall model

[0097] In summary, the establishment of the error model can be divided into: 1) Establishment of offset model: including horizontal position offset Δx and vertical position offset Δy, 2) Establishment of diffusion model: the diffusion of a single pixel point is represented by a Gaussian function SPF function, but due to the different diffusion degrees of different pixel positions, in order to accurately describe this diffusion, Δσ parameterized Gaussian is used to fit the point spread process, σ represents the variance, for different σ, the degree of point spread is different.

[0098] Through the above, the differentiable modeling of the error is completed, and the establishment of the error model is decomposed into the solution of the above parameters Δx, Δy, Δσ. The above parameters are not consistent for different pixel points and different spectral bands on the entire image. In order to be more consistent with the actual physical situation, TV constraint is added to the model parameters of adjacent pixel points, so that the TV constraint is as smooth as possible.

[0099] (2) Correction of spectral imaging system error (offline calibration)

[0100] The offline calibration process is to train the network using the data set to realize the correction and adjustment of the network to the error, which mainly includes the following steps:

[0101] (2.1) Make a simulation data set:

[0102] Select a public data set, a CAVE data set containing 32 groups of indoor scene hyperspectral images, which contains 31 spectral images (HR-HSI) and a corresponding color image (HR-MSI);

[0103] (2.1.1) Get low-resolution image

[0104] The low-resolution image rule commonly used in the field of remote sensing is adopted, first, a 7*7 Gaussian blur kernel is used to filter the HR-HSI, the mean value of the blur kernel is 0, and the variance is 2, then the pixels are obtained in the row and column directions of the hyperspectral image with a step of the downsampling multiple, then maximum and minimum normalization is performed, and a low-resolution hyperspectral image (LR-HSI) is obtained;

[0105] (2.1.2) Obtain the CCD image

[0106] For the obtained low-resolution image, first, a sampling template is defined, the sampling points of the template are taken every 48*48, and when the spectral dimension is 31, that is, there is only 1 sampling point in the 48*48 block and 31 spectral channels are dispersed. Then, according to the preset sampling template, the corresponding point coordinates are restored to the arrangement obtained on the CCD, that is, the CCD image, which is a GT image without error, and then according to the established model, the band progressive tilt and Gaussian filtering operation of the pixel points on the CCD. For different bands of the same pixel point, the higher the wavelength band tilt, the lower the diffusion filtering. Through this operation, the error formation process of the imaging system can be simulated. The tilt range of the maximum tilt of each pixel point is 20 pixel grids, the adjacent pixel points are gradually transformed, and the step length of each change is half of the maximum tilt angle / pixel width, and the Gaussian filtering kernel size linearly changes from 3-10 according to the number of bands;

[0107] (2.2) Color difference correction through the network;

[0108] (2.2.1) The patch operation is performed on the above-mentioned CCD error data set, and the size of the patch is 48*48;

[0109] (2.2.2) The patch is sent into the network, the input channel number of the first layer convolutional network is 1, the output channel number is 16, the 3*3 convolutional kernel is used in the convolutional layer, LeakyRelu() is used as the activation function, then the pooling layer is used, and the step length of the pooling layer is set to 2;

[0110] (2.2.3) Then the obtained feature map is input into the network, the input channel number of the network is the output channel number of the previous layer, and the output channel number is twice the input channel number, then the activation function and the pooling layer are used, the activation function is LeakyRelu(), the step length of the pooling layer is 2, and a total of 5 convolutional networks are set;

[0111] (2.2.4) The feature map output by the last convolutional network is connected twice, the size of the output of the first full connection is 512, and the size of the output of the second full connection is 31 spectral channels;

[0112] (2.2.5) Train by setting the loss function as mean square error loss MSE_LOSS, and the optimizer as SGD with momentum 0.9, and train 500 times to obtain the preliminary result after error correction;

[0113] (2.2.6) Save the corrected parameters of the network, and obtain the correction result of the whole image through forward inference;

[0114] (3) Complete the calculation of spectral reconstruction;

[0115] According to the corrected low spatial resolution spectral image and high spatial resolution RGB image, a high-resolution spectral image is obtained by fusion.

[0116] (3.1) First, the image set is preprocessed, divided into small patches with size 128 and step 64, then UP_HSI is obtained by interpolation from LR_HSI to the same multiple as HR_MSI, and UP_HSI and HR_MSI are taken as the input of the network respectively;

[0117] (3.2) The first 8 channels are taken from UP-HSI, combined with the three channels of HR_MSI, and the output is obtained through the image fusion sub-network, and taken as the input of the next layer;

[0118] (3.3) The second 16 channels are taken from UP-HSI, combined with the three channels of HR_MSI, and the output is obtained through the image fusion sub-network, and taken as the input of the next layer;

[0119] (3.4) The next layer takes out 31 channels, combined with the three channels of HR_MSI, and the input of the previous layer fusion sub-network, and the output result obtained by the fusion sub-network is added to UP-HIS pixel by pixel to obtain the output;

[0120] (3.4) The output of the previous step is input into the purification network to obtain the final output result. The purification network includes four convolutional networks with kernel sizes of 3, 3, 1 and 1 respectively, and the final output is the reconstructed HR-HSI.

[0121] (3.5) Save the network parameters.

[0122] (4) Joint error correction for spectral reconstruction:

[0123] (4.1) Imaging system detection;

[0124] For the whole spectral imaging system, the basic principle is to first use the imaging system to detect and obtain a low-resolution chromatic spectral image (LR-HSI) and a high spatial resolution RGB image (HR-MSI) obtained by the RGB camera.

[0125] (4.2) Error correction;

[0126] For the low-resolution chromatic spectral image obtained by the imaging system, after pre-processing the data, the offline calibrated network parameters pre-trained in step (2) are loaded and the network parameters are fixed, forward inference is performed, and preliminary error correction is performed on the low-resolution spectral image with errors.

[0127] (4.3) Training spectral fusion network;

[0128] For the spectral image (LR-HSI) after preliminary error correction and the high spatial resolution spectral image obtained by the RGB camera, the low resolution spectral image is linearly interpolated, and the RGB image is used as the input of the network. Through the network parameters of the reconstruction network saved in step (3), the preliminary reconstructed target spectral image (HR-HSI) of the scene is obtained.

[0129] (4.4) Establishing a degradation model between space and spectrum;

[0130] (a) Establishment of spatial degradation model;

[0131] The main purpose of the spatial degradation model is to degrade the high-resolution preliminary reconstructed target spectral image (HR-HSI) of the scene to LR-HSI image. In the specific implementation process, the convolution is set, the step length of the convolution kernel is set to the multiple of the down-sampled scene, and for example, the 8 times super-resolution is taken as an example. The step length of the convolution is 8, and the size of the convolution kernel is 8. The high-resolution image is convolved to obtain a low-resolution spectral image degraded by 8 times down-sampling.

[0132] (b) Establishment of spectral degradation model;

[0133] The purpose of establishing the spectral degradation model is to degrade the preliminary reconstructed target spectral image (HR-HSI) of the scene to an RGB image through the model. For the RGB camera, we can consider that the spectral response function (SRF) is known, so we only need to multiply the spectral response matrix by the reconstructed spectral image to realize the spectral down-sampling. The size of the spectral response matrix is 31*3, the size of the spectral image is 512*512*31, and the multiplication of the two obtains a spectral image with a size of 512*512*3.

[0134] (4.4) Establishing an imaging degradation model

[0135] The modeling process of error degradation is shown in step (1), and deep separable convolution and deformable convolution are used to simulate the point spread function and displacement offset process, respectively.

[0136] (4.4.1) Simulation of point spread function: first, the low-resolution image is enlarged from 1*1 to 10*10 in position relationship by deconvolution, the up-sampling multiple of deconvolution is 10 times, then a depth separable convolution is used, the input channel number and the output channel number of the convolution kernel are both 31, the size of the convolution kernel is 7*7, and the filtering offset process is simulated through convolution;

[0137] (4.4.2) Simulate displacement by deformable convolution, the input and output channel numbers of deformable convolution remain unchanged, the size of the convolution kernel is set to 48, the step is set to 1, and the padding is set to 1;

[0138] (4.5) Correct the reconstruction results and model parameters by online observation

[0139] The preliminary reconstructed image is subjected to the degradation model in (4.2), (4.3) and (4.4) above, to obtain a spectral image 1, an RGB image 2 and an error CCD spectral image 3, the image 1 and the image 2 are subjected to loss with the LR-HSI and the HR-MSI, and the image 3 is subjected to loss with the CCD image, and through the backward propagation of the network, the spectral reconstruction algorithm and the error correction network are jointly optimized.

[0140] As shown in Figure 2 : it is a schematic diagram of the establishment of the overall model of the application, the leftmost diagram is the spectral dispersion distribution under the ideal condition, the middle diagram is obtained through the offset in the horizontal and vertical directions, and the right diagram is obtained through the point spread function, and the distribution of the right diagram is the actual observation image distribution;

[0141] As shown in Figure 3 : Figure 3 is a schematic diagram of a dual-channel spectral imaging system based on a DMD, light enters the imaging system, is reflected by a digital micromirror array after passing through optical devices such as a filter, the light is divided into two paths by controlling the rotation angle of the digital micromirror unit, one path is received by a color camera to obtain an RGB color image, and the other path is received by a gray-scale camera to obtain a spectral image after being split by a prism;

[0142] As shown in Figure 4 :

[0143] As shown in Figure 5 : Figure 5is the joint error correction spectral image calculation reconstruction diagram, is the framework diagram of the whole calculation spectral imaging chromatic aberration correction technology, the low resolution chromatic aberration spectral image Y that we shoot is carried out chromatic aberration correction through Net1, obtains low resolution spectral image X, then image X and RGB image are fused through the reconstruction network Net2 and obtain the high resolution spectral image of preliminary reconstruction, then through the space and chromatic aberration degradation, obtain the estimated image Y*, through the inter-spectral degradation, obtain the estimated image Z*, through the constraint of the observed scene X, Y, Z, optimize the reconstruction result.

[0144] The above is only to illustrate the technical idea of the present application, and cannot limit the protection scope of the present application. Any modification made according to the technical idea of the present application on the basis of the technical scheme falls within the protection scope of the claims of the present application.

Claims

1. A method for correcting errors in computed spectral imaging based on deep learning, characterized in that, Comprising the following steps: Step (1): analysis and modeling of the error of the low-resolution spectral image acquired by the imaging system; Step (2): building a neural network to correct the error of the low-resolution spectral image, i.e. offline calibration; Step (3): taking the corrected low-resolution spectral image and the RGB image captured by the color camera of the imaging system as input, building an image fusion reconstruction network through deep learning method, and calculating the spectral reconstruction to obtain the preliminary reconstructed target image; Step (4): performing imaging system degradation on the preliminary reconstructed target image, and performing constraint with the actual observation and the degraded image to realize joint optimization of the color difference correction network and the spectral fusion network, and realize spectral imaging error correction; The step (2) is specifically: The offline calibration process is to train the network with the data set to realize the correction and adjustment of the network to the error, mainly including the following steps: (2.1) making a simulation data set: Select a public data set, including a CAVE data set containing 32 groups of indoor scene hyperspectral images, 31 spectral images HR-HSI and a corresponding color image HR-MSI; (2.1.1) obtain low-resolution image The low-resolution image is obtained by using the commonly used low-resolution image making rule in the field of remote sensing. First, a 7*7 Gaussian blur kernel is used to filter the HR-HSI, the mean value of the blur kernel is 0, and the variance is 2. Then, the pixels are obtained by downsampling in the row and column directions of the hyperspectral image with a step of 2. Then, the maximum and minimum normalization is performed to obtain the low-resolution hyperspectral image LR-HSI; (2.1.2) obtain CCD image For the obtained low-resolution image, first, define a sampling template, and the sampling points are every 48*48. When the spectral dimension is 31, there is only one sampling point in the 48*48 block, and the 31 spectral channels are dispersed. Then, the CCD image is obtained by arranging the points in the corresponding position according to the preset sampling template. The CCD image is a GT image without error. Then, according to the established model, the waveband progressive tilt and Gaussian filtering operation are performed on the pixel points on the CCD. For different wavebands of the same pixel point, the higher the waveband tilt, the lower the diffusion filtering. Through this operation, the error formation process of the imaging system is simulated. The tilt range of the maximum tilt angle of each pixel point is 20 pixel grids, and the adjacent pixel points are gradually changed, with a step of half of the maximum tilt angle / pixel width. The size of the Gaussian filter kernel changes linearly from 3-10 according to the number of wavebands; (2.2) color difference correction through network; (2.2.1) patch operation is performed on the above-mentioned CCD error data set, and the size of the patch is 48*48; (2.2.2) send the patch into the network, the input channel number of the first layer convolution network is 1, the output channel number is 16, the convolution kernel of the convolution layer is 3*3, LeakyRelu() is used as the activation function, then the pooling layer is used, and the step of the pooling layer is set to 2; (2.2.3) Then the obtained feature map is input into the network, the input channel number of the network is the output channel number of the previous layer, the output channel number is twice the input channel number, then through the activation function and the pooling layer, the activation function is LeakyRelu(), the pooling layer step is 2, a total of 5 convolutional networks are set; (2.2.4) The feature map output by the last convolutional network is connected twice, the first full connection outputs 512, and the second full connection outputs 31 spectral channels; (2.2.5) The loss function is set as mean square error loss MSE-LOSS for training, the optimizer is selected as SGD, the momentum is set as 0.9, and the training is performed 500 times to obtain the preliminary result after error correction; (2.2.6) The corrected parameters of the network are saved, and the forward inference is performed to obtain the correction result of the whole image; The step (3) is specifically: According to the corrected low spatial resolution spectral image and the high spatial resolution RGB image, a high-resolution spectral image is obtained by fusion; (3.1) First, the image set is preprocessed, divided into a plurality of small patches with a size of 128 and a step of 64, then the LR-HSI is interpolated to the same multiple as the HR-MSI to obtain the UP-HSI, and the UP-HSI and the HR-MSI are respectively taken as the input of the network; (3.2) The first time, 8 channels are taken out from the UP-HSI, combined with the three channels of the HR-MSI, and the output is obtained through the image fusion sub-network and taken as the input of the next layer; (3.3) The second time, 16 channels are taken out from the UP-HSI, combined with the three channels of the HR-MSI, and the output is obtained through the image fusion sub-network and taken as the input of the next layer; (3.4) The next layer takes out 31 channels, combines the three channels of the HR-MSI, and the input of the previous layer fusion sub-network as the input, and the output result obtained by the fusion sub-network is added pixel by pixel with the UP-HIS to obtain the output; (3.4) The output before is purified through the purification network to obtain the final output result, the purification network includes four convolutional networks, the convolution kernel size is 3, 3, 1, 1 respectively, and the last output is the reconstructed HR-HSI; (3.5) Save the network parameters.

2. The deep learning based computational spectral imaging error correction method of claim 1, wherein, The step (1) is specifically: Modeling the imaging system error, obtaining the actual spectral imaging system observation and the theoretical design model, and analyzing the composition principle of the error; (1.1) Obtain the actual engineering imaging system observation and analyze it; (1.2) Model the displacement offset; (1.3) Model the point spread phenomenon; (1.4) Establishment of the overall model.

3. The deep learning based computational spectral imaging error correction method of claim 2, wherein, The step (1.1) is: First, the imaging system is built to capture the scene, extract the actual observation on the CCD detector, and compare and analyze with the theoretical design model, the actual observation image has pixel position offset and non-rigid diffusion phenomenon for each waveband pixel point; The step (1.2) is: First the spectral image I M*N is subjected to a sampling and combining operation: For the result of sparse sampling and block combination of spectral images, Where k*k is the number of sampling points; J is the offset image, A is the affine transformation, and the degree of freedom dF(A) of A is 2; The step (1.3) is: For the hyperspectral image I λ (x,y), the observed RGB image J c (x,y) can be expressed as: J c (x,y) = ∫∫∫Ω c (λ) I λ (μ,v) p λ (x-μ,y-v) dμdvdλ where Ω c (λ) is the spectral response curve, c e {r, g, b}, p λ (x, y) is a point spread function related to position and wavelength, with Δσ describing the variance of the Gaussian distribution of the point spread function; using a parameterized Gaussian for the estimation of the PSF increases the constraints between points.

4. The deep learning based computational spectral imaging error correction method of claim 3, wherein, The step (1.4) is: (1.4.1) Establishment of offset model: offset Δx of horizontal position and offset Δy of vertical position are contained; (1.4.2) Establishment of diffusion model: the diffusion of a single pixel point is expressed by using a SPF function conforming to a Gaussian function, a Gaussian parameterized by Δσ is used to fit the process of point diffusion, Δσ represents variance, and the establishment of the error model is decomposed into the solving of Δx, Δy and Δσ.

5. The deep learning based computational spectral imaging error correction method of claim 1, wherein, The step (2.1) is specifically: (2.1.1) Obtain a low-resolution spectral image; First, a Gaussian blur kernel is used to filter the HR-HSI, then the pixels are obtained in the row and column directions of the hyperspectral image with a step of the downsampling multiple, and then maximum and minimum normalization is performed to obtain a low-resolution spectral image LR-HSI; (2.1.2) Obtain a CCD image; For the obtained low-resolution spectral image, a sampling template is first defined; then the corresponding position point coordinates are restored to the arrangement obtained on the CCD, that is, the CCD image, according to the preset sampling template; then the band progressive tilt and Gaussian filtering operation of the pixel points on the CCD are performed according to the established model; for different bands of the same pixel point, the higher the wavelength band tilt, the lower the diffusion filtering; the adjacent pixel points are progressively transformed, and the step length of each change is half of the maximum tilt angle / pixel width, and the Gaussian filtering kernel size linearly changes from 3-10 according to the number of bands.

6. The deep learning based computational spectral imaging error correction method of claim 1, wherein, The step (4) is specifically: (4.1) Imaging system detection; The imaging system detection is used to obtain a low-resolution chromatic aberration spectral image LR-HSI-1 and a high spatial resolution RGB image obtained by an RGB camera; (4.2) Error correction technology; After the low-resolution chromatic aberration spectral image obtained by the imaging system is preprocessed, the network parameters of the offline calibration pre-trained in step (2) are loaded, and the network parameters are fixed, forward inference is performed, the error of the low-resolution spectral image is preliminarily corrected, and a spectral image LR-HSI-2 after error preliminary correction is obtained; (4.3) Training of spectral fusion network; For the spectral image LR-HSI-2 after error preliminary correction and the high spatial resolution spectral image HR-MSI-1 obtained by the RGB camera, the low-resolution spectral image is linearly interpolated, and the image is used as the input of the network together with the HR-MSI-1 image, the network parameters of the reconstruction network saved in step (3) are used to obtain a preliminary reconstructed target spectral image HR-HSI-1 of the scene; (4.4) Establishment of spatial and spectral degradation model; The spatial degradation model is established by using a convolutional network, and the spectral degradation is simulated by multiplying the camera response curve, and a low spatial resolution image LR-MSI and a low spectral resolution image LR-HSI-3 are obtained; (4.5) Establishment of imaging error degradation model; The error degradation modeling process is shown in step (1), and a depth separable convolution and a deformable convolution are used to simulate a point spread function and a displacement offset process, respectively. (4.6) Correcting the reconstruction results and model parameters by online observation; The preliminary reconstructed image is used to obtain the spectral image LR-HSI-2, the target spectral image HR-HSI-1 and the low spectral resolution image LR-HSI-3 through the degradation model in steps (4.2), (4.3) and (4.4) above, the spectral image LR-HSI-3 and the RGB image LR-MSI are made loss with LR-HSI and HR-MSI, the error CCD spectral image HR-MSI-1 is made loss with the CCD image HR-MSI, the spectral reconstruction algorithm and the error correction network are jointly optimized through the back propagation of the network, the self-supervised learning strategy is adopted to perform online fine adjustment on the reconstruction network parameters and the error model parameters, the joint optimization of reconstruction, online re-calibration and error correction is realized according to the online prior in the real scene.

7. The deep learning based computational spectral imaging error correction method of claim 6, wherein, The step (4.4) is specifically: Simulation of point spread function: deep separable convolution is adopted to realize different point spread functions for different spectral channels of the scene; Simulation of displacement: the displacement is simulated through deformable convolution, the input and output channel numbers of the deformable convolution remain unchanged, and the offset estimation is performed for each point.

Images