A multispectral image reconstruction method for rotational diffraction

Abstract

The present invention discloses a method for reconstructing a multispectral image for rotational diffraction. The method comprises: obtaining a rotational diffraction multispectral image data set; building a convolutional neural network for spectral image reconstruction, wherein the convolutional neural network comprises an encoding part, a decoding part, and a spectral upsampling residual connection part; inputting the rotational diffraction multispectral image data set into the convolutional neural network and training the convolutional neural network to obtain a trained convolutional neural network; inputting a rotational diffraction blurred image to be reconstructed into the trained convolutional neural network, and the trained convolutional neural network outputs a reconstructed hyperspectral image. The present invention does not require the assistance of a point spread function or iterative solution during the reconstruction process, and has the characteristics of fast speed, low consumption of computing resources, high spatial resolution and spectral accuracy of the reconstruction result, and flexible number of output spectral channels.

Description

Technical Field

[0001] The present invention relates to a multispectral image reconstruction method in the field of spectral imaging technology and deep learning, and specifically relates to a multispectral image reconstruction method for rotational diffraction. Background Art

[0002] As a multi-dimensional information acquisition technology that combines imaging technology and spectral technology, spectral imaging technology has the advantage of being able to detect and obtain two-dimensional spatial information and one-dimensional spectral information of the target compared to traditional imaging technology. It is the main information acquisition method for studying material composition types, thermal radiation characteristics, etc. Since the birth of spectral imaging technology in the last century, it has been widely used in agriculture, food, resource exploration, environmental protection, biomedicine, military and other fields.

[0003] However, traditional hyperspectral imaging systems have problems such as complex optical systems, precision moving devices, and long exposure times, which greatly limit their application in many dynamic scenes. In recent years, hyperspectral imaging systems have developed towards lightweight, miniaturized, and snapshot measurements. Especially with the development of computer resources, many lightweight, miniaturized snapshot hyperspectral imaging systems based on computational reconstruction have emerged, and the rotational diffraction hyperspectral imaging system is one of them.

[0004] In 2019, Jeon et al. proposed a snapshot hyperspectral imaging system based on rotational diffraction. This imaging system can achieve snapshot spectral imaging using only one diffraction element supplemented by a corresponding reconstruction algorithm. It has the characteristics of light weight, snapshot type, and simple structure. Finally, it achieved 10nm interval and 25-channel hyperspectral imaging in the 420nm-660nm spectrum. However, in the reconstruction algorithm proposed by this system, the assistance of the point spread function of the diffraction element is required. The number of output spectral channels must be consistent with the number of spectral channels of the calibrated point spread function. In addition, there are update and iteration steps in the reconstruction process, which has high computational cost and large consumption of computing resources. Summary of the invention

[0005] In order to solve the technical problems existing in the background technology, the existing reconstruction algorithm needs the assistance of point spread function, the number of spectral channels of the reconstruction result must be consistent with the number of spectral channels of the point spread function, iterative update is required in the reconstruction process, and the computing resource consumption is large. The present invention proposes a multispectral image reconstruction method for rotational diffraction, which does not require the assistance of point spread function in the reconstruction process, has a flexible number of spectral channels of the reconstruction result, does not require iterative update, and has a low computing cost.

[0006] The present invention is achieved through the following technical solutions:

[0007] The present invention comprises the following steps:

[0008] S1: Obtain a rotational diffraction multispectral image dataset, which consists of a rotational diffraction blurred image and a corresponding hyperspectral real image;

[0009] S2: Build a convolutional neural network for spectral image reconstruction. The convolutional neural network includes an encoding part, a decoding part, and a spectral upsampling residual connection part. The input of the convolutional neural network is respectively input into the encoding part and the spectral upsampling residual connection part, the output of the encoding part is input into the decoding part, and the output of the decoding part and the output of the spectral upsampling residual connection part are element-wise added to each other to obtain the output of the convolutional neural network.

[0010] S3: input the rotational diffraction multispectral image data set in S1 into the convolutional neural network in S2 and train the convolutional neural network to obtain a trained convolutional neural network;

[0011] S4: Input the rotational diffraction blurred image to be reconstructed into the trained convolutional neural network, and the trained convolutional neural network outputs the reconstructed hyperspectral image.

[0012] The method for acquiring the rotational diffraction multispectral image data set in S1 includes:

[0013] A rotational diffraction data set shooting system based on a translation stage is used to collect multiple sets of rotational diffraction blurred images and corresponding hyperspectral real images, thereby forming a rotational diffraction multispectral image data set;

[0014] Alternatively, a rotational diffraction blurred image is generated by calculating a known hyperspectral real image dataset and a point spread function of a rotational diffraction hyperspectral imaging system, and a rotational diffraction multispectral image dataset is composed of the known hyperspectral real image dataset and the corresponding rotational diffraction blurred image.

[0015] The encoding part includes five residual convolution blocks and four downsampling layers. The input of the encoding part is input to the first residual convolution block. The first residual convolution block is connected to the second residual convolution block after passing through the first downsampling layer, the second residual convolution block, the second downsampling layer, the third residual convolution block, the third downsampling layer, the fourth residual convolution block and the fourth downsampling layer in sequence. The output of the first residual convolution block is used as the first output of the encoding part, the output of the second residual convolution block is used as the second output of the encoding part, the output of the third residual convolution block is used as the third output of the encoding part, the output of the fourth residual convolution block is used as the fourth output of the encoding part, and the output of the fifth residual convolution block is used as the fifth output of the encoding part. The first to fifth outputs of the encoding part are respectively input to the first to fifth inputs of the decoding part in sequence.

[0016] The decoding part includes four transposed convolutional layers, four residual convolutional blocks and a first 2D convolutional layer,

[0017] The fifth input of the decoding part is input to the first transposed convolution layer, the output of the first transposed convolution layer is cascaded with the fourth input of the decoding part and input to the sixth residual convolution block, the output of the sixth residual convolution block is connected to the second transposed convolution layer, the output of the second transposed convolution layer is cascaded with the third input of the decoding part and input to the seventh residual convolution block, the output of the seventh residual convolution block is connected to the third transposed convolution layer, the output of the third transposed convolution layer is cascaded with the second input of the decoding part and input to the eighth residual convolution block, the output of the eighth residual convolution block is connected to the fourth transposed convolution layer, the output of the fourth transposed convolution layer is cascaded with the first input of the decoding part and input to the ninth residual convolution block, the output of the ninth residual convolution block is connected to the first 2D convolution layer, and the output of the first 2D convolution layer is used as the output of the decoding part.

[0018] The spectral upsampling residual connection part includes a spectral upsampling residual module, and the input and output of the spectral upsampling residual connection part are respectively the input and output of the spectral upsampling residual module.

[0019] The residual convolution block includes 3 2D convolution layers, 3 batch normalization layers, 2 activation layers and an SE attention module. The inputs of the residual convolution block are respectively input into the third 2D convolution layer and the fourth 2D convolution layer. The outputs of the third 2D convolution layer are sequentially taken as attention input features through the first activation layer, the first batch normalization layer and the fifth 2D convolution layer. The outputs of the attention input features after being input into the SE attention module are then multiplied by the channels with the attention input features as the intermediate features. The outputs of the fourth 2D convolution layer after the second batch normalization layer and the intermediate features are element-wise added and input into the second activation layer. The outputs of the second activation layer after the third batch normalization layer are taken as the outputs of the residual convolution block.

[0020] The formula of the total loss function used in the convolutional neural network training in S3 is:

[0021] L=αL1+βL SSIM +γL Grad

[0022] L1=|XY|

[0023]

[0024]

[0025] Among them, L is the total loss function value, L1 is the content loss function value, and L SSIM is the structural similarity loss function value, L Gradis the value of the gradient difference loss function, a, β, and γ represent the weights of the content loss function, the structural similarity loss function, and the gradient difference loss function, respectively. X is the reconstructed hyperspectral image, Y is the real hyperspectral image, and u X 、u Y are the means of the reconstructed hyperspectral image X and the real hyperspectral image Y, respectively. are the variances of the reconstructed hyperspectral image X and the real hyperspectral image Y, σ XY is the covariance of the reconstructed hyperspectral image X and the real hyperspectral image Y, C1 and C2 are the first and second smoothing factors respectively; is the gradient operator of the reconstructed hyperspectral image; is the gradient operator of the real hyperspectral image, and || represents the absolute value operation.

[0026] The generation formula of the rotational diffraction blurred image is as follows:

[0027]

[0028] Among them, J c (x, y) represents the rotational diffraction blurred image under the c color channel, c represents the color channel, satisfying c∈{r,g,b}, r,g,b are red, green, and blue channels respectively, m represents the number of spectral channels of the real hyperspectral image, I(x,y;λ i ) represents the hyperspectral real image slice in the i-th spectral channel, represents the convolution operation, k c (x,y;λ i ) represents the point spread function of the c color channel under the i-th spectral channel.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) The reconstruction method proposed in the present invention does not require the assistance of a point spread function during the reconstruction process;

[0031] (2) The number of channels of the output hyperspectral image of the present invention can be flexibly changed with the number of channels of the rotational diffraction data set;

[0032] (3) The present invention does not require iterative updates during the reconstruction process, consumes relatively less computing resources, and the network training process takes less time. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the network structure of the present invention;

[0034] Figure 2 Schematic diagram of the residual convolution block of the present invention;

[0035] Figure 3 Visualization images of the three channels of the generated RGB image with rotational diffraction blur features;

[0036] Figure 4 Schematic diagram of the system for building an uncalibrated rotational diffraction multispectral dataset;

[0037] Figure 5 The images of the reconstruction results of the present invention in different spectral bands;

[0038] Figure 6 It is a spectrum curve diagram of a part of the color block of the reconstruction result of the present invention;

[0039] Figure 7 The images of reconstruction results of the comparison methods in different spectral bands;

[0040] Figure 8 The spectral curve diagram of some color blocks of the reconstruction results of the comparison method;

[0041] Fig. 9 This is a comparison diagram of the spatial resolution details of the reconstruction results of the present invention and the comparative method. DETAILED DESCRIPTION

[0042] The present invention is described in further detail below in conjunction with the accompanying drawings.

[0043] According to the inventive content of the present invention, the complete implementation example and its implementation process are as follows:

[0044] The present invention comprises the following steps:

[0045] S1: Obtain a rotational diffraction multispectral image dataset, which consists of a rotational diffraction blurred image and a corresponding hyperspectral real image;

[0046] The method for acquiring the rotational diffraction multispectral image dataset in S1 includes:

[0047] A rotational diffraction data set shooting system based on a translation stage is used to collect multiple sets of rotational diffraction blurred images and corresponding hyperspectral real images, thereby forming a rotational diffraction multispectral image data set; a hyperspectral image is a spectral image with a spectral resolution higher than 10nm.

[0048] like Figure 4As shown, the rotational diffraction data set shooting system based on the translation stage includes a wide-spectrum light source 1, a wide-spectrum light source 2, a rotational diffraction hyperspectral imaging system 3, a hyperspectral camera 4 and a translation stage 5. The spectral range of the emitted light of the wide-spectrum light source 1 and the wide-spectrum light source 2 covers at least the 480nm to 900nm spectrum. The rotational diffraction hyperspectral imaging system 3 and the hyperspectral camera 4 are arranged on the translation stage 5. The optical system of the rotational diffraction hyperspectral imaging system 3 and the hyperspectral camera 4 has the same field of view angle and magnification, and the detectors used by the two have the same spatial resolution. The optical axis direction of the rotational diffraction hyperspectral imaging system 3 and the hyperspectral camera 4 is parallel and perpendicular to the moving direction of the translation stage 5.

[0049] The displacement stage 5 is adjusted so that the optical axis of the rotational diffraction hyperspectral imaging system 3 faces the shooting scene, the object in the shooting scene is located in the middle of the system image plane, and a rotational diffraction blurred image is taken. The displacement stage 5 is adjusted again so that the optical axis of the hyperspectral camera 4 faces the shooting scene, the image plane position of the captured object is consistent with the image plane position of the object in the rotational diffraction blurred image, and the corresponding hyperspectral real image is taken.

[0050] Alternatively, a rotational diffraction blurred image is generated by calculating a known hyperspectral real image dataset and a point spread function of a rotational diffraction hyperspectral imaging system, and a rotational diffraction multispectral image dataset is composed of the known hyperspectral real image dataset and the corresponding rotational diffraction blurred image.

[0051] The generation formula of the rotational diffraction blurred image is as follows:

[0052]

[0053] Among them, J c (x, y) represents the rotational diffraction blurred image under the c color channel, c represents the color channel, satisfying c∈{r,g,b}, r,g,b are red, green, and blue channels respectively, m represents the number of spectral channels of the real hyperspectral image, I(x,y;λ i ) represents the hyperspectral real image slice in the i-th spectral channel, represents the convolution operation, k c (x,y;λ i ) represents the point spread function of the c color channel under the i-th spectral channel.

[0054] In this embodiment, a computational simulation method is used to construct a data pair. First, the point spread function of the system is calibrated, and the point spread function is cropped, registered, and denoised. Then, 61 sets of hyperspectral real image data sets with a spectral range of 480 to 900 nm, a spectral interval of 10 nm, 43 channels, and a resolution of 2048*2048 are established. The point spread function is convolved with the hyperspectral real image data set to generate a rotational diffraction blurred image with a rotational diffraction blur feature. The rotational diffraction blurred image and the corresponding hyperspectral real image form a data pair, and multiple sets of data pairs constitute a rotational diffraction multispectral data set. 56 sets of data pairs are selected from the 61 sets for training, and the remaining 5 sets are used for verification and testing. Figure 3 The images shown in (a), (b), and (c) are the three channels of the RGB images in the test set. The data pairs with a resolution of 2048*2048 are cropped into 256*256 image blocks using the same cropping method, which can be used to train the neural network.

[0055] S2: Build a convolutional neural network for spectral image reconstruction, such as Figure 1 As shown, the convolutional neural network includes an encoding part, a decoding part and a spectral upsampling residual connection part; the input of the convolutional neural network is respectively input into the encoding part and the spectral upsampling residual connection part, the output of the encoding part is input into the decoding part, and the output of the decoding part and the output of the spectral upsampling residual connection part are element-wise added to each other to obtain the output of the convolutional neural network;

[0056] The encoding part includes five residual convolution blocks and four downsampling layers. In a specific implementation, the pooling function of the downsampling layer is a maximum pooling function. The input of the encoding part is input to the first residual convolution block. The first residual convolution block is sequentially connected to the second residual convolution block after passing through the first downsampling layer, the second residual convolution block, the second downsampling layer, the third residual convolution block, the third downsampling layer, the fourth residual convolution block and the fourth downsampling layer. The output of the first residual convolution block is used as the first output of the encoding part, the output of the second residual convolution block is used as the second output of the encoding part, the output of the third residual convolution block is used as the third output of the encoding part, the output of the fourth residual convolution block is used as the fourth output of the encoding part, and the output of the fifth residual convolution block is used as the fifth output of the encoding part. The first to fifth outputs of the encoding part are respectively input to the first to fifth inputs of the decoding part.

[0057] The decoding part consists of four transposed convolutional layers, four residual convolutional blocks, and the first 2D convolutional layer.

[0058] The fifth input of the decoding part is input to the first transposed convolution layer, the output of the first transposed convolution layer is cascaded with the fourth input of the decoding part and input to the sixth residual convolution block, the output of the sixth residual convolution block is connected to the second transposed convolution layer, the output of the second transposed convolution layer is cascaded with the third input of the decoding part and input to the seventh residual convolution block, the output of the seventh residual convolution block is connected to the third transposed convolution layer, the output of the third transposed convolution layer is cascaded with the second input of the decoding part and input to the eighth residual convolution block, the output of the eighth residual convolution block is connected to the fourth transposed convolution layer, the output of the fourth transposed convolution layer is cascaded with the first input of the decoding part and input to the ninth residual convolution block, the output of the ninth residual convolution block is connected to the first 2D convolution layer, and the output of the first 2D convolution layer is used as the output of the decoding part. In a specific implementation, the feature output of the decoding part is 43 channels.

[0059] The input image first passes through a residual convolution block and becomes a 256×256×32 feature x in , and then after the first maxpooling downsampling and residual convolution block, it becomes a 128×128×64 feature x down1 , and then after the second maxpooling downsampling and residual convolution block, it becomes a 64×64×128 feature x down2 , and then after the third maxpooling downsampling and residual convolution block, it becomes a 32×32×256 feature x down3 , and finally after the fourth maxpooling downsampling and residual convolution block, it becomes a 16×16×256 feature x down4 , the encoding work is now completed.

[0060] During the decoding process, x down4 After transposed convolution upsampling, and x down3 Perform a jump connection and then input the residual convolution block to obtain a 32×32×128 feature x up1 , x up1 After transposed convolution upsampling, and x down2 Perform a jump connection and then input the residual convolution block to obtain a 64×64×64 feature x up2 , x up2 After transposed convolution upsampling, and x down1 Perform a jump connection and then input the residual convolution block to obtain a 128×128×32 feature x up3 , x up3 After transposed convolution upsampling, and x in Perform a jump connection and then input the residual convolution block to obtain a 256×256×64 feature x up4 , x up4Finally, a 1×1 convolution block is used to obtain a decoding result of 256×256×43. The decoding result is added to the spectral upsampling result to obtain the reconstructed hyperspectral image.

[0061] The spectral upsampling residual connection part includes a spectral upsampling residual module. In the specific implementation, the spectral upsampling residual module is the second 2D convolution layer. The input and output of the spectral upsampling residual connection part are the input and output of the spectral upsampling residual module respectively. The spectral upsampling residual connection part outputs a spectral upsampling result with a scale of 256×256×43.

[0062] like Figure 2 As shown in the figure, the residual convolution block includes 3 2D convolution layers, 3 batch normalization layers, 2 activation layers and SE attention module. In the specific implementation, the activation function of the activation layer is ReLu. The input of the residual convolution block is respectively input into the third 2D convolution layer and the fourth 2D convolution layer. The output of the third 2D convolution layer is successively taken as the attention input feature through the first activation layer, the first batch normalization layer and the fifth 2D convolution layer. The output of the attention input feature after being input into the SE attention module is multiplied by the channel with the attention input feature and the output is taken as the intermediate feature. The output of the fourth 2D convolution layer after the second batch normalization layer and the intermediate feature are added element by element and input into the second activation layer. The output of the second activation layer after the third batch normalization layer is taken as the output of the residual convolution block.

[0063] Assuming the input image is represented as x, the role of the residual convolution block can be described as follows:

[0064] The feature after the first convolution block can be expressed as x1=BN(F act (F1(x))), where F1(·) represents a two-dimensional convolution with a convolution kernel of 3×3, and F act (·) is the ReLU activation function, and BN is the batch normalization operation.

[0065] Then the feature of x1 after the attention mechanism can be described as x2 = F2(x1)·F SE (F2(x1)), where F2(·) represents a two-dimensional convolution with a convolution kernel of 3×3, and F SE (·) is the SE Attention operation. SE Attention stands for Squeeze-and-Excitation Attention, which is a widely used attention mechanism module.

[0066] The feature after the residual operation can be described as x res =BN(F3(x)), where F3(·) represents a two-dimensional convolution with a convolution kernel of 3×3.

[0067] The input of the final residual convolution block can be expressed as:

[0068] x out =BN(F act (x res +x2))

[0069] Among them, F act (·) is the ReLU activation function, and BN is the batch normalization operation.

[0070] S3: input the rotational diffraction multispectral image data set in S1 into the convolutional neural network in S2 and train the convolutional neural network to obtain the trained convolutional neural network for training;

[0071] During training, the Adam optimizer is used and the learning rate is set to 10. -4 , the learning rate drops by half every 30 epochs, the batch size is set to 8, and the training ends when the number of epochs is equal to 150. The formula of the total loss function used in the convolutional neural network training in S3 is:

[0072] L=aL1+βL SSIM +γL Grad

[0073] L1=|XY|

[0074]

[0075]

[0076] Among them, L is the total loss function value, L1 is the content loss function value, and L SSIM is the structural similarity loss function value, L Grad is the value of the gradient difference loss function, a, β, and γ represent the weights of the content loss function, the structural similarity loss function, and the gradient difference loss function, respectively. X is the reconstructed hyperspectral image, Y is the real hyperspectral image, and u X 、u Y are the means of the reconstructed hyperspectral image X and the real hyperspectral image Y, respectively. are the variances of the reconstructed hyperspectral image X and the real hyperspectral image Y, σ XY is the covariance of the reconstructed hyperspectral image X and the real hyperspectral image Y, C1 and C2 are the first and second smoothing factors, respectively, which are constant terms set to prevent the denominator from being zero; is the gradient operator of the reconstructed hyperspectral image; is the gradient operator of the real hyperspectral image, and || represents the absolute value operation.

[0077] S4: Input the rotational diffraction blurred image to be reconstructed into the trained convolutional neural network, and the trained convolutional neural network outputs the reconstructed hyperspectral image.

[0078] The convolutional neural network does not require the assistance of a point spread function during the reconstruction process, does not require iterative updates, has a fast reconstruction speed, and has a low computational cost. It uses a 1×1 convolution block to change the number of spectral channels for spectral upsampling. The number of spectral channels of the reconstruction result can be flexibly changed with different rotational diffraction data sets.

[0079] Select one of the reconstruction results, and the performance of the reconstructed hyperspectral image in each spectral band is as follows: Figure 5 As shown in the figure, it can be seen that the image spatial information of each spectral band has been well restored. Several color blocks in the reconstructed hyperspectral image are selected to evaluate the spectral accuracy. The results are shown in Figure 6 As shown, Figure 6 (a) is the selected reconstructed hyperspectral image. Figure 6 (b)-(e) are the spectral curves of the first to fourth color blocks in the selected reconstructed hyperspectral image, respectively. It can be seen from the figure that the spectral curve of the reconstruction result of the corresponding color block is consistent with the label in the data set, and the root mean square error (RMSE) between the spectral curve of the selected color block and the label is 0.01531, 0.02444, 0.01485, and 0.01798, respectively. Table 1 is a comparison of objective evaluation indicators of the present invention and other reconstruction methods in constructing data sets. The evaluation indicators are peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and spectral angle mapping (SAM), wherein the higher the PSNR and SSIM, the better, and the lower the SAM value, the better. Compared with other reconstruction methods, the present invention has achieved better results in all indicators.

[0080] Table 1 Comparison of objective evaluation indicators of the present invention and other reconstruction methods on the constructed data set

[0081]

[0082] In order to verify the advancedness of the method of the present invention, the reconstruction results of the traditional Jeon et al. method are compared with the reconstruction results of the present invention. Peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and spectral angle mapping (SAM) are used as evaluation indicators, where the higher the PSNR and SSIM, the better, and the lower the SAM value, the better. The comparison results are shown in Table 1. It can be seen from Table 1 that the present invention has achieved better results in all indicators. In addition to the comparison of evaluation indicators, Figure 7 The results of the reconstruction of the contrast method in different spectral bands are shown in Figure 2. From the figure, it can be seen that the clarity of the restored image in each spectral band is not as good as that of Figure 5 , Figure 8The spectral curves of some color blocks of the reconstruction results of the contrast method are shown. Figure 8 (a) is the selected reconstructed hyperspectral image. Figure 8 (b)-(e) are the spectral curves of the first to fourth color blocks in the selected reconstructed hyperspectral image. It can be seen from the figure that although the reconstruction result is roughly correct, the root mean square error (RMSE) between the selected color block spectral curve and the label is 0.09163, 0.07776, 0.08523, and 0.08566, respectively, which is lower than the accuracy of the method of the present invention. Fig. 9 The spatial resolution detail comparison diagram of the reconstruction results of the present invention and the comparative method is shown. Fig. 9 (a) and (c) are the reconstruction results of the present invention and the comparative method at the 500nm spectral band, respectively. Fig. 9 (b) and (d) are the reconstruction results of the present invention and the comparative method at the 800nm ​​spectral band, respectively. It can be seen from the figures that the reconstruction results of the present invention contain richer spatial information.

Claims

1. A multispectral image reconstruction method for rotational diffraction, characterized in that: The following steps are involved: S1: Obtain a rotational diffraction multispectral image dataset, which consists of a rotational diffraction blurred image and a corresponding hyperspectral real image; S2: Build a convolutional neural network for spectral image reconstruction. The convolutional neural network includes an encoding part, a decoding part, and a spectral upsampling residual connection part. The input of the convolutional neural network is respectively input into the encoding part and the spectral upsampling residual connection part, the output of the encoding part is input into the decoding part, and the output of the decoding part and the output of the spectral upsampling residual connection part are element-wise added to each other to obtain the output of the convolutional neural network. S3: input the rotational diffraction multispectral image data set in S1 into the convolutional neural network in S2 and train the convolutional neural network to obtain a trained convolutional neural network; S4: Input the rotational diffraction blurred image to be reconstructed into the trained convolutional neural network, and the trained convolutional neural network outputs the reconstructed hyperspectral image.

2. A multispectral image reconstruction method for rotational diffraction according to claim 1, characterized in that: The method for acquiring the rotational diffraction multispectral image data set in S1 includes: A rotational diffraction data set shooting system based on a translation stage is used to collect multiple sets of rotational diffraction blurred images and corresponding hyperspectral real images, thereby forming a rotational diffraction multispectral image data set; Alternatively, a rotational diffraction blurred image is generated by calculating a known hyperspectral real image dataset and a point spread function of a rotational diffraction hyperspectral imaging system, and a rotational diffraction multispectral image dataset is composed of the known hyperspectral real image dataset and the corresponding rotational diffraction blurred image.

3. The multispectral image reconstruction method for rotational diffraction according to claim 1, characterized in that: The encoding part includes five residual convolution blocks and four downsampling layers. The input of the encoding part is input to the first residual convolution block. The first residual convolution block is connected to the second residual convolution block after passing through the first downsampling layer, the second residual convolution block, the second downsampling layer, the third residual convolution block, the third downsampling layer, the fourth residual convolution block and the fourth downsampling layer in sequence. The output of the first residual convolution block is used as the first output of the encoding part, the output of the second residual convolution block is used as the second output of the encoding part, the output of the third residual convolution block is used as the third output of the encoding part, the output of the fourth residual convolution block is used as the fourth output of the encoding part, and the output of the fifth residual convolution block is used as the fifth output of the encoding part. The first to fifth outputs of the encoding part are respectively input to the first to fifth inputs of the decoding part in sequence.

4. The multispectral image reconstruction method for rotational diffraction according to claim 1, characterized in that: The decoding part includes four transposed convolutional layers, four residual convolutional blocks and a first 2D convolutional layer, The fifth input of the decoding part is input to the first transposed convolution layer, the output of the first transposed convolution layer is cascaded with the fourth input of the decoding part and input to the sixth residual convolution block, the output of the sixth residual convolution block is connected to the second transposed convolution layer, the output of the second transposed convolution layer is cascaded with the third input of the decoding part and input to the seventh residual convolution block, the output of the seventh residual convolution block is connected to the third transposed convolution layer, the output of the third transposed convolution layer is cascaded with the second input of the decoding part and input to the eighth residual convolution block, the output of the eighth residual convolution block is connected to the fourth transposed convolution layer, the output of the fourth transposed convolution layer is cascaded with the first input of the decoding part and input to the ninth residual convolution block, the output of the ninth residual convolution block is connected to the first 2D convolution layer, and the output of the first 2D convolution layer is used as the output of the decoding part.

5. The multispectral image reconstruction method for rotational diffraction according to claim 1, characterized in that: The spectral upsampling residual connection part includes a spectral upsampling residual module, and the input and output of the spectral upsampling residual connection part are respectively the input and output of the spectral upsampling residual module.

6. A multispectral image reconstruction method for rotational diffraction according to claim 3 or 4, characterized in that: The residual convolution block includes 3 2D convolution layers, 3 batch normalization layers, 2 activation layers and an SE attention module. The inputs of the residual convolution block are respectively input into the third 2D convolution layer and the fourth 2D convolution layer. The outputs of the third 2D convolution layer are sequentially taken as attention input features through the first activation layer, the first batch normalization layer and the fifth 2D convolution layer. The outputs of the attention input features after being input into the SE attention module are then multiplied by the channels with the attention input features as the intermediate features. The outputs of the fourth 2D convolution layer after the second batch normalization layer and the intermediate features are element-wise added and input into the second activation layer. The outputs of the second activation layer after the third batch normalization layer are taken as the outputs of the residual convolution block.

7. The multispectral image reconstruction method for rotational diffraction according to claim 1, characterized in that: The formula of the total loss function used in the convolutional neural network training in S3 is: L=αL1+βL SSIM +γL Grad L1=|XY| Among them, L is the total loss function value, L1 is the content loss function value, and L SSIM is the structural similarity loss function value, L Grad is the value of the gradient difference loss function, α, β, and γ represent the weights of the content loss function, the structural similarity loss function, and the gradient difference loss function, respectively. X is the reconstructed hyperspectral image, Y is the real hyperspectral image, and u X 、u Y are the means of the reconstructed hyperspectral image X and the real hyperspectral image Y, respectively. are the variances of the reconstructed hyperspectral image X and the real hyperspectral image Y, σ XY is the covariance of the reconstructed hyperspectral image X and the real hyperspectral image Y, C1 and C2 are the first and second smoothing factors respectively; is the gradient operator of the reconstructed hyperspectral image; is the gradient operator of the real hyperspectral image, and || represents the absolute value operation.

8. The multispectral image reconstruction method for rotational diffraction according to claim 2, characterized in that: The generation formula of the rotational diffraction blurred image is as follows: Among them, J c (x, y) represents the rotational diffraction blurred image under the c color channel, c represents the color channel, satisfying c∈{r,g,b}, r,g,b are red, green, and blue channels respectively, m represents the number of spectral channels of the real hyperspectral image, I(x,y;λ i ) represents the hyperspectral real image slice in the i-th spectral channel, represents the convolution operation, k c (x,y;λ i ) represents the point spread function of the c color channel under the i-th spectral channel.

Images