A fusion-based rotational diffraction hyperspectral imaging system and method

Abstract

The present invention discloses a rotational diffraction hyperspectral imaging system and method based on fusion. It comprises: building a dual-branch imaging system, wherein one branch is a color clear imaging system, and the other branch is a rotational diffraction hyperspectral imaging system; building a convolutional neural network based on image fusion, inputting the images collected by the two imaging systems into the convolutional neural network, and after three stages of feature extraction, feature fusion and spectral reconstruction, a hyperspectral image with high spatial resolution and spectral accuracy can be obtained. The present invention adds a color clear imaging branch to the traditional rotational diffraction hyperspectral imaging system, and uses the imaging results of the branch to carry out fusion reconstruction of hyperspectral images, which can effectively improve the spatial resolution and spectral accuracy of the reconstruction results.

Description

Technical Field

[0001] The present invention relates to a spectral encoding imaging system and method in spectral imaging technology, and in particular to a fusion-based rotational diffraction hyperspectral imaging system and method. Background Art

[0002] Spectral imaging, a multidimensional information acquisition technology that combines imaging and spectroscopy, offers advantages over traditional imaging techniques in its ability to detect and obtain both two-dimensional spatial information and one-dimensional spectral information about the target. It is a key information acquisition tool for studying material composition and thermal radiation properties. Since its inception in the last century, spectral imaging has been widely applied in agriculture, food, resource exploration, environmental protection, biomedicine, and military fields.

[0003] Traditional hyperspectral imaging systems require scanning to obtain a three-dimensional data cube. Common scanning methods include scanning along the spectral dimension and scanning along the spatial dimension. Due to the required scanning steps, these systems are generally unsuitable for dynamic imaging, and the scanning process requires complex components such as precision mechanical movement. To overcome the shortcomings of complex optomechanical systems and long imaging times, spectral imaging technologies based on coded reconstruction have attracted widespread attention in recent years.

[0004] In 2019, Jeon et al. proposed a snapshot hyperspectral imaging system based on rotational diffraction. This imaging system can achieve snapshot spectral imaging using only one diffraction element supplemented by a corresponding reconstruction algorithm. It has the characteristics of being lightweight, snapshot-type, and simple in structure. However, due to the large scale of the point spread function of the system, the initial image obtained is severely degraded, and the quality of the reconstructed hyperspectral image needs to be improved. Specifically, the spatial resolution is low, the spectral accuracy is not high enough, and there are some pseudo-textures in the image. Domestic scholars have optimized the rotational diffraction hyperspectral imaging system, such as optimizing the design parameters of the diffraction element, using a refractive-diffractive hybrid optical system to improve the system's sensitivity, and using a front-end telescope system to improve the system's detection capability for distant targets. However, the quality of the reconstruction results has not been significantly improved. Summary of the Invention

[0005] In order to solve the technical problems existing in the background technology, the reconstructed spectral image quality of the rotational diffraction hyperspectral imaging system is poor, there are problems such as pseudo-texture, spectral accuracy and spatial resolution that need to be improved. The present invention proposes a fusion-based rotational diffraction hyperspectral imaging system and method. On the basis of the traditional rotational diffraction imaging system, an additional clear imaging branch is added by using a beam splitter, and the clear imaging result is used as auxiliary information to reconstruct the rotational diffraction blurred image, thereby improving the spatial resolution and spectral accuracy of the reconstruction result.

[0006] The technical solution for achieving the purpose of the present invention is:

[0007] 1. A fusion-based rotational diffraction hyperspectral imaging system

[0008] The system includes a beam splitter, a coded diffraction element, a first image sensor, an imaging lens, and a second image sensor;

[0009] After the parallel light is incident on the beam splitter, it is transmitted and reflected. The transmitted light of the beam splitter is encoded by the coding diffraction element to form a rotational diffraction blurred image on the first image sensor, and the reflected light of the beam splitter is converged by the imaging lens to form a clear image on the second image sensor; the incident end face of the coding diffraction element is set as the coding end face.

[0010] The incident end face of the coding diffraction element is set as the coding end face by the following formula:

[0011]

[0012]

[0013] Where λ(θ) represents the design wavelength of the coding diffraction element at angle θ, and the operating wavelength of the coding diffraction element is [λ min ,λ max ],λ min Indicates the minimum operating wavelength of the coded diffraction element, λ max represents the maximum operating wavelength of the coded diffraction element, N is the number of repetition periods, Indicates the angle of the coded diffraction element The working wavelength at the position of the coding diffraction element is h(r,θ), h(r,θ) represents the height of the coding diffraction element at an angle of θ and a radius of r, h0 is the height of the base material of the coding diffraction element, and n λ and n0 are the refractive indices of light with wavelength λ in the current coding diffraction element and in the air, respectively. DOE is the focal length of the coding diffraction element, and m is the diffraction order.

[0014] The focal length of the coding diffraction element is equal to that of the imaging lens; the image plane size and resolution of the first image sensor and the second image sensor are the same.

[0015] 2. A fusion-based rotational diffraction hyperspectral imaging method

[0016] 1) Build a rotational diffraction hyperspectral imaging system;

[0017] 2) Using a rotational diffraction hyperspectral imaging system to collect rotational diffraction blurred images and corresponding clear images;

[0018] 3) The rotational diffraction blurred image and the corresponding clear image are simultaneously input into the rotational diffraction hyperspectral convolutional neural network for spectral reconstruction to obtain a hyperspectral image.

[0019] The rotating diffraction hyperspectral imaging system includes a beam splitter, a coded diffraction element, a first image sensor, an imaging lens, and a second image sensor;

[0020] After the parallel light is incident on the beam splitter, it is transmitted and reflected. The transmitted light of the beam splitter is encoded by the coding diffraction element to form a rotational diffraction blurred image on the first image sensor, and the reflected light of the beam splitter is converged by the imaging lens to form a clear image on the second image sensor; the incident end face of the coding diffraction element is set as the coding end face.

[0021] The rotational diffraction hyperspectral convolutional neural network includes a feature extraction module, a feature fusion module, and a hyperspectral image reconstruction module; the rotational diffraction blurred image is input into the first feature extraction module, and the clear image is input into the second feature extraction module. The first output of the first feature extraction module is used as the first input of the hyperspectral image reconstruction module, and the first output of the second feature extraction module is used as the second input of the hyperspectral image reconstruction module. The second output of the first feature extraction module and the second output of the second feature extraction module are channel-concatenated and then input into the feature fusion module. The first output of the feature fusion module is used as the third input of the hyperspectral image reconstruction module, and the second output of the feature fusion module is used as the fourth input of the hyperspectral image reconstruction module. The output of the hyperspectral image reconstruction module is recorded as a hyperspectral image.

[0022] The first feature extraction module and the second feature extraction module have the same structure, and are mainly composed of a feature extraction block, a first two-dimensional convolutional layer and a first activation layer connected in sequence. The input of the feature extraction module serves as the input of the feature extraction block, the output of the feature extraction block serves as the first output of the feature extraction module, and the output of the first activation layer serves as the second output of the feature extraction module.

[0023] The feature fusion module includes a first double residual block, a second two-dimensional convolutional layer and a second activation layer. The input of the feature fusion module serves as the input of the first double residual block. The first double residual block is connected to the second activation layer after passing through the second two-dimensional convolutional layer. The output of the first double residual block serves as the first output of the feature fusion module, and the output of the second activation layer serves as the second output of the feature fusion module.

[0024] The hyperspectral image reconstruction module includes two double residual blocks, two transposed convolution layers, five activation layers and three two-dimensional convolution layers; the fourth input of the hyperspectral image reconstruction module is used as the input of the second double residual block, the second double residual block is connected to the third activation layer after passing through the first transposed convolution layer, the output of the third activation layer is channel-concatenated with the third input of the hyperspectral image reconstruction module and then input into the third double residual block, the third double residual block is connected to the fourth activation layer after passing through the second transposed convolution layer, the output of the fourth activation layer is channel-concatenated with the first input and the second input of the hyperspectral image reconstruction module and then input into the third two-dimensional convolution layer, the third two-dimensional convolution layer is connected to the seventh activation layer after passing through the fifth activation layer, the fourth two-dimensional convolution layer, the sixth activation layer and the fifth two-dimensional convolution layer in sequence, and the output of the seventh activation layer is used as the output of the hyperspectral image reconstruction module.

[0025] The feature extraction block includes six two-dimensional convolutional layers and an eighth activation layer. The input of the feature extraction block is used as the input of the sixth two-dimensional convolutional layer, and the output of the sixth two-dimensional convolutional layer is used as the input of the thirteenth to sixteenth two-dimensional convolutional layers. The outputs of the thirteenth to sixteenth two-dimensional convolutional layers are spliced ​​in the channel dimension and input into the seventh two-dimensional convolutional layer. The seventh two-dimensional convolutional layer is connected to the eighth activation layer, and the output of the eighth activation layer is used as the output of the feature extraction block.

[0026] The double residual block includes three two-dimensional convolutional layers and three activation layers. The input of the double residual block is used as the input of the eighth two-dimensional convolutional layer. The eighth two-dimensional convolutional layer is connected to the tenth activation layer after passing through the ninth activation layer and the ninth two-dimensional convolutional layer in sequence. The output of the tenth activation layer is spliced ​​with the input of the double residual block in the channel dimension and then input into the tenth two-dimensional convolutional layer. The tenth two-dimensional convolutional layer is connected to the eleventh activation layer. The output of the eleventh activation layer and the output of the ninth activation layer are spliced ​​in the channel dimension as the output of the double residual block.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] (1) An additional imaging branch is added by using a beam splitter to improve the quality of reconstructed hyperspectral images;

[0029] (2) A fusion-based hyperspectral image reconstruction method is proposed. Using a dual-branch network structure, the rotational diffraction blurred image and the clear image are input into the convolutional neural network, which can significantly improve the spatial resolution and spectral accuracy of the reconstructed hyperspectral image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of the fusion-based rotational diffraction hyperspectral imaging system;

[0031] Figure 2 Schematic diagram of the network structure of the fusion-based rotational diffraction hyperspectral image reconstruction method;

[0032] Figure 3 It is a structural diagram of the feature extraction block;

[0033] Figure 4 Schematic diagram of the structure of the double residual block;

[0034] Figure 5 Comparison of spatial details between the reconstruction results of the present invention and the traditional non-fusion method;

[0035] Figure 6 The spectral curves of the reconstruction results of the present invention are compared with those of the traditional non-fusion method;

[0036] In the figure: 1. Beam splitter, 2. Encoded diffraction element, 3. First image sensor, 4. Imaging lens, 5. Second image sensor. DETAILED DESCRIPTION

[0037] The present invention will be described in further detail below with reference to the accompanying drawings.

[0038] According to the invention content of the present invention, the complete implementation embodiment and its implementation process are as follows:

[0039] like Figure 1 As shown, the system includes a beam splitter 1, a coded diffraction element 2, a first image sensor 3, an imaging lens 4 and a second image sensor 5;

[0040] After being incident on the beam splitter 1, parallel light is transmitted and reflected. The transmission light axis of the beam splitter 1 is coaxial with the encoding diffraction element 2 and the first image sensor 3. The reflection light axis of the beam splitter 1 is coaxial with the imaging lens 4 and the second image sensor 5. The transmitted light of the beam splitter 1 is encoded by the encoding diffraction element 2, forming a rotationally diffracted blurred image on the first image sensor 3. The reflected light of the beam splitter 1 is converged by the imaging lens 4, forming a clear image on the second image sensor 5. The energy of the transmitted light and the reflected light through the beam splitter 1 is the same. The first image sensor 3 is located at the focus of the encoding diffraction element 2, and the second image sensor 5 is located at the focus of the imaging lens 4. The incident end face of the encoding diffraction element 2 (i.e., the end face of the encoding diffraction element 2 close to the beam splitter 1) is set as the encoding end face.

[0041] The output end face of the coding diffraction element 2 is a plane, and the incident end face of the coding diffraction element 2 is also a plane before coding. The incident end face of the coding diffraction element 2 (i.e., the end face of the coding diffraction element 2 close to the beam splitter 1) is set as the coding end face by the following formula:

[0042]

[0043]

[0044] Wherein, λ(θ) represents the design wavelength of the coding diffraction element 2 at the angle θ, and the working wavelength of the coding diffraction element 2 is [λ min ,λ max ],λ min represents the minimum operating wavelength of the coding diffraction element 2, λ max represents the maximum operating wavelength of the coded diffraction element 2, N is the number of repetition periods, represents the angle of the coding diffraction element 2 The working wavelength at the position of the coding diffraction element 2 is h(r,θ), the height of the coding diffraction element 2 at the angle θ and the radius r, h0 is the height of the base material of the coding diffraction element 2, and n λ and n0 are the refractive indices of light with wavelength λ in the current coding diffraction element 2 and air, respectively. DOE is the focal length of the coding diffraction element 2, and m is the diffraction order.

[0045] The focal length of the coding diffraction element 2 is equal to that of the imaging lens 4. The image plane size and resolution of the first image sensor 3 and the second image sensor 5 are the same.

[0046] The imaging model of the rotational diffraction imaging branch composed of the beam splitter 1, the coding diffraction element 2, and the first image sensor 3 can be described as follows:

[0047] Assume that a point light source at infinite distance illuminates the imaging system. The incident light field in front of the imaging system can be approximated as a plane wave light field. The complex amplitude expression of the incident light field is:

[0048]

[0049] Where u0(x,y) represents the complex amplitude of the incident light field, A0 is the amplitude of the incident light field, and φ0 is the phase of the incident light field. Beam splitter 1 splits the incident light energy into two without changing the phase. The complex amplitude of the light field on the front surface of the coding diffraction element 2 is:

[0050]

[0051] Among them, u 11 (x, y) represents the complex amplitude of the light field on the front surface of the coding diffraction element 2.

[0052] The coding diffraction element 2 modulates the phase of the incident light field. The phase of the coding diffraction element 2 can be expressed as:

[0053]

[0054] Among them, φ h (x, y) represents the phase modulation term brought by the coding diffraction element 2, and h(x, y) represents the surface profile height of the coding diffraction element 2.

[0055] The complex amplitude of the light field on the rear surface of the coding diffraction element 2 can be expressed as:

[0056]

[0057] Among them, u 12 (x, y) represents the complex amplitude of the light field on the rear surface of the coding diffraction element 2.

[0058] After this light field propagates through Fresnel diffraction at a distance z and reaches the sensor plane, the complex amplitude of the light field on the plane where the first image sensor 3 is located can be described as:

[0059]

[0060] Among them, u 13 (x', y') represents the complex amplitude of the light field on the first image sensor plane, z represents the distance between the rear surface of the coding diffraction element 2 and the first image sensor 3, and k = 2π / λ is the wave vector. The intensity collected by the sensor plane is the point spread function of the system, which is expressed as:

[0061]

[0062] Among them, p λ (x', y') represents the point spread function of the system at wavelength λ collected by the first image sensor 3, Represents a Fourier transform operation.

[0063] The formation model of the rotational diffraction blurred image is as follows:

[0064]

[0065] Among them J 1c (x, y) represents the rotational diffraction blurred image under the c color channel, c represents the color channel, and satisfies c∈{r, g, b}, r, g, b are red, green, and blue channels respectively, s represents the number of spectral channels of the hyperspectral real image, which is set to 31 in this embodiment, I(x, y; λ i ) represents the hyperspectral real image slice in the i-th spectral channel, represents the convolution operation, p 1c (x,y;λ i ) represents the point spread function of the rotational diffraction imaging branch in the c color channel of the i-th spectral channel.

[0066] The clear image formation model of the clear imaging branch composed of the beam splitter 1, the imaging lens 4, and the second image sensor 5 can be described as follows:

[0067]

[0068] Among them J 2c (x, y) represents the clear image under the c color channel, c represents the color channel, and satisfies c∈{r, g, b}, r, g, b are red, green, and blue channels respectively, s represents the number of spectral channels of the real hyperspectral image, I(x, y; λ i ) represents the hyperspectral real image slice in the i-th spectral channel, represents the convolution operation, p 2c (x,y;λ i ) represents the point spread function of the clear imaging branch under the c color channel of the i-th spectral channel.

[0069] Using the fusion-based rotational diffraction hyperspectral imaging system, the rotational diffraction blurred image J can be obtained simultaneously. 1c (x,y) and clear image J 2c (x, y). Different spectral channels in a rotationally diffraction-blurred image correspond to blur characteristics in different directions, providing a foundation for neural network reconstruction of hyperspectral images. Clear images have a smaller point spread function, higher image resolution, and contain more spatial detail information.

[0070] The rotational diffraction hyperspectral imaging method includes the following steps:

[0071] 1) Build a rotational diffraction hyperspectral imaging system;

[0072] The rotating diffraction hyperspectral imaging system includes a beam splitter 1, a coded diffraction element 2, a first image sensor 3, an imaging lens 4 and a second image sensor 5;

[0073] After being incident on the beam splitter 1, parallel light is transmitted and reflected. The transmission light axis of the beam splitter 1 is coaxial with the encoding diffraction element 2 and the first image sensor 3. The reflection light axis of the beam splitter 1 is coaxial with the imaging lens 4 and the second image sensor 5. The transmitted light of the beam splitter 1 is encoded by the encoding diffraction element 2, forming a rotationally diffracted blurred image on the first image sensor 3. The reflected light of the beam splitter 1 is converged by the imaging lens 4, forming a clear image on the second image sensor 5. The energy of the transmitted light and the reflected light through the beam splitter 1 is the same. The first image sensor 3 is located at the focus of the encoding diffraction element 2, and the second image sensor 5 is located at the focus of the imaging lens 4. The incident end face of the encoding diffraction element 2 (i.e., the end face of the encoding diffraction element 2 close to the beam splitter 1) is set as the encoding end face.

[0074] 2) Using a rotational diffraction hyperspectral imaging system to collect rotational diffraction blurred images and corresponding clear images;

[0075] 3) The rotational diffraction blurred image and the corresponding clear image are simultaneously input into the rotational diffraction hyperspectral convolutional neural network for spectral reconstruction to obtain a hyperspectral image.

[0076] like Figure 2 As shown, the rotational diffraction hyperspectral convolutional neural network includes a feature extraction module, a feature fusion module, and a hyperspectral image reconstruction module; the rotational diffraction blurred image is input into the first feature extraction module, and the clear image is input into the second feature extraction module. The first output of the first feature extraction module is used as the first input of the hyperspectral image reconstruction module, and the first output of the second feature extraction module is used as the second input of the hyperspectral image reconstruction module. The second output of the first feature extraction module and the second output of the second feature extraction module are channel-concatenated and then input into the feature fusion module. The first output of the feature fusion module is used as the third input of the hyperspectral image reconstruction module, and the second output of the feature fusion module is used as the fourth input of the hyperspectral image reconstruction module. The output of the hyperspectral image reconstruction module is recorded as a hyperspectral image.

[0077] The first and second feature extraction modules have the same structure, consisting primarily of a feature extraction block, a first two-dimensional convolutional layer, and a first activation layer, connected in sequence. The input of the feature extraction module serves as the input of the feature extraction block, the output of the feature extraction block serves as the first output of the feature extraction module, and the output of the first activation layer serves as the second output of the feature extraction module. Specifically, the first two-dimensional convolutional layer has a convolution kernel size of 2×2 and a stride of 2, and the activation function of the first activation layer is PReLU.

[0078] The feature fusion module includes a first dual residual block, a second two-dimensional convolutional layer, and a second activation layer. The input of the feature fusion module serves as the input of the first dual residual block. The first dual residual block is connected to the second activation layer after passing through the second two-dimensional convolutional layer. The output of the first dual residual block serves as the first output of the feature fusion module, and the output of the second activation layer serves as the second output of the feature fusion module. In a specific implementation, the second two-dimensional convolutional layer has a convolution kernel size of 2×2 and a stride of 2, and the activation function of the second activation layer is PReLU.

[0079] like Figure 3 As shown, the feature extraction block includes six 2D convolutional layers and an eighth activation layer. The input of the feature extraction block serves as the input of the sixth 2D convolutional layer, and the output of the sixth 2D convolutional layer serves as the input of the 13th to 16th 2D convolutional layers. The outputs of the 13th to 16th 2D convolutional layers are concatenated in the channel dimension and input into the seventh 2D convolutional layer. The seventh 2D convolutional layer is connected to the eighth activation layer, and the output of the eighth activation layer serves as the output of the feature extraction block. In the specific implementation, the convolution kernel size of the sixth to seventh 2D convolutional layers is 3×3 with a stride of 1. The convolution kernel sizes of the 13th to 16th 2D convolutional layers are 7×7, 5×5, 3×3, and 1×1, respectively. The activation function of the eighth activation layer is PReLU.

[0080] The hyperspectral image reconstruction module includes two double residual blocks, two transposed convolutional layers, five activation layers, and three two-dimensional convolutional layers. The fourth input of the hyperspectral image reconstruction module serves as the input of the second double residual block. The second double residual block is connected to the third activation layer after passing through the first transposed convolutional layer. The output of the third activation layer is channel-concatenated with the third input of the hyperspectral image reconstruction module and then input into the third double residual block. The third double residual block is connected to the fourth activation layer after passing through the second transposed convolutional layer. The output of the fourth activation layer is channel-concatenated with the first and second inputs of the hyperspectral image reconstruction module and then input into the third two-dimensional convolutional layer. The third two-dimensional convolutional layer is connected to the seventh activation layer after passing through the fifth activation layer, the fourth two-dimensional convolutional layer, the sixth activation layer, and the fifth two-dimensional convolutional layer. The output of the seventh activation layer serves as the output of the hyperspectral image reconstruction module. In a specific implementation, the third to fifth two-dimensional convolutional layers have a convolution kernel size of 3×3 and a stride of 1. The activation function of the third to seventh activation layers is PReLU.

[0081] like Figure 4 As shown, the first to third dual residual blocks have the same structure. They consist of three 2D convolutional layers and three activation layers. The input of the dual residual block serves as the input of the eighth 2D convolutional layer. The eighth 2D convolutional layer is sequentially connected to the ninth and ninth 2D convolutional layers before being connected to the tenth activation layer. The output of the tenth activation layer is concatenated with the input of the dual residual block in the channel dimension and then input into the 11th 2D convolutional layer. The 11th 2D convolutional layer is connected to the 11th activation layer. The output of the 11th activation layer is concatenated with the output of the ninth activation layer in the channel dimension and serves as the output of the dual residual block. In a specific implementation, the eighth to tenth 2D convolutional layers have a convolution kernel size of 3×3 and a stride of 1. The activation function of the ninth to eleventh activation layers is PReLU.

[0082] In this embodiment, the data set used for training is set up in detail as follows: 21 hyperspectral images are selected from the known public hyperspectral KAIST data set, 31 hyperspectral images are selected from the CAVE data set, and 158 hyperspectral images are selected from the ICVL data set as the training set. Another 9 hyperspectral images are selected from the known public hyperspectral KAIST data set, and 20 hyperspectral images are selected from the ICVL data set as the test set for evaluating the quality of the reconstruction results. Using the aforementioned rotational diffraction blurred image formation model, the hyperspectral image and the point spread function of the rotational diffraction imaging branch generate a rotational diffraction blurred image. Using the aforementioned clear image formation model, the hyperspectral image and the point spread function of the clear imaging branch generate a clear image. A hyperspectral image and the corresponding rotational diffraction blurred image, the clear image, form a data pair, and multiple data pairs form the data set for training and testing.

[0083] Using the training set constructed above, Figure 2The convolutional neural network shown in the figure is trained. During the training process, Gaussian noise with a mean of 0 and a standard deviation of 0.01 is applied to all input images. The Adam optimizer is used and the learning rate is set to 10 -4 , the learning rate is reduced by half every 20 epochs, the batch size is set to 16, and the training ends when the number of epochs is equal to 100. The loss function formula used in convolutional neural network training is:

[0084] L=|XY|

[0085] Where X is the reconstructed hyperspectral image, Y is the real hyperspectral image, and || represents the absolute value operation.

[0086] The rotational diffraction blurred image and the clear image in the test dataset are input into the trained neural network to obtain a 31-channel hyperspectral image output.

[0087] In order to verify the effectiveness of the present invention, Figure 2 Only one of the two input branches of the neural network shown above was retained, and only one of the rotated diffraction image or the clear image was used as input to reconstruct the hyperspectral image. The reconstruction results were compared with those of the present invention, as shown in Table 1. The evaluation metrics were peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and spectral angle mapping (SAM). PSNR and SSIM reflect spatial resolution to a certain extent, with higher values ​​being better. SAM is used to evaluate spectral accuracy, with lower values ​​being better. Compared to other reconstruction methods, the present invention achieved superior results in all metrics.

[0088] Table 1 Comparison of reconstruction results between the proposed method and the non-fusion method

[0089]

[0090]

[0091] As can be seen from Table 1, the performance of the hyperspectral image reconstructed using only one image in terms of PSNR, SSIM, and SAM is not as good as that of the method of the present invention, indicating that the fusion method proposed in the present invention can effectively improve the spatial resolution and spectral accuracy of the reconstruction results.

[0092] In addition to quantitatively evaluating the reconstruction results, we also present the spatial details and light curves of the reconstructed hyperspectral images. Figure 5 (a1) and (a2) are two grayscale visualization images of the true value. Figure 5 (b1) and (b2) are two grayscale visualization images of the reconstruction results using only the rotational diffraction blurred image. Figure 5(c1) and (c2) are two grayscale visualization images of the fusion reconstruction results of the method of the present invention. It can be seen that the reconstruction results of the method of the present invention contain more spatial detail information and have better spatial resolution. Figure 6 (a), (b) and (c) are the spectral curves of the three color blocks respectively. It can be seen from the figure that the spectral curve of the fusion method of the present invention is closer to the true value than the spectral curve of the traditional non-fusion method.

Claims

1. A fusion-based rotational diffraction hyperspectral imaging system, characterized in that: It comprises a beam splitter (1), a coding diffraction element (2), a first image sensor (3), an imaging lens (4) and a second image sensor (5); After the parallel light is incident on the beam splitter (1), transmission and reflection occur. The transmitted light of the beam splitter (1) is encoded by the encoding diffraction element (2) to form a rotational diffraction blurred image on the first image sensor (3). The reflected light of the beam splitter (1) is converged by the imaging lens (4) to form a clear image on the second image sensor (5). The incident end face of the encoding diffraction element (2) is set as the encoding end face. The incident end face of the coding diffraction element (2) is set as the coding end face by the following formula: Wherein, λ(θ) represents the design wavelength of the coding diffraction element (2) at the angle θ, and the working wavelength of the coding diffraction element (2) is [λ min ,λ max ],λ min represents the minimum operating wavelength of the coding diffraction element (2), λ max represents the maximum operating wavelength of the coded diffraction element (2), N is the number of repetition periods, represents the angle of the coding diffraction element (2) The working wavelength at θ is θ, h(r,θ) represents the height of the coding diffraction element (2) at an angle of θ and a radius of r, h0 is the height of the base material of the coding diffraction element (2), and n λ and n0 are the refractive indices of the light with wavelength λ in the current coding diffraction element (2) and in the air, respectively. DOE is the focal length of the coding diffraction element (2), and m is the diffraction order.

2. The fusion-based rotational diffraction hyperspectral imaging system according to claim 1, characterized in that: The focal length of the coding diffraction element (2) is equal to that of the imaging lens (4); the image plane size and resolution of the first image sensor (3) and the second image sensor (5) are the same.

3. A fusion-based rotational diffraction hyperspectral imaging method, characterized in that: The following steps are involved: 1) Building the rotational diffraction hyperspectral imaging system according to claim 1; 2) Using a rotational diffraction hyperspectral imaging system to collect rotational diffraction blurred images and corresponding clear images; 3) The rotational diffraction blurred image and the corresponding clear image are simultaneously input into the rotational diffraction hyperspectral convolutional neural network for spectral reconstruction to obtain a hyperspectral image.

4. The method for fusion-based rotational diffraction hyperspectral imaging according to claim 3, characterized in that: The rotational diffraction hyperspectral convolutional neural network includes a feature extraction module, a feature fusion module, and a hyperspectral image reconstruction module; the rotational diffraction blurred image is input into the first feature extraction module, and the clear image is input into the second feature extraction module. The first output of the first feature extraction module is used as the first input of the hyperspectral image reconstruction module, and the first output of the second feature extraction module is used as the second input of the hyperspectral image reconstruction module. The second output of the first feature extraction module and the second output of the second feature extraction module are channel-concatenated and then input into the feature fusion module. The first output of the feature fusion module is used as the third input of the hyperspectral image reconstruction module, and the second output of the feature fusion module is used as the fourth input of the hyperspectral image reconstruction module. The output of the hyperspectral image reconstruction module is recorded as a hyperspectral image.

5. The method for fusion-based rotational diffraction hyperspectral imaging according to claim 4, characterized in that: The first feature extraction module and the second feature extraction module have the same structure, and are mainly composed of a feature extraction block, a first two-dimensional convolutional layer and a first activation layer connected in sequence. The input of the feature extraction module serves as the input of the feature extraction block, the output of the feature extraction block serves as the first output of the feature extraction module, and the output of the first activation layer serves as the second output of the feature extraction module.

6. The method for fusion-based rotational diffraction hyperspectral imaging according to claim 4, characterized in that: The feature fusion module includes a first double residual block, a second two-dimensional convolutional layer and a second activation layer. The input of the feature fusion module serves as the input of the first double residual block. The first double residual block is connected to the second activation layer after passing through the second two-dimensional convolutional layer. The output of the first double residual block serves as the first output of the feature fusion module, and the output of the second activation layer serves as the second output of the feature fusion module. The hyperspectral image reconstruction module includes two double residual blocks, two transposed convolutional layers, five activation layers and three two-dimensional convolutional layers. The fourth input of the hyperspectral image reconstruction module is used as the input of the second double residual block. The second double residual block is connected to the third activation layer after passing through the first transposed convolution layer. The output of the third activation layer is channel-concatenated with the third input of the hyperspectral image reconstruction module and then input into the third double residual block. The third double residual block is connected to the fourth activation layer after passing through the second transposed convolution layer. The output of the fourth activation layer is channel-concatenated with the first input and the second input of the hyperspectral image reconstruction module and then input into the third two-dimensional convolution layer. The third two-dimensional convolution layer is connected to the seventh activation layer after passing through the fifth activation layer, the fourth two-dimensional convolution layer, the sixth activation layer and the fifth two-dimensional convolution layer in sequence. The output of the seventh activation layer is used as the output of the hyperspectral image reconstruction module.

7. The method for fusion-based rotational diffraction hyperspectral imaging according to claim 5, characterized in that: The feature extraction block includes six two-dimensional convolutional layers and an eighth activation layer. The input of the feature extraction block is used as the input of the sixth two-dimensional convolutional layer, and the output of the sixth two-dimensional convolutional layer is used as the input of the thirteenth to sixteenth two-dimensional convolutional layers. The outputs of the thirteenth to sixteenth two-dimensional convolutional layers are spliced ​​in the channel dimension and input into the seventh two-dimensional convolutional layer. The seventh two-dimensional convolutional layer is connected to the eighth activation layer, and the output of the eighth activation layer is used as the output of the feature extraction block.

8. The method for fusion-based rotational diffraction hyperspectral imaging according to claim 6, characterized in that: The double residual block includes three two-dimensional convolutional layers and three activation layers. The input of the double residual block is used as the input of the eighth two-dimensional convolutional layer. The eighth two-dimensional convolutional layer is connected to the tenth activation layer after passing through the ninth activation layer and the ninth two-dimensional convolutional layer in sequence. The output of the tenth activation layer is spliced ​​with the input of the double residual block in the channel dimension and then input into the tenth two-dimensional convolutional layer. The tenth two-dimensional convolutional layer is connected to the eleventh activation layer. The output of the eleventh activation layer and the output of the ninth activation layer are spliced ​​in the channel dimension as the output of the double residual block.

Images