Refraction-diffraction hybrid double-focus image fusion multispectral imaging system and method

Abstract

The present invention discloses a refractive-diffractive hybrid twice-focused image fusion multispectral imaging system and method. The system is mainly composed of a coded diffraction element, a positive lens and a detector placed in sequence along the optical axis. The uncoded end face of the coded diffraction element is in contact with the positive lens and fixedly connected. The coded diffraction element and the positive lens together form a refractive-diffractive hybrid module. The refractive-diffractive hybrid module and the detector are arranged at intervals. The incident light is sequentially converged by the coded diffraction element and the positive lens and then imaged on the detector; by adjusting the imaging position of the detector, the system acquires two RGB images of the imaging object after two focusing; the method fuses and reconstructs the two RGB images acquired by the system to obtain a spectral data cube of the imaging object. Compared with the traditional method, the method of the present invention obtains more information for use in the reconstruction process, and can effectively improve the spatial resolution and spectral accuracy of the reconstruction result.

Description

Technical Field

[0001] The present invention belongs to a spectral coding imaging system and method in spectral imaging technology, and specifically relates to a refraction-diffraction mixed twice-focus image fusion multi-spectral imaging system and method. Background Art

[0002] Spectral imaging technology is a new type of technology that uses single or multiple spectral channels for spectral data collection and processing, image display and analysis and interpretation. Since spectral imaging combines spectroscopy and imaging technology, it can simultaneously obtain the two-dimensional spatial information and one-dimensional spectral information of the target to be measured, so it has the advantages of both spectroscopy and imaging. At present, it plays a vital role in agriculture, resource exploration, remote sensing detection, military reconnaissance and other fields.

[0003] Traditional three-dimensional spectral cube data acquisition can be divided into two methods: (1) Place a narrowband filter in front of the imaging system, obtain 2D narrowband spectral features in a single exposure, replace filters with different spectral passbands, and perform multiple exposures to obtain three-dimensional spectral data; (2) Use prisms, gratings and other dispersive elements to distinguish light of different wavelengths, and only obtain one-dimensional spectral information and one-dimensional spatial information in one imaging. Scan the target through push scanning, swing scanning and other methods, and splice the one-dimensional spectral information and spatial information of multiple imaging to obtain three-dimensional spectral data. This traditional data acquisition method sacrifices time and is difficult to achieve dynamic scene measurement, which limits the application of spectral imaging technology in many scenarios. In recent years, spectral imaging technology has developed in the direction of lightweight, small, and snapshot imaging.

[0004] In 2019, Daniel S. Jeon et al. proposed a diffractive optical element with a rotating PSF characteristic, which combined with deep neural network image restoration technology to achieve hyperspectral imaging with a spectral interval of 10nm in the visible light band of 420-660nm. Since the system only uses one diffractive optical element to replace the complex optical system in the traditional spectral imaging system, it has the advantages of light weight and snapshot imaging. However, the system also has disadvantages such as large initial image blur, limited spatial resolution of reconstruction results, large consumption of computing resources, and long network training time.

[0005] Domestic scholars have optimized the system based on the rotational diffraction mechanism and proposed an optimization design method for diffraction optical elements; proposed the use of a refractive-diffractive hybrid system to shorten the back intercept of the system and reduce the amount of initial image blur; proposed a new image reconstruction algorithm; proposed adding a front optical system in front of the diffraction imaging system for long-distance imaging and other improvements.

[0006] In addition, in the field of remote sensing multispectral imaging, some scholars have also conducted research on spatial-spectral image fusion, proposing to fuse high spatial resolution panchromatic images with high spectral resolution multispectral images to obtain multispectral images with high spatial resolution and high spectral resolution. However, this type of research generally improves image quality after obtaining multispectral data. This type of work is usually based on simulation data sets, and the number of spectral channels of multispectral images is generally small, making it difficult to apply them in actual scenarios. Summary of the invention

[0007] In order to solve the technical problems existing in the background technology, aimed at the problem of limited spatial resolution of the reconstruction results of the existing rotational diffraction system, the present invention draws on the idea of ​​improving the image quality by fusion of spatial spectrum images, and proposes a refractive-diffraction mixed twice-focus image fusion multi-spectral imaging system and method. With the refractive-diffraction mixed imaging system as the hardware basis, two-focus imaging is performed to obtain a blurred image with rotational diffraction spectral characteristics and a clear image with a higher spatial resolution, and fusion imaging is performed, which can effectively improve the spectral image reconstruction quality of the rotational diffraction spectral imaging system.

[0008] The technical solution for achieving the purpose of the present invention is:

[0009] 1. A refractive-diffractive hybrid double-focus image fusion multispectral imaging system

[0010] The system includes a coded diffractive element, a positive lens and a detector;

[0011] The coded diffraction element, the positive lens and the detector are placed in sequence along the optical axis, the uncoded end face of the coded diffraction element is in contact with and fixedly connected to the positive lens, the coded diffraction element and the positive lens together form a refractive-diffraction hybrid module, the refractive-diffraction hybrid module and the detector are arranged at intervals, and the incident light is sequentially converged by the coded diffraction element and the positive lens and then imaged on the detector; the detector is moved along the optical axis, the distance between the detector and the refractive-diffraction hybrid module is adjusted, when the detector is located at a first focusing imaging position, the detector collects and obtains an RGB image with a rotational diffraction blur feature; when the detector is located at a second focusing imaging position, the detector collects and obtains a clear RGB image.

[0012] The coding end face of the coding diffraction element is set by the following formula:

[0013]

[0014]

[0015] Where λ(θ) represents the wavelength at the angle θ of the coding diffraction element. The working wavelength of the coding diffraction element is [λ min ,λ max ],λmin represents the minimum operating wavelength of the coded diffraction element, λ max represents the maximum operating wavelength of the coded diffraction element, N is the number of repetition periods, Indicates the angle of the coded diffraction element The working wavelength at the position of the coding diffraction element is h(r,θ), h(r,θ) represents the height of the coding diffraction element at an angle of θ and a radius of r, h0 represents the height of the base material of the coding diffraction element, and n λ and n0 are the refractive indices of the light with wavelength λ in the current coding diffraction element and in the air, respectively. DOE is the focal length of the coding diffraction element, and m is the diffraction order.

[0016] When the detector is located at the first focusing imaging position, the distance between the detector and the refraction-diffraction hybrid module is z1, and the calculation formula is as follows:

[0017]

[0018] When the detector is located at the second focusing imaging position, the distance between the detector and the refraction-diffraction hybrid module is z2, and the calculation formula is as follows:

[0019] z2=f lens

[0020] Among them, f DOE is the focal length of the coding diffraction element, f lens is the focal length of the positive lens.

[0021] The focal length of the positive lens satisfies the following formula:

[0022]

[0023] Where z is the focus distance when the coding diffraction element plays a major role, satisfying z <f DOE , z=z1.

[0024] 2. A double-focus image fusion multispectral imaging method with mixed refraction and diffraction

[0025] The method comprises the following steps:

[0026] S1: Build a multispectral imaging system, which includes a coded diffraction element, a positive lens and a detector;

[0027] The coded diffraction element, the positive lens and the detector are placed in sequence along the optical axis, the uncoded end surface of the coded diffraction element is in contact with and fixedly connected to the positive lens, the coded diffraction element and the positive lens together form a refractive-diffractive hybrid module, the refractive-diffractive hybrid module and the detector are arranged at intervals, and the incident light is sequentially converged by the coded diffraction element and the positive lens and then imaged on the detector;

[0028] S2: Use the multispectral imaging system to shoot the imaging object, move the detector along the optical axis, and adjust the distance between the detector and the refraction-diffraction hybrid module. When the detector is located at the first focus imaging position, the detector acquires an RGB image with rotational diffraction blur characteristics; when the detector is located at the second focus imaging position, the detector acquires a clear RGB image.

[0029] S3: Use the fusion reconstruction algorithm to fuse and reconstruct the two RGB images collected in S2 to obtain the spectral data cube of the imaging object.

[0030] The coding end face of the coding diffraction element is set by the following formula:

[0031]

[0032]

[0033] Where λ(θ) represents the wavelength at the angle θ of the coding diffraction element. The working wavelength of the coding diffraction element is [λ min ,λ max ],λ min represents the minimum operating wavelength of the coded diffraction element, λ max represents the maximum operating wavelength of the coded diffraction element, N is the number of repetition periods, Indicates the angle of the coded diffraction element The working wavelength at the position of the coding diffraction element is h(r,θ), h(r,θ) represents the height of the coding diffraction element at an angle of θ and a radius of r, h0 represents the height of the base material of the coding diffraction element, and n λ and n0 are the refractive indices of the light with wavelength λ in the current coding diffraction element and in the air, respectively. DOE is the focal length of the coding diffraction element, and m is the diffraction order.

[0034] When the detector is located at the first focusing imaging position, the distance between the detector and the refraction-diffraction hybrid module is z1, and the calculation formula is as follows:

[0035]

[0036] When the detector is located at the second focusing imaging position, the distance between the detector and the refraction-diffraction hybrid module is z2, and the calculation formula is as follows:

[0037] z2=f lens

[0038] Among them, f DOE is the focal length of the coding diffraction element, f lens is the focal length of the positive lens.

[0039] The focal length of the positive lens satisfies the following formula:

[0040]

[0041] Where z is the focus distance when the coding diffraction element plays a major role, satisfying z <f DOE , z=z1.

[0042] The fusion reconstruction algorithm includes a spatial-spectral image fusion algorithm, a neural network based on a dual-branch structure, a neural network based on two-stage reconstruction, a fusion neural network based on a multi-scale convolution kernel, and a deep convolutional neural network.

[0043] The deep convolutional neural network is specifically:

[0044] The input of the convolutional neural network is an image of the RGB images collected by the two focusing operations merged in the channel dimension. The deep convolutional neural network includes an encoding part, a decoding part, and a spectral upsampling residual connection part. The input of the deep convolutional neural network is respectively input into the encoding part and the spectral upsampling residual connection part, and the output of the encoding part is input into the decoding part. The output of the decoding part and the output of the spectral upsampling residual connection part are element-wise added to each other to obtain the output of the deep convolutional neural network.

[0045] The encoding part includes five residual convolution blocks and four downsampling layers. In a specific implementation, the pooling function of the downsampling layer is a maximum pooling function. The input of the encoding part is input to the first residual convolution block. The first residual convolution block is sequentially connected to the second residual convolution block after passing through the first downsampling layer, the second residual convolution block, the second downsampling layer, the third residual convolution block, the third downsampling layer, the fourth residual convolution block and the fourth downsampling layer. The output of the first residual convolution block is used as the first output of the encoding part, the output of the second residual convolution block is used as the second output of the encoding part, the output of the third residual convolution block is used as the third output of the encoding part, the output of the fourth residual convolution block is used as the fourth output of the encoding part, and the output of the fifth residual convolution block is used as the fifth output of the encoding part. The first to fifth outputs of the encoding part are respectively input to the first to fifth inputs of the decoding part in sequence;

[0046] The decoding part includes four transposed convolution layers, four residual convolution blocks and a first 2D convolution layer, the fifth input of the decoding part is input to the first transposed convolution layer, the output of the first transposed convolution layer is cascaded with the fourth input of the decoding part and input to the sixth residual convolution block, the output of the sixth residual convolution block is connected to the second transposed convolution layer, the output of the second transposed convolution layer is cascaded with the third input of the decoding part and input to the seventh residual convolution block, the output of the seventh residual convolution block is connected to the third transposed convolution layer, the output of the third transposed convolution layer is cascaded with the second input of the decoding part and input to the eighth residual convolution block, the output of the eighth residual convolution block is connected to the fourth transposed convolution layer, the output of the fourth transposed convolution layer is cascaded with the first input of the decoding part and input to the ninth residual convolution block, the output of the ninth residual convolution block is connected to the first 2D convolution layer, and the output of the first 2D convolution layer is used as the output of the decoding part;

[0047] The spectral upsampling residual connection part includes a spectral upsampling residual module. In a specific implementation, the spectral upsampling residual module is a second 2D convolution layer. The input and output of the spectral upsampling residual connection part are respectively the input and output of the spectral upsampling residual module.

[0048] The structures of the 9 residual convolution blocks are the same. Each residual convolution block includes 3 2D convolution layers, 3 batch normalization layers, 2 activation layers and an SE attention module. In the specific implementation, the activation function of the activation layer is ReLu. The inputs of the residual convolution blocks are respectively input into the third 2D convolution layer and the fourth 2D convolution layer. The outputs of the third 2D convolution layer are successively taken as the attention input features through the first activation layer, the first batch normalization layer and the fifth 2D convolution layer. The outputs of the attention input features after being input into the SE attention module are multiplied by the channels with the attention input features and the outputs are taken as the intermediate features. The outputs of the fourth 2D convolution layer after the second batch normalization layer and the intermediate features are element-wise added and input into the second activation layer. The outputs of the second activation layer after the third batch normalization layer are taken as the outputs of the residual convolution block.

[0049] Since different angles and directions are designed according to wavelengths during the design of the diffraction element, the point spread function under the coded diffraction element exhibits a rotation characteristic. The point spread function has a similar shape in the entire spectrum, but the rotation angles are different in different spectrums. The rotation characteristic of the point spread function is reflected in the imaging result at a distance of z1. Specifically, the blurred features in different directions of the image represent the corresponding spectral features. If the blurred image is directly used for reconstruction, there will usually be a problem of compromise between spatial resolution and spectral resolution, and the spatial resolution of the reconstruction result is limited.

[0050] If the clear imaging result at distance z2 is directly used for reconstruction, since the image does not have spectral blur characteristics, the reconstructed result has a high spatial resolution and contains more spatial detail information, but the spectral accuracy is poor.

[0051] After the above steps, the images obtained by the two focusing operations can be fused and reconstructed to improve the spatial resolution and spectral accuracy of the reconstructed spectral data cube.

[0052] Compared with the prior art, the present invention has the following significant advantages:

[0053] (1) The present invention studies the imaging model of the refractive-diffractive hybrid imaging system and explores new application methods of the refractive-diffractive hybrid system;

[0054] (2) The present invention obtains a blurred image with a rotational diffraction blur feature and a clear image with more spatial detail information by focusing and imaging twice;

[0055] (3) The two focused images obtained by the present invention can provide more effective information for the reconstruction algorithm, and can significantly improve the quality of the reconstruction results of the rotational diffraction spectral imaging system. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 The figure is a flow chart of the multi-spectral imaging method of mixing two focused images by refraction and diffraction.

[0057] Figure 2 Schematic diagram of the imaging model of the refractive-diffractive hybrid optical system.

[0058] Figure 3 Schematic diagram of the refractive-diffractive hybrid spectral encoding imaging system.

[0059] Figure 4 This is a simulation diagram of the system point spread function at two focusing positions.

[0060] Figure 5 Schematic diagram of the fusion reconstruction network structure.

[0061] Figure 6 Schematic diagram of the residual convolution block structure in the fusion reconstruction network.

[0062] Figure 7 The reconstruction results of the two-focus image fusion of some test set scenes are compared with the traditional reconstruction results in some spectral details.

[0063] In the figure: 1. Diffraction element, 2. Positive lens, 3. Detector. DETAILED DESCRIPTION

[0064] The present invention is described in further detail below in conjunction with the accompanying drawings.

[0065] According to the inventive content of the present invention, the complete implementation example and its implementation process are as follows:

[0066] like Figure 2 As shown, the imaging system of the present invention comprises a coding diffraction element 1, a positive lens 2 and a detector 3;

[0067] The coding diffraction element 1, the positive lens 2 and the detector 3 are sequentially placed along the optical axis, the uncoded end face of the coding diffraction element 1 is in contact with the positive lens 2 and fixedly connected, the coding diffraction element 1 and the positive lens 2 form a refractive-diffractive hybrid module, the refractive-diffractive hybrid module and the detector 3 are arranged at intervals, the optical axis of the refractive-diffractive hybrid module is perpendicular to the detector image plane and passes through the center of the detector image plane, and the refractive-diffractive hybrid module is placed in a manner that the positive lens faces the detector. The incident light is sequentially converged by the coding diffraction element 1 and the positive lens 2 and then imaged on the detector 3; the detector 3 is moved along the optical axis to adjust the distance between the detector 3 and the refractive-diffractive hybrid module, when the detector 3 is located at the first focusing imaging position, the diffraction coding element encoding plays a major role at this position, and the detector 3 collects and obtains an RGB image with a rotational diffraction blur feature; when the detector 3 is located at the second focusing imaging position, the positive lens convergence plays a major role at this position, and the detector 3 collects and obtains a clear RGB image. After two focusing imagings, two RGB images can be obtained, one of which contains the blur features of different spectral channels, and the other one has no spectral blur features but has richer spatial detail information.

[0068] The coding end face of the coding diffractive element 1 is set by the following formula:

[0069]

[0070]

[0071] Wherein, λ(θ) represents the wavelength at the angle θ of the coding diffraction element 1, and the working wavelength of the coding diffraction element 1 is [λ min ,λ max ],λ min represents the minimum operating wavelength of the coding diffraction element 1, λ max represents the maximum operating wavelength of the coding diffraction element 1, and N is the number of repetition periods, that is, 360° of the circular coding diffraction element 1 is divided into N equal parts. represents the angle of the coding diffraction element 1 The working wavelength at the position of the coding diffraction element 1 is θ, h(r,θ) represents the height of the coding diffraction element 1 at an angle of θ and a radius of r, h0 represents the height of the base material of the coding diffraction element 1, and n λ and n0 are the refractive indices of the light with wavelength λ in the current coding diffraction element 1 (substrate material) and in the air, respectively. DOEis the focal length of the coding diffraction element 1, and m is the diffraction order.

[0072] The focal length of positive lens 2 satisfies the following formula:

[0073]

[0074] Where z is the focus distance when the coding diffraction element plays a major role, satisfying z <f DOE , z=z1, that is, when the detector 3 is located at the first focusing imaging position, the coding diffraction element plays a major role.

[0075] In this embodiment, the designed working band is 460nm~700nm, and the focal length of the coding diffraction element 1 is designed to be f DOE =100mm, the number of repetition periods is designed to be N=2, and the focal length of positive lens 2 is f lens =40mm.

[0076] The imaging model of the refractive-diffractive hybrid module can be described as follows, assuming that the complex amplitude expression of the incident light field on the front surface of the diffractive element is:

[0077]

[0078] The diffraction element modulates the phase of the incident light field. The phase of the diffraction element can be expressed as:

[0079]

[0080] The light field on the back surface of the diffractive element can be expressed as:

[0081]

[0082] After phase modulation by the positive lens, the light field on the rear surface of the positive lens can be expressed as:

[0083]

[0084] Take a positive lens with a focal length of f lens The lens phase modulation term in the above formula can be described as:

[0085]

[0086] Where k = 2π / l is the wave vector. The light field on the back surface of the positive lens reaches the sensor plane after propagating through Fresnel diffraction at a distance of z. The light field on the sensor plane can be described as:

[0087]

[0088] Take parallel light field incident, the intensity image collected by the sensor plane is the point spread function of the system. Simplifying the above formula, the expression of the point spread function of the system can be obtained as follows:

[0089]

[0090] When the detector 3 is located at the first focusing imaging position, the distance between the detector 3 and the refraction-diffraction hybrid module is z1, and the calculation formula is as follows:

[0091]

[0092] In this embodiment, z1≈28.57 mm. According to the derivation of the imaging model, the point spread function of the system can be described as:

[0093]

[0094] It can be seen that the point spread function of the system remains the same as the traditional diffraction element imaging model without a positive lens, and is mainly modulated by the coding diffraction element. Figure 4 As shown in the left column of each column, the point spread function exhibits a 180° rotation feature.

[0095] When the detector 3 is located at the second focusing imaging position, the distance between the detector 3 and the refraction-diffraction hybrid module is z2, and the calculation formula is as follows:

[0096] z2=f lens

[0097] Among them, f DOE is the focal length of the coding diffraction element 1, f lens is the focal length of the positive lens.

[0098] In this embodiment, z2=40 mm. According to the derivation of the imaging model, the point spread function of the system can be described as:

[0099]

[0100] Compared with the traditional diffraction element imaging model, the influence of the secondary phase factor is reduced, the blur kernel size is smaller, and the point spread function of the system is as follows: Figure 4 As shown in the right column of each column, the imaging result will also contain more spatial detail information.

[0101] like Figure 2 and Figure 3 As shown, the imaging method comprises the following steps:

[0102] S1: Build a multispectral imaging system, which includes a coded diffraction element 1, a positive lens 2 and a detector 3;

[0103] The coded diffraction element 1, the positive lens 2 and the detector 3 are sequentially placed along the optical axis, the uncoded end surface of the coded diffraction element 1 is in contact with and fixedly connected to the positive lens 2, the coded diffraction element 1 and the positive lens 2 form a refractive-diffractive hybrid module, the refractive-diffractive hybrid module and the detector 3 are arranged at intervals, and the incident light is sequentially converged by the coded diffraction element 1 and the positive lens 2 and then imaged on the detector 3;

[0104] S2: Use the multispectral imaging system to photograph the imaging object, move the detector 3 along the optical axis, and adjust the distance between the detector 3 and the refractive-diffractive hybrid module. When the detector 3 is located at the first focusing imaging position, the diffraction coding element encoding plays a major role at this position, and the detector 3 collects and obtains an RGB image with rotational diffraction blur characteristics; when the detector 3 is located at the second focusing imaging position, the positive lens convergence plays a major role at this position, and the detector 3 collects and obtains a clear RGB image. After two focusing imagings, two RGB images can be obtained, one of which contains blur characteristics of different spectral channels, and the other one does not have spectral blur characteristics, but has richer spatial detail information.

[0105] S3: The two RGB images collected in S2 are fused and reconstructed using a fusion reconstruction algorithm to obtain a high-resolution spectral data cube of the imaged object. The fusion reconstruction algorithm simultaneously utilizes the diffraction blur characteristics of the rotational diffraction blurred image and the spatial detail information of the sharp focus image.

[0106] The fusion reconstruction algorithm includes but is not limited to the fusion reconstruction network, where the structural diagrams of the fusion reconstruction network are as follows: Figure 5 and Figure 6 .

[0107] The rotational diffraction blur feature collected at the focus position z1 contains rich rotational spectral features, but lacks spatial detail information. The clear image collected at the focus position z2 does not have rotational spectral features, but contains rich spatial detail information. It is difficult to restore a multispectral image with high spatial resolution and high spectral accuracy using only one of these two images. However, if the above two images with different features can be obtained at the same time, the input of the reconstruction algorithm can be enriched, and the spatial resolution and spectral accuracy of the reconstruction results can be effectively improved. There are many methods for fusion reconstruction algorithms, such as traditional spatial spectrum image fusion algorithms, neural networks based on dual-branch structures, neural networks based on two-stage reconstruction, fusion neural networks based on multi-scale convolution kernels, etc. Only one example is introduced in detail below as an embodiment.

[0108] In this embodiment, a deep convolutional neural network with a residual connection structure of encoding, decoding and spectral upsampling is used. The deep convolutional neural network is as follows: Figure 5As shown. The two RGB images collected by the two focusing operations are combined in the channel dimension as the input of the convolutional neural network. The deep convolutional neural network includes an encoding part, a decoding part, and a spectral upsampling residual connection part. The input of the deep convolutional neural network is respectively input into the encoding part and the spectral upsampling residual connection part, and the output of the encoding part is input into the decoding part. The output of the decoding part and the output of the spectral upsampling residual connection part are element-wise added to each other and the output is used as the output of the deep convolutional neural network.

[0109] The encoding part includes five residual convolution blocks and four downsampling layers. In a specific implementation, the pooling function of the downsampling layer is a maximum pooling function. The input of the encoding part is input to the first residual convolution block. The first residual convolution block is sequentially connected to the second residual convolution block after passing through the first downsampling layer, the second residual convolution block, the second downsampling layer, the third residual convolution block, the third downsampling layer, the fourth residual convolution block and the fourth downsampling layer. The output of the first residual convolution block is used as the first output of the encoding part, the output of the second residual convolution block is used as the second output of the encoding part, the output of the third residual convolution block is used as the third output of the encoding part, the output of the fourth residual convolution block is used as the fourth output of the encoding part, and the output of the fifth residual convolution block is used as the fifth output of the encoding part. The first to fifth outputs of the encoding part are respectively input to the first to fifth inputs of the decoding part.

[0110] The decoding part includes four transposed convolution layers, four residual convolution blocks and a first 2D convolution layer. The fifth input of the decoding part is input to the first transposed convolution layer, and the output of the first transposed convolution layer is cascaded with the fourth input of the decoding part and input to the sixth residual convolution block. The output of the sixth residual convolution block is connected to the second transposed convolution layer, and the output of the second transposed convolution layer is cascaded with the third input of the decoding part and input to the seventh residual convolution block. The output of the seventh residual convolution block is connected to the third transposed convolution layer, and the output of the third transposed convolution layer is cascaded with the second input of the decoding part and input to the eighth residual convolution block. The output of the eighth residual convolution block is connected to the fourth transposed convolution layer, and the output of the fourth transposed convolution layer is cascaded with the first input of the decoding part and input to the ninth residual convolution block. The output of the ninth residual convolution block is connected to the first 2D convolution layer, and the output of the first 2D convolution layer is used as the output of the decoding part. In a specific implementation, the feature output of the decoding part is 25 channels.

[0111] The input image first passes through a residual convolution block and becomes a 256×256×32 feature x in , and then after the first maximum pooling downsampling and residual convolution block, it becomes a 128×128×64 feature x down1 , and then after the second maximum pooling downsampling and residual convolution block, it becomes a 64×64×128 feature x down2, and then after the third maximum pooling downsampling and residual convolution block, it becomes a 32×32×256 feature x down3 , and finally after the fourth maximum pooling downsampling and residual convolution block, it becomes a 16×16×256 feature x down4 , the encoding work is now completed.

[0112] During the decoding process, x down4 After transposed convolution upsampling, and x down3 Perform a jump connection and then input the residual convolution block to obtain a 32×32×128 feature x up1 , x up1 After transposed convolution upsampling, and x down2 Perform a jump connection and then input the residual convolution block to obtain a 64×64×64 feature x up2 , x up2 After transposed convolution upsampling, and x down1 Perform a jump connection and then input the residual convolution block to obtain a 128×128×32 feature x up3 , x up3 After transposed convolution upsampling, and x in Perform a jump connection and then input the residual convolution block to obtain a 256×256×64 feature x up4 , x up4 Finally, a 1×1 convolution block is used to obtain a decoding result of 256×256×25. The decoding result is added to the spectral upsampling result to obtain the reconstructed hyperspectral image.

[0113] The spectral upsampling residual connection part includes a spectral upsampling residual module. In the specific implementation, the spectral upsampling residual module is the second 2D convolution layer. The input and output of the spectral upsampling residual connection part are the input and output of the spectral upsampling residual module respectively. The spectral upsampling residual connection part outputs a spectral upsampling result with a scale of 256×256×25.

[0114] like Figure 6 As shown in the figure, the residual convolution block includes 3 2D convolution layers, 3 batch normalization layers, 2 activation layers and SE attention module. In the specific implementation, the activation function of the activation layer is ReLu. The input of the residual convolution block is respectively input into the third 2D convolution layer and the fourth 2D convolution layer. The output of the third 2D convolution layer is successively taken as the attention input feature through the first activation layer, the first batch normalization layer and the fifth 2D convolution layer. The output of the attention input feature after being input into the SE attention module is multiplied by the channel with the attention input feature and the output is taken as the intermediate feature. The output of the fourth 2D convolution layer after the second batch normalization layer and the intermediate feature are added element by element and input into the second activation layer. The output of the second activation layer after the third batch normalization layer is taken as the output of the residual convolution block.

[0115] In the process of training the network, the data set is set as follows: the 460-700nm spectral band images in the known public hyperspectral KAIST data set are used as the real hyperspectral image references in the training set. The scale of the real hyperspectral images in the training set is 2704×3376×25, and the spectral interval is 10nm. The point spread function of the real hyperspectral image data set and the rotational diffraction spectral imaging system at the focus distance z1 is used to generate the rotational diffraction blurred image, and the point spread function of the real hyperspectral image data set and the rotational diffraction spectral imaging system at the focus distance z2 is used to generate the clear image. The rotational diffraction blurred images generated by multiple groups of simulations, the corresponding clear images and the real hyperspectral images constitute the training set part of the twice-focused image fusion multispectral data set.

[0116] The generation formula of the rotation diffraction blurred image or clear image is as follows:

[0117]

[0118]

[0119] Among them, J 1c (x,y) and J 2c (x, y) represents the rotational diffraction blurred image and the clear image under the c color channel, respectively. c represents the color channel, satisfying c∈{r,g,b}, r,g,b are red, green, and blue channels, respectively. m represents the number of spectral channels of the real hyperspectral image. I(x,y;λ i ) represents the hyperspectral real image slice in the i-th spectral channel, represents the convolution operation, k 1c (x,y;λ i ) represents the point spread function of the c color channel under the i-th spectral channel when the focus distance is z1, k 2c (x,y;λ i ) represents the point spread function of the c color channel under the i-th spectral channel when the focus distance is z2.

[0120] The 460-700nm spectral images in the known public hyperspectral CAVE dataset are selected as the real hyperspectral image references in the test set. The scale of the real hyperspectral images in the test set is 512×512×25, and the spectral interval is 10nm. The point spread function of the real hyperspectral image dataset and the rotational diffraction spectral imaging system at the focus distance z1 is used to generate the rotational diffraction blurred image, and the point spread function of the real hyperspectral image dataset and the rotational diffraction spectral imaging system at the focus distance z2 is used to generate the clear image. The rotational diffraction blurred images generated by multiple groups of simulations, the corresponding clear images and the real hyperspectral images constitute the test set part of the twice-focused image fusion multispectral dataset.

[0121] Using the training set constructed above, Figure 5 The deep neural network shown in the figure is trained. The Adam optimizer is used during the training process and the learning rate is set to 10 -4 , the learning rate drops by half every 50 epochs, the batch size is set to 12, and the condition for the end of training is that the number of epochs is equal to 60. The formula of the total loss function used in convolutional neural network training is:

[0122] L=aL1+βL SSIM +γL Grad

[0123] L1=|XY|

[0124]

[0125]

[0126] Among them, L is the total loss function value, L1 is the content loss function value, and L SSIM is the structural similarity loss function value, L Grad is the value of the gradient difference loss function, a, β, and γ represent the weights of the content loss function, the structural similarity loss function, and the gradient difference loss function, respectively. X is the reconstructed hyperspectral image, Y is the real hyperspectral image, and u X 、u Y are the means of the reconstructed hyperspectral image X and the real hyperspectral image Y, respectively. are the variances of the reconstructed hyperspectral image X and the real hyperspectral image Y, σ XY is the covariance of the reconstructed hyperspectral image X and the real hyperspectral image Y, C1 and C2 are the first and second smoothing factors, respectively, which are constant terms set to prevent the denominator from being zero; is the gradient operator of the reconstructed hyperspectral image; is the gradient operator of the real hyperspectral image, and || represents the absolute value operation.

[0127] The two focused images are fused with the multispectral test dataset and input into the trained convolutional neural network, which outputs the reconstructed hyperspectral image.

[0128]

[0129] In order to verify the advancedness of the method of the present invention, the reconstruction result of the traditional reconstruction using only a single rotation diffraction blurred image is compared with the reconstruction result of the present invention. The two reconstruction results use the same neural network structure and the same training strategy. The input of the traditional single rotation diffraction blurred image reconstruction result is a 3-channel RGB image, and the input of the reconstruction result of the present invention is a 6-channel input composed of a rotation diffraction blurred RGB image and a clear and focused RGB image. The objective evaluation parameters of the present invention and other reconstruction methods in constructing data sets are compared, and the comparison results are shown in Table 1. The peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and spectral angle mapping (SAM) are used as evaluation parameters in the table, where the higher the PSNR and SSIM, the better, and the lower the SAM value, the better. It can be seen from Table 1 that the present invention has achieved better results in all indicators.

[0130] In addition to the comparison of evaluation parameters, Figure 7 The local details of the reconstruction results of the method of the present invention and the traditional single rotation diffraction image at 460nm, 560nm and 660nm spectral bands are compared in the figure. Figure 7 (a1)-(i1) are the reconstruction results and their local details of the method of the present invention, respectively. Figure 7 (a2)-(i2) are the reconstruction results of a traditional single rotating diffraction image and its local details. From the comparison of the two groups of images, it can be seen that the spatial resolution of the reconstruction result of the method of the present invention is significantly higher, and richer spatial detail information is reconstructed.

Claims

1. A refractive-diffractive hybrid double-focus image fusion multispectral imaging system, characterized in that: It comprises a coded diffraction element (1), a positive lens (2) and a detector (3); The coded diffraction element (1), the positive lens (2) and the detector (3) are sequentially placed along the optical axis; the uncoded end face of the coded diffraction element (1) is in contact with and fixedly connected to the positive lens (2); the coded diffraction element (1) and the positive lens (2) together form a refraction-diffraction hybrid module; the refraction-diffraction hybrid module and the detector (3) are arranged at intervals; incident light rays are sequentially converged by the coded diffraction element (1) and the positive lens (2) and then imaged on the detector (3); the detector (3) is moved along the optical axis to adjust the distance between the detector (3) and the refraction-diffraction hybrid module; when the detector (3) is located at a first focusing imaging position, the detector (3) collects and obtains an RGB image with a rotational diffraction blur feature; when the detector (3) is located at a second focusing imaging position, the detector (3) collects and obtains a clear RGB image; The coding end face of the coding diffraction element (1) is set by the following formula: Wherein, λ(θ) represents the wavelength at the angle θ of the coding diffraction element (1), and the working wavelength of the coding diffraction element (1) is [λ min ,λ max ],θ min represents the minimum operating wavelength of the coded diffraction element (1), λ max represents the maximum operating wavelength of the coded diffraction element (1), N is the number of repetition periods, represents the angle of the coding diffraction element (1) The operating wavelength at the position of the coding diffraction element (1), h(r,θ) represents the height of the coding diffraction element (1) at an angle of θ and a radius of r, h0 represents the height of the base material of the coding diffraction element (1), and n λ and n0 are the refractive indices of the light with wavelength λ in the current coding diffraction element (1) and in the air, respectively. DOE is the focal length of the coding diffraction element (1), m is the diffraction order; When the detector (3) is located at the first focusing imaging position, the distance between the detector (3) and the refraction-diffraction hybrid module is z1, and the calculation formula is as follows: When the detector (3) is located at the second focusing imaging position, the distance between the detector (3) and the refraction-diffraction hybrid module is z2, and the calculation formula is as follows: z2=f lens Among them, f DOE is the focal length of the coding diffraction element (1), f lens is the focal length of the positive lens; The focal length of the positive lens (2) satisfies the following formula: Where z is the focal distance when the coding diffraction element plays a major role, satisfying z<f DOE , z=z1.

2. A refraction-diffraction hybrid double-focus image fusion multispectral imaging method, characterized in that: The following steps are involved: S1: Building a multispectral imaging system, the multispectral imaging system comprising a coded diffraction element (1), a positive lens (2) and a detector (3); The coded diffraction element (1), the positive lens (2) and the detector (3) are sequentially placed along the optical axis; the uncoded end surface of the coded diffraction element (1) is in contact with and fixedly connected to the positive lens (2); the coded diffraction element (1) and the positive lens (2) together form a refractive-diffractive hybrid module; the refractive-diffractive hybrid module and the detector (3) are arranged at intervals; incident light rays are sequentially converged by the coded diffraction element (1) and the positive lens (2) and then imaged on the detector (3); S2: Using a multispectral imaging system to photograph an imaging object, moving the detector (3) along the optical axis, and adjusting the distance between the detector (3) and the refraction-diffraction hybrid module; when the detector (3) is located at a first focusing imaging position, the detector (3) acquires an RGB image with a rotational diffraction blur feature; when the detector (3) is located at a second focusing imaging position, the detector (3) acquires a clear RGB image. S3: Use the fusion reconstruction algorithm to fuse and reconstruct the two RGB images collected in S2 to obtain the spectral data cube of the imaging object; The coding end face of the coding diffraction element (1) is set by the following formula: Wherein, λ(θ) represents the wavelength at the angle θ of the coding diffraction element (1), and the working wavelength of the coding diffraction element (1) is [λ min ,λ max ],λ min represents the minimum operating wavelength of the coded diffraction element (1), λ max represents the maximum operating wavelength of the coded diffraction element (1), N is the number of repetition periods, represents the angle of the coding diffraction element (1) The operating wavelength at the position of the coding diffraction element (1), h(r,θ) represents the height of the coding diffraction element (1) at an angle of θ and a radius of r, h0 represents the height of the base material of the coding diffraction element (1), and n λ and n0 are the refractive indices of the light with wavelength λ in the current coding diffraction element (1) and in the air, respectively. DOE is the focal length of the coding diffraction element (1), m is the diffraction order; When the detector (3) is located at the first focusing imaging position, the distance between the detector (3) and the refraction-diffraction hybrid module is z1, and the calculation formula is as follows: When the detector (3) is located at the second focusing imaging position, the distance between the detector (3) and the refraction-diffraction hybrid module is z2, and the calculation formula is as follows: z2=f lens Among them, f DOE is the focal length of the coding diffraction element (1), f lens is the focal length of the positive lens; The focal length of the positive lens (2) satisfies the following formula: Where z is the focal distance when the coding diffraction element plays a major role, satisfying z<f DOE , z=z1.

3. The method for fusion of two-focus images of a refraction-diffraction hybrid multispectral imaging according to claim 2, characterized in that: The fusion reconstruction algorithm includes a spatial-spectral image fusion algorithm, a neural network based on a dual-branch structure, a neural network based on two-stage reconstruction, a fusion neural network based on a multi-scale convolution kernel, and a deep convolutional neural network.

4. The method for fusion of double-focus images with mixed refraction and diffraction according to claim 3 is characterized in that: The deep convolutional neural network is specifically: The input of the convolutional neural network is an image of the RGB images collected by the two focusing operations merged in the channel dimension. The deep convolutional neural network includes an encoding part, a decoding part, and a spectral upsampling residual connection part. The input of the deep convolutional neural network is respectively input into the encoding part and the spectral upsampling residual connection part, and the output of the encoding part is input into the decoding part. The output of the decoding part and the output of the spectral upsampling residual connection part are element-wise added to each other to obtain the output of the deep convolutional neural network. The encoding part includes five residual convolution blocks and four downsampling layers. The input of the encoding part is input to the first residual convolution block. The first residual convolution block is connected to the second residual convolution block after passing through the first downsampling layer, the second residual convolution block, the second downsampling layer, the third residual convolution block, the third downsampling layer, the fourth residual convolution block and the fourth downsampling layer in sequence. The output of the first residual convolution block is used as the first output of the encoding part, the output of the second residual convolution block is used as the second output of the encoding part, the output of the third residual convolution block is used as the third output of the encoding part, the output of the fourth residual convolution block is used as the fourth output of the encoding part, and the output of the fifth residual convolution block is used as the fifth output of the encoding part. The first to fifth outputs of the encoding part are respectively input to the first to fifth inputs of the decoding part in sequence; The decoding part includes four transposed convolution layers, four residual convolution blocks and a first 2D convolution layer, the fifth input of the decoding part is input to the first transposed convolution layer, the output of the first transposed convolution layer is cascaded with the fourth input of the decoding part and input to the sixth residual convolution block, the output of the sixth residual convolution block is connected to the second transposed convolution layer, the output of the second transposed convolution layer is cascaded with the third input of the decoding part and input to the seventh residual convolution block, the output of the seventh residual convolution block is connected to the third transposed convolution layer, the output of the third transposed convolution layer is cascaded with the second input of the decoding part and input to the eighth residual convolution block, the output of the eighth residual convolution block is connected to the fourth transposed convolution layer, the output of the fourth transposed convolution layer is cascaded with the first input of the decoding part and input to the ninth residual convolution block, the output of the ninth residual convolution block is connected to the first 2D convolution layer, and the output of the first 2D convolution layer is used as the output of the decoding part; The spectral upsampling residual connection part includes a spectral upsampling residual module, and the input and output of the spectral upsampling residual connection part are the input and output of the spectral upsampling residual module respectively.

5. The method for multi-spectral imaging of a refractive-diffractive hybrid double-focus image fusion according to claim 4, characterized in that: The structures of the 9 residual convolution blocks are the same. Each residual convolution block includes 3 2D convolution layers, 3 batch normalization layers, 2 activation layers and an SE attention module. The inputs of the convolution blocks are respectively input into the third and fourth 2D convolution layers. The outputs of the third 2D convolution layer are successively taken as the attention input features through the first activation layer, the first batch normalization layer and the fifth 2D convolution layer. The outputs of the attention input features after being input into the SE attention module are multiplied with the attention input features by channel and then used as the intermediate features. The outputs of the fourth 2D convolution layer after the second batch normalization layer and the intermediate features are element-wise added and input into the second activation layer. The outputs of the second activation layer after the third batch normalization layer are taken as the outputs of the residual convolution block.

Images