Multi-frame reconstruction multispectral imaging system based on dynamic space-time joint coding

Abstract

The invention discloses a multi-frame reconstruction multispectral imaging system based on dynamic space-time joint coding. The system comprises a filtering window module used for limiting the spectral range of imaging; the spatial code modulation module is used for providing spectral codes in a spatial dimension; the time coding modulation module is used for carrying out inter-frame modulation on the spectrum codes provided by the space coding modulation module; the optical imaging module is used for imaging the light field modulated by the time coding modulation module on the color acquisition module; the color acquisition module is used for carrying out further spectral coding on the light field to obtain a measurement image; a calculation reconstruction module; the method is used for recovering a high-dimensional multispectral image data cube by using a measurement image. The method has the advantages of being light in weight, low in cost, high in underqualitative relieving capacity, efficient and stable in reconstruction method, high in adaptability and practicability and the like.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of spectral imaging technology, and particularly relates to a multi-frame joint reconstruction and space-time joint modulation coding multispectral imaging system. Specifically, the present application combines spatial coding (color mask) and time coding (dynamic LCD mask), and realizes reconstruction of a high-dimensional spectral data cube through multi-frame observation, and belongs to the cross field of computational spectral imaging and compressed sensing imaging. BACKGROUND

[0002] High-spectral / multi-spectral imaging technology can obtain information of a target scene on tens or even hundreds of continuous wave bands by combining spatial imaging and spectral measurement, and form a three-dimensional data cube containing two-dimensional spatial distribution and one-dimensional spectral information. Compared with traditional RGB color imaging (only 3 wide wave bands), high-spectral imaging provides more abundant spectral details, and has important application value in the fields of vegetation monitoring, atmospheric detection, medical diagnosis, remote sensing, etc.

[0003] In recent years, in order to improve the efficiency of obtaining high-spectral images, computational spectral imaging technology based on compressed sensing has appeared. Among them, more representative technologies are: integral field spectrometer (IFS), coded aperture snapshot spectral imaging (CASSI), prism-mask video imaging spectrometry (Prism-mask Video Imaging Spectrometry), computed tomography imaging spectrometry (Computed Tomography Imaging Spectrometry). These technologies are based on the principles of prism dispersion or grating diffraction spectrometry, and need a collimating mirror group to obtain sufficient dispersion distance, which greatly increases the volume and complexity of the system. Coded aperture snapshot spectral imaging (CASSI) is one of the typical representatives. The CASSI system uses a random coded aperture and a dispersion element to modulate the incident light, compresses and projects the three-dimensional spectral data onto a two-dimensional detector, and realizes single-exposure acquisition of high-spectral information. However, the traditional CASSI has inherent defects: the inverse problem formed by single-frame observation is seriously underdetermined (the unknown spectral information is much more than the observation value), and needs to assume that the scene is sparse in a certain transform domain (such as the spectral dimension) to be reconstructed. In addition, the system is complex (needs high-precision calibration of dispersion relationship), and the reconstruction quality decreases obviously when facing a scene that is not sparse enough.

[0004] To alleviate the ill-posedness of single-frame imaging, researchers have proposed a multi-frame spectral imaging scheme, that is, to obtain multiple observation images with different encoding modulations to increase the effective observation. The multi-frame scheme greatly reduces the ill-conditioned nature of the reconstruction problem by providing more linearly independent measurements, which improves the accuracy of data cube reconstruction. However, the previous multi-frame spectral imaging usually requires the replacement of different physical encoding elements (such as the replacement of different patterned encoding apertures or filters), which increases the system complexity and operation cost, limiting its practicality.

[0005] Another related technology is spectral imaging based on color mask coding. This method uses a specially designed color filter mask (also known as a multi-spectral filter array, similar to the color filter array of an RGB camera but with more wavebands) covering the sensor to encode the spectral components at different pixel positions. The color mask scheme has simple optical path and low cost, but also has some shortcomings: each pixel only measures the waveband transmitted by the corresponding filter, resulting in sparse sampling of high-dimensional spectra, and severe undersampling of waveband components in space, which requires interpolation or prior constraints (underdetermined problem) for reconstruction. At the same time, the color filter usually has a relatively wide spectral passband, and the modulation correlation between different wavebands is strong, which limits the spectral reconstruction accuracy. To obtain higher resolution spectral information, additional sampling is required in space or time.

[0006] In summary, there is still room for improvement in the existing technology in terms of improving the quality and speed of spectral imaging reconstruction. There is an urgent need for a new imaging system and method design that can increase observation information, improve encoding methods, and optimize reconstruction algorithms to alleviate underdetermination, improve reconstruction accuracy, and speed up reconstruction. SUMMARY

[0007] In view of the defects in the prior art, the purpose of the present application is to provide a compact integrated low-cost multi-frame reconstruction multi-spectral imaging system based on dynamic space-time joint coding.

[0008] The technical scheme adopted by the present application is as follows:

[0009] The multi-frame reconstruction multi-spectral imaging system based on dynamic space-time joint coding comprises:

[0010] A filter window module for limiting the spectral range of imaging;

[0011] A spatial encoding modulation module for providing spectral encoding in the spatial dimension;

[0012] A time encoding modulation module for inter-frame modulation of the spectral encoding provided by the spatial encoding modulation module;

[0013] An optical imaging module for imaging the light field modulated by the time encoding modulation module on a color acquisition module;

[0014] The color acquisition module is used for further spectral encoding of the light field to obtain a measurement image.

[0015] The calculation reconstruction module is used for recovering a high-dimensional multi-spectral image data cube by using the measurement image.

[0016] The application further provides a multi-frame reconstruction multi-spectral imaging method based on dynamic space-time joint coding, which utilizes the imaging system, and steps of the imaging method include:

[0017] The incident light is first filtered by the filtering window module to remove incoherent wave bands, then modulated by the space encoding modulation module and the time encoding modulation module, and then imaged on the color acquisition module by the optical imaging module to obtain a measurement image; the space encoding modulation module and the time encoding modulation module are jointly calibrated with the optical imaging module to obtain a spectral point spread function, and the spectral point spread function and the measurement image are input into the calculation reconstruction module for reconstruction calculation.

[0018] The main points of the application include the following three points:

[0019] (1) Coding imaging structure design based on space-time joint modulation

[0020] Traditional multi-spectral imaging techniques rely on optical elements such as gratings, prisms, and filter wheels to achieve wavelength separation, or use fixed color filter arrays (such as Bayer filters) for channel sampling. These techniques either have a large system size and complex structure, or have sparse observation information and limited reconstruction accuracy. The space-time joint coding structure proposed in the application combines a space encoding modulation module (fixed color random mask) with a time encoding modulation module (programmable black-and-white LCD dynamic mask) to achieve joint modulation of spectral information in the spatial and temporal dimensions, and constructs point spread functions (PSFs) that are different between frames. This structure only relies on static and color inkjet masks and black-and-white LCDs, which are two easily available elements, and can be seamlessly integrated into the front end of a traditional camera, with the advantages of compact structure, high coding freedom, and large information compression ratio.

[0021] (2) Multi-frame joint reconstruction method for underdetermination alleviation

[0022] Traditional single-frame reconstruction methods are limited by the serious underdetermination problem, and the reconstruction results depend on strong sparse priors, with poor stability. The application actively introduces frame-to-frame coding differences in the imaging process, integrates temporal information through a multi-frame joint reconstruction method, improves the rank and redundancy at the observation matrix level, and significantly improves the ill-conditioning of the reconstruction problem. While accelerating the convergence speed, the computational burden of large-scale optimization problems is reduced. The method framework has strong compatibility and iteration stability, and can adaptively adjust the balance between coding complexity and reconstruction accuracy.

[0023] (3)High integration, low cost spectral imaging scheme

[0024] Compared with the system based on dispersive elements or multi-stage filtering devices, the present application does not rely on complex optical splitting structure, and only through mask modulation and lens imaging to realize the compression observation of high-dimensional spectral data, greatly compressing the system volume and construction complexity, with good integration and manufacturing feasibility. In implementation, the color mask can be prepared by low-cost methods such as printing and dye spraying, and the LCD mask is a commercial mature device, which is suitable for multi-platform deployment (including industrial cameras, portable devices, etc.). The imaging structure of the present application has high scalability and hardware adaptation flexibility in actual deployment, and is convenient for large-scale popularization and application.

[0025] Therefore, compared with the prior art, the present application has the following advantages:

[0026] (1)Lightweight and low cost: the system does not rely on dispersive optical elements and high-precision motion devices, and realizes spatio-temporal joint modulation through spatial coding modulation module and temporal coding modulation module, which significantly reduces the system cost and improves the integration.

[0027] (2)Strong underdetermined alleviation ability: by introducing time domain modulation to obtain multiple sets of complementary observation, the solvability and robustness of the reconstruction problem are effectively improved, which is especially suitable for low light, low signal-to-noise ratio and other scenes.

[0028] (3)High efficiency and stability of reconstruction method: the improved FISTA algorithm shows better iteration stability and reconstruction accuracy under the joint coding model, and supports fast recovery of more complex spectral structure.

[0029] (4)Strong adaptability and practicality: the system is suitable for standard RGB sensor platform, and is also compatible with gray CMOS, with good platform universality and landing feasibility. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. The drawings in the following description are only some embodiments of the present application.

[0031] Figure 1 It is a structural framework diagram of the multispectral imaging system of the present application.

[0032] Figure 2 It is a structural composition diagram of the multispectral imaging system of the present application.

[0033] Figure 3 It is an example diagram of (a) color mask parameters and (b) filter response in the embodiment of the present application.

[0034] Figure 4The PSF actually calibrated in different spectral channels at the same time in the embodiment of the present application.

[0035] Figure 5 The black and white mask parameter schematic diagram in the embodiment of the present application.

[0036] Figure 6 The OTF actually calibrated in different frame numbers in the same spectral channel in the embodiment of the present application.

[0037] Figure 7 The synthetic RGB image of (a) the measurement image collected and (b) the reconstructed spectral image in the embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present application clearer, the implementation method and results of the present application will be further described in detail below in combination with the drawings. The specific implementation of the present application will be described in detail below in combination with the drawings. It should be understood that the following embodiments are intended to help understand the principles of the present application, but do not constitute a limitation on the claims. Some changes or equivalent replacements can be made to the implementation process without departing from the spirit of the present application.

[0039] As shown in Figure 1 , the multi-frame reconstruction multispectral imaging system based on dynamic space-time joint coding provided by the embodiment includes the following modules: a filter window module, a spatial coding modulation module, a time coding modulation module, an optical imaging module, a color acquisition module and a calculation reconstruction module. Among them, the filter window module, the spatial coding modulation module, the time coding modulation module, the optical imaging module and the color acquisition module are connected in sequence, the spatial coding modulation module and the optical imaging module, and the time coding modulation module are jointly calibrated to obtain a spectral point spread function, and the spectral point spread function and the measurement image collected by the color acquisition module are input into the calculation reconstruction module for reconstruction calculation.

[0040] As shown in Figure 2 , from left to right are the filter window module, the spatial coding modulation module, the time coding modulation module, the optical imaging module and the color acquisition module, the incident light is filtered through the filter window module to remove the incoherent waveband, then modulated through the spatial coding modulation module and the time coding modulation module, and then imaged on the color acquisition module through the optical imaging module, and finally forms a measurement image.

[0041] The filter window module is used to limit the spectral range of imaging, so that the light of the scene passes through the specified waveband. The waveband range concerned by the selected hyperspectral imaging is selected, and the light emitted by the scene first passes through the filter window module. The band-pass filter F is used in the embodiment to filter out stray light other than the target waveband, to prevent other wavebands of light from bringing crosstalk to affect the reconstruction quality and improve the spectral fidelity.

[0042] The spatial coding modulation module is a static color mask with spatially specific filter response, such as Figure 3 As shown in (a), different spatial positions have different colors, and different colors have different spectral transmittance characteristics, such as Figure 3 As shown in (b), it provides spectral encoding in the spatial dimension. The color mask produces differential transmission modulation of light of different wavelength bands at different spatial positions, as shown in Figure 4 As shown, the spatial coding modulation module has different shapes of point spread functions (PSF) for different bands, which shows that it realizes spectral compression coding in the spatial dimension of the scene, and its transmittance function is recorded as T λ (x S ,y S ), where (x S ,y S ) is the spatial position, and λ is the wavelength. In implementation, the color distribution of the color mask is generated using Gaussian noise. However, the design of the color mask is not limited to this embodiment. Any device that has spectral filtering units with different filtering responses distributed at different spatial positions can serve as a color mask.

[0043] The time-coded modulation module is a spectral filter unit device with a spatial filter response having time specificity, which is called a dynamic black and white LCD (Liquid Crystal Display) mask. Figure 5 As shown, it is used to display different binary patterns during the multi-frame exposure acquisition process to change its transmission pattern in a time series and perform inter-frame modulation on the spatially encoded light field. Each frame t corresponds to a different LCD pattern, and its transmittance function is recorded as P t (x T ,y T ), where (x T ,y T ) is the spatial position on the LCD mask. The module's spatial filtering response continuously changes between multiple frames collected continuously, that is, there are coding differences between each frame, which constitutes a time domain modulation of the light signal passing through the color mask, so that the image recorded in each frame has independent coding information between frames, such as Figure 5As shown, the optical transfer functions (OTFs) of different frames are different in size and shape, indicating that there are differences in their spatial position filtering responses between different time frames, and the spatial filtering responses of different frames cover different intervals in the spatial frequency domain. The design of the black and white mask must ensure that the filtering responses of the patterns of each frame are complementary in the frequency domain. In a specific implementation, a classic binary coding pattern such as a uniformly redundant array (URA) can be used to generate an OTF that is differentiated between frames by adjusting its scale or phase, but it is not limited to this. As long as the device has the characteristic of a spectral filtering unit whose spatial position filtering response is different between different time frames, it can be used as a dynamic mask for this solution.

[0044] In this embodiment, the color mask and the LCD mask are aligned and act together on the incident light field. The transmittance function formed by the two is recorded as T t,λ (x,y)=T λ (x,y)·P t (x, y), the modulated light field propagates over a certain distance and is then sent to the optical imaging module (e.g. Figure 2 The convex lens in the image is focused to form a point spread function at different frames and wavelengths, which is recorded as PSF. t,λ (x, y), where (x, y) is the spatial position. This point spread function comprehensively expresses the diffraction and modulation characteristics under the combined effects of spatial encoding and temporal modulation.

[0045] The optical imaging module can be a single convex lens, a composite lens group, or an apochromatic lens with aberration correction function or a miniaturized board-level lens structure; it can also include a component with a fixed aperture or a variable aperture. As long as it has the function of imaging the jointly encoded light field onto the sensor and can maintain the consistency of the point spread function in space, it can be used as the optical imaging module described in the present invention. In this embodiment, a convex lens is used to focus the encoded light field, which has been jointly modulated by the color mask and the LCD mask, onto the collection surface of the color collection module to ensure that the spatial mapping relationship is not distorted and the energy is focused. Its imaging distance and field of view can be adjusted according to the mask size and sensor size to ensure the imaging accuracy of the system.

[0046] The color acquisition module is a multispectral image sensor with a spectral filter array, which is used to capture the image after the optical imaging module focuses. The filter array can be in the form of a 2×2 Bayer filter, or various wide-spectrum filter forms such as 3×3, 4×4, etc.; a grayscale sensor can also be used, that is, each pixel has the same response to the spectrum and has only one spectral channel. This embodiment uses a Bayer filter array, and the response function of the Bayer filter array is recorded as L c(λ), where c is the number of spectral channels, c ∈ R, G, B are different channels. This module down-samples the scene information in both spatial and spectral dimensions, each spectral channel has a different filter response form. After the convex lens imaging, the filtered array is recorded by the sensor, and multiple frames of measurement images Y c,t (x,y), the generation process satisfies the following model:

[0047] Y c,t (x,y)=Poisson(∑L c (λ)·PSF t,λ (x,y)*o t (λ,x,y))

[0048] Where Poisson represents the Poisson noise, o t (λ,x,y) represents the spatial spectral distribution of the real scene at wavelength λ at time frame t, * is a two-dimensional convolution operation.

[0049] The calculation reconstruction module runs on a computing medium, such as MATLAB / NumPy program on a PC platform, uses FISTA iterative algorithm, uses multiple frame information to model and get gradient, and constrains the reconstruction solution. In order to further improve the speed and accuracy of the reconstruction, the FISTA algorithm is optimized in the embodiment, which specifically includes:

[0050] The calculation reconstruction module takes multiple frames of measurement images Y c,t , point spread function PSF t,λ (x,y), and filter response function L c (λ) as input, establishes a multi-frame joint reconstruction model based on the shot noise model, and assumes a negative log-likelihood function under the Poisson distribution noise. Under this Poisson noise assumption, the gradient is:

[0051]

[0052] Where ⊙ is matrix point multiplication, A = ∑L c (λ)·PSF t,λ (x,y)*o t (λ,x,y) is the hypothetical measurement image under the ideal noiseless condition, is the transpose of the point spread function, and Y is the measurement image under the actual condition.

[0053] By using the FISTA-based optimization algorithm, the negative log-likelihood function under the Poisson distribution noise is minimized. This process continuously performs gradient descent on A. When the negative log-Poisson likelihood function reaches the minimum value, it is considered that the real spectral data cube is successfully reconstructed.

[0054] The reconstructed scene reconstructs a high-dimensional data cube in spatial and spectral dimensions. The pseudo-color image after synthesis of the reconstruction result in the embodiment is as shown in FIG. 1. Figure 7 The FIGS. (a) and (b) show the original scene and the reconstructed scene, and the good reconstruction effect can be seen.

[0055] Preferably, the band-pass filter F used by the filtering window module is compactly and tightly arranged with the color mask CM and the dynamic LCD mask LCD, so as to improve the system integration. The optical imaging module is located 1-5 cm downstream of the filtering window module, and the color acquisition module is located behind the optical imaging module. The photosensitive surface of the sensor is parallel to the color mask, and the entire optical path is highly compact, which is suitable for modular or miniaturized integration.

[0056] Compared with the past snapshot spectral imaging system and the larger imaging system constructed by using prisms, gratings, diffractive optical elements, super surfaces and other optical devices, the embodiment adopts random color masks and random black and white masks for joint modulation at the pixel level, has the characteristics of compact structure, flexible coding and superior reconstruction performance, and significantly improves the multispectral imaging capability under the underdetermined condition, thereby creating an integrated paradigm for the new generation of snapshot spectral imaging.

[0057] Obviously, the above embodiments are only part of the embodiments of the present application, rather than all the embodiments. The above embodiments are only used for explaining the present application, and do not constitute a limitation on the protection scope of the present application. Based on the above embodiments, all other embodiments obtained by those skilled in the art without creative labor, i.e. all modifications, equivalent replacements and improvements made within the spirit and principles of the present application, fall within the protection scope of the present application.

Claims

1. A multi-frame reconstruction multispectral imaging system based on dynamic space-time joint coding, characterized in that: The system includes: Filter window module, used to limit the spectral range of imaging; A spatial coding modulation module is used to provide spectral coding in the spatial dimension; A time coding modulation module, configured to perform inter-frame modulation on the spectral coding provided by the spatial coding modulation module; An optical imaging module, configured to image the light field modulated by the time coding modulation module onto a color acquisition module; The color acquisition module is used to further perform spectral encoding on the light field to obtain a measurement image; A calculation and reconstruction module is used to restore a high-dimensional multispectral image data cube using the measured image.

2. The multi-frame reconstruction multispectral imaging system based on dynamic space-time joint coding according to claim 1, characterized in that: The spatial coding modulation module is a static color mask with spatially specific filtering response, which produces differential transmission modulation for light of different wavelength bands at different spatial positions.

3. The multi-frame reconstruction multispectral imaging system based on dynamic space-time joint coding according to claim 1, characterized in that: The time coding modulation module is a dynamic black and white LCD mask with time-specific spatial filtering response. The spatial filtering response of the dynamic black and white LCD mask continuously changes between multiple frames collected continuously, that is, there are coding differences between each frame, forming time domain modulation.

4. The multi-frame reconstruction multispectral imaging system based on dynamic space-time joint coding according to claim 1, characterized in that: The filter window module, the spatial coding modulation module and the time coding modulation module are closely arranged and placed.

5. A multi-frame reconstruction multispectral imaging method based on dynamic space-time joint coding, using the imaging system according to any one of claims 1 to 4, characterized in that: The steps of the imaging method include: The incident light is first filtered out of irrelevant bands by the filter window module, modulated by the spatial coding modulation module and the time coding modulation module, and then imaged on the color acquisition module by the optical imaging module to obtain a measurement image; the spatial coding modulation module, the time coding modulation module, and the optical imaging module are jointly calibrated to obtain a spectral point spread function, and the spectral point spread function and the measurement image are input into the calculation and reconstruction module for reconstruction calculation.

6. The multi-frame reconstruction multispectral imaging method based on dynamic space-time joint coding according to claim 5, characterized in that: The spatial coding modulation module is a static color mask with spatially specific filtering response. The color distribution of the color mask is generated by Gaussian noise. The spatial coding modulation module has a point spread function with different shapes for different bands. Its transmittance function is recorded as T λ (x S ,y S ), where (x S ,y S ) is the spatial position and λ is the wavelength.

7. The multi-frame reconstruction multispectral imaging method based on dynamic space-time joint coding according to claim 6, characterized in that: The temporal coding modulation module is a dynamic black-and-white LCD mask with a time-specific spatial filtering response. The dynamic black-and-white LCD mask displays different binary patterns during the multi-frame exposure acquisition process to change its transmission pattern in a time sequence and perform inter-frame modulation on the spatially coded light field. Each frame t corresponds to a different LCD pattern, and its transmittance function is recorded as P t (x T ,y T ), where (x T ,y T ) is the spatial position on the LCD mask.

8. The multi-frame reconstruction multispectral imaging method based on dynamic space-time joint coding according to claim 7, characterized in that: The color mask and the LCD mask are arranged in a laminated manner and act together on the incident light field; the transmittance function formed by the two is recorded as T t,λ (x,y)=T λ (x,y)·P t (x, y), the modulated light field is focused and imaged by the optical imaging module to form a point spread function at different frames and wavelengths, which is recorded as PSF t,λ (x,y), where (x,y) is the spatial position.

9. The multi-frame reconstruction multispectral imaging method based on dynamic space-time joint coding according to claim 8, characterized in that: The optical imaging module focuses and images, which are then recorded by the color acquisition module to obtain multiple frames of measurement images Y c,t (x,y), its generation process satisfies the following model: Y c,t (x,y)=Poisson(∑L c (λ)·PSF t,λ (x,y)*o t (λ,x,y)) Among them, Poisson represents Poisson noise, o t (λ,x,y) represents the spatial spectral distribution of the real scene at wavelength λ in time frame t, and * is a two-dimensional convolution operation.

10. The multi-frame reconstruction multispectral imaging method based on dynamic space-time joint coding according to claim 9, characterized in that: The calculation and reconstruction module converts the multi-frame measurement image Y c,t , point spread function PSF t,λ (x,y) and filter response function L c (λ) is used as input, and a multi-frame joint reconstruction model is established based on the shot noise model. It is assumed to be a negative log-likelihood function under Poisson distribution noise. Under this assumption, the gradient is: Where ⊙ is the matrix dot product, A=∑L c (λ)·PSF t,λ (x,y)*o t (λ,x,y) is the hypothetical measurement image under ideal noise-free conditions, is the transpose of the point spread function, and Y is the measured image in actual conditions; The negative log-likelihood function under Poisson distribution noise is minimized, and the hypothetical measurement image A is continuously gradient descended. When the negative log-likelihood function reaches the minimum value, the real spectral data cube is reconstructed as the output.

Images