Snapshot spectral imaging method for dynamic scene

By using a snapshot-type spectral imaging method, and employing diffractive optical elements and area array photodetectors for light field modulation and data reconstruction, the problems of low energy utilization, complex structure, and low sensitivity of existing spectral imagers are solved, and efficient dynamic scene three-dimensional spectral imaging is achieved.

CN118961605APending Publication Date: 2024-11-15CHANGCHUN INST OF ELECTRONIC TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410750660.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-12
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing spectral imagers suffer from low energy efficiency, complex system structure, low detection sensitivity, and huge data volume, making it difficult to achieve dynamic scene imaging with high spectral resolution and high signal-to-noise ratio.

Method used

A snapshot-type spectral imaging method is adopted, which uses diffractive optical elements to randomly modulate the light field, and combines a planar array photodetector and a high-performance server to reconstruct three-dimensional data, thereby realizing spectral imaging of dynamic scenes.

Benefits of technology

It achieves efficient dynamic scene 3D spectral image reconstruction with an imaging rate of 30fps/s, meeting the system requirements and improving the fidelity of spectral information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118961605A_ABST
    Figure CN118961605A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of spectral imaging, and discloses a dynamic scene-oriented snapshot spectral imaging method. The method comprises the following steps of: placing a target object on a focal plane of an imaging lens, imaging the target object on a front CCD (charge coupled device) 1 imaging surface on which an adjustable diaphragm is positioned through a spectroscope, and obtaining a broadband image on the front imaging surface; the diffractive optical element performs random phase modulation on a light field emitted by each point on the broadband image on the front imaging surface, a speckle pattern in which two-dimensional spatial information and one-dimensional spectral information are overlapped is formed after modulation, and the speckle pattern is imaged on the area array photoelectric detector CCD2; collecting speckle light intensity distribution information of each position of the diffractive optical element by using monochromatic light, and arranging the speckle light intensity distribution information according to a certain sequence to obtain a measurement matrix of the system; and according to the detection signal and the measurement matrix, performing three-dimensional data reconstruction on the detection signal by using a high-performance server in a computational imaging mode. The result appearance shape and spectrum information reduction degree is high, and the system use requirement is met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of optoelectronic imaging technology, and in particular to a snapshot spectral imaging method for dynamic scenes. Background Art

[0002] Spectral imaging technology can reveal the fluctuating nature of light information of natural substances. Spectral images have the ability to depict scene information in detail. They have great application prospects in many fields such as meteorological observation, land planning, medical diagnosis, vegetation classification, military reconnaissance, resource survey, earthquake monitoring, and disaster prevention. Multispectral imaging technology can be divided into scanning methods based on multiple exposures and computational imaging methods based on single exposure from the imaging time dimension. Traditional scanning methods can generally only obtain static spectral images. Computational imaging methods can obtain multispectral data in dynamic scenes by using single exposure and computational reconstruction of modulated spectral data, thus realizing the expansion of spectral imaging from "static" to "dynamic". As a new computational imaging method, correlation imaging technology mainly modulates the target by referring to the light field, and obtains the target image information by using the fluctuation and high-order correlation of the light field, which can achieve higher information acquisition efficiency. Compressed sensing theory is a new signal acquisition and encoding and decoding theory. The application of compressed sensing theory to multispectral imaging technology has been widely studied. At present, correlation imaging technology mostly uses active light sources, and using sunlight as the light source requires real-time acquisition of sunlight fluctuation information, which has great limitations in the imaging process. Combining correlation imaging technology with compressed sensing theory and applying it to multi-spectral imaging systems, the spatial random phase modulator is used to achieve three-dimensional random compression sampling of the spectral dimension and the spatial two-dimensional dimension, which greatly improves the information acquisition capability and efficiency of the system. In recent years, many scholars have proposed solutions to analyze the spectral imaging of moving targets using computational imaging theory.

[0003] The existing spectral imagers have the following problems:

[0004] (1) Low energy utilization, making it difficult to simultaneously achieve high spectral resolution and high signal-to-noise ratio. The energy utilization of traditional prism dispersion, grating diffraction, and spatial modulation interference spectral imagers is reduced.

[0005] (2) The system structure is complex and cannot perform real-time imaging of moving targets. Michelson-type or large-aperture static interferometric spectroscopy imagers use time modulation, which requires long exposure time and extremely high platform stability.

[0006] (3) Low detection sensitivity and huge data volume. Traditional spectrometers need to obtain a complete spectral image of the target by scanning the spatial or spectral dimension. The structure is relatively complex, and the imaging time for a single target point is short, resulting in low detection sensitivity. And with the improvement of spatial resolution and spectral resolution, the amount of spectral image data is getting larger and larger, and the transmission and storage of large amounts of data has also become a problem. Summary of the invention

[0007] The purpose of the present invention is to provide a snapshot spectral imaging method for dynamic scenes to solve the problems of low energy utilization, complex system structure, low detection sensitivity and huge data volume.

[0008] To achieve this object, the present invention adopts the following technical solutions:

[0009] A snapshot spectral imaging method for dynamic scenes, the method comprising the following steps:

[0010] The target object is placed on the focal plane of the imaging lens, and is imaged onto the front CCD1 imaging plane where the adjustable aperture is located through the beam splitter, and a wide-band image is obtained on the front imaging plane;

[0011] The diffractive optical element performs random phase modulation on the light field emitted from each point on the wide-band image on the front imaging plane, and after modulation, a speckle pattern with two-dimensional spatial information and one-dimensional spectral information overlaps is formed. The speckle pattern is imaged on the area array photodetector CCD2;

[0012] Monochromatic light is used to collect the speckle intensity distribution information at each position of the diffractive optical element, and the system measurement matrix is ​​obtained after arranging it in a certain order;

[0013] According to the detection signal and the measurement matrix, the detection signal is reconstructed into three-dimensional data by means of computational imaging using a high-performance server.

[0014] Preferably, the CCD1 is an RGB camera for determining a target shooting area.

[0015] Preferably, CCD2 is a grayscale camera for capturing the light intensity value of the light field reflected or transmitted by the detection target.

[0016] Preferably, the method further includes: before image reconstruction, the camera needs to be calibrated.

[0017] Preferably, the input-output relationship of the imaging system can be expressed as:

[0018]

[0019] In the formula, It represents the multispectral image information reconstructed by the pixels in the mth row and nth column on the front imaging surface in d spectral bands. It represents the light intensity value obtained by the pixel in the pth row and qth column on the CCD detection surface when the calibration point light source of the imaging system moves on the pixel in the mth row and nth column on the object surface; y pq represents the intensity distribution of the detection signal in the pth row and qth column on the detection surface during imaging; ε pq Represents the detection signal noise.

[0020] Preferably, the reconstruction of the multispectral image can be achieved by solving the following optimization problem:

[0021]

[0022] Here, τ is a non-negative parameter and ||v||2 is the 2-norm of v.

[0023] Beneficial effects of the present invention: This paper uses snapshot spectral imaging to achieve three-dimensional spectral image reconstruction of dynamic scenes at a video-level imaging rate of 30fps / s. The imaging principle and spectral calibration method of snapshot spectral imaging are explained. A spectral imaging experiment was conducted with three small fish of different colors in motion as the target, and the color fusion image of 15 spectral channels was compared with the RGB image. The experimental results show that the appearance shape and spectral information of the small fish are highly restored, meeting the system usage requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the contents of the embodiments of the present invention and these drawings without paying any creative work.

[0025] Figure 1 It is a snapshot spectral imaging optical path diagram of the present invention;

[0026] Figure 2 It is the snapshot type spectral imaging calibration optical path diagram of the present invention;

[0027] Figure 3 It is the experimental result of snapshot video spectral imaging of the present invention;

[0028] Figure 4 is the spectral distribution of the target in the commercial spectrometer and GISC camera of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0030] In the description of the present invention, unless otherwise clearly specified and limited, the terms "connected", "connected", and "fixed" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0031] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0032] In the description of this embodiment, the terms "upper", "lower", "left", "right" and other directions or positional relationships are based on the directions or positional relationships shown in the drawings, and are only for the convenience of description and simplification of operation, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operate in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used to distinguish in the description and have no special meaning.

[0033] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0034] like Figure 1-4 As shown, a snapshot spectral imaging method for dynamic scenes comprises the following steps:

[0035] The target object is placed on the focal plane of the imaging lens, and is imaged onto the front CCD1 imaging plane where the adjustable aperture is located through the beam splitter, and a wide-band image is obtained on the front imaging plane;

[0036] The diffractive optical element performs random phase modulation on the light field emitted from each point on the wide-band image on the front imaging plane, and after modulation, a speckle pattern with two-dimensional spatial information and one-dimensional spectral information overlaps is formed. The speckle pattern is imaged on the area array photodetector CCD2;

[0037] Monochromatic light is used to collect the speckle intensity distribution information at each position of the diffractive optical element, and the system measurement matrix is ​​obtained after arranging it in a certain order;

[0038] According to the detection signal and the measurement matrix, the detection signal is reconstructed into three-dimensional data by means of computational imaging using a high-performance server.

[0039] Preferably, the CCD1 is an RGB camera for determining a target shooting area.

[0040] Preferably, CCD2 is a grayscale camera for capturing the light intensity value of the light field reflected or transmitted by the detection target.

[0041] Preferably, the method further includes: before image reconstruction, the camera needs to be calibrated.

[0042] Preferably, the input-output relationship of the imaging system can be expressed as:

[0043]

[0044] In the formula, It represents the multispectral image information reconstructed by the pixels in the mth row and nth column on the front imaging surface in d spectral bands. It represents the light intensity value obtained by the pixel in the pth row and qth column on the CCD detection surface when the calibration point light source of the imaging system moves on the pixel in the mth row and nth column on the object surface; y pq represents the intensity distribution of the detection signal in the pth row and qth column on the detection surface during imaging; ε pq Represents the detection signal noise.

[0045] Preferably, the reconstruction of the multispectral image can be achieved by solving the following optimization problem:

[0046]

[0047] Here, τ is a non-negative parameter and ||v||2 is the 2-norm of v.

[0048] The snapshot video spectral imaging experiment for dynamic scenes was completed in a laboratory environment. The imaging system consists of a snapshot spectral imaging and an RGB camera fixed by a ground tripod. The experimental parameters are set as follows: the spectral center wavelengths are 461nm, 474nm, 487nm, 501nm, 515nm, 530nm, 545nm, 561nm, 578nm, 590nm, 614nm, 633nm, 653nm, 674nm and 698nm, and the pixel resolution of the snapshot spectral imaging is greater than 0.5mrad. The temporal resolution of the snapshot spectral imaging and RGB camera is 30fps / s. Daylight and LED lights are used as illumination sources, and the target object to be measured is 1.6m away from the snapshot spectral imaging.

[0049] The snapshot video spectral imaging experiment results are as follows: Figure 3 As shown in Figure 1, 3 swimming fish are used as imaging targets, and the imaging results obtained by 3 consecutive frames of RGB camera at a rate of 30fps / s are shown in Figure 1. Figure 3 (a) Figure 3 (f) and Figure 3 As shown in (k), the speckle pattern obtained by snapshot spectral imaging at the same time is as follows Figure 3 (b) Figure 3 (g) and Figure 3 As shown in (i), the multispectral grayscale image obtained by the image reconstruction algorithm is as follows Figure 3 (d) Figure 3 (i) and Figure 3 (n) shows that the multi-spectral pseudo-color image is Figure 3 (e) Figure 3 (j) and Figure 3 As shown in (o), the multi-spectral fusion image is Figure 3 (c) Figure 3 (h) and Figure 3 (m) shown.

[0050] Commercial fiber optic spectrometer is used to Figure 4 The spectral distribution of the 400nm to 700nm band in (a) was tested, and the multispectral grayscale image reconstructed by the GISC snapshot spectral camera was shown in Figure 2. Figure 4 (b) shows the comparison of the spectral distribution curve of the target tested by the commercial fiber optic spectrometer and the spectral distribution curve in the GISC snapshot spectral camera. Figure 4 (c) as shown.

[0051] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, various obvious changes, readjustments and substitutions can be made without departing from the protection scope of the present invention. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A snapshot spectral imaging method for dynamic scenes, characterized in that: The method comprises the following steps: The target object is placed on the focal plane of the imaging lens, and is imaged onto the front CCD1 imaging plane where the adjustable aperture is located through the beam splitter, and a wide-band image is obtained on the front imaging plane; The diffractive optical element performs random phase modulation on the light field emitted from each point on the wide-band image on the front imaging plane, and after modulation, a speckle pattern with two-dimensional spatial information and one-dimensional spectral information overlaps is formed. The speckle pattern is imaged on the area array photodetector CCD2; Monochromatic light is used to collect the speckle intensity distribution information at each position of the diffractive optical element, and the system measurement matrix is ​​obtained after arranging it in a certain order; According to the detection signal and the measurement matrix, the detection signal is reconstructed into three-dimensional data by means of computational imaging using a high-performance server.

2. The snapshot spectral imaging method for dynamic scenes according to claim 1, characterized in that: The CCD1 is an RGB camera used to determine the target shooting area.

3. The snapshot spectral imaging method for dynamic scenes according to claim 1, characterized in that: CCD2 is a grayscale camera used to capture the light intensity value of the light field reflected or transmitted by the detection target.

4. The snapshot spectral imaging method for dynamic scenes according to claim 1, characterized in that: Also includes: Before image reconstruction, the camera needs to be calibrated.

5. The snapshot spectral imaging method for dynamic scenes according to claim 1, characterized in that: The input-output relationship of the imaging system can be expressed as: In the formula, It represents the multispectral image information reconstructed by the pixels in the mth row and nth column on the front imaging surface in d spectral bands. It represents the light intensity value obtained by the pixel in the pth row and qth column on the CCD detection surface when the calibration point light source of the imaging system moves on the pixel in the mth row and nth column on the object surface; y pq represents the intensity distribution of the detection signal in the pth row and qth column on the detection surface during imaging; ε pq Represents the detection signal noise.

6. The snapshot spectral imaging method for dynamic scenes according to claim 1, characterized in that: The reconstruction of multispectral images can be achieved by solving the following optimization problem: Here, τ is a non-negative parameter and ||v||2 is the 2-norm of v.