Multi-snapshot hyperspectral compression imaging system

By using displacement actuators and spectral domain reconstruction neural network models in hyperspectral compression imaging system, moving pixel filters collect image data from different locations and perform spatial combinations, the images in multi-snapshot hyperspectral compression imaging system are solved. The problem of low reconstruction accuracy has achieved high-precision spatial-spectral data reconstruction.

CN118896689AActive Publication Date: 2024-11-05CHANGCHUN INST OF OPTICS FINE MECHANICS & PHYSICS CHINESE ACAD OF SCI
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411013471.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-05
Estimated Expiration
2044-07-26

AI Technical Summary

Technical Problem

The existing hyperspectral compression imaging system has the problem of low reconstruction accuracy of multi-snapshot image, especially in hyperspectral resolution application scenarios, and the reconstruction quality is difficult to guarantee.

Method used

A multi-snapshot hyperspectral compression imaging system is used to move pixel filters through a displacement actuator to acquire image data at different locations, and a neural network model is used to spatially combine the spectral domain to reconstruct it into a space-spectral data cube. The system includes an optical lens, a pixel filter, a plane array camera and a displacement actuator. The filter pixel grid is the same as the imaging pixel grid, and image data acquisition with different filter effects is achieved through a variety of moving solutions.

Benefits of technology

The accuracy of the space-spectral data cube reconstruction data is improved, and the problem of low image reconstruction accuracy of high spectral compression imaging systems in the prior art is solved, and remote sensing applications with high information throughput, high sensitivity and high integration are realized.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118896689A_ABST
    Figure CN118896689A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of spectral imaging, and discloses a multi-snapshot type hyperspectral compression imaging system, which comprises an optical lens, a pixel optical filter, an area-array camera and a displacement actuator, the displacement actuator moves the pixel optical filter according to a preset moving mode so that the detector can collect multiple pieces of image data of the optical filter at different positions and having different filtering effects, and the upper computer carries out spatial combination through a spectral domain reconstruction neural network model. To reconstruct the plurality of image data into spatial-spectral data cube reconstruction data. At the moment, image processing is carried out through the original image acquisition equipment, so that the reconstruction data accuracy of the space-spectral data cube reconstructed through the scheme is high, and the technical problem that the reconstruction image accuracy of a hyperspectral compression imaging system in the prior art is not high is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of spectral imaging, and in particular to a multi-snapshot hyperspectral compression imaging system. Background Art

[0002] Spectral imaging technology has the ability to collect scene space-spectral data cubes. The added high-resolution spectral channel information provides more refined differentiated features for target detection, identification, and analysis. It has been widely used in military security, environmental monitoring, biological sciences, medical diagnosis, food inspection and other fields.

[0003] In the framework of the traditional Nyquist-Shannon complete sampling theory, in order to realize the acquisition of three-dimensional space-spectral data cubes with two-dimensional image sensors, spectral imagers mainly adopt three types of acquisition methods: First, slice and time-share the data cube, such as push-broom spectral imagers, filter spectral imagers, Fourier transform spectral imagers, etc. This type of architecture usually requires additional scanning or switching mechanisms, sacrificing time resolution. Second, remap and reduce the dimension of the data cube, such as spectral beam splitting spectral imagers, fiber recombination spectral imagers, multi-aperture or microlens array spectral imagers, etc. This type of architecture can achieve snapshot acquisition, but it increases the demand for sensor array size, and the spectral resolution is generally not high. Third, sample and reduce the dimension of the data cube, such as pixel filter array spectral imagers, micro-slicer spectral imagers, etc. This type of architecture can still achieve snapshot acquisition, but it sacrifices spatial resolution, and discrete sampling also leads to decreased sensitivity. In summary: Traditional spectral imagers need to make compromises between spatial, spectral, and temporal resolution, which becomes increasingly difficult to reconcile in remote sensing application scenarios that are developing towards high information throughput, high sensitivity, and high integration.

[0004] Based on the theoretical framework of compressed sensing, compressed spectral imaging technology can solve the contradiction between spatial, spectral and temporal resolution by implementing coded modulation and dimensionality reduction sampling on the spatial-spectral data cube, and reconstruct the original signal based on the prior characteristics of the natural scene with a collection data volume far lower than the Nyquist requirement. In the typical hyperspectral compression imaging CASSI architecture, the hardware combination of coded aperture and dispersion element hinders the miniaturization of its system. In contrast, the wide spectrum pixel filter architecture MSFA architecture only uses pixel filters installed on the surface of the image sensor to complete the effective modulation of spatial spectrum information, which greatly simplifies the composition of the hyperspectral compression imaging system and brings the advantages of high robustness and miniaturization. However, under this framework, the spectral transmittance of each pixel is fixed, so only one snapshot sampling can be performed. The limited data makes it difficult to support the reconstruction of spectral data, especially in the application scenarios of high spectral resolution, the reconstruction quality is difficult to guarantee. Summary of the invention

[0005] In view of the above problems, an embodiment of the present invention provides a multi-snapshot hyperspectral compression imaging system, which is used to solve the technical problem of low image reconstruction accuracy of the multi-snapshot hyperspectral compression imaging system in the prior art.

[0006] According to one aspect of an embodiment of the present invention, a multi-snapshot hyperspectral compression imaging system is provided, comprising:

[0007] An optical lens for acquiring image data;

[0008] A pixel filter having a plurality of filter pixel grids for filtering the image data;

[0009] An area array camera, the area array camera comprising a detector having a plurality of imaging pixel grids, the filter pixel grid having the same specification as the imaging pixel grid and being used to collect the image data;

[0010] A displacement actuator, on which the pixel filter is disposed, and used to move the pixel filter in a preset movement manner so that the detector collects a plurality of image data with filters at different positions and with different filtering effects;

[0011] A host computer is equipped with a spectral domain reconstruction neural network model, and spatial combination is performed through the spectral domain reconstruction neural network model to reconstruct the multiple image data into spatial-spectral data cube reconstruction data.

[0012] In an optional manner, the filter pixel grids of the filter are arranged in an array, the pixel filter includes a plurality of filter areas, and the colors of the filter pixel grids in the same filter area are all different.

[0013] In an optional manner, the number of filter pixel grids in any of the filter areas is 9 and is arranged as a 3*3 array.

[0014] In an optional manner, the preset movement mode consists of multiple movement schemes, including:

[0015] A first movement scheme, controlling the pixel filter to move one imaging pixel grid along a first direction;

[0016] A second movement scheme, controlling the pixel filter to move one imaging pixel grid along a second direction, and acquiring the image data when the filtering pixel grid of the pixel filter coincides with the imaging pixel grid; the second direction is perpendicular to the first direction, and the number of grids moved in the second direction is accumulated once;

[0017] The third movement scheme is to control the pixel filter to move one imaging pixel grid along the third direction, and obtain the image data when the filtering pixel grid of the pixel filter coincides with the imaging pixel grid; the third direction is opposite to the second direction, and the number of grids moved in the third direction is accumulated once.

[0018] In an optional manner, the preset moving mode is performed as follows:

[0019] Step S1, repeatedly executing the second movement scheme until the number of movement grids in the second direction is 3, and the number of translations is accumulated to 3 times;

[0020] Step S2, executing the first movement plan and clearing the data of the number of movement grids in the second direction;

[0021] Step S3, repeatedly executing the third movement scheme until the number of movement grids in the third direction is 3, and the number of translations is accumulated to 3 times;

[0022] Step S4, executing the first movement plan and clearing the data of the number of movement grids in the second direction;

[0023] Step S5, repeatedly executing the second movement scheme until the number of movement grids in the second direction is 3, and the number of translations is accumulated to 3 times;

[0024] Step S6: When the number of translation times reaches the target number, stop the image data acquisition process.

[0025] In an optional manner, when the pixel filter is in a motion initial position or a motion cutoff position, each of the filtering pixel grids coincides with an imaging pixel grid of the detector.

[0026] In an optional manner, the timing constraint relationship between the exposure of the detector, the reading of the pixel data, and the displacement actuator driving the pixel filter to shift is:

[0027] When the exposure of the detector is completed, the displacement actuator is started to drive the pixel filter to move, and the displacement process of one imaging pixel grid needs to be completed before the next exposure starts;

[0028] Pixel data readout can only be started after the detector exposure is completed, and all pixel readout operations need to be completed before the next exposure ends.

[0029] In an optional manner, the shutter of the area array camera is a global electronic shutter.

[0030] The multi-snapshot hyperspectral compression imaging system of the present application moves the pixel filter in a preset movement mode through a displacement actuator so that the detector collects multiple image data with filters in different positions and different filtering effects, and the host computer performs spatial combination through the spectral domain reconstruction neural network model to reconstruct the multiple image data into spatial-spectral data cube reconstruction data. At this time, by setting up image processing on the original image acquisition device, the spatial-spectral data cube reconstruction data reconstructed by the above scheme has high accuracy, so as to solve the technical problem of low image reconstruction accuracy of the hyperspectral compression imaging system in the prior art.

[0031] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0033] Figure 1 The schematic diagram of the structure of the multi-snapshot hyperspectral compression imaging system provided by the present invention is shown;

[0034] Figure 2 A schematic diagram of two-dimensional displacement of pixel filters in a multi-snapshot hyperspectral compression imaging system provided by the present invention is shown;

[0035] Figure 3 A schematic diagram showing the timing constraint relationship between the detector operation and the piezoelectric drive displacement in the multi-snapshot hyperspectral compression imaging system provided by the present invention is shown;

[0036] Figure 4 A schematic diagram showing the comparison between the reconstructed image and the original image of the multi-snapshot hyperspectral compression imaging system provided by the present invention in the 550nm spectrum band is shown;

[0037] Figure 5 A schematic diagram showing comparison of spectral curves at random positions of a target scene in a multi-snapshot hyperspectral compression imaging system provided by the present invention is shown;

[0038] Figure 6 A schematic diagram of the displacement process of the pixel filter in the multi-snapshot hyperspectral compression imaging system provided by the present invention is shown. DETAILED DESCRIPTION

[0039] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0040] The present application proposes a multi-snapshot hyperspectral compression imaging system, which is used to solve the technical problem of low image reconstructed accuracy of the hyperspectral compression imaging system in the prior art.

[0041] In an alternative embodiment, referring to Figure 1 As shown, a multi-snapshot hyperspectral compression imaging system includes an optical lens, a pixel filter, an array camera, and a displacement actuator.

[0042] The pixel filter has a plurality of filter pixel grids, the area array camera includes a detector, the detector has a plurality of imaging pixel grids, the filter pixel grids have the same specifications as the imaging pixel grids, and the pixel filter is disposed on the displacement actuator.

[0043] In the above embodiment, the optical lens acquires image data for filtering the image data, the array camera collects the image data, and the displacement actuator moves the pixel filter in a preset movement mode so that the detector collects multiple image data with filters in different positions and different filtering effects. The host computer is equipped with a spectral domain reconstruction neural network model, and the spectral domain reconstruction neural network model is used for spatial combination to reconstruct the multiple image data into spatial-spectral data cube reconstruction data.

[0044] Among them, the multi-snapshot hyperspectral compression imaging system of the present application moves the pixel filter in a preset moving manner through a displacement actuator so that the detector collects multiple image data with filters in different positions and with different filtering effects, and the host computer performs spatial combination through the spectral domain reconstruction neural network model to reconstruct the multiple image data into spatial-spectral data cube reconstruction data. At this time, by setting up image processing on the original image acquisition device, the spatial-spectral data cube reconstruction data reconstructed by the above scheme has high accuracy, so as to solve the technical problem of low image reconstruction accuracy of the hyperspectral compression imaging system in the prior art.

[0045] Optionally, refer to Figure 2 As shown, the filter pixels of the filter are arranged in an array, the pixel filter includes a plurality of filter areas, and the colors of the filter pixels in the same filter area are all different.

[0046] Through the above settings, the length of each side of the obtained image data can be made uniform, so that the subsequent pixel-by-pixel changes and data collection can be convenient, so that the changes and data collection processes can be accurate and controllable.

[0047] In addition, by repeatedly setting the filter area and setting different colors in the filter area, the requirements for the filter production process can be reduced. At the same time, by taking the colors of the filter pixels in the same filter area to be different as a setting requirement, it can be ensured that in the process of pixel-by-pixel movement, the filtering effect of the same imaging pixel grid is not consistent each time it is moved, thereby improving the accuracy of the final image data reconstruction into the spatial-spectral data cube reconstruction data.

[0048] Optionally, the number of filter pixel grids in any of the filter areas is 9 and is arranged as a 3*3 array.

[0049] Through the above settings, the length of each side of the obtained image data can be made uniform, so that the subsequent pixel-by-pixel changes and data collection can be convenient, so that the changes and data collection processes can be accurate and controllable.

[0050] Optionally, the preset movement mode is composed of multiple movement schemes, including:

[0051] The first movement scheme controls the pixel filter to move one imaging pixel grid along a first direction.

[0052] The second movement scheme is to control the pixel filter to move one imaging pixel grid along the second direction, and obtain the image data when the filtering pixel grid of the pixel filter coincides with the imaging pixel grid; the second direction is perpendicular to the first direction, and the number of grids moved in the second direction is accumulated once.

[0053] The third movement scheme is to control the pixel filter to move one imaging pixel grid along the third direction, and obtain the image data when the filtering pixel grid of the pixel filter coincides with the imaging pixel grid; the third direction is opposite to the second direction, and the number of grids moved in the third direction is accumulated once.

[0054] In the first, second and third movement schemes, the displacement is a modulation of each captured image, so that each captured image data is related, so that for the overall captured image data, there are overlapping imaging pixel grids, so that the filtering effects of different imaging pixel grids assist each other in reconstruction, and high-precision image data reconstruction is achieved.

[0055] Optionally, the number of the imaging pixel grids in the first direction is greater than or equal to the number of the pixel filters.

[0056] Optionally, the number of the imaging pixel grids in the second direction is greater than or equal to the number of the pixel filters.

[0057] Through the above optional implementation scheme, the amount of initially collected data can be increased to avoid invalid reconstruction due to too little data or parameters.

[0058] In an alternative embodiment, referring to Figure 6 As shown, the preset moving mode is performed as follows:

[0059] Step S1, repeatedly executing the second movement scheme until the number of movement grids in the second direction is 3, and the number of translations is accumulated to 3 times;

[0060] Step S2, executing the first movement plan and clearing the data of the number of movement grids in the second direction;

[0061] Step S3, repeatedly executing the third movement scheme until the number of movement grids in the third direction is 3, and the number of translations is accumulated to 3 times;

[0062] Step S4, executing the first movement plan and clearing the data of the number of movement grids in the second direction;

[0063] Step S5, repeatedly executing the second movement scheme until the number of movement grids in the second direction is 3, and the number of translations is accumulated to 3 times;

[0064] Step S6: When the number of translation times reaches the target number, stop the image data acquisition process.

[0065] In the above scheme, for each spatial pixel point, a total of 9 image data modulated by different spectral transmittance characteristics are collected, and the data is input into the spectral domain reconstruction neural network model to obtain 151 spectral bands of hyperspectral reconstruction data. All spatial pixels are reconstructed in this way in turn, and spatially combined to obtain the final spatial-spectral data cube reconstruction data.

[0066] Based on the above preset movement mode, the spectral range is set to 400-700nm, the spectral resolution is 2nm, and the reconstructed image in the 550nm spectral band is similar to the original image. Figure 4 As shown, the spectral curves at random positions in the scene are compared. Figure 5 As shown, the PSNR of the reconstructed image is 47.94dB.

[0067] Optionally, when the pixel filter is in a motion initial position or a motion cutoff position, each of the filtering pixel grids coincides with an imaging pixel grid of the detector.

[0068] By ensuring that each filter pixel grid coincides with an imaging pixel grid of the detector, the image data in the same imaging pixel grid is unified, avoiding filter effect errors caused by pixel misalignment, thereby affecting the reconstruction result. The reconstruction accuracy is further improved by controlling the overlap.

[0069] It should be noted that the piezoelectric displacement actuator has closed-loop control capabilities, and each displacement can be controlled with an accuracy of several nanometers, which can ensure that the filter pixels coincide with the detector imaging pixels.

[0070] Optionally, refer to Figure 3 As shown, the timing constraint relationship between the exposure of the detector, the reading of the pixel data, and the displacement actuator driving the pixel filter to shift is:

[0071] When the exposure of the detector is completed, the displacement actuator is started to drive the pixel filter to move, and the displacement process of one imaging pixel grid needs to be completed before the next exposure starts;

[0072] Pixel data readout can only be started after the detector exposure is completed, and all pixel readout operations need to be completed before the next exposure ends.

[0073] Through the above constraint relationship, it can be ensured that the imaging pixel grid and the filtering pixel grid of the pixel filter coincide with each other during each exposure, and it can be ensured that each exposure is at a different position to avoid duplication of collected pixel data.

[0074] Optionally, the shutter of the area array camera is a global electronic shutter.

[0075] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. Similarly, in order to simplify the present invention and help understand one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. Wherein, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself is a separate embodiment of the present invention.

[0076] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and further may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive.

[0077] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A multi-snapshot hyperspectral compression imaging system, characterized in that: include: An optical lens for acquiring image data; A pixel filter having a plurality of filter pixel grids for filtering the image data; An area array camera, the area array camera comprising a detector having a plurality of imaging pixel grids, the filter pixel grid having the same specification as the imaging pixel grid and being used to collect the image data; A displacement actuator, the pixel filter is arranged on the displacement actuator, and is used to move the pixel filter in a preset movement mode so that the detector collects a plurality of image data with filters at different positions and with different filtering effects; A host computer is equipped with a spectral domain reconstruction neural network model, and spatial combination is performed through the spectral domain reconstruction neural network model to reconstruct the multiple image data into spatial-spectral data cube reconstruction data.

2. The multi-snapshot hyperspectral compression imaging system according to claim 1, characterized in that: The filter pixels of the filter are arranged in an array. The pixel filter includes a plurality of filter areas. The colors of the filter pixels in the same filter area are all different.

3. The multi-snapshot hyperspectral compression imaging system according to claim 2, characterized in that: The number of filter pixel grids in any of the filter areas is 9 and is arranged in a 3*3 array.

4. The multi-snapshot hyperspectral compression imaging system according to claim 1, characterized in that: The preset movement mode consists of multiple movement schemes, including: A first movement scheme, controlling the pixel filter to move one imaging pixel grid along a first direction; A second movement scheme, controlling the pixel filter to move one imaging pixel grid along a second direction, and acquiring the image data when the filtering pixel grid of the pixel filter coincides with the imaging pixel grid; the second direction is perpendicular to the first direction, and the number of grids moved in the second direction is accumulated once; The third movement scheme is to control the pixel filter to move one imaging pixel grid along the third direction, and obtain the image data when the filtering pixel grid of the pixel filter coincides with the imaging pixel grid; the third direction is opposite to the second direction, and the number of grids moved in the third direction is accumulated once.

5. The multi-snapshot hyperspectral compression imaging system according to claim 4, characterized in that: The preset moving mode is executed as follows: Step S1, repeatedly executing the second movement scheme until the number of movement grids in the second direction is 3, and the number of translations is accumulated to 3 times; Step S2, executing the first movement plan and clearing the data of the number of movement grids in the second direction; Step S3, repeatedly executing the third movement scheme until the number of movement grids in the third direction is 3, and the number of translations is accumulated to 3 times; Step S4, executing the first movement plan and clearing the data of the number of movement grids in the second direction; Step S5, repeatedly executing the second movement scheme until the number of movement grids in the second direction is 3, and the number of translations is accumulated to 3 times; Step S6: When the number of translation times reaches the target number, stop the image data acquisition process.

6. The multi-snapshot hyperspectral compression imaging system according to claim 4, characterized in that: When the pixel filter is in the motion initial position or the motion cutoff position, each of the filter pixel grids coincides with the imaging pixel grid of the detector.

7. The multi-snapshot hyperspectral compression imaging system according to claim 1, characterized in that: The timing constraint relationship between the exposure of the detector, the reading of the pixel data, and the displacement actuator driving the pixel filter to shift is: When the exposure of the detector is completed, the displacement actuator is started to drive the pixel filter to move, and the displacement process of one imaging pixel grid needs to be completed before the next exposure starts; Pixel data readout can only be started after the detector exposure is completed, and all pixel readout operations need to be completed before the next exposure ends.

8. The multi-snapshot hyperspectral compression imaging system according to claim 1, characterized in that: The shutter of the area array camera is a global electronic shutter.

9. The multi-snapshot hyperspectral compression imaging system according to claim 2, characterized in that: The color of each pixel grid is different.

Citation Information

Patent Citations

  • Spectral imaging system based on micro-nano optical filter array and scanning method thereof

    CN114112041A

  • Hyperspectral calculation imaging method and system for multispectral acquisition and related equipment

    CN114218857A

  • Passive multispectral compressed sensing imaging device and imaging method

    CN114674436A

  • Snapshot spectral imaging method and system based on joint optimization

    CN114972803A

  • Snapshot type multispectral imaging system and manufacturing method thereof

    CN116295831A