A multi-snapshot hyperspectral compressive imaging system
By using displacement actuators and spectral domain reconstruction neural network models in a hyperspectral compressed imaging system, high-precision image reconstruction under multiple snapshot acquisitions was achieved, solving the problem of low image reconstruction accuracy in existing technologies and meeting the needs of remote sensing applications with high information throughput and high sensitivity.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHANGCHUN INST OF OPTICS FINE MECHANICS & PHYSICS CHINESE ACAD OF SCI
- Filing Date
- 2024-07-26
- Publication Date
- 2026-07-14
Smart Images
Figure CN118896689B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of spectral imaging technology, specifically to a multi-snapshot hyperspectral compressed imaging system. Background Technology
[0002] Spectral imaging technology has the ability to acquire scene space-spectral data cubes. The added high-resolution spectral channel information provides more refined differential features for target detection, identification, and analysis, and has been widely used in fields such as military security, environmental monitoring, biological science, medical diagnosis, and food inspection.
[0003] Within the framework of the traditional Nyquist-Shannon complete sampling theory, spectral imagers primarily employ three acquisition methods to acquire three-dimensional spatial-spectral data cubes using two-dimensional image sensors: First, they slice and time-division process the data cube, such as pushbroom spectral imagers, filter spectral imagers, and Fourier transform spectral imagers. These architectures typically require additional scanning or switching mechanisms, sacrificing temporal resolution. Second, they remap and reduce the dimensionality of the data cube, such as beam splitting spectral imagers, fiber reconstructed spectral imagers, and multi-aperture or microlens array spectral imagers. These architectures can achieve snapshot-like acquisition, but they drastically increase the sensor array size requirement, and the spectral resolution is generally low. Third, they sample and reduce the dimensionality of the data cube, such as pixel filter array spectral imagers and microslicer spectral imagers. These architectures can still achieve snapshot-like acquisition, but they sacrifice spatial resolution, and discrete sampling also leads to decreased sensitivity. In summary, traditional spectral imagers need to make compromises between spatial, spectral, and temporal resolution, which is becoming increasingly difficult to reconcile in remote sensing applications that are developing towards high information throughput, high sensitivity, and high integration.
[0004] Based on the compressed sensing theoretical framework, compressed spectral imaging technology resolves the contradictions between spatial, spectral, and temporal resolutions by encoding, modulating, and dimensionality-reducing the spatial-spectral data cube. It also reconstructs the original signal based on prior features of the natural scene, requiring far less data than the Nyquist method. In the typical hyperspectral compressed imaging CASSI architecture, the hardware combination of the encoding aperture and dispersive elements hinders system miniaturization. In contrast, the wide-spectrum pixel filter architecture (MSFA) achieves effective modulation of spatial-spectral information using only pixel filters mounted on the image sensor surface, greatly simplifying the hyperspectral compressed imaging system and offering advantages in robustness and miniaturization. However, within this framework, the spectral transmittance of each pixel is fixed, thus only one snapshot sampling can be performed. The limited data makes spectral data reconstruction difficult, especially in high-spectral-resolution applications, where reconstruction quality is hard to guarantee. Summary of the Invention
[0005] In view of the above problems, embodiments of the present invention provide a multi-snapshot hyperspectral compression imaging system to solve the technical problem of low image reconstruction accuracy in existing multi-snapshot hyperspectral compression imaging systems.
[0006] According to one aspect of the present invention, a multi-snapshot hyperspectral compressed imaging system is provided, comprising:
[0007] Optical lenses are used to acquire image data;
[0008] A pixel filter has multiple filter pixels for filtering the image data;
[0009] An area array camera, comprising a detector having multiple imaging pixel grids, wherein the filter pixel grids have the same specifications as the imaging pixel grids and are used to acquire the image data;
[0010] A displacement actuator, wherein the pixel filter is disposed on the displacement actuator, and is used to move the pixel filter according to a preset movement method so that the detector acquires multiple image data with different filtering effects at different positions of the filter;
[0011] The host computer is equipped with a spectral domain reconstruction neural network model, and spatial combination is performed through the spectral domain reconstruction neural network model to reconstruct multiple image data into spatial-spectral data cube reconstruction data.
[0012] In one alternative embodiment, the filter pixel grid is configured as an array, the pixel filter includes multiple filter areas, and the color of the filter pixel grid in the same filter area is different.
[0013] In one alternative approach, the number of filter pixels in any of the filter regions is 9, and they are configured as a 3x3 array.
[0014] In one alternative approach, the preset movement mode comprises multiple movement schemes, including:
[0015] The first movement scheme involves controlling the pixel filter to move along a first direction by one imaging pixel grid.
[0016] The second movement scheme involves controlling the pixel filter to move along a second direction by one imaging pixel grid, and acquiring the image data when the filter pixel grid of the pixel filter coincides with the imaging pixel grid; the second direction is perpendicular to the first direction, and the number of grids moved in the second direction is accumulated once;
[0017] The third movement scheme involves controlling the pixel filter to move along a third direction by one imaging pixel grid, and acquiring the image data when the filter pixel grid of the pixel filter coincides with the imaging pixel grid; the third direction is opposite to the second direction, and the number of third-direction movement grids is accumulated once.
[0018] In one alternative approach, the preset movement method is executed as follows:
[0019] Step S1: Repeat the second movement scheme until the number of squares moved in the second direction is 3, and accumulate to 3 translation counts;
[0020] Step S2: Execute the first movement plan and clear the data of the number of squares moved in the second direction;
[0021] Step S3: Repeat the third movement scheme until the third-direction movement grid number is 3, and accumulate to 3 translation times;
[0022] Step S4: Execute the first movement plan and clear the data of the number of movement grids in the second direction;
[0023] Step S5: Repeat the second movement scheme until the number of squares moved in the second direction is 3, and accumulate to 3 translation counts;
[0024] Step S6: When the number of translations reaches the target number, stop the image data acquisition process.
[0025] In one alternative approach, when the pixel filter is in the initial motion position or the motion stop position, each of the filter pixel grids coincides with the imaging pixel grid of the detector.
[0026] In one alternative approach, the timing constraints between the detector's exposure, the pixel data readout, and the displacement actuator driving the pixel filter displacement are as follows:
[0027] The displacement actuator is activated after the detector exposure ends to drive the pixel filter displacement, and the displacement process of one imaging pixel grid needs to be completed before the next exposure begins.
[0028] Pixel data readout can only begin after the detector exposure is complete, and all pixel readout operations must be completed before the next exposure ends.
[0029] In one alternative approach, the shutter of the area array camera is a global electronic shutter.
[0030] This application's multi-snapshot hyperspectral compressed imaging system moves the pixel filter according to a preset movement pattern using a displacement actuator, causing the detector to acquire multiple image data with different filter effects at different positions. The host computer spatially combines these images using a spectral domain reconstruction neural network model to reconstruct a spatial-spectral data cube reconstructed data set. By performing image processing on the original image acquisition device, the accuracy of the reconstructed spatial-spectral data cube is achieved, thus solving the technical problem of low image reconstruction accuracy in existing hyperspectral compressed imaging systems.
[0031] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description
[0032] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0033] Figure 1 A schematic diagram of the structure of the multi-snapshot hyperspectral compressed imaging system provided by the present invention is shown;
[0034] Figure 2 This invention illustrates a schematic diagram of the two-dimensional displacement of the pixel filter in the multi-snapshot hyperspectral compressed imaging system provided by the present invention.
[0035] Figure 3 A schematic diagram illustrating the temporal constraint relationship between detector operation and piezoelectric driven displacement in the multi-snapshot hyperspectral compressed imaging system provided by the present invention is shown.
[0036] Figure 4 This diagram illustrates a comparison between the reconstructed image and the original image in the 550nm spectral band of the multi-snapshot hyperspectral compressed imaging system provided by the present invention.
[0037] Figure 5 This invention provides a schematic diagram showing a comparison of spectral curves at random locations in a target scene using the multi-snapshot hyperspectral compressed imaging system.
[0038] Figure 6 A schematic diagram illustrating the displacement process of the pixel filter in the multi-snapshot hyperspectral compressed imaging system provided by the present invention is shown. Detailed Implementation
[0039] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.
[0040] This application proposes a multi-snapshot hyperspectral compression imaging system to solve the technical problem of low image reconstruction accuracy in existing hyperspectral compression imaging systems.
[0041] In an alternative embodiment, refer to Figure 1 As shown, a multi-snapshot hyperspectral compressed imaging system includes an optical lens, a pixel filter, an area array camera, and a displacement actuator.
[0042] The pixel filter has multiple filter pixels, and the area array camera includes a detector with multiple imaging pixels. The filter pixels have the same specifications as the imaging pixels. The pixel filter is disposed on the displacement actuator.
[0043] In the above embodiments, an optical lens acquires image data for filtering, an area scan camera acquires the image data, and a displacement actuator moves the pixel filter according to a preset movement method so that the detector acquires multiple image data with different filter positions and different filtering effects. A host computer is equipped with a spectral domain reconstruction neural network model, and spatial combination is performed through the spectral domain reconstruction neural network model to reconstruct multiple image data into spatial-spectral data cube reconstruction data.
[0044] In this application, the multi-snapshot hyperspectral compressed imaging system moves the pixel filter according to a preset movement method via a displacement actuator, so that the detector acquires multiple image data with different filter effects at different positions of the filter. The host computer performs spatial combination through the spectral domain reconstruction neural network model to reconstruct the multiple image data into spatial-spectral data cube reconstruction data. At this time, by setting up image processing on the original image acquisition device, the spatial-spectral data cube reconstruction data reconstructed by the above scheme has high accuracy, thereby solving the technical problem of low image reconstruction accuracy in existing hyperspectral compressed imaging systems.
[0045] Optionally, refer to Figure 2 As shown, the filter pixel grid of the filter is set as an array, and the pixel filter includes multiple filter areas, and the color of the filter pixel grid in the same filter area is different.
[0046] By setting the above parameters, each side of the obtained image data can be made uniform, which facilitates subsequent pixel-by-pixel changes and data acquisition, making the changes and data acquisition process accurate and controllable.
[0047] In addition, by repeating the filter area and setting different colors within the filter area, the requirements for the filter manufacturing process can be reduced. At the same time, by setting the requirement that the colors of the filter pixels in the same filter area are all different, it can be ensured that the filtering effect of moving the same imaging pixel is not consistent each time during the pixel-by-pixel movement, thereby improving the accuracy of the final image data reconstruction into spatial-spectral data cube reconstruction data.
[0048] Optionally, the number of filter pixels in any of the filter areas is 9, and they are set as a 3*3 array.
[0049] By setting the above parameters, each side of the obtained image data can be made uniform, which facilitates subsequent pixel-by-pixel changes and data acquisition, making the changes and data acquisition process accurate and controllable.
[0050] Optionally, the preset movement mode comprises multiple movement schemes, including:
[0051] The first movement scheme involves controlling the pixel filter to move along a first direction by one imaging pixel grid.
[0052] The second movement scheme involves controlling the pixel filter to move along a second direction by one imaging pixel grid, and acquiring the image data when the filter pixel grid of the pixel filter coincides with the imaging pixel grid; the second direction is perpendicular to the first direction, and the number of grids moved in the second direction is accumulated once.
[0053] The third movement scheme involves controlling the pixel filter to move along a third direction by one imaging pixel grid, and acquiring the image data when the filter pixel grid of the pixel filter coincides with the imaging pixel grid; the third direction is opposite to the second direction, and the number of third-direction movement grids is accumulated once.
[0054] In the first, second, and third moving schemes described above, the displacement is the modulation of each acquired image. This ensures that the image data acquired each time is correlated, resulting in overlapping imaging pixel grids for the overall acquired image data. This allows the filtering effects of imaging pixel grids from different acquisitions to mutually assist in reconstruction, achieving high-precision image data reconstruction.
[0055] Optionally, the number of imaging pixel grids in the first direction is greater than or equal to the number of pixel filters.
[0056] Optionally, the number of imaging pixel grids in the second direction is greater than or equal to the number of pixel filters.
[0057] The above optional implementation schemes can increase the amount of initial data collected, avoiding invalid reconstruction due to insufficient data or parameters.
[0058] In an alternative embodiment, refer to Figure 6 As shown, the preset movement mode is executed as follows:
[0059] Step S1: Repeat the second movement scheme until the number of squares moved in the second direction is 3, and accumulate to 3 translation counts;
[0060] Step S2: Execute the first movement plan and clear the data of the number of squares moved in the second direction;
[0061] Step S3: Repeat the third movement scheme until the third-direction movement grid number is 3, and accumulate to 3 translation times;
[0062] Step S4: Execute the first movement plan and clear the data of the number of movement grids in the second direction;
[0063] Step S5: Repeat the second movement scheme until the number of squares moved in the second direction is 3, and accumulate to 3 translation counts;
[0064] Step S6: When the number of translations reaches the target number, stop the image data acquisition process.
[0065] In the above scheme, for each spatial pixel, image data modulated with nine different spectral transmittance characteristics are collected. This data is input into the spectral domain reconstruction neural network model, which yields 151 spectral bands of hyperspectral reconstruction data. All spatial pixels are reconstructed in this manner sequentially and then spatially combined to obtain the final spatial-spectral data cube reconstruction data.
[0066] Based on the aforementioned preset movement method, the spectral range is set to 400-700nm, and the spectral resolution is 2nm. The reconstructed image in the 550nm spectral band is similar to the original image. Figure 4 As shown, spectral curves at random locations in the scene are compared, for example... Figure 5 As shown, the PSNR of the reconstructed image is 47.94dB.
[0067] Optionally, when the pixel filter is in the initial motion position or the motion stop position, each of the filter pixel grids coincides with the imaging pixel grid of the detector.
[0068] By ensuring that each filter pixel grid coincides with an imaging pixel grid of the detector, image data within the same imaging pixel grid is made consistent, avoiding errors in the filtering effect caused by pixel misalignment, which in turn affects the reconstruction result. Controlling the overlap further improves the reconstruction accuracy.
[0069] It should be noted that the piezoelectric displacement actuator has closed-loop control capability, and each displacement can be controlled with a precision of several nanometers, which can ensure that the filter pixels and the detector imaging pixels coincide.
[0070] Optionally, refer to Figure 3 As shown, the timing constraints between the detector's exposure, the pixel data readout, and the displacement actuator driving the pixel filter displacement are as follows:
[0071] The displacement actuator is activated after the detector exposure ends to drive the pixel filter displacement, and the displacement process of one imaging pixel grid needs to be completed before the next exposure begins.
[0072] Pixel data readout can only begin after the detector exposure is complete, and all pixel readout operations must be completed before the next exposure ends.
[0073] The above constraints ensure that the imaging pixel grid and the filter pixel grid overlap during each exposure. Furthermore, they guarantee that each exposure occurs at a different location, preventing duplicate pixel data acquisition.
[0074] Optionally, the shutter of the area array camera is a global electronic shutter.
[0075] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. Similarly, for the sake of brevity and to aid in understanding one or more aspects of the invention, in the description of exemplary embodiments of the invention above, various features of the embodiments are sometimes grouped together in a single embodiment, figure, or description thereof. The claims, which follow the detailed description, are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.
[0076] Those skilled in the art will understand that the modules in the device of the embodiment can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiment can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components, except that at least some of such features and / or processes or units are mutually exclusive.
[0077] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.
Claims
1. A multi-snapshot hyperspectral compressed imaging system, characterized in that, include: Optical lenses are used to acquire image data; A pixel filter has multiple filter pixels for filtering the image data; An area array camera, comprising a detector having multiple imaging pixel grids, wherein the filter pixel grids have the same specifications as the imaging pixel grids and are used to acquire the image data; A displacement actuator, wherein the pixel filter is disposed on the displacement actuator, and is used to move the pixel filter according to a preset movement method so that the detector acquires multiple image data with different filtering effects at different positions of the filter; The host computer is equipped with a spectral domain reconstruction neural network model, and spatial combination is performed through the spectral domain reconstruction neural network model to reconstruct multiple image data into spatial-spectral data cube reconstruction data; The displacement actuator operates according to the preset movement mode, moving the pixel filter in steps of a single imaging pixel grid, so that the multiple image data collected by the detector are compressed observation results with different spatial-spectral modulation codes provided by the same target scene at different spatial positions through the pixel filter; The spectral domain reconstruction neural network model is configured to receive the multiple image data and directly reconstruct the spatial-spectral data cube reconstruction data of the target scene by jointly decoding and nonlinearly mapping the compressed coding information contained in the multiple image data.
2. The multi-snapshot hyperspectral compressed imaging system as described in claim 1, characterized in that, The filter pixel grid is configured as an array, and the pixel filter includes multiple filter areas, with each filter pixel grid in the same filter area having a different color.
3. The multi-snapshot hyperspectral compressed imaging system as described in claim 2, characterized in that, The number of filter pixels in any of the filter regions is 9, and they are set as a 3*3 array.
4. The multi-snapshot hyperspectral compressed imaging system as described in claim 1, characterized in that, The preset movement mode consists of multiple movement schemes, including: The first movement scheme involves controlling the pixel filter to move along a first direction by one imaging pixel grid. The second movement scheme involves controlling the pixel filter to move along a second direction by one imaging pixel grid, and acquiring the image data when the filter pixel grid of the pixel filter coincides with the imaging pixel grid; the second direction is perpendicular to the first direction, and the number of grids moved in the second direction is accumulated once; The third movement scheme involves controlling the pixel filter to move along a third direction by one imaging pixel grid, and acquiring the image data when the filter pixel grid of the pixel filter coincides with the imaging pixel grid; the third direction is opposite to the second direction, and the number of third-direction movement grids is accumulated once.
5. The multi-snapshot hyperspectral compressed imaging system as described in claim 4, characterized in that, The preset movement mode is executed as follows: Step S1: Repeat the second movement scheme until the number of squares moved in the second direction is 3, and accumulate to 3 translation counts; Step S2: Execute the first movement plan and clear the data of the number of squares moved in the second direction; Step S3: Repeat the third movement scheme until the third-direction movement grid number is 3, and accumulate to 3 translation times; Step S4: Execute the first movement plan and clear the data of the number of movement grids in the second direction; Step S5: Repeat the second movement scheme until the number of squares moved in the second direction is 3, and accumulate to 3 translation counts; Step S6: When the number of translations reaches the target number, stop the image data acquisition process.
6. The multi-snapshot hyperspectral compressed imaging system as described in claim 4, characterized in that, When the pixel filter is in the initial or final position of motion, each of the filter pixels coincides with the imaging pixel grid of the detector.
7. The multi-snapshot hyperspectral compressed imaging system as described in claim 1, characterized in that, The timing constraints between the detector exposure, the pixel data readout, and the displacement actuator driving the pixel filter displacement are as follows: The displacement actuator is activated after the detector exposure ends to drive the pixel filter displacement, and the displacement process of one imaging pixel grid needs to be completed before the next exposure begins. Pixel data readout can only begin after the detector exposure is complete, and all pixel readout operations must be completed before the next exposure ends.
8. The multi-snapshot hyperspectral compressed imaging system as described in claim 1, characterized in that, The shutter of the area scan camera is a global electronic shutter.
9. The multi-snapshot hyperspectral compressed imaging system as described in claim 2, characterized in that, Each pixel has a different color.