Spectral imaging method and system based on hyperspectral low-rankness and binocular stereo matching
By employing a method based on hyperspectral low-rank and binocular stereo matching, and utilizing the SVD algorithm and GwcNet neural network for image correction, hyperspectral images can be rapidly reconstructed. This solves the problems of slow reconstruction speed and insufficient spectral image quality in existing technologies, achieving highly efficient spectral imaging results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIV
- Filing Date
- 2023-03-28
- Publication Date
- 2026-04-14
AI Technical Summary
Existing binocular stereo matching-based spectral imaging systems have shortcomings in reconstruction speed and spectral image quality. In particular, dictionary learning methods require a long time in dynamic scene applications, and the spectral image quality of panchromatic image reconstruction needs to be improved.
A method based on hyperspectral low-rank and binocular stereo matching is adopted. By acquiring images of the target scene from the CASSI branch and the RGB camera, the orthogonal spectral basis and coefficients are updated using the SVD algorithm. Parallax correction is performed by combining the GwcNet neural network, and the hyperspectral image is quickly reconstructed using the low-rank algorithm to eliminate aberrations caused by prism dispersion.
It greatly improves imaging speed, enhances spectral reconstruction accuracy and quality, eliminates aberrations through two-stage dispersion, and improves system debugging accuracy and imaging effect.
Smart Images
Figure CN116380246B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of spectral imaging technology, and in particular to a spectral imaging method and system based on hyperspectral low-rank properties and binocular stereo matching. Background Technology
[0002] Hyperspectral images consist of two spatial dimensions and one spectral dimension. Compared with the commonly used RGB color images, hyperspectral images can obtain more spectral channels, thus obtaining more details and features of the target. Therefore, they are widely used in medical imaging, remote sensing and other fields.
[0003] Ordinary cameras can only acquire two spatial dimensions and one temporal dimension of light information, completely ignoring the different dimensions of depth and spectrum. Existing dual-camera systems with depth acquisition capabilities can achieve high spatial and spectral resolution for simple scenes, but these systems mostly sacrifice temporal resolution to obtain scene depth information. To overcome this shortcoming, a depth-sensing spectral imaging system based on binocular stereo matching has been proposed. It simultaneously acquires depth and spectral information, and then uses disparity estimation to acquire spectral information simultaneously with depth information. The combination of the two yields richer spatial information, thereby improving the accuracy of spectral reconstruction.
[0004] Improving the reconstruction speed of spectral images is one of the key points of binocular stereo matching spectral imaging. To this end, researchers have designed different reconstruction algorithms to speed up the reconstruction. Among them, dictionary-based algorithms are a commonly used reconstruction method. However, this method requires a long time to learn the dictionary, and its application in dynamic scenes still needs to be optimized. Summary of the Invention
[0005] The purpose of this application is to overcome the shortcomings of the prior art and provide a spectral imaging method and system based on hyperspectral low-rank and binocular stereo matching.
[0006] Firstly, a spectral imaging method based on hyperspectral low-rank property and binocular stereo matching is provided, including:
[0007] S100: Acquire the encoded image from the CASSI branch of the target scene and the color image from the RGB camera, and represent the low-rank hyperspectral image as follows: ,in, It is an orthogonal spectral basis, where F represents the coefficients;
[0008] S200, Orthogonal spectral base based on SVD algorithm The coefficients F are updated, and the hyperspectral image is initialized to obtain an initial hyperspectral image;
[0009] S400: Synthesize a color image from the CASSI branch viewpoint using the initial hyperspectral image;
[0010] S500 obtains the disparity between the color image under the CASSI branch view and the color image under the RGB camera view through the GwcNet neural network, and aligns the color image of the RGB camera to the CASSI branch according to the disparity to form a new color image.
[0011] S600: The new color image and the original encoded image are used again to quickly reconstruct the hyperspectral image using a low-order algorithm;
[0012] S700: Determine whether the peak signal-to-noise ratio of the reconstructed hyperspectral image reaches the threshold. If the determination result is otherwise, return to step S400. If the determination result is yes, output the reconstructed hyperspectral image.
[0013] Furthermore, acquiring the color image and encoded image of the target scene includes:
[0014] Constructing a sensing model based on a binocular matched-coded aperture snapshot system:
[0015]
[0016]
[0017] in, These are encoded images acquired by the CASSI branch. Represents the encoding template matrix, For Hadamah accumulation; It is a color image captured by an RGB camera. It is the spectral response function of the RGB detector;
[0018] The low-rank expression of the hyperspectral image: Substituting the above sensing model, we obtain the new sensing model as follows:
[0019]
[0020]
[0021] in, yes In matrix form, , represent Vectorization, .
[0022] Furthermore, based on the SVD algorithm, orthogonal spectral bases are... Sum of coefficients The updates include:
[0023] Encoded image Y, color image and The size of the segment is , , For each overlapping block, the spectral image is rapidly reconstructed by updating the orthogonal basis and coefficients. The model for the orthogonal basis and coefficients is represented as follows:
[0024]
[0025]
[0026] in, represent Vectorization, It is an orthogonal spectral basis, and F represents the coefficients.
[0027] Furthermore, initializing the hyperspectral image includes: for each updated block, the size is... All blocks are processed by the formula Reconstruction is performed, and all blocks are aggregated into the final initial hyperspectral image. .
[0028] Furthermore, synthesizing a color image from the CASSI perspective using the initial hyperspectral image includes: using the initial spectral image Synthesized color images from the CASSI branch perspective ,in, Represents the number of iterations. .
[0029] Furthermore, aligning the color images from the RGB camera to the CASSI branch based on parallax includes:
[0030] Color images from RGB cameras Corrected to the CASSI branch viewpoint, resulting in a corrected color image. The principle of disparity matching between corresponding pixels is expressed by the following formula:
[0031] in, This represents the cost of matching an image patch. This represents the disparity value.
[0032] Furthermore, the process of quickly reconstructing the hyperspectral image from the new color image and the original encoded image using a low-rank algorithm includes: replacing the color image in step S100 with the new color image, and then using the low-rank algorithm from steps S100 to S200 in conjunction with the observation image from the CASSI branch to quickly reconstruct the new hyperspectral image.
[0033] Secondly, a spectral imaging system based on hyperspectral low-rank properties and binocular stereo matching is provided, including:
[0034] RGB camera, used to acquire color images of the target scene. ;
[0035] The CASSI branch system, arranged from left to right, includes an objective lens, a double Amish prism, a relay lens, an encoding module, a relay lens, a double Amish prism, a relay lens, and a grayscale camera, used to acquire the encoded image Y of the target scene;
[0036] Computer, used to receive color images of the target scene. The encoded image Y is then processed, and a computer program is executed to implement spectral imaging methods such as those based on hyperspectral low-rank and binocular stereo matching.
[0037] Thirdly, a computer-readable storage medium is provided that stores program code for execution by a device, the program code including steps for performing a method as described in any implementation of the first aspect.
[0038] Fourthly, an electronic device is provided, the electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the method as in any of the implementations of the first aspect.
[0039] This application has the following beneficial effects: It fully utilizes the low-rank characteristics of spectral images, greatly improving the imaging speed. Furthermore, addressing the issue that the quality of spectral images reconstructed from panchromatic images used in past binocular stereo matching spectral imaging systems needs improvement, this application proposes an improved binocular stereo matching spectral imaging system. This system uses an RGB camera as the imaging device, uses color images to guide the reconstruction of spectral images, and eliminates the aberrations that may be caused by the first prism dispersion through two dispersion processes, thereby improving the accuracy of spectral reconstruction to a certain extent. Attached Figure Description
[0040] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.
[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 This is a flowchart of the spectral imaging method based on hyperspectral low-rank and binocular stereo matching according to Embodiment 1 of this application;
[0043] Figure 2 This is a schematic diagram of a spectral imaging system based on hyperspectral low-rank and binocular stereo matching, according to an embodiment of this application.
[0044] Figure label:
[0045] 1. RGB camera; 2. CASSI branch system; 201. Objective lens; 202. Double Amishi prism; 203. Relay lens; 204. Encoding template; 205. Grayscale camera; 3. Computer. Detailed Implementation
[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0047] Example 1
[0048] The spectral imaging method based on hyperspectral low-rank and binocular stereo matching according to Embodiment 1 of this application includes: S100, acquiring the encoded image obtained by the CASSI branch of the target scene and the color image obtained by the RGB camera 1, and representing the low-rank of the hyperspectral image as: ,in, It is an orthogonal spectral basis, where F represents the coefficients; S200, based on the SVD algorithm for orthogonal spectral basis. The process involves updating the coefficient F and initializing the hyperspectral image to obtain an initial hyperspectral image; S400: synthesizing a color image from the CASSI branch perspective using the initial hyperspectral image; S500: obtaining the disparity between the color image from the CASSI branch perspective and the color image from the RGB camera 1 perspective using the GwcNet neural network, and aligning the color image from the RGB camera 1 to the CASSI branch based on the disparity to form a new color image; S600: rapidly reconstructing the hyperspectral image again using a low-rank algorithm with the new color image and the original encoded image; S700: determining whether the peak signal-to-noise ratio of the reconstructed hyperspectral image reaches the threshold. If the determination result is "no", the process returns to step S400; if the determination result is "yes", the reconstructed hyperspectral image is output. This application fully exploits the low-rank characteristics of spectral images, greatly improving the imaging speed.
[0049] Specifically, Figure 1 A flowchart of the spectral imaging method based on hyperspectral low-rank property and binocular stereo matching in Embodiment 1 of the application is shown, including:
[0050] S100: Acquire the encoded image acquired by the CASSI branch of the target scene and the color image (i.e., RGB image) acquired by RGB camera 1, and represent the low-rank hyperspectral image as follows: ,in, It is an orthogonal spectral basis, where F represents the coefficients;
[0051] For details, please refer to Figure 2 The light from the target scene is captured from two branches. One branch enters the CASSI branch system 2, which consists of components such as the encoding template 204, dual Amish prisms 202, grayscale camera 205, and repeater 203. The other branch enters the RGB camera 1. The optical axes of the grayscale camera 205 and the RGB camera 1 are parallel and perpendicular to the baseline. The grayscale camera 205 and the RGB camera 1 are placed parallel to each other. The dispersion of the CASSI branch system 2 uses dual Amish prisms 202. After the incident light is dispersed by these prisms, the center wavelength light still exits the prism in its original direction. It is then encoded by the encoding template 204 and undergoes a second dispersion by an Amish prism placed in the opposite direction, eliminating the aberrations introduced by the first prism. Finally, it is imaged on the grayscale camera 205 by the repeater 203. After thorough calibration, this structure only exhibits parallax in the horizontal direction, making system debugging easier and reducing errors. The light is captured by the RGB camera 1 after entering the system.
[0052] Assuming the target scenario is ,X spatial dimensions The observed image (i.e., the encoded image) of CASSI branch system 2 is as follows:
[0053]
[0054] Where Y is the encoded image with a size of , The 204-matrix encoding template encodes the spatial and spectral dimensions of the target scene, with a size of [missing information]. , This is the Hadamard product, a linear transformation that can be written in matrix form as follows:
[0055]
[0056] in, It is the forward response matrix of CASSI branch system 2, with size . X is a vector representation of the target scene with a size of Similarly, the observed image (i.e., the color image) of RGB camera 1 is represented as:
[0057]
[0058] in, It is an observed color image, with a size of A is the spectral response function of the RGB detector, with dimensions of... This is a linear transformation, which can be written in the following matrix form:
[0059]
[0060] in, This is the image observed by RGB camera 1, which is a vector composed of the concatenation of image vectors from three channels. The size is , It is a matrix obtained based on the spectral curve of RGB camera 1, with a size of Combining CASSI branch system 2 and RGB camera 1, the model of the entire system is as follows:
[0061]
[0062] make The model of the entire system can be written as Once the system model is in place, the next step is to quickly reconstruct the spectral image of the target scene by observing the images and utilizing the low-rank characteristics of the spectrum.
[0063] After obtaining the observation images from two cameras, utilizing the low-rank characteristic of hyperspectral images, the hyperspectral image can be represented in low-rank as the product of orthogonal spectral bases and coefficients. The hyperspectral image is represented as follows:
[0064]
[0065] in, It is an orthogonal spectral basis with a size of , Representative coefficient, size is Where k represents the rank of the hyperspectral image, we then substitute the above equation into the observation models of CASSI branch system 2 and RGB camera 1. The model of CASSI branch system 2 becomes:
[0066]
[0067] in, It is by and The size was jointly decided upon. , represent Vectorization, The expression is:
[0068]
[0069] in, represent Vectorization, .
[0070] Similarly, the model of RGB camera 1 becomes: ;
[0071] S200, Orthogonal spectral base based on SVD algorithm Sum of coefficients The process involves updating and initializing the hyperspectral image to obtain an initial hyperspectral image.
[0072] Specifically, will , and The size of the segment is , , The overlapping blocks are used to quickly reconstruct the spectral image for each block by updating the orthogonal basis and coefficients. The models for both are represented as follows:
[0073] ,
[0074]
[0075] SVD is used to update the coefficient F. Next, the following formula will be used to... Substitute into orthogonal basis Update: ;
[0076] For each block after the update, the size is... All blocks are processed by the formula Reconstruction is performed, and all blocks are aggregated into the final initial hyperspectral image. .
[0077] S400: Synthesize a color image from the CASSI branch viewpoint using the initial hyperspectral image;
[0078] Specifically, using the initial spectral image Synthesized color images from the CASSI branch perspective .in Represents the number of iterations. .
[0079] S500 obtains the disparity between the color image under the CASSI branch view and the color image under the RGB camera 1 view through the GwcNet neural network, and aligns the color image of RGB camera 1 to the CASSI branch according to the disparity to form a new color image.
[0080] Specifically, the input to the GwcNet neural network is a color image from the perspective of the CASSI branch. Color images captured by RGB camera 1 The disparity maps of the two color images were obtained. Thus, the disparity matrix is obtained, and the disparity matching principle for each pixel can be expressed by the following formula:
[0081]
[0082] in, This represents the cost of matching an image patch. This represents the parallax value, which aims to maximize the overlap area of color images from two different viewpoints, i.e., minimize the parallax value.
[0083] After calculating the pixel position in the RGB branch color image corresponding to each pixel in the CASSI branch color image, the pixel value is used as the pixel value of the RGB branch color image, and the resulting calibrated color image is the new color image. .
[0084] S600: The new color image and the original encoded image are used again to quickly reconstruct the hyperspectral image using a low-order algorithm;
[0085] Specifically, the process of quickly reconstructing the hyperspectral image from the new color image and the original encoded image using a low-rank algorithm includes: replacing the color image in step S100 with the new color image, combining the observation image from the CASSI branch, and then using the low-rank algorithm from steps S100 to S200 to quickly reconstruct the new hyperspectral image.
[0086] S700: Determine whether the peak signal-to-noise ratio (PSNR) of the reconstructed hyperspectral image has reached the threshold. It should be noted that the threshold of the peak signal-to-noise ratio is set according to the application requirements. When the peak signal-to-noise ratio reaches the preset threshold, the current hyperspectral image has met the application requirements. If the determination result is "no", return to step S400. If the determination result is "yes", output the reconstructed hyperspectral image.
[0087] Example 2
[0088] like Figure 2 As shown, the spectral imaging system based on hyperspectral low-rank and binocular stereo matching involved in Embodiment 2 of this application includes:
[0089] RGB camera 1, used to acquire color images of the target scene. ;
[0090] CASSI branch system 2 includes, from left to right, an objective lens 201, a double Amish prism 202, a relay lens 203, an encoding module, a relay lens 203, a double Amish prism 202, a relay lens 203, and a grayscale camera 205, used to acquire an encoded image Y of the target scene. In CASSI branch system 2, the dispersion is achieved using a double Amish prism 202. After the incident light is dispersed by this prism, the light of the center wavelength still exits the prism in the original direction. Then, after being encoded by the encoding template 204, it undergoes a second dispersion through an Amish prism placed in the opposite direction to eliminate the aberrations caused by the first prism. Finally, the image is formed on the grayscale camera 205 through the relay lens 203.
[0091] Computer 3 is used to receive color images of the target scene. The encoded image Y is then processed, and a computer program is executed to implement a spectral imaging method based on hyperspectral low-rank and binocular stereo matching.
[0092] It should be noted that, in response to the problem that the quality of the reconstructed spectral image using panchromatic images in past binocular stereo matching spectral imaging systems needs to be improved, an improved binocular stereo matching spectral imaging system is proposed. This system uses an RGB camera 1 as the imaging device, uses color images to guide the reconstruction of the spectral image, and eliminates the aberrations that may be caused by the first prism dispersion through two dispersions, which can improve the accuracy of spectral reconstruction to a certain extent.
[0093] Example 3
[0094] The present application discloses a computer-readable storage medium that stores program code for execution by a device, the program code including steps for performing the method in any implementation of the present application, as described in the first embodiment of the present application.
[0095] The computer-readable storage medium may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM); the computer-readable storage medium may store program code, and when the program stored in the computer-readable storage medium is executed by a processor, the processor is used to perform the steps of the method in any of the implementations of Embodiment 1 of this application.
[0096] Example 4
[0097] An electronic device according to Embodiment 4 of this application includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the method in any of the implementations in Embodiment 1 of this application.
[0098] The processor can be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to implement the method in any of the implementations of Embodiment 1 of this application.
[0099] The processor can also be an integrated circuit electronic device with signal processing capabilities. In implementation, each step of the method in any of the implementations of Embodiment 1 of this application can be completed by the integrated logic circuitry in the processor's hardware or by software instructions.
[0100] The aforementioned processor can also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the functions required by the units included in the data processing apparatus of the embodiments of this application, or executes the methods in any implementation of Embodiment 1 of this application.
[0101] The above are merely preferred embodiments of this application; however, the scope of protection of this application is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in this application, based on the technical solution and its improved concept, should be covered within the scope of protection of this application.
Claims
1. A spectral imaging method based on hyperspectral low-rank property and binocular stereo matching, characterized in that, include: S100: Acquire the encoded image from the CASSI branch of the target scene and the color image from the RGB camera, and represent the low-rank hyperspectral image as follows: ,in, It is an orthogonal spectral basis, where F represents the coefficients; Acquiring the color image and encoded image of the target scene includes: Constructing a sensing model based on a binocular matched-coded aperture snapshot system: ； ； Where Y is the encoded image acquired by the CASSI branch. Represents the encoding template matrix. For Hadamah accumulation; It is a color image captured by an RGB camera. It is the spectral response function of the RGB detector; The low-rank expression of the hyperspectral image: Substituting the above sensing model, we obtain the new sensing model as follows: ； ； in, yes In matrix form, , represent Vectorization, ; S200, Orthogonal spectral base based on SVD algorithm Sum of coefficients The process involves updating and initializing the hyperspectral image to obtain an initial hyperspectral image. S400: Synthesize a color image from the CASSI branch viewpoint using the initial hyperspectral image; S500 obtains the disparity between the color image under the CASSI branch view and the color image under the RGB camera view through the GwcNet neural network, and aligns the color image of the RGB camera to the CASSI branch according to the disparity to form a new color image. S600: The new color image and the original encoded image are used again to quickly reconstruct the hyperspectral image using a low-order algorithm; S700: Determine whether the peak signal-to-noise ratio of the reconstructed hyperspectral image reaches the threshold. If the determination result is otherwise, return to step S400. If the determination result is yes, output the reconstructed hyperspectral image.
2. The spectral imaging method based on hyperspectral low-rank property and binocular stereo matching according to claim 1, characterized in that, Orthogonal spectral bases based on SVD algorithm Sum of coefficients The updates include: Encoded image Y, color image and The size of the segment is , , For each overlapping block, the spectral image is rapidly reconstructed by updating the orthogonal basis and coefficients. The model for the orthogonal basis and coefficients is represented as follows: ； ； in, represent Vectorization, It is an orthogonal spectral basis, and F represents the coefficients.
3. The spectral imaging method based on hyperspectral low-rank property and binocular stereo matching according to claim 2, characterized in that, Initializing the hyperspectral image includes: for each updated block, the size is... All blocks are processed by the formula Reconstruction is performed, and all blocks are aggregated into the final initial hyperspectral image. .
4. The spectral imaging method based on hyperspectral low-rank property and binocular stereo matching according to claim 3, characterized in that, Synthesizing a color image from the CASSI perspective using an initial hyperspectral image includes: using the initial spectral image Synthesized color images from the CASSI branch perspective ,in, Represents the number of iterations. .
5. The spectral imaging method based on hyperspectral low-rank property and binocular stereo matching according to claim 4, characterized in that, Aligning the color images from the RGB camera to the CASSI branch based on parallax includes: Color images from RGB cameras Corrected to the CASSI branch viewpoint, resulting in a corrected color image. The principle of disparity matching between corresponding pixels is expressed by the following formula: ； in, This represents the cost of matching an image patch. This represents the disparity value.
6. The spectral imaging method based on hyperspectral low-rank property and binocular stereo matching according to claim 5, characterized in that, The process of quickly reconstructing a hyperspectral image from the new color image and the original encoded image using a low-rank algorithm includes: replacing the color image in step S100 with the new color image, combining the observation image from the CASSI branch, and then using the low-rank algorithm from steps S100 to S200 to quickly reconstruct the new hyperspectral image.
7. A spectral imaging system based on hyperspectral low-rank properties and binocular stereo matching, characterized in that, include: RGB camera, used to acquire color images of the target scene. ; The CASSI branch system, arranged from left to right, includes an objective lens, a double Amish prism, a relay lens, an encoding module, a relay lens, a double Amish prism, a relay lens, and a grayscale camera, used to acquire the encoded image Y of the target scene; Computer, used to receive color images of the target scene. The encoded image Y is then processed, and a computer program is executed to implement the method as described in any one of claims 1-6.
8. A computer-readable storage medium, characterized in that, The computer-readable medium stores program code for execution by the device, the program code including steps for performing the method as described in any one of claims 1-6.
9. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the method as described in any one of claims 1-6.