A spectral imaging method based on rgb camera and broadband filter encoding

Abstract

The application discloses a spectral imaging method based on an RGB camera and a broadband filter coding. According to a camera spectral response function, an optimized parameter of a broadband filter and a spectral reconstruction network construction design stage, an end-to-end optimization framework is constructed. Then, the end-to-end optimization framework is trained according to a hyperspectral dataset to obtain a trained end-to-end optimization framework of the design stage and obtain a theoretical design parameter of the broadband filter. Then, an actual spectral transmittance function of the combined broadband filter and a final spectral reconstruction network are obtained. Finally, an RGB imaging system is built and a scene image is obtained, which is input into the final spectral reconstruction network to obtain a hyperspectral image. The application is based on the wavelength coding spectral imaging principle, a specially designed broadband filter is added in the imaging light path, and the input image of the reconstruction algorithm is increased from 3 channels to multiple channels through multiple imaging, so that the spectral accuracy of the reconstruction result can be effectively improved.

Description

Technical Field

[0001] This invention relates to a wavelength-coded spectral imaging system and method in the field of computational spectral imaging, specifically a spectral imaging method based on an RGB camera and broadband filter coding. Background Technology

[0002] Spectral imaging, a multidimensional information acquisition technique combining imaging and spectral technologies, offers advantages over traditional imaging methods in obtaining both two-dimensional spatial and one-dimensional spectral information of the target object. Since different substances possess unique spectral characteristics, similar to human fingerprints, spectral imaging is a primary means of acquiring information for studying material composition, thermal radiation characteristics, and other related fields, playing a crucial role in food safety, agricultural production, water quality monitoring, biomedicine, and resource exploration. Traditional spectral data cube acquisition can be divided into two methods: 1) Placing narrowband filters in front of a conventional two-dimensional imaging system, switching between filters with different passband ranges, and performing multiple simultaneous images to complete the spectral dimension scan and obtain a three-dimensional data cube, with the number of spectral channels matching the number of narrowband filters; 2) Adding dispersive elements such as gratings or prisms to the imaging system, arranging the incident light sequentially on the sensor according to wavelength, obtaining one-dimensional spatial and spectral information on the image plane, and acquiring a three-dimensional data cube through push-broom techniques. Both of these imaging systems, due to the need for scanning steps, are typically characterized by system complexity, numerous precision moving parts, and long processing times, severely limiting their application scenarios.

[0003] With the development of computer resources, computational coding-based reconstruction of spectral imaging methods has emerged. These methods allow for the design of more sophisticated hardware systems through coding and other means, followed by reconstruction algorithms to reconstruct degraded images and obtain spectral data cubes. Compared to traditional spectral imaging methods, these coding-based reconstruction systems generally feature small size, light weight, and high throughput. However, due to the need for complex algorithms and even large amounts of data, these techniques also suffer from poor robustness, low spectral resolution, and shortcomings in algorithm stability and generalization. Coding-based reconstruction of spectral imaging methods can be mainly divided into three categories based on the coding method: amplitude-coded, phase-coded, and wavelength-coded. Amplitude- or phase-coded spectral imaging methods typically result in severely degraded initial images, leading to very limited spatial resolution in the reconstruction results. Compared to these two methods, wavelength-coded systems, without spatial resolution degradation, offer more satisfactory spatial resolution in the reconstruction results.

[0004] In conventional wavelength-coded spectral imaging systems, materials such as filters, quantum dots, or metasurfaces are typically used as wavelength-coding elements to modulate spectral transmittance. Based on the placement of the wavelength-coding element, systems can be categorized into switching-type and image-plane integrated types. Switching-type systems typically place the encoding element in front of the light source or imaging system, acquiring multiple images through multiple switching and simultaneous captures, while a single imaging session yields only one image, resulting in slower imaging speeds. Image-plane integrated systems, on the other hand, integrate the wavelength-coding element onto the image plane, using multiple small pixels to form a measurement unit, which sacrifices the spatial resolution of the imaging result.

[0005] In color cameras, a Bayer filter array is installed in front of the CMOS sensor. This array typically consists of three filters with different spectral transmittances, corresponding to the R, G, and B color channels of the output image. These filters can be considered as wavelength encoding of the incident light spectrum. Therefore, the process of reconstructing the spectral data cube from an image from a conventional RGB color camera can also be considered a wavelength-encoded spectral imaging process. However, in conventional RGB cameras, the filter transmittance function is designed to match human vision. This visually-oriented design is not necessarily beneficial for the final spectral reconstruction task. Reconstructing a hyperspectral image from a conventional RGB image remains a highly problematic task, and the spectral accuracy and precision of the reconstructed result need improvement. Summary of the Invention

[0006] To address the technical problems existing in the background art, and considering the limited spectral accuracy of RGB image reconstruction and the low image acquisition efficiency of other wavelength-coded spectral imaging systems, this invention proposes to combine a conventional RGB imaging system with a broadband filter based on the principle of wavelength coding. By utilizing the wavelength coding potential of the Bayer filter built into the RGB camera and the external broadband coding filter, the number of initial image channels can be multiplied, effectively improving the spectral accuracy of the reconstructed spectral image.

[0007] The technical solution for achieving the objective of this invention is as follows:

[0008] 1) Set the initial optimization parameters corresponding to m broadband filters;

[0009] 2) Based on the spectral response function S of the RGB camera, the optimized parameters of each broadband filter, and the spectral reconstruction network, an end-to-end optimization framework is constructed for the design phase.

[0010] 3) Train the end-to-end optimization framework for the design stage based on the hyperspectral dataset to obtain the trained end-to-end optimization framework for the design stage. Use the optimization parameters of each broadband filter in the trained end-to-end optimization framework for the design stage as the theoretical design parameters of each broadband filter.

[0011] 4) Based on the theoretical design parameters of each broadband filter, obtain the actual spectral transmittance function of each broadband filter and the final spectral reconstruction network;

[0012] 5) Based on the actual spectral transmittance function of each broadband filter and the RGB camera, build an RGB imaging system. The RGB imaging system actually captures scene images, and the corresponding hyperspectral images are reconstructed and output through the final spectral reconstruction network based on the scene images.

[0013] Specifically, 4) refers to:

[0014] Based on the theoretical design parameters of m broadband filters, the actual spectral transmittance functions of the m broadband filters are processed and calibrated. The actual spectral transmittance functions of the m broadband filters are used to replace the spectral transmittance functions corresponding to the optimization parameters of each broadband filter in the trained end-to-end optimization framework of the design stage, and these functions are set as non-trainable parameters. The experimental stage end-to-end optimization framework is then obtained, and the network is trained to obtain the trained experimental stage end-to-end optimization framework. The spectral reconstruction network in the trained experimental stage end-to-end optimization framework is denoted as the final spectral reconstruction network.

[0015] In step 5), n broadband filters are selected from m broadband filters and combined to obtain a combined broadband filter, where n ≤ m. The combined broadband filter and the RGB camera are arranged sequentially along the optical axis to form an RGB imaging system.

[0016] In step 5), the three-channel RGB images obtained by the RGB imaging system under different broadband filter combinations are all stitched together to obtain the scene image, which is then used as the input to the final spectral reconstruction network.

[0017] The scene images actually captured by the RGB imaging system satisfy the following formula:

[0018]

[0019]

[0020] Where the subscript c represents the three color channels output by the RGB camera, satisfying c∈{R, G, B}, R, G, B represent the R, G, B color channels output by the RGB camera respectively, and the subscript i represents the i-th shot, J ci (x, y) represents the scene image actually captured by the RGB imaging system, λ1 and λ2 represent the start and end wavelengths of the spectral imaging, respectively, I(x, y, λ) represents the incident spectral information of the scene, and T i (λ) represents the spectral transmittance function of the i-th combination of broadband filters, S c(λ) represents the spectral response function of the RGB camera in the c color channel; t q (λ) is the spectral transmittance function of the broadband filter (3)q, and Π(·) represents the multiplication symbol.

[0021] In step 2), the optimization parameters of each broadband filter in the end-to-end optimization framework during the design phase are set as trainable parameters. The optimization parameters of each broadband filter are mapped to the spectral transmittance function of the broadband filter using the following formula:

[0022]

[0023] Where T is the spectral transmittance function of each broadband filter; Ω is the optimized parameter of each broadband filter.

[0024] The formula for the total loss function of the end-to-end optimization framework is as follows:

[0025] L total =αL img +βL filter

[0026] L img =||I * -I||1

[0027]

[0028] Among them, L total L represents the total loss function value, α and β are the first and second balancing coefficients, respectively. img To reconstruct the spectral image I * The loss function value between the input hyperspectral image I and L filter The loss function value constrains the smoothness of the filter, where k represents the number of spectral channels, o represents the total number of spectral channels, and t... q (k) is the spectral transmittance function of the broadband filter (3)q, where |||||1 represents the 1-norm and || represents the absolute value.

[0029] Compared with the prior art, the beneficial effects of this invention are as follows:

[0030] (1) A wavelength encoding method combining an RGB camera and a broadband filter is used, which can obtain three-channel images in each shot, thus improving image acquisition efficiency;

[0031] (2) Use an end-to-end optimization framework to jointly optimize the transmittance function of the broadband filter and the weights of the neural network to achieve the optimal design in a specific scenario;

[0032] (3) Unlike existing technologies that reconstruct hyperspectral images in the RGB domain, this invention reconstructs hyperspectral images in the RAW domain, avoiding the influence of nonlinear post-processing from the RAW domain to the RGB domain of the image. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of a spectral imaging method based on an RGB camera and broadband filter encoding.

[0034] Figure 2 This is a schematic diagram of the imaging system structure.

[0035] Figure 3 This is a schematic diagram of the imaging process of the imaging system.

[0036] Figure 4 This is the spectral response function of the imaging system calibrated in this embodiment.

[0037] Figure 5 This is a graph showing the mapping relationship between the optimized parameters Ω of a broadband filter and the spectral transmittance function T.

[0038] Figure 6 This is a schematic diagram of the end-to-end optimization framework.

[0039] Figure 7 This is a graph of the transmittance function of the broadband filter optimized by the end-to-end optimization framework.

[0040] Figure 8 This is a schematic diagram of the spectral reconstruction results.

[0041] Figure 9 This is a comparison chart of the spectral accuracy between the method of this invention and the traditional RGB spectral reconstruction method.

[0042] Figure 10 This is a comparison of the spectral curves of the method of the present invention and the traditional RGB spectral reconstruction method.

[0043] In the image: 1. RGB camera, 2. Imaging lens, 3. Broadband filter. Detailed Implementation

[0044] The present invention will now be described in further detail with reference to the accompanying drawings.

[0045] According to the invention, the complete implementation embodiments and their implementation processes are as follows:

[0046] In this embodiment, a Sony A7M3 camera is selected as the RGB camera, a Tamron 28-75mm F2.8 lens is selected as the imaging lens, the number of broadband filters is m=1, the selected imaging spectral range is 420nm-660nm, with spectral intervals of 10nm, for a total of 25 spectral channels, and the publicly available KAIST and ICVL datasets are used as the training and testing sets for the training of the neural network.

[0047] like Figure 1 and Figure 3 As shown, the implementation process of this invention can be divided into two stages: a training and optimization stage and an imaging experiment stage. The training and optimization stage mainly completes the optimal design of the broadband filter and the optimization of the reconstruction network weights. The imaging experiment stage mainly completes the calibration of the broadband filter, the fine-tuning of the reconstruction network weights, and the spectral reconstruction of the imaging results. Specifically, it includes the following steps:

[0048] 1) Set the initial optimization parameters for m broadband filters; the spectral encoding range of the broadband filters is from the start wavelength to the end wavelength of spectral imaging. Different spectral transmittance encodings can be achieved by combining 0 to m broadband filters 3. There are N possible combinations of broadband filters 3, which can be represented as:

[0049]

[0050] in, The expression indicates that n filters are selected from m filters for combination, and ! indicates a factorial operation. Since RGB camera 1 can obtain three color channels (R, G, B) in each shot, it can obtain images with a maximum of 3·N channels for spectral reconstruction.

[0051] 2) Based on the spectral response function S of the RGB camera, the optimized parameters of each broadband filter, and the spectral reconstruction network, an end-to-end optimization framework is constructed for the design phase, such as... Figure 6 As shown, the input of the end-to-end optimization framework in the design phase is combined with the spectral transmittance function of m broadband filters and the spectral response function S of the camera to form a degradation model. The degradation image is output and input into the spectral reconstruction network. The optimization parameters of the broadband filters in the end-to-end optimization framework in the design phase and the parameters in the spectral reconstruction network are set as trainable parameters, while the spectral response function S of the RGB camera is a non-trainable parameter.

[0052] The optimized parameter Ω of broadband filter 3 is a trainable parameter with a value range of (-∞, ∞), which obviously does not meet the requirement that the spectral transmittance of the filter is [0, 1]. Therefore, a mapping function is needed to map the trainable parameter Ω directly optimized by the network to the actual spectral transmittance function T. The following formula is used to map the optimized parameter of each broadband filter to the spectral transmittance function of the broadband filter. The correspondence between the two is as follows: Figure 5 As shown:

[0053]

[0054] Where T is the spectral transmittance function of each broadband filter; Ω is the optimized parameter of each broadband filter.

[0055] In subsequent step 4), the optimized parameters of each broadband filter in the trained end-to-end optimization framework of the design phase also satisfy the above mapping relationship with the corresponding theoretical spectral transmittance function. During the manufacturing process, there will be discrepancies between the theoretical spectral transmittance function and the actual spectral transmittance function of the broadband filter.

[0056] In practice, the spectral response function of the basic imaging system consisting of the RGB camera and the imaging lens is first calibrated. The calibrated spectral response function is as follows: Figure 4 As shown. Then, the optimized parameters Ω of the broadband filter are initialized, specifically by initializing the values ​​of Ω in each channel to 1 and setting them as trainable parameters. The optimized parameters Ω of the filter are then mapped to the spectral transmittance function T. The setup is as follows... Figure 6 The end-to-end optimization framework shown utilizes the spectral response function of the basic imaging system, the hyperspectral image, and the spectral transmittance function of the broadband filter combination to perform image degradation, obtaining a wavelength-coded output image, i.e., the degraded image. This wavelength-coded output image is then input into the spectral reconstruction network to obtain the spectral reconstruction result. The loss function between the spectral reconstruction result and the ground truth hyperspectral image is calculated, gradient backpropagation is performed, and the weight parameters of the spectral reconstruction network and the broadband filter optimization parameters are updated.

[0057] 3) The end-to-end optimization framework for the design phase is trained based on the hyperspectral dataset to obtain a trained end-to-end optimization framework for the design phase. Each hyperspectral image is processed by an RGB imaging system consisting of m broadband filters and an RGB camera, outputting a degraded image. In the neural network-based end-to-end framework, the spectral transmittance function of broadband filter 3 and the network weights of the spectral reconstruction algorithm are simultaneously optimized. The forward working model of the end-to-end optimization framework can be expressed by the following equation:

[0058] I * =Model(I, S, Ω)

[0059] Where I, S, and Ω represent the input hyperspectral image, the spectral response function of the RGB camera, and the optimized parameters of the combined broadband filter, respectively. * Let represent the reconstructed spectral image, and Model(·) represent the mapping of the neural network. The optimized parameters Ω of the combined broadband filters in the trained initial end-to-end optimization framework are used as the theoretical design parameters of the combined broadband filters. The spectral reconstruction network is denoted as the coarse spectral reconstruction network.

[0060] In this embodiment, the spectral reconstruction network used is the open-source MST++ network (Multi-stage Spectral-wise Transformer). The input image of the network has a scale of 256×256×6, where 256 represents the width and height of the image, and 6 represents the number of channels in the input image. The output image of the spectral reconstruction network has a scale of 256×256×25.

[0061] The total loss function of the end-to-end optimization framework consists of two main parts: one is the loss function between the reconstructed spectral image and the ground truth, which is used to reduce the error of the reconstruction result and constrain the spectral reconstruction.

[0062] The other part involves imposing constraints on the smoothness of the broadband filter 3 to achieve a smooth filter design, facilitating subsequent research and manufacturing. The specific formula for the total loss function is as follows:

[0063] L total =αL img +βL filter

[0064] L img =||I * -I||1

[0065]

[0066] Among them, L total L represents the total loss function value, α and β are the first and second balancing coefficients, respectively. img To reconstruct the spectral image (i.e., the output of the spectral reconstruction network) I * The loss function value L between the input hyperspectral image I (i.e., the true value) and the input hyperspectral image I (i.e., the ground truth) filter The loss function value constrains the smoothness of the filter, where k represents the number of spectral channels, o represents the total number of spectral channels, and t... q (k) is the spectral transmittance function of the broadband filter 3q, where ||||||1 represents the 1-norm, and ||| represents the absolute value. In this embodiment, m = 1, there are 25 spectral channels in the 420-660nm range, and o = 25.

[0067] 4) Based on the theoretical design parameters of each broadband filter, obtain the actual spectral transmittance function of each broadband filter and the final spectral reconstruction network;

[0068] 4) Specifically:

[0069] Based on the theoretical design parameters of m broadband filters, the actual spectral transmittance functions of the m broadband filters are processed and calibrated. The actual spectral transmittance functions of the m broadband filters are then used to replace the spectral transmittance functions corresponding to the optimized parameters of each broadband filter in the trained end-to-end optimization framework of the design stage. After setting these actual spectral transmittance functions as untrainable parameters, the optimized parameters are no longer needed. The experimental end-to-end optimization framework is then obtained, and the network is trained. This involves fine-tuning the weights of the spectral reconstruction network to eliminate the influence of broadband filter processing errors. The trained experimental end-to-end optimization framework is then obtained, and the spectral reconstruction network in the trained experimental end-to-end optimization framework is denoted as the final spectral reconstruction network.

[0070] 5) Based on the actual spectral transmittance function of each broadband filter and the RGB camera, construct an RGB imaging system, such as... Figure 2 As shown, the RGB imaging system actually captures the scene image, and the corresponding hyperspectral image is reconstructed and output through the final spectral reconstruction network based on the scene image.

[0071] After selecting n broadband filters from m broadband filters and combining them, a combined broadband filter is obtained, where n ≤ m. The combined broadband filter and the RGB camera are arranged sequentially along the optical axis to form an RGB imaging system.

[0072] In practice, an imaging lens 2 is mounted on an RGB camera 1 to acquire RAW images of the scene. The relative positions of the RGB camera 1 and the scene are adjusted and fixed, keeping their relative positions constant. 0 to n filters are selected from m broadband filters and combined to form new broadband filters with different spectral transmittance functions, which are denoted as combined broadband filters. The combined broadband filters are placed in front of the imaging lens, and RAW images of the scene under different broadband filter combinations are captured using the same shooting parameters. Each scene RAW image is de-mosaiced and converted into a corresponding three-channel RGB image. The image obtained by stitching together all three-channel RGB images under different broadband filter combinations along the channel dimension is used as the final scene image and input into the final spectral reconstruction network. The final spectral reconstruction network reconstructs a spectral data cube.

[0073] In this embodiment, since the number of broadband filters m=1, the imaging system can capture two types of RGB images: RGB images without broadband filter encoding and RGB images with broadband filters added. The spectral encoding information of the former comes from the Bayer filter array in front of the camera CMOS, and the spectral encoding information of the latter comes from the multiplication of the transmittance function of the Bayer filter array and the broadband filter.

[0074] The scene images actually captured by the RGB imaging system satisfy the following formula:

[0075]

[0076]

[0077] Wherein, the subscript c represents the three color channels output by the RGB camera, satisfying c∈{R, G, B}, where R, G, and B represent the R, G, and B color channels output by the RGB camera, respectively, and the subscript i represents the i-th shot. In this embodiment, i={1, 2}, J ci (x, y) represents the scene image actually captured by the RGB imaging system, i.e., the degraded image; λ1 and λ2 represent the start and end wavelengths of the spectral imaging, respectively. In this embodiment, the start and end wavelengths are 420 nm and 660 nm, respectively; I(x, y, λ) represents the incident spectral information of the scene; T i (λ) represents the spectral transmittance function of the i-th broadband filter combination, S c (λ) represents the spectral response function of the RGB camera in the c color channel; t q (λ) is the spectral transmittance function of the broadband filter 3q, and Π(·) represents the multiplication sign. When no broadband filter is placed in the imaging optical path of the imaging system, T i (λ) is always equal to 1.

[0078] The first image is captured without a broadband filter. Then, a second image is captured with a broadband filter placed in front of the imaging lens, using the same shooting parameters. These two images are then de-mosaiced to obtain two three-channel images. These images are then stitched together along the channel dimensions to obtain the image shown below. Figure 8 The 6-channel image is shown. Using this 6-channel image as input to a spectral reconstruction network, the reconstructed image is as follows. Figure 9 The hyperspectral image shown.

[0079] To verify the superiority of the present invention, a comparative experiment was conducted using direct reconstruction of hyperspectral images from traditional RGB images. In the comparative experiment, other equipment and parameters remained unchanged. The peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and spectral angle mapping (SAM) of the reconstruction results were compared. PSNR and SSIM represent the spatial resolution of the reconstruction results to a certain extent, and the higher these two indicators are, the better. SAM represents the spectral accuracy of the reconstruction results to a certain extent, and the lower this indicator is, the better. The comparison results are shown in Table 1.

[0080] Table 1 compares the accuracy of reconstruction results between the method of the present invention and the comparative methods.

[0081]

[0082]

[0083] The table above shows that the reconstruction results of this invention are superior to traditional methods in both spatial resolution and spectral accuracy. More intuitively, we selected color patches from the two images, plotted their spectral curves, and compared them with the true values. The comparison results are as follows... Figure 10 As shown, Ground Truth represents the true value of the simulation process, RGB2HSI represents the comparison method of directly reconstructing the hyperspectral image from the RGB image, and Ours represents the method of this invention. It can be seen that the spectral curve of the method of this invention is closer to the true value. To quantitatively illustrate the accuracy of the spectral curve, the mean absolute error is used to calculate the deviation between the reconstructed spectral curve and the true value. Specific data are shown in Table 2. The table shows that the method of this invention has superior spectral accuracy.

[0084] Table 2 shows the average absolute error of the spectral curves.

[0085]

[0086] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A spectral imaging method based on an RGB camera and broadband filter encoding, characterized in that, Includes the following steps: 1) Set the initial optimization parameters corresponding to m broadband filters; 2) Based on the spectral response function S of the RGB camera, the optimized parameters of each broadband filter, and the spectral reconstruction network, an end-to-end optimization framework is constructed for the design phase. 3) Train the end-to-end optimization framework for the design stage based on the hyperspectral dataset to obtain the trained end-to-end optimization framework for the design stage. Use the optimization parameters of each broadband filter in the trained end-to-end optimization framework for the design stage as the theoretical design parameters of each broadband filter. 4) Based on the theoretical design parameters of each broadband filter, obtain the actual spectral transmittance function of each broadband filter and the final spectral reconstruction network; 5) Based on the actual spectral transmittance function of each broadband filter and the RGB camera, build an RGB imaging system. The RGB imaging system actually captures scene images, and the corresponding hyperspectral images are reconstructed and output through the final spectral reconstruction network based on the scene images.

2. The spectral imaging method based on an RGB camera and broadband filter encoding according to claim 1, characterized in that, Specifically, 4) refers to: Based on the theoretical design parameters of m broadband filters, the actual spectral transmittance functions of the m broadband filters are processed and calibrated. The actual spectral transmittance functions of the m broadband filters are used to replace the spectral transmittance functions corresponding to the optimization parameters of each broadband filter in the trained end-to-end optimization framework of the design stage, and these functions are set as non-trainable parameters. The experimental stage end-to-end optimization framework is then obtained, and the network is trained to obtain the trained experimental stage end-to-end optimization framework. The spectral reconstruction network in the trained experimental stage end-to-end optimization framework is denoted as the final spectral reconstruction network.

3. The spectral imaging method based on an RGB camera and broadband filter encoding according to claim 1, characterized in that, In step 5), n broadband filters are selected from m broadband filters and combined to obtain a combined broadband filter, where n ≤ m. The combined broadband filter and the RGB camera are arranged sequentially along the optical axis to form an RGB imaging system.

4. The spectral imaging method based on an RGB camera and broadband filter encoding according to claim 1, characterized in that, In step 5), the three-channel RGB images obtained by the RGB imaging system under different broadband filter combinations are all stitched together to obtain the scene image, which is then used as the input to the final spectral reconstruction network.

5. The spectral imaging method based on an RGB camera and broadband filter encoding according to claim 1, characterized in that, The scene images actually captured by the RGB imaging system satisfy the following formula: Where the subscript c represents the three color channels output by the RGB camera, satisfying c∈{R, G, B}, R, G, B represent the R, G, B color channels output by the RGB camera respectively, and the subscript i represents the i-th shot, J ci (x, y) represents the scene image actually captured by the RGB imaging system, λ1 and λ2 represent the start and end wavelengths of the spectral imaging, respectively, I(x, y, λ) represents the incident spectral information of the scene, and T i (λ) represents the spectral transmittance function of the i-th combination of broadband filters, S c (λ) represents the spectral response function of the RGB camera in the c color channel; t q (λ) is the spectral transmittance function of the broadband filter (3)q, and Π(·) represents the multiplication symbol.

6. The spectral imaging method based on an RGB camera and broadband filter encoding according to claim 1, characterized in that, In step 2), the optimization parameters of each broadband filter in the end-to-end optimization framework during the design phase are set as trainable parameters. The optimization parameters of each broadband filter are mapped to the spectral transmittance function of the broadband filter using the following formula: Where T is the spectral transmittance function of each broadband filter; Ω is the optimized parameter of each broadband filter.

7. The spectral imaging method based on an RGB camera and broadband filter encoding according to claim 1, characterized in that, The formula for the total loss function of the end-to-end optimization framework is as follows: L total =αL img +βL filter L img =||I * -I||1 Among them, L total L represents the total loss function value, α and β are the first and second balancing coefficients, respectively. img To reconstruct the spectral image I * The loss function value between the input hyperspectral image I and L filter The loss function value constrains the smoothness of the filter, where k represents the number of spectral channels, o represents the total number of spectral channels, and t... q (k) is the spectral transmittance function of the broadband filter (3)q, where |||||1 represents the 1-norm and || represents the absolute value.

Images