A spectral estimation method and system based on a single-pixel sensor-assisted spectral imaging device

Abstract

This invention discloses a spectral estimation method and system based on a single-pixel sensor-assisted spectral imaging device, belonging to the field of digital image signal processing technology. It integrates an RGB single-pixel sensor with a spectral imaging device through multi-sensor fusion for estimating the light source spectrum of a spectral image. This provides a new method for downstream spectral reflectance calculation and color constancy correction, expands the application scenarios of spectral imaging devices, improves the accuracy of light source spectral estimation, and eliminates the architectural limitation of requiring a white balance color chart for scene capture in spectral imaging devices. This invention uses an RGB single-pixel sensor, enabling rapid and convenient estimation of ambient light sources within the shooting scene, overcoming the limitations of existing spectral imaging devices in light source spectral correction, and expanding the application scenarios of spectral imaging devices.

Description

Technical Field

[0001] This invention relates to the field of digital image signal processing technology, and more specifically to a spectral estimation method and system based on a single-pixel sensor-assisted spectral imaging device. Background Technology

[0002] Currently, with the continuous development of spectral imaging technology, snapshot spectral imaging devices have broad application prospects. However, the imaging quality of spectral image sensors is affected by the illumination source, thus requiring the measurement of the light source in the scene to eliminate color cast. A common method is to use a standard white balance color chart to capture scene light source information. However, in actual shooting scenarios, due to the limitations of the shooting scene, it is difficult to set up the white balance color chart, resulting in a mismatch between the light source parameters and the actual captured scene, thus leading to errors in color calibration and spectral reflectance calculation.

[0003] Therefore, improving the accuracy of light source spectrum estimation is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the present invention provides a spectral estimation method and system based on a single-pixel sensor-assisted spectral imaging device. By fusing RGB single-pixel sensors with spectral imaging devices through multi-sensor fusion, it is used for light source spectral estimation of spectral images, providing a new method for downstream spectral reflectance calculation and color constancy correction, expanding the application scenarios of spectral imaging devices, and improving the accuracy of light source spectral estimation.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A spectral estimation method based on a single-pixel sensor-assisted spectral imaging device includes:

[0007] Step 1: Collect the response values ​​of the white balance color card under different preset correlated color temperature illuminations using an RGB single-pixel sensor and a spectral imaging device, and calculate the mean spectrum of each channel by selecting the grayscale region of the white balance color card to construct a spectral dataset;

[0008] Step 2: Based on the spectral dataset, establish a mapping model using a multilayer perceptron feedforward neural network;

[0009] Step 3: Place the RGB single-pixel sensor next to the imaging spectral device and keep it at the same horizontal line to obtain the spectral image to be processed and the response value of the RGB single-pixel sensor;

[0010] Step 4: Use the mapping model to map the response value of the RGB single pixel sensor to the spectral device space and use it as the light source spectral estimation result; downsample the spectral image to be processed to H×W by max pooling, and use the weighted sum of the spectral angle error and L2 error of the spectral pixel value and the light source spectral estimation result as the similarity index;

[0011] Step 5: Sort the spectral pixel values ​​in ascending order of similarity index, and take the top n% of pixels as the light source spectrum screening result. Then, take the maximum value of each channel in the light source spectrum screening result as the final illumination spectrum estimation result.

[0012] Optionally, step 1 is as follows:

[0013] Step 101: Register a CCT light source with an interval of no more than 500K (optional 300K) and a Duv light source with an interval of 0.005 (optional 0.003), with the value range being -0.005 to +0.005 and between 1800K and 18000K, using a multi-channel adjustable light source lighting box;

[0014] Step 102: Using the white balance color chart as the reference image and the light source selected in step 101 as the illumination source, acquire a spectral image using a spectral imaging device;

[0015] Step 103: Using the white balance color chart as the reference image, the light source selected in step 101 is the illumination source, and the RGB three-channel response is acquired using an RGB single-pixel sensor;

[0016] Step 104: In the spectral image of step 102, the grayscale region of the white balance color card is selected to calculate the mean spectrum of each channel, and normalization is performed by the maximum and minimum values ​​in each channel;

[0017] Step 105: Under the various lighting conditions preset in Step 101, repeat the operations in Steps 102-104 to construct a spectral dataset.

[0018] Optionally, the normalization calculation formula is as follows:

[0019]

[0020] In the formula, S represents the normalized spectral value of the i-th channel. i S represents the spectral value of the i-th channel of the mean spectrum. max With S min These represent the maximum and minimum values ​​of the mean spectrum.

[0021] Optionally, step 2 is as follows:

[0022] Step 201: Take the constructed spectral dataset, randomly select it, and divide it into a dataset, test set, and validation set in a ratio of 8:1:1;

[0023] Step 202: Optimize the n-layer multilayer perceptron feedforward neural network by using the weighted sum of spectral angle error and L2 error as the loss function.

[0024] Optionally, the formula for the loss function is:

[0025]

[0026] L=λ1·L Ang +λ2·L2;

[0027] In the formula, C is the total number of channels in the spectral image, and S and Let S represent the spectral tensor of the spectral image pixels and the spectral tensor of the estimated light source spectrum, respectively. i and λ1 and λ2 represent the spectral values ​​of the i-th channel in the spectral image pixel and the estimated light source spectrum, respectively, with weighting coefficients.

[0028] Optionally, step 4 is as follows:

[0029] Step 401: Input the channel data captured by the RGB single-pixel sensor into the mapping model established by the multilayer perceptron feedforward neural network to obtain the light source spectrum estimation results in the spectral sensor space;

[0030] Step 402: Downsample the acquired spectral image to H×W using max pooling;

[0031] Step 403: Calculate the loss error pixel by pixel between the light source spectrum estimation result and the downsampled image as a similarity index and record it.

[0032] A spectral estimation system based on a single-pixel sensor-assisted spectral imaging device includes:

[0033] The dataset construction module collects the response values ​​of the white balance color card under different preset correlated color temperature illuminations using an RGB single-pixel sensor and a spectral imaging device, and calculates the mean spectrum of each channel by selecting the grayscale region of the white balance color card, thereby constructing a spectral dataset.

[0034] The mapping model building module establishes a mapping model based on a spectral dataset using a multilayer perceptron feedforward neural network.

[0035] The response value calculation module places the RGB single-pixel sensor next to the imaging spectral device and keeps it at the same horizontal line to obtain the spectral image to be processed and the response value of the RGB single-pixel sensor.

[0036] The similarity index calculation module uses the mapping model to map the response value of the RGB single-pixel sensor to the spectral device space and uses it as the light source spectral estimation result; the spectral image to be processed is downsampled to H×W by max pooling, and the weighted sum of the spectral angle error and L2 error of the spectral pixel value and the light source spectral estimation result is used as the similarity index;

[0037] The spectral estimation result output module sorts the spectral pixel values ​​in ascending order according to the similarity index, takes the top n% of pixels as the light source spectrum screening result, and takes the maximum value of each channel in the light source spectrum screening result as the final illumination spectrum estimation result.

[0038] As can be seen from the above technical solution, compared with the prior art, this invention discloses a spectral estimation method and system based on a single-pixel sensor-assisted spectral imaging device. By using an external single-pixel sensor to assist the spectral imaging device in estimating the light source spectrum, it overcomes the architectural limitation of requiring a white balance color chart for scene capture in spectral image devices. In the tasks of solving the reflectance of spectral images and generating spectral images, the illumination source causes color shift in the image captured by the imaging device. However, it is difficult to estimate the light source parameters in the scene using single image data, often requiring a white balance color chart to capture the light source parameters for image correction. Due to different shooting scenes and device architecture limitations, it is often difficult to capture the light source parameters in the scene. This invention uses an RGB single-pixel sensor, which can quickly and conveniently estimate the ambient light source in the shooting scene, overcoming the limitations of existing spectral imaging devices in light source spectrum correction and expanding the application scenarios of spectral imaging devices. Furthermore, it has the following beneficial effects:

[0039] High integration: Existing single-pixel sensors are small in size, light in weight, easy to integrate, and have fast acquisition speed and high stability. In addition, the single-pixel sensor in this invention is mounted on the same side as the spectral imaging device, without the need for additional adjustments in the scene, and has a high degree of integration.

[0040] High reliability: Conventional light source spectrum estimation methods based on white balance color charts are limited by the equipment itself, and scene limitations directly affect the accuracy of the scene's light source spectrum. This invention uses an RGB single-pixel sensor to directly utilize ambient light information for light source spectrum estimation, avoiding errors introduced by the deviation between the white balance color chart and the scene's light source information. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the method flow provided by the present invention;

[0043] Figure 2 A schematic diagram of the distribution of the simulated light source provided by this invention in the CIE 1931 xy chromaticity diagram;

[0044] Figure 3 A graph showing the comparison between the light source spectrum estimation results provided by this invention and the actual light source spectrum;

[0045] Figure 4 This is a schematic diagram of the direct connection of the device provided by the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] This invention discloses a spectral estimation method for a single-pixel sensor-assisted spectral imaging device, such as... Figure 1 Shown, including:

[0048] Step 1: Collect the response values ​​of the white balance color card under different preset correlated color temperature illuminations using an RGB single-pixel sensor and a spectral imaging device, and calculate the mean spectrum of each channel by selecting the grayscale region of the white balance color card to construct a spectral dataset;

[0049] Step 2: Based on the spectral dataset, establish a mapping model using a multilayer perceptron feedforward neural network;

[0050] Step 3: Place the RGB single-pixel sensor next to the imaging spectral device and keep it at the same horizontal line to obtain the spectral image to be processed and the response value of the RGB single-pixel sensor;

[0051] Step 4: Use the mapping model to map the response value of the RGB single pixel sensor to the spectral device space and use it as the light source spectral estimation result; downsample the spectral image to be processed to H×W by max pooling, and use the weighted sum of the spectral angle error and L2 error of the spectral pixel value and the light source spectral estimation result as the similarity index;

[0052] Step 5: Sort the spectral pixel values ​​in ascending order of similarity index, and take the top n% of pixels as the light source spectrum screening results. Then, take the maximum value of each channel in the light source spectrum screening results as the final illumination spectrum estimation result.

[0053] In a specific embodiment, step 1 is as follows:

[0054] Step 101: Register a light source with CCT at 500K intervals and Duv at 0.005 intervals, with values ​​ranging from -0.005 to +0.005, and between 1800K and 18000K, using a multi-channel adjustable light source lighting box;

[0055] Step 102: Using the white balance color chart as the reference image and the light source selected in step 101 as the illumination source, acquire a spectral image using a spectral imaging device;

[0056] Step 103: Using the white balance color chart as the reference image, the light source selected in step 101 is the illumination source, and the RGB three-channel response is acquired using an RGB single-pixel sensor;

[0057] Step 104: In the spectral image of step 102, the grayscale region of the white balance color card is selected to calculate the mean spectrum of each channel, and normalization is performed by the maximum and minimum values ​​in each channel;

[0058] Step 105: Under the various lighting conditions preset in Step 101, repeat the operations in Steps 102-104 to construct a spectral dataset.

[0059] In a specific embodiment, the normalization calculation formula is as follows:

[0060]

[0061] In the formula, S represents the normalized spectral value of the i-th channel. i S represents the spectral value of the i-th channel of the mean spectrum. max With S min These represent the maximum and minimum values ​​of the mean spectrum.

[0062] In a specific embodiment, step 2 is as follows:

[0063] Step 201: Take the constructed spectral dataset, randomly select it, and divide it into a dataset, test set, and validation set in a ratio of 8:1:1;

[0064] Step 202: Optimize the n-layer multilayer perceptron feedforward neural network by using the weighted sum of spectral angle error and L2 error as the loss function.

[0065] In a specific embodiment, the formula for the loss function is:

[0066]

[0067] L=λ1·L Ang +λ2·L2;

[0068] In the formula, C is the total number of channels in the spectral image, and S and Let S represent the spectral tensor of the spectral image pixels and the spectral tensor of the estimated light source spectrum, respectively. i and λ1 and λ2 represent the spectral values ​​of the i-th channel in the spectral image pixel and the estimated light source spectrum, respectively, with weighting coefficients.

[0069] In one specific embodiment, step 4 is as follows:

[0070] Step 401: Input the channel data captured by the RGB single-pixel sensor into the mapping model established by the multilayer perceptron feedforward neural network to obtain the light source spectrum estimation results in the spectral sensor space;

[0071] Step 402: Downsample the acquired spectral image to H×W using max pooling;

[0072] Step 403: Calculate the loss error pixel by pixel between the light source spectrum estimation result and the downsampled image as a similarity index and record it.

[0073] The following specific example further illustrates the method of the present invention.

[0074] This example uses a hyperspectral image sensor module to demonstrate a method for spectral estimation in an RGB single-pixel sensor-assisted spectral imaging device. The hyperspectral image sensor module has a resolution of 512×512, a spectral range of 420-720nm, and a spectral resolution of 3nm.

[0075] Specifically, the following steps are included:

[0076] Configure preset light sources in the light source simulation box, such as... Figure 2 The distribution of simulated light sources on the CIE 1931 xy chromaticity diagram is shown. The light source samples are spaced at 500K intervals, and the Duv values ​​are spaced at 0.005 intervals, with light sources ranging from -0.005 to +0.005 and from 2300K to 10000K.

[0077] Training set samples are acquired under a preset light source. Using the preset light source as the illumination source, a white balance color chart is placed in the scene. The hyperspectral image sensor module is perpendicular to the plane of the color chart, and the RGB single-pixel sensor is placed parallel to the hyperspectral image sensor module on the same horizontal plane and exposed simultaneously.

[0078] The mean value of each channel is calculated from the grayscale region of the captured hyperspectral image, and the spectrum is normalized to its maximum and minimum values. Using the RGB single-pixel sensor response value as input and the normalized spectrum as output, an 8-layer MLP feedforward neural network is trained. The weighted sum of angle error and L2 error is used as the loss error for optimization, and the mapping model is obtained through training.

[0079] A hyperspectral image to be processed is captured using a hyperspectral image sensor, and the response values ​​of RGB single-pixel sensors are obtained simultaneously. The RGB values ​​are then mapped to the hyperspectral sensor space using a trained mapping model, which serves as the result of the light source spectrum estimation.

[0080] The image is downsampled to 128×128 sub-images using max pooling with a 4×4 convolution kernel. The loss error is calculated as a similarity index based on the pixel spectrum and the light source spectrum estimation results, and recorded. In this example, λ1 = 0.66 and λ2 = 0.33.

[0081]

[0082] L=λ1·LAng+λ2·L2

[0083] The 128×128 pixels are sorted in ascending order of similarity index, and the top n% of pixels are used as the light source spectrum selection results. In this example, n=3.5, and the maximum value of each channel is used as the final illumination spectrum estimate.

[0084] like Figure 3 The figure shows a comparison between the light source spectral estimation results of this invention and the actual light source spectrum. As can be seen from the figure, the light source spectral estimation method proposed in this invention can accurately estimate the light source parameters at the time of image capture, thereby assisting in solving for the spectral reflectance image of an object and the color constancy performance of RGB images generated based on hyperspectral images.

[0085] The connection relationships between devices are as follows Figure 4 As shown, the light source is a standard light source box; the registered lighting source is a CCT light source with a 300K interval between 1800K and 18000K.

[0086] Single-pixel sensor: RGB three-channel single-pixel sensor; measures spectral images, acquires ambient light parameters, and filters spectral features. Spectral imaging equipment: used to acquire raw spectral images of a scene.

[0087] A spectral estimation system based on a single-pixel sensor-assisted spectral imaging device includes:

[0088] The dataset construction module collects the response values ​​of the white balance color card under different preset correlated color temperature illuminations using an RGB single-pixel sensor and a spectral imaging device, and calculates the mean spectrum of each channel by selecting the grayscale region of the white balance color card, thereby constructing a spectral dataset.

[0089] The mapping model building module establishes a mapping model based on a spectral dataset using a multilayer perceptron feedforward neural network.

[0090] The response value calculation module places the RGB single-pixel sensor next to the imaging spectral device and keeps it at the same horizontal line to obtain the spectral image to be processed and the response value of the RGB single-pixel sensor.

[0091] The similarity index calculation module uses a mapping model to map the response value of the RGB single-pixel sensor to the spectral device space and uses it as the light source spectral estimation result; the spectral image to be processed is downsampled to H×W by max pooling, and the weighted sum of the spectral angle error and L2 error of the spectral pixel value and the light source spectral estimation result is used as the similarity index;

[0092] The spectral estimation result output module sorts the spectral pixel values ​​in ascending order according to the similarity index, and uses the top n% of pixels as the light source spectrum screening result. The maximum value of each channel in the light source spectrum screening result is then used as the final illumination spectrum estimation result.

[0093] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0094] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A spectral estimation method for a single-pixel sensor-assisted spectral imaging device, characterized in that, include: Step 1: Collect the response values ​​of the white balance color card under different preset correlated color temperature illuminations using an RGB single-pixel sensor and a spectral imaging device, and calculate the mean spectrum of each channel by selecting the grayscale region of the white balance color card to construct a spectral dataset; Step 2: Based on the spectral dataset, establish a mapping model using a multilayer perceptron feedforward neural network; Step 3: Place the RGB single-pixel sensor next to the imaging spectroscopy device and keep it at the same horizontal line to obtain the spectral image to be processed and the response value of the RGB single-pixel sensor; Step 4: Use the mapping model to map the response value of the RGB single-pixel sensor to the spectral device space, and use it as the light source spectrum estimation result; The spectral image to be processed is downsampled to H×W by max pooling, and the weighted sum of the spectral pixel values ​​and the spectral angle error and L2 error of the light source spectrum estimation results is used as the similarity index. Step 5: Sort the spectral pixel values ​​in ascending order of similarity index, and take the top n% of pixels as the light source spectrum screening result. Then, take the maximum value of each channel in the light source spectrum screening result as the final illumination spectrum estimation result.

2. The spectral estimation method for a single-pixel sensor-assisted spectral imaging device according to claim 1, characterized in that, Step 1 is described in detail as follows: Step 101: Register a CCT light source with an interval of no more than 500K and a Duv light source with an interval of 0.005, with a value range of -0.005 to +0.005 and a value range of 1800K to 18000K using a multi-channel adjustable light source lighting box; Step 102: Using the white balance color chart as the reference image and the light source selected in step 101 as the illumination source, acquire a spectral image using a spectral imaging device; Step 103: Using the white balance color chart as the reference image, the light source selected in step 101 is the illumination source, and the RGB three-channel response is acquired using an RGB single-pixel sensor; Step 104: In the spectral image of step 102, the grayscale region of the white balance color card is selected to calculate the mean spectrum of each channel, and normalization is performed by the maximum and minimum values ​​in each channel; Step 105: Under the various lighting conditions preset in Step 101, repeat the operations in Steps 102-104 to construct a spectral dataset.

3. The spectral estimation method for a single-pixel sensor-assisted spectral imaging device according to claim 2, characterized in that, The normalization calculation formula is as follows: In the formula, S represents the normalized spectral value of the i-th channel. i S represents the spectral value of the i-th channel of the mean spectrum. max With S min These represent the maximum and minimum values ​​of the mean spectrum.

4. The spectral estimation method for a single-pixel sensor-assisted spectral imaging device according to claim 1, characterized in that, Step 2 is described in detail below: Step 201: Take the constructed spectral dataset, randomly select it, and divide it into a dataset, test set, and validation set in a ratio of 8:1:1; Step 202: Optimize the n-layer multilayer perceptron feedforward neural network by using the weighted sum of spectral angle error and L2 error as the loss function.

5. The spectral estimation method for a single-pixel sensor-assisted spectral imaging device according to claim 4, characterized in that, The formula for the loss function is: L=λ1·L Ang +λ2·L2; In the formula, C is the total number of channels in the spectral image, and S and Let S represent the spectral tensor of the spectral image pixels and the spectral tensor of the estimated light source spectrum, respectively. i and λ1 and λ2 represent the spectral values ​​of the i-th channel in the spectral image pixel and the estimated light source spectrum, respectively, with weighting coefficients.

6. The spectral estimation method for a single-pixel sensor-assisted spectral imaging device according to claim 1, characterized in that, Step 4 is described in detail below: Step 401: Input the channel data captured by the RGB single-pixel sensor into the mapping model established by the multilayer perceptron feedforward neural network to obtain the light source spectrum estimation results in the spectral sensor space; Step 402: Downsample the acquired spectral image to H×W using max pooling; Step 403: Calculate the loss error pixel by pixel between the light source spectrum estimation result and the downsampled image as a similarity index and record it.

7. A spectral estimation system based on a single-pixel sensor-assisted spectral imaging device, characterized in that, The spectral estimation method based on a single-pixel sensor-assisted spectral imaging device according to any one of claims 1-6 includes: The dataset construction module collects the response values ​​of the white balance color card under different preset correlated color temperature illuminations using an RGB single-pixel sensor and a spectral imaging device, and calculates the mean spectrum of each channel by selecting the grayscale region of the white balance color card, thereby constructing a spectral dataset. The mapping model building module establishes a mapping model based on a spectral dataset using a multilayer perceptron feedforward neural network. The response value calculation module places the RGB single-pixel sensor next to the imaging spectral device and keeps it at the same horizontal line to obtain the spectral image to be processed and the response value of the RGB single-pixel sensor. The similarity index calculation module uses the mapping model to map the response value of the RGB single-pixel sensor to the spectral device space and uses it as the light source spectral estimation result; the spectral image to be processed is downsampled to H×W by max pooling, and the weighted sum of the spectral angle error and L2 error of the spectral pixel value and the light source spectral estimation result is used as the similarity index; The spectral estimation result output module sorts the spectral pixel values ​​in ascending order according to the similarity index, takes the top n% of pixels as the light source spectrum screening result, and takes the maximum value of each channel in the light source spectrum screening result as the final illumination spectrum estimation result.

Images