Spectrum estimation method and system based on single-pixel sensor auxiliary spectral imaging equipment

Abstract

The invention discloses a spectrum estimation method and system based on a single-pixel sensor auxiliary spectrum imaging device, relates to the technical field of digital image signal processing, and fuses an RGB single-pixel sensor and a spectrum imaging device through multi-sensor fusion for light source spectrum estimation of a spectrum image. A new method is provided for downstream spectral reflectivity calculation and color constancy correction, the application scene of spectral imaging equipment is expanded, the accuracy of light source spectrum estimation is improved, and the framework limitation that a white balance color card needs to be erected for scene capture of the spectral imaging equipment is eliminated. According to the invention, the RGB single-pixel sensor is used, the ambient light source in the shooting scene can be estimated quickly and conveniently, the limitation of light source spectrum correction of the existing spectral imaging equipment is eliminated, and the application scene of the spectral imaging equipment is expanded.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image signal processing, and more particularly to a spectrum estimation method and system based on a single-pixel sensor-assisted spectrum imaging device. Background Art

[0002] At present, with the continuous development of spectral imaging equipment technology, snapshot spectral imaging equipment has broad application prospects. However, the imaging quality of spectral image sensors is affected by the lighting source, so it is necessary to measure the light source in the scene to eliminate color deviation. The commonly used method uses a standard white balance color card to capture the scene light source information. However, in actual shooting scenes, due to the limitations of the shooting scene, the white balance color card is difficult to set up, resulting in the mismatch between the light source parameters and the actual captured scene, which leads to errors in color calibration and spectral reflectance calculation.

[0003] Therefore, how to improve the accuracy of light source spectrum estimation is a problem that technical personnel in this field need to solve urgently. Summary of the invention

[0004] In view of this, the present invention provides a spectral estimation method and system based on a single-pixel sensor-assisted spectral imaging device, which fuses the RGB single-pixel sensor with the spectral imaging device through multi-sensor fusion and is used for light source spectrum estimation of spectral images. It provides a new method for downstream spectral reflectance calculation and color constancy correction, expands the application scenarios of spectral imaging devices, and improves the accuracy of light source spectrum estimation.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A spectrum estimation method based on a single-pixel sensor-assisted spectrum imaging device, comprising:

[0007] Step 1: Use the RGB single-pixel sensor and spectral imaging device to collect the response values ​​of the white balance color card under preset different correlated color temperature illumination, and select the grayscale area of ​​the white balance color card to calculate the mean spectrum of each channel to construct a spectral data set;

[0008] Step 2: Based on the spectral data set, a mapping model is established through a multi-layer perceptron feedforward neural network;

[0009] Step 3: Place the RGB single pixel sensor next to the imaging spectrum device and keep them at the same horizontal line to obtain the spectral image to be processed and the response value of the RGB single pixel sensor;

[0010] Step 4: Use the mapping model to map the response value of the RGB single-pixel sensor to the spectral device space, and use it as the light source spectrum estimation result; downsample the spectral image to be processed to H×W through maximum pooling, and use the weighted sum of the spectral angle error and L2 error of the spectral pixel value and the light source spectrum estimation result as the similarity index;

[0011] Step 5: Arrange the spectral pixel values ​​in ascending order according to the similarity index, take the first n% of pixels as the light source spectrum screening result, and take the maximum value of each channel in the light source spectrum screening result as the final illumination spectrum estimation result.

[0012] Optionally, the step 1 is specifically as follows:

[0013] Step 101: Use a multi-channel adjustable light source lighting box to align CCT with an interval of no more than 500K (optionally 300K), Duv with an interval of 0.005 (optionally 0.003), a value range of -0.005 to +0.005, and a light source between 1800K and 18000K;

[0014] Step 102: using the white balance color card as a reference image, using the light source selected in step 101 as an illumination light source, and using a spectral imaging device to collect a spectral image;

[0015] Step 103: using the white balance color card as a reference image, the light source selected in step 101 as an illumination light source, and using an RGB single pixel sensor to collect RGB three-channel responses;

[0016] Step 104: the spectral image in step 102 selects the grayscale area of ​​the white balance color card to calculate the mean spectrum of each channel, and normalizes it by the maximum and minimum values ​​in each channel;

[0017] Step 105: Repeat the operations in steps 102 to 104 under the various lighting conditions preset in step 101 to construct a spectral data set.

[0018] Optionally, the normalized calculation formula is:

[0019]

[0020] In the formula, is the normalized spectral value of the ith channel, S i is the spectral value of the i-th channel of the mean spectrum, S max With S min are the maximum and minimum values ​​of the mean spectrum.

[0021] Optionally, the step 2 is specifically as follows:

[0022] Step 201: Take the constructed spectral data set, randomly select, and divide it into a data set, a test set, and a validation set in a ratio of 8:1:1;

[0023] Step 202: Optimize the n-layer multilayer perceptron feedforward neural network using the weighted sum of the spectral angle error and the L2 error as the loss function.

[0024] Optionally, the formula of the loss function is:

[0025]

[0026] L=λ1·L Ang +λ2·L2;

[0027] Where C is the total number of channels of the spectral image, S and They represent the spectral tensor of the spectral image pixel and the spectral tensor of the estimated light source spectrum, S i and They are respectively represented as the spectral image pixels and the spectral values ​​of the i-th channel in the estimated light source spectrum, and λ1 and λ2 are weighting coefficients.

[0028] Optionally, step 4 is specifically as follows:

[0029] Step 401: inputting the channel data captured by the RGB single pixel sensor into a mapping model established by a multi-layer perceptron feed-forward neural network to obtain a light source spectrum estimation result in the spectral sensor space;

[0030] Step 402: downsampling the collected spectral image to H×W through maximum pooling;

[0031] Step 403: Calculate the loss error pixel by pixel between the light source spectrum estimation result and the image obtained by downsampling as a similarity index and record it.

[0032] A spectral estimation system based on a single-pixel sensor-assisted spectral imaging device, comprising:

[0033] The dataset construction module collects the response values ​​of the white balance color card under preset different correlated color temperature illuminations through RGB single-pixel sensors and spectral imaging devices, and selects the grayscale area of ​​the white balance color card to calculate the mean spectrum of each channel, thereby constructing a spectral dataset;

[0034] The mapping model building module builds the mapping model based on the spectral data set through a multi-layer perceptron feedforward neural network;

[0035] The response value calculation module places the RGB single pixel sensor next to the imaging spectrum device and keeps it at the same horizontal line to obtain the spectral image to be processed and the response value of the RGB single pixel sensor;

[0036] The similarity index calculation module maps the response value of the RGB single pixel sensor to the spectral device space using the mapping model, and uses it as the light source spectrum estimation result; downsamples the spectral image to be processed to H×W through maximum pooling, and uses the weighted sum of the spectral angle error and the L2 error of the spectral pixel value and the light source spectrum estimation result as the similarity index;

[0037] The spectrum estimation result output module arranges the spectrum pixel values ​​in ascending order according to the similarity index, takes the first n% of pixels as the light source spectrum screening result, and takes the maximum value of each channel in the light source spectrum screening result as the final illumination spectrum estimation result.

[0038] It can be known from the above technical solutions that, compared with the prior art, the present invention discloses a spectral estimation method and system based on a single-pixel sensor-assisted spectral imaging device, which uses an external single-pixel sensor to assist the spectral imaging device in estimating the light source spectrum, thus getting rid of the architectural limitation that the spectral imaging device needs to set up a white balance color card for scene capture. In the reflectance solution of the spectral image and the image generation task of the spectral image, the illumination light source will cause the image captured by the imaging device to produce color deviation. However, it is difficult to estimate the light source parameters in the scene based on a single image data, and it is often necessary to set up a white balance color card to capture the light source parameters in the scene to correct the image. Due to different shooting scenes and equipment architecture limitations, it is often difficult to capture the light source parameters in the scene. The present invention uses an RGB single-pixel sensor, which can quickly and conveniently estimate the ambient light source in the shooting scene, getting rid of the limitations of the existing light source spectrum correction of spectral imaging equipment, and expanding the application scenarios of spectral imaging equipment. And it has the following beneficial effects:

[0039] High integration: the existing single-pixel sensor is small in size, light in weight, easy to integrate, and has a fast acquisition speed and high stability. The single-pixel sensor in the present invention is installed on the same side as the spectral imaging device and does not require additional adjustment in the scene, so it has a high degree of integration.

[0040] High reliability: Conventional light source spectrum estimation methods based on white balance color cards are limited by the equipment itself, and the limitations of the scene directly affect the accuracy of the scene light source spectrum. The present invention uses an RGB single-pixel sensor to directly use ambient light information for light source spectrum estimation, avoiding the error introduced by the deviation of the white balance color card and the scene for light source information. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0042] Figure 1 A schematic diagram of the method provided by the present invention;

[0043] Figure 2 A schematic diagram showing the distribution of the simulated light source provided by the present invention in the CIE 1931xy chromaticity diagram;

[0044] Figure 3 A curve diagram showing the comparison between the light source spectrum estimation result and the real light source spectrum provided by the present invention;

[0045] Figure 4 This is a schematic diagram of direct connection of the equipment provided by the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The embodiment of the present invention discloses a spectrum estimation method based on a single pixel sensor assisted spectrum imaging device, such as Figure 1 As shown, including:

[0048] Step 1: Use the RGB single-pixel sensor and spectral imaging device to collect the response values ​​of the white balance color card under preset different correlated color temperature illumination, and select the grayscale area of ​​the white balance color card to calculate the mean spectrum of each channel to construct a spectral data set;

[0049] Step 2: Based on the spectral data set, a mapping model is established through a multi-layer perceptron feedforward neural network;

[0050] Step 3: Place the RGB single pixel sensor next to the imaging spectrum device and keep them at the same horizontal line to obtain the spectral image to be processed and the response value of the RGB single pixel sensor;

[0051] Step 4: Use the mapping model to map the response value of the RGB single-pixel sensor to the spectral device space and use it as the light source spectrum estimation result; downsample the spectral image to be processed to H×W through maximum pooling, and use the weighted sum of the spectral angle error and L2 error of the spectral pixel value and the light source spectrum estimation result as the similarity index;

[0052] Step 5: Arrange the spectral pixel values ​​in ascending order according to the similarity index, and use the first n% of pixels as the light source spectrum screening result, and use the maximum value of each channel in the light source spectrum screening result as the final illumination spectrum estimation result.

[0053] In a specific embodiment, step 1 is as follows:

[0054] Step 101: Use a multi-channel adjustable light source lighting box to align light sources with CCT at intervals of 500K, Duv at intervals of 0.005, a value range of -0.005 to +0.005, and 1800K to 18000K;

[0055] Step 102: using the white balance color card as a reference image, using the light source selected in step 101 as an illumination light source, and using a spectral imaging device to collect a spectral image;

[0056] Step 103: using the white balance color card as a reference image, the light source selected in step 101 as an illumination light source, and using an RGB single pixel sensor to collect RGB three-channel responses;

[0057] Step 104: the spectral image in step 102 selects the grayscale area of ​​the white balance color card to calculate the mean spectrum of each channel, and normalizes it by the maximum and minimum values ​​in each channel;

[0058] Step 105: Repeat the operations in steps 102 to 104 under the various lighting conditions preset in step 101 to construct a spectral data set.

[0059] In a specific embodiment, the normalized calculation formula is:

[0060]

[0061] In the formula, is the normalized spectral value of the ith channel, S i is the spectral value of the i-th channel of the mean spectrum, S max With S min are the maximum and minimum values ​​of the mean spectrum.

[0062] In a specific embodiment, step 2 is as follows:

[0063] Step 201: Take the constructed spectral data set, randomly select, and divide it into a data set, a test set, and a validation set in a ratio of 8:1:1;

[0064] Step 202: Optimize the n-layer multilayer perceptron feedforward neural network using the weighted sum of the spectral angle error and the L2 error as the loss function.

[0065] In a specific embodiment, the formula of the loss function is:

[0066]

[0067] L=λ1·L Ang +λ2·L2;

[0068] Where C is the total number of channels of the spectral image, S and They represent the spectral tensor of the spectral image pixel and the spectral tensor of the estimated light source spectrum, S i and They are respectively represented as the spectral image pixels and the spectral values ​​of the i-th channel in the estimated light source spectrum, and λ1 and λ2 are weighting coefficients.

[0069] In a specific embodiment, step 4 is as follows:

[0070] Step 401: inputting the channel data captured by the RGB single pixel sensor into a mapping model established by a multi-layer perceptron feed-forward neural network to obtain a light source spectrum estimation result in the spectral sensor space;

[0071] Step 402: downsampling the collected spectral image to H×W through maximum pooling;

[0072] Step 403: Calculate the loss error pixel by pixel between the light source spectrum estimation result and the image obtained by downsampling as a similarity index and record it.

[0073] A specific example is introduced below to further illustrate the method of the present invention.

[0074] This example takes a hyperspectral image sensor module as an example, and uses a parameter RGB single pixel sensor to assist the spectral estimation method of the spectral imaging device. The hyperspectral image sensor module has a resolution of 512×512, a spectral range of 420-720nm, and a spectral resolution of 3nm.

[0075] The specific steps include:

[0076] Configure the preset light source in the light source simulation box, such as Figure 2 The distribution of simulated light sources on the CIE 1931xy chromaticity diagram is shown. The light source samples are spaced at 500K, Duv is spaced at 0.005, and the value range is -0.005 to +0.005. The light source is between 2300K and 10000K.

[0077] The training set samples are obtained under the preset light source. The preset light source is used as the illumination source, a white balance color card is placed in the scene, the hyperspectral image sensor module is perpendicular to the color card plane, and the RGB single pixel sensor is placed parallel to the same horizontal plane of the hyperspectral image sensor module and exposed at the same time.

[0078] The grayscale area of ​​the captured hyperspectral image is selected to obtain the mean of each channel, and the spectrum is normalized to the maximum and minimum values. The RGB single-pixel sensor response value is used as input, and the normalized spectrum is used as output to train an 8-layer MLP feedforward neural network. The weighted sum of the angle error and the L2 error is used as the loss error, and the optimization is performed to train and obtain the mapping model.

[0079] A hyperspectral image sensor is used to capture the hyperspectral image to be processed, and the response value of the RGB single-pixel sensor is obtained at the same time. The trained mapping model is used to map the RGB value to the hyperspectral sensor space as the light source spectrum estimation result.

[0080] Use a 4×4 convolution kernel to perform maximum pooling on the image and downsample it to a 128×128 sub-image. Calculate the loss error between the pixel spectrum and the light source spectrum estimation result as the similarity index and record it. In this example, λ1=0.66,λ2=0.33:

[0081]

[0082] L=λ1·LAng+λ2·L2

[0083] The 128×128 pixels are arranged in ascending order according to the similarity index, and the first n% of the pixels are used as the light source spectrum screening result. In this example, n=3.5, and the maximum value of each channel is used as the final illumination spectrum estimation result.

[0084] like Figure 3 The comparison results of the light source spectrum estimation results of the present invention and the real light source spectrum are shown. It can be seen from the figure that the light source spectrum estimation method proposed in the present invention can better estimate the light source parameters when the image is captured, thereby assisting in solving the object spectral reflectance image and the color constancy performance of the RGB image generated based on the hyperspectral image.

[0085] The connection relationship between devices is as follows Figure 4 As shown, light source: standard light box; registration lighting source CC T with 300K as interval between 1800K-18000K.

[0086] Single pixel sensor: RGB three-channel single pixel sensor; measures spectral image acquisition ambient light parameters to filter spectral features. Spectral imaging device: used to acquire the original spectral image in the scene.

[0087] A spectral estimation system based on a single-pixel sensor-assisted spectral imaging device, comprising:

[0088] The data set construction module collects the response values ​​of the white balance color card under preset different correlated color temperature illuminations through the RGB single pixel sensor and the spectral imaging device, and selects the grayscale area of ​​the white balance color card to calculate the mean spectrum of each channel, thereby constructing a spectral data set;

[0089] The mapping model building module builds the mapping model based on the spectral data set through a multi-layer perceptron feedforward neural network;

[0090] The response value calculation module places the RGB single pixel sensor next to the imaging spectrum device and keeps it at the same horizontal line to obtain the spectral image to be processed and the response value of the RGB single pixel sensor;

[0091] The similarity index calculation module uses a mapping model to map the response value of the RGB single-pixel sensor to the spectral device space and uses it as the light source spectrum estimation result; the spectral image to be processed is downsampled to H×W through maximum pooling, and the weighted sum of the spectral angle error and L2 error of the spectral pixel value and the light source spectrum estimation result is used as the similarity index;

[0092] The spectrum estimation result output module arranges the spectrum pixel values ​​in ascending order according to the similarity index, takes the first n% of pixels as the light source spectrum screening result, and takes the maximum value of each channel in the light source spectrum screening result as the final illumination spectrum estimation result.

[0093] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0094] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A spectral estimation method based on a single-pixel sensor-assisted spectral imaging device, characterized in that: include: Step 1: Use the RGB single-pixel sensor and spectral imaging device to collect the response values ​​of the white balance color card under preset different correlated color temperature illumination, and select the grayscale area of ​​the white balance color card to calculate the mean spectrum of each channel to construct a spectral data set; Step 2: Based on the spectral data set, a mapping model is established through a multi-layer perceptron feedforward neural network; Step 3: Place the RGB single pixel sensor next to the imaging spectrum device and keep them at the same horizontal line to obtain the spectral image to be processed and the response value of the RGB single pixel sensor; Step 4: Map the response value of the RGB single pixel sensor to the spectral device space using the mapping model, and use it as the light source spectrum estimation result; The spectral image to be processed is downsampled to H×W by maximum pooling, and the weighted sum of the spectral angle error and the L2 error of the spectral pixel value and the light source spectrum estimation result is used as the similarity index; Step 5: Arrange the spectral pixel values ​​in ascending order according to the similarity index, take the first n% of pixels as the light source spectrum screening result, and take the maximum value of each channel in the light source spectrum screening result as the final illumination spectrum estimation result.

2. The spectral estimation method based on a single-pixel sensor-assisted spectral imaging device according to claim 1, characterized in that: The step 1 is specifically as follows: Step 101: Use a multi-channel adjustable light source lighting box to align a light source with a CCT interval of no more than 500K, a Duv interval of 0.005, a value range of -0.005 to +0.005, and a light source between 1800K and 18000K; Step 102: using the white balance color card as a reference image, using the light source selected in step 101 as an illumination light source, and using a spectral imaging device to collect a spectral image; Step 103: using the white balance color card as a reference image, the light source selected in step 101 as an illumination light source, and using an RGB single pixel sensor to collect RGB three-channel responses; Step 104: the spectral image in step 102 selects the grayscale area of ​​the white balance color card to calculate the mean spectrum of each channel, and normalizes it by the maximum and minimum values ​​in each channel; Step 105: Repeat the operations in steps 102 to 104 under the various lighting conditions preset in step 101 to construct a spectral data set.

3. The spectral estimation method based on a single-pixel sensor-assisted spectral imaging device according to claim 2, characterized in that: The normalized calculation formula is: In the formula, is the normalized spectral value of the ith channel, S i is the spectral value of the i-th channel of the mean spectrum, S max With S min are the maximum and minimum values ​​of the mean spectrum.

4. The spectral estimation method based on a single-pixel sensor-assisted spectral imaging device according to claim 1, characterized in that: The step 2 is specifically as follows: Step 201: Take the constructed spectral data set, randomly select, and divide it into a data set, a test set, and a validation set in a ratio of 8:1:1; Step 202: Optimize the n-layer multilayer perceptron feedforward neural network using the weighted sum of the spectral angle error and the L2 error as the loss function.

5. The spectral estimation method based on a single-pixel sensor-assisted spectral imaging device according to claim 4, characterized in that: The formula of the loss function is: L=λ1·L Ang +λ2·L2; Where C is the total number of channels of the spectral image, S and They represent the spectral tensor of the spectral image pixel and the spectral tensor of the estimated light source spectrum, S i and They are respectively represented as the spectral image pixels and the spectral values ​​of the i-th channel in the estimated light source spectrum, and λ1 and λ2 are weighting coefficients.

6. The spectral estimation method based on a single-pixel sensor-assisted spectral imaging device according to claim 1, characterized in that: The step 4 is specifically as follows: Step 401: inputting the channel data captured by the RGB single pixel sensor into a mapping model established by a multi-layer perceptron feed-forward neural network to obtain a light source spectrum estimation result in the spectral sensor space; Step 402: downsampling the collected spectral image to H×W through maximum pooling; Step 403: Calculate the loss error pixel by pixel between the light source spectrum estimation result and the image obtained by downsampling as a similarity index and record it.

7. A spectral estimation system based on a single-pixel sensor-assisted spectral imaging device, characterized in that: A spectral estimation method based on a single-pixel sensor-assisted spectral imaging device according to any one of claims 1 to 6, comprising: The data set construction module collects the response values ​​of the white balance color card under preset different correlated color temperature illuminations through the RGB single pixel sensor and the spectral imaging device, and selects the grayscale area of ​​the white balance color card to calculate the mean spectrum of each channel, thereby constructing a spectral data set; The mapping model building module builds the mapping model based on the spectral data set through a multi-layer perceptron feedforward neural network; The response value calculation module places the RGB single pixel sensor next to the imaging spectrum device and keeps it at the same horizontal line to obtain the spectral image to be processed and the response value of the RGB single pixel sensor; The similarity index calculation module maps the response value of the RGB single pixel sensor to the spectral device space using the mapping model, and uses it as the light source spectrum estimation result; downsamples the spectral image to be processed to H×W through maximum pooling, and uses the weighted sum of the spectral angle error and the L2 error of the spectral pixel value and the light source spectrum estimation result as the similarity index; The spectrum estimation result output module arranges the spectrum pixel values ​​in ascending order according to the similarity index, takes the first n% of pixels as the light source spectrum screening result, and takes the maximum value of each channel in the light source spectrum screening result as the final illumination spectrum estimation result.

Images