A spectrum reconstruction method with customized filter channel combination

Abstract

This paper proposes a spectral reconstruction method with customized filter channel combination, aiming to reduce data redundancy in the reconstruction process. This method is achieved by minimizing two objective functions: one objective function is the difference between the red, green and blue components of the scene and the filter channel in the sRGB color space; the other function is the correlation between the filter channels; secondly, the customized filter channel is integrated with the camera to capture the scene image with high signal-to-noise ratio, and the deep learning algorithm is used for hyperspectral reconstruction. Compared with the traditional RGB channel, our optimization method can reduce the root mean square error and mean relative absolute error in the reconstruction result, and improve the peak signal-to-noise ratio, thereby enhancing the accuracy of spectral reconstruction.

Description

Technical Field

[0001] The present invention belongs to the field of spectral imaging, and mainly relates to spectral imaging technology and spectral image multi-band fusion processing technology. Background Art

[0002] Reconstructing hyperspectral data from RGB cameras has been proven to be an efficient strategy for spectral data acquisition. Many scholars have explored the use of compressed sensing technology to reconstruct hyperspectral images [Opt. Express, 7, 24, 2016]. However, compressed sensing technology has limitations in the accurate fitting and generalization capabilities of the objective function. Therefore, deep learning-based RGB to hyperspectral image (RGB-HSI) reconstruction methods have emerged, providing a revolutionary solution [PhotonicsRes., 2, 9, 2021]. Deep learning methods can extract general features from large amounts of data, achieving real-time processing capabilities, strong robustness and high accuracy. Galliani et al. first used convolutional neural networks (CNNs) for RGB-HSI reconstruction [in CORR, 09470, 2017]. Since then, the Image Restoration and Enhancement Challenge (NTIRE) has greatly stimulated the enthusiasm of researchers in the field of deep learning, prompting them to design and implement network architectures such as hierarchical regression networks and attention mechanisms, and are committed to the reconstruction of spectral data [in CVPR, 422, 2020].

[0003] Subsequently, the focus of research further shifted to optimizing the camera spectral response (CSR) to improve reconstruction performance. Arad et al. emphasized the key role of CSR in RGB-HSI reconstruction and confirmed that differences in CSR can lead to up to 33% variation in reconstruction accuracy [in ICCV, 3153, 2017]. Based on this finding, researchers represented by Yang and Fu proposed a variety of improved convolutional networks [in ICIP, 2933, 2021; in ECCV, 788, 2018], aiming to extract the optimal CSR curve from existing RGB cameras. Nie et al. further promoted the development of this field [in CVPR, 4767, 2018], who extended the search space to an infinite non-negative function space and applied non-negative 1×1 convolution to design arbitrary CSR curves. However, an obvious disadvantage of this approach is that arbitrarily designed CSR curves may be difficult to process.

[0004] In summary, existing spectral reconstruction studies are mostly limited to the use of RGB cameras, where the RGB channels are designed to mimic human visual perception and are not necessarily the optimal channels for spectral reconstruction. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to improve the accuracy of spectral reconstruction in different scenarios and customize a spectral reconstruction method with an optimal filter channel combination according to the scenario. The present invention provides a spectral reconstruction method with customized filter channel combination.

[0006] The present invention provides a spectrum reconstruction method for customized filter channel combination, comprising the following steps:

[0007] Step 1: Obtain scene images by photographing the scene with a smartphone, and obtain the scene images through clustering. n color groups and calculate the sRGB value of each color group;

[0008] Step 2. Preset a visible light band, and randomly generate a plurality of filter chip transmittance curves with equally spaced central wavelengths within the visible light band;

[0009] Step 3. Construct a filter channel combination, wherein the filter channel combination has three filter channels, numbered as filter channel No. 1, filter channel No. 2 and filter channel No. 3, each filter channel is provided with a filter chip, and the preset visible light band is evenly divided into three frequency bands, each frequency band corresponds to a filter channel, and the visible light band range of the frequency band is from m The filter chip corresponding to the filter channel is selected from the filter chips with different central wavelengths, and the corresponding filter chips are No. 1, No. 2 and No. 3. The central wavelength of the filter chip in each filter channel falls within the corresponding frequency band, and the combination is 1. a filter channel combination, and number the filter channel combination;

[0010] Step 4. Calculate the sRGB value of each filter chip based on the filter chip transmittance curve obtained in step 2;

[0011] Step 5. Evaluate the difference between each filter channel combination and the sRGB value of each color group in the scene image;

[0012] Step 6. Calculate the correlation coefficient between the three filter chips in each filter channel combination ;

[0013] Step 7. Calculate the difference between the filter channel combination obtained in step 5 and the sRGB value of the color group in the scene image and the correlation coefficient between the three filter chips in each filter channel combination obtained in step 6 The total value from The filter channel combination with the smallest total value is selected from the filter channel combinations as the customized filter channel combination for the scene.

[0014] Compared with the prior art, the patent of the present invention has the following advantages: Compared with the traditional RGB channel, the present invention is based on the optimization method of customized spectral reconstruction filter channel, by calculating the sRGB value of the color group of the scene image, and calculating the difference between the sRGB value of the filter chip and the color group in the filter channel of the three frequency bands, combined with the correlation coefficient between the filter chips in the filter channel, effectively reducing the root mean square error (RMSE) and the mean relative absolute error (MRAE), while improving the peak signal-to-noise ratio (PSNR), thereby improving the reconstruction accuracy; in addition, compared with the traditional hyperspectral imaging system, the spectral reconstruction system constructed with customized filter channels meets the future needs of miniaturization, lightweight and integration, and significantly reduces the cost.

[0015] In a possible implementation, the step 1 uses a smart phone to capture a scene image, and obtains a cluster of the scene images. n color groups, and calculate the sRGB value of each color group specifically including:

[0016] Step 101. Obtain a scene image by photographing the scene with a smart phone, and extract sRGB values ​​of the scene image pixel by pixel;

[0017] Step 102: Based on the sRGB value of each pixel, the K-means clustering algorithm is used to perform color classification on the scene image to achieve automatic pixel clustering and obtain n color groups and number the color groups;

[0018] Step 103: Calculate the sRGB average value of the pixels in each color group as the sRGB value of the corresponding color group.

[0019] In a possible implementation, step 6 calculates the sRGB value of each filter chip based on the filter chip transmittance curve obtained in step 4. The specific process is as follows: First, the filter chip transmittance curve is converted into the tristimulus value of the CIE 1931XYZ color space. X ( k j ), Y ( k j ), Z ( k j ), the conversion formula is as follows:

[0020] ;

[0021] In the formula, P λ Represents the spectral power distribution of standard illuminant D65; , , is the color matching function of the CIE 1931 XYZ color space; next, the tristimulus values ​​are converted to red, green, and blue components, denoted as s R' ( k j ),s G' ( k j ),s B' ( k j ), as follows:

[0022] ;

[0023] Finally, to accommodate the nonlinear response of the human visual system, sR ′( k j ) is gamma corrected to obtain sR ( k j ), the correction formula is as follows:

[0024] ;

[0025] ;

[0026] .

[0027] In one possible implementation, the calculation formula for evaluating the difference between each filter channel combination and the sRGB value of each color group in the scene image in step 7 is:

[0028] ;

[0029] In the formula, sR i , sG i , sB i Represents the first i The red, green, and blue component values ​​of a color group; R ( k j ),s G ( k j ),s B ( k j ) respectively correspond to k The first filter channel combination j The red, green, and blue component values ​​of each filter chip; nIndicates the number of color clusters in the scene image; each component value in the sRGB value is normalized, that is, each value is divided by 255.

[0030] In a possible implementation, in step 8, the correlation coefficients between the three filter chips in each filter channel combination are calculated. The calculation formula is:

[0031] ;

[0032] In the formula, j, j' Respectively represent any two filter chips in the channel combination; , Respectively represent k The first filter channel combination j and j' The filter chip has a wavelength λ The transmittance at , Respectively represent k The first filter channel combination j and j' The average transmittance of a filter chip in the visible light band.

[0033] In one possible implementation, step 9 calculates the difference between the filter channel combination obtained in step 7 and the sRGB value of the color group in the scene image and the total value obtained in step 8, from The calculation formula for selecting the smallest total value from the filter channel combinations as the customized filter channel combination for the scene is:

[0034] ;

[0035] In the formula, r 12 Represents the correlation coefficient between filter chip No. 1 and filter chip No. 2, r 13 Represents the correlation coefficient between filter chip No. 1 and filter chip No. 3, r 23 Represents the correlation coefficient between filter chip No. 2 and filter chip No. 3, select F ( f , r ) The filter channel combination corresponding to the smallest value is used as the custom filter channel combination for the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of the optimization method of the present invention;

[0037] Figure 2 Structural diagram of the system for spectral reconstruction;

[0038] Figure 3 An image obtained based on the customized filter channel combination of the present invention;

[0039] Figure 4 Part a in the figure is a schematic diagram of the real and fake car scene;

[0040] Figure 4 Parts b1, b2, and b3 are comparison diagrams of the reconstruction results of the custom filter channel and the RGB channel;

[0041] Figure 5 Compare the quantitative reconstruction results of the customized filter channel and the RGB channel. DETAILED DESCRIPTION

[0042] First, those skilled in the art should understand that these implementations are only used to explain the technical principles of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. Those skilled in the art can make adjustments to them as needed to adapt to specific application scenarios.

[0043] The present application is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] See also Figure 1 to Figure 5 As shown, this embodiment discloses a spectrum reconstruction method with customized filter channel combination, including:

[0045] Step 1: Obtain scene images by photographing the scene with a smartphone, and obtain the scene images through clustering. n color groups and calculate the sRGB value of each color group; specifically including:

[0046] Step 101. Obtain a scene image by photographing the scene with a smart phone, and extract sRGB values ​​of the scene image pixel by pixel;

[0047] Step 102: Based on the sRGB value of each pixel, the K-means clustering algorithm is used to perform color classification on the scene image to achieve automatic pixel clustering and obtain n color groups and number the color groups;

[0048] Step 103: Calculate the sRGB average value of the pixels in each color group as the sRGB value of the corresponding color group.

[0049] Step 2. Preset a visible light band, and randomly generate a plurality of filter chip transmittance curves with central wavelengths equally spaced within the visible light band; the preset visible light band in this specific embodiment is 400-700nm.

[0050] Step 3. Construct a filter channel combination, wherein the filter channel combination has three filter channels, numbered as filter channel No. 1, filter channel No. 2 and filter channel No. 3, each filter channel is provided with a filter chip, and the preset visible light band is evenly divided into three frequency bands, each frequency band corresponds to a filter channel, and the visible light band range of the frequency band is from m The filter chip corresponding to the filter channel is selected from the filter chips with different central wavelengths, and the corresponding filter chips are No. 1, No. 2 and No. 3. The central wavelength of the filter chip in each filter channel falls within the corresponding frequency band, and the combination is kinds of filter channel combinations and number the filter channel combinations; in this specific embodiment, the existing filter chips on hand are used to form a variety of different channel combinations, and eight groups of filter channel combinations are obtained: [400, 500, 600], [400, 500, 650], [450, 500, 600], [450, 500, 650], [400,550, 600], [400, 550, 650], [450, 550, 600] and [450, 550, 650].

[0051] Step 4. Calculate the sRGB value of each filter chip based on the filter chip transmittance curve in step 2. The specific process is as follows:

[0052] First, convert the filter chip transmittance curve into the tristimulus values ​​of the CIE 1931 XYZ color space X ( k j ), Y ( k j ), Z ( k j ), the conversion formula is as follows:

[0053] ;

[0054] In the formula, P λ Represents the spectral power distribution of standard illuminant D65; , , is the color matching function of the CIE 1931 XYZ color space;

[0055] Next, the tristimulus values ​​are converted into red, green, and blue components, denoted as s R' ( k j ),s G' ( k j ),sB' ( k j ), as follows:

[0056] ;

[0057] Finally, to accommodate the nonlinear response of the human visual system, sR ′( k j ) is gamma corrected to obtain sR ( k j ), the correction formula is as follows:

[0058] ;

[0059] ;

[0060] .

[0061] Step 5. Evaluate the difference between different filter channel combinations and the sRGB values ​​of the color population in the scene image;

[0062] ;

[0063] In the formula, sR i , sG i , sB i Represents the first i The red, green, and blue component values ​​of a color cluster; R ( k j ),s G ( k j ),s B ( k j ) respectively correspond to k The first filter channel combination j The red, green, and blue component values ​​of each filter chip; n Indicates the number of color clusters in the scene image; each component value in the sRGB value is normalized, that is, each value is divided by 255.

[0064] Step 6. Calculate the correlation coefficient between the three filter chips in each filter channel combination ;

[0065] ;

[0066] In the formula, j, j'Respectively represent any two filter chips in the channel combination; , Respectively represent k The first filter channel combination j and j' The filter chip has a wavelength λ The transmittance at , Respectively represent k The first filter channel combination j and j' The average transmittance of a filter chip in the visible light band.

[0067] Step 7. Calculate the difference between each filter channel combination obtained in step 5 and the sRGB value of the color group in the scene image and the correlation coefficient between the three filter chips in each filter channel combination obtained in step 6 The total value from The filter channel combination with the smallest total value is selected from the filter channel combinations as the customized filter channel combination for the scene. The calculation formula is:

[0068] ;

[0069] In the formula, r 12 Represents the correlation coefficient between filter chip No. 1 and filter chip No. 2, r 13 Represents the correlation coefficient between filter chip No. 1 and filter chip No. 3, r 23 Represents the correlation coefficient between filter chip No. 2 and filter chip No. 3, select F ( f , r ) The filter channel combination corresponding to the smallest value is used as the customized filter channel combination for the scene;

[0070] After the calculation in step 9, the filter channel combination [450, 500, 650] is obtained in this specific embodiment as the stator filter channel combination of this scenario, and the filter channel combination [450, 500, 650] is integrated into Figure 2 In the spectral reconstruction system shown, the system includes a CCD camera, a lens, a filter wheel, and a motor driving the filter wheel. When capturing an image, the lens needs to be aligned with the customized filter channel. The computer synchronously controls the motor and the camera, so that the system can capture three images of the scene at one time, such as Figure 3 As shown in the figure, the three captured images are reconstructed into 31 hyperspectral images through the MST++ algorithm. The reconstruction results are shown in Figure 4 a, b1, b2, b3 and Figure 5As shown; as a control, the real spectrum was measured and obtained by the GaiaField-V10 push-broom hyperspectral imaging system.

[0071] The results show that compared with the traditional RGB channel, the reconstructed spectral curve obtained by the customized filter channel of the present invention is closer to the real spectrum. While effectively reducing the root mean square error (RMSE) and the mean relative absolute error (MRAE), the present invention significantly improves the peak signal-to-noise ratio (PSNR), thereby enhancing the accuracy of spectral reconstruction.

[0072] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A spectrum reconstruction method with customized filter channel combination, characterized in that: The following steps are involved: Step 1: Obtain scene images by photographing the scene with a smartphone, and obtain the scene images through clustering. n color groups and calculate the sRGB value of each color group; Step 2. Preset a visible light band, and randomly generate a plurality of filter chip transmittance curves with equally spaced central wavelengths within the visible light band; Step 3. Construct a filter channel combination, wherein the filter channel combination has three filter channels, numbered as filter channel No. 1, filter channel No. 2 and filter channel No. 3, each filter channel is provided with a filter chip, and the preset visible light band is evenly divided into three frequency bands, each frequency band corresponds to a filter channel, and the visible light band range of the frequency band is from m The filter chip corresponding to the filter channel is selected from the filter chips with different central wavelengths, and the corresponding filter chips are No. 1, No. 2 and No.

3. The central wavelength of the filter chip in each filter channel falls within the corresponding frequency band, and the combination is 1. a filter channel combination, and number the filter channel combination; Step 4. Calculate the sRGB value of each filter chip based on the filter chip transmittance curve obtained in step 2; Step 5. Evaluate the difference between each filter channel combination and the sRGB value of each color group in the scene image; Step 6. Calculate the correlation coefficient between the three filter chips in each filter channel combination ; Correlation coefficient The calculation formula is: ； In the formula, j, j' Respectively represent any two filter chips in the channel combination; , Respectively represent k The first filter channel combination j and j' The filter chip has a wavelength λ The transmittance at , Respectively represent k The first filter channel combination j and j' The average transmittance of each filter chip in the visible light band; Step 7. Calculate the difference between each filter channel combination obtained in step 5 and the sRGB value of the color group in the scene image and the correlation coefficient between the three filter chips in each filter channel combination obtained in step 6 The total value from The filter channel combination with the smallest total value is selected from the filter channel combinations as the customized filter channel combination for the scene.

2. The spectral reconstruction method of filter channel combination customization according to claim 1, characterized in that: In step 1, a scene image is captured by a smart phone, and a clustering method is used to obtain the scene image. n color groups, and calculate the sRGB value of each color group specifically including: Step 101. Obtain a scene image by photographing the scene with a smart phone, and extract sRGB values ​​of the scene image pixel by pixel; Step 102: Based on the sRGB value of each pixel, the K-means clustering algorithm is used to perform color classification on the scene image to achieve automatic pixel clustering and obtain n color groups and number the color groups; Step 103: Calculate the sRGB average value of the pixels in each color group as the sRGB value of the corresponding color group.

3. The spectral reconstruction method of filter channel combination customization according to claim 1, characterized in that: Step 4 calculates the sRGB value of each filter chip based on the filter chip transmittance curve obtained in step 2. The specific process is as follows: First, convert the filter chip transmittance curve into the tristimulus values ​​of the CIE 1931 XYZ color space X ( k j ), Y ( k j ), Z ( k j ), the conversion formula is as follows: ； In the formula, P λ Represents the spectral power distribution of standard illuminant D65; , , is the color matching function of the CIE 1931 XYZ color space; Next, the tristimulus values ​​are converted into red, green, and blue components, denoted as s R' ( k j ),s G' ( k j ),s B' ( k j ), as follows: ； Finally, to accommodate the nonlinear response of the human visual system, sR ′( k j ) is gamma corrected to obtain sR ( k j ), the correction formula is as follows: ； ； 。 4. The spectral reconstruction method of filter channel combination customization according to claim 1, characterized in that: The calculation formula for evaluating the difference between each filter channel combination and the sRGB value of each color group in the scene image in step 5 is: ； In the formula, sR i , sG i , sB i Represents the first i The red, green, and blue component values ​​of a color group; R ( k j ),s G ( k j ),s B ( k j ) respectively correspond to k The first filter channel combination j The red, green, and blue component values ​​of each filter chip; n Indicates the number of color clusters in the scene image; each component value in the sRGB value is normalized, that is, each value is divided by 255.

5. The spectral reconstruction method of filter channel combination customization according to claim 4, characterized in that: In step 7, the difference between each filter channel combination obtained in step 5 and the sRGB value of the color group in the scene and the correlation coefficient between the three filter chips in each filter channel combination obtained in step 6 are calculated. The total value from The calculation formula for selecting the filter channel combination with the smallest total value from the filter channel combinations as the customized filter channel combination for the scene is: ； In the formula, r 12 Represents the correlation coefficient between filter chip No. 1 and filter chip No. 2, r 13 Indicates the correlation coefficient between filter chip No. 1 and filter chip No. 3, r 23 Represents the correlation coefficient between filter chip No. 2 and filter chip No. 3, select F ( f , r ) The filter channel combination corresponding to the smallest value is used as the custom filter channel combination for the scene.

Images