Spectral reconstruction method and system based on optimizing digital camera spectral response function

Abstract

The present invention provides a spectral reconstruction method and system based on optimizing the spectral response function of a digital camera, comprising: measuring the spectral reflectance of a training sample; photographing and obtaining the original format response values of the test sample under different imaging scenes and recording the camera imaging parameters; synchronously measuring the spectral power distribution of the light source of the imaging scene; then, based on a full-parameter imaging model of the digital camera, predicting the raw response values of a training sample set under the same imaging conditions, performing maximum and minimum normalization on the original format response values of the test sample and the predicted response values of the training sample set in three RGB channels, performing spectral characteristic modeling on the camera, and finally performing multi-spectral reconstruction on the test sample under the same imaging conditions to obtain reconstructed spectral data of the test sample.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer digital image processing, and in particular relates to a spectrum reconstruction method and system based on optimizing the spectral response function of a digital camera. Background Art

[0002] A multispectral image is an image with data from multiple spectral channels. It contains both spatial and spectral information about the scene. Unlike an RGB image, where a single pixel contains color information from three channels, a single pixel in a multispectral image can include spectral reflectance information from dozens of channels. When a light source shines on an object, it selectively absorbs it and produces a reflectance spectrum. This reflectance spectrum is independent of the observer and the light source, representing an inherent physical property of the object. Therefore, the spectrum is a fingerprint of color information and an important characteristic that characterizes the physical and chemical properties of an object. It has important applications in color reproduction, cultural relic preservation, medical diagnosis, computer vision, remote sensing, and other fields.

[0003] Multispectral imaging technology, based on high-spatial-resolution monochrome or color imaging sensors, establishes various imaging systems by configuring bandpass filters, multi-primary color filter arrays, or multicolor light sources. These systems record digital images of object surfaces and generate multispectral images of objects through sample set optimization and spectral reconstruction. This overcomes the metamerism problem inherent in color imaging, accurately characterizing and distinguishing objects. Spectral color management techniques enable high-fidelity acquisition and replication of color information. Compared to expensive hyperspectral cameras, multispectral imaging offers a more universal application advantage and has rapidly developed in recent decades, becoming a key frontier in current scientific and technological development. Spectral reconstruction, the key to multispectral imaging, involves reconstructing the object response signal obtained by the imaging system into its corresponding accurate spectral information. Therefore, the choice and design of spectral reconstruction methods directly determine the accuracy of multispectral image acquisition, which in turn impacts its performance in the aforementioned fields.

[0004] However, existing spectral reconstruction methods are sensitive to changes in imaging conditions (such as light source, illumination, imaging parameters, etc.). That is, the spectral reconstruction matrix established under a certain imaging condition cannot be directly applied to other imaging conditions for spectral reconstruction. Otherwise, the reconstructed spectral curve will be deformed and its spectral characteristic information will be lost. In addition, when multi-spectral reconstruction is applied in an open environment, the uncontrollable and uneven illumination and changes in camera imaging parameters (such as exposure time, sensitivity ISO, etc.) will cause the imaging conditions to change, thereby increasing the spectral reconstruction error and causing the reconstructed spectrum to deviate from the true value, making it impossible to apply it in practice. In response to the challenges of multi-spectral reconstruction applications in open environments with changing imaging conditions, neither the academic community nor the industrial community at home and abroad has proposed a good solution. Summary of the Invention

[0005] The purpose of the present invention is to solve the problem described in the background technology and to propose a spectrum reconstruction method based on optimizing the spectral response function of a digital camera.

[0006] In response to the problems existing in the above-mentioned existing research, the present invention proposes a method to solve the problem. First, the spectral reflectance of the training sample is measured, the original format response value of the test sample under different imaging scenes is photographed and the camera imaging parameters are recorded, the spectral power distribution of the light source of the imaging scene is synchronously measured, and the spectral response function optimization method is used to estimate the spectral response function of the digital camera. Then, based on the full-parameter imaging model of the digital camera, the raw response value of the training sample set under the same imaging conditions is predicted, and the original format response value of the test sample and the predicted training sample set response value are respectively divided into RGB three channels for maximum and minimum normalization, the camera is spectrally characterized and modeled, and a spectral reconstruction model is constructed using a spectral reconstruction algorithm. Finally, multi-spectral reconstruction is performed on the test sample under the same imaging conditions to obtain a reconstructed spectrum of the test sample. The technical solution of the present invention is a spectral reconstruction method based on optimizing the spectral response function of a digital camera, which specifically includes the following steps:

[0007] Step 1: Measure and obtain the spectral reflectance of the training sample;

[0008] Step 2: photograph and obtain the original digital response values of the test samples under different imaging scenes, and record the camera imaging parameters;

[0009] Step 3, measuring the spectral power distribution of the light source during the shooting process;

[0010] Step 4: combining the above-mentioned training sample spectral reflectance, camera imaging parameters and light source spectral power distribution, using the spectral response function optimization method to estimate the digital camera spectral response function, obtain the camera spectral response function, and obtain the nonlinear parameters of the digital camera full-parameter imaging model;

[0011] Step 5: Based on the digital camera full-parameter imaging model, combined with the camera spectral response function, camera imaging parameters and nonlinear parameters, predict the raw response value of the training sample under any imaging conditions;

[0012] Step 6: Normalize the original format response value of the test sample and the predicted training sample response value in RGB channels respectively;

[0013] Step 7, combining the spectral reflectance of the training sample and the two normalized response values in step 6, and constructing a spectral reconstruction model using a spectral reconstruction algorithm;

[0014] Step 8: Reconstruct the test sample using the spectrum reconstruction model to obtain the test sample spectrum.

[0015] Furthermore, the specific implementation of step 4 is as follows:

[0016] Step 41: Collect the existing public camera spectral response function database and normalize the camera spectral response function by three channels of RGB;

[0017] Step 42: combining the spectral reflectance of the training sample and the spectral power distribution of the light source, using principal component analysis to calculate the principal components of the camera spectral response function database by channel, and using a pseudo-inverse algorithm to calculate the initial spectral response function;

[0018] Step 43, based on the initial digital camera imaging model, combined with the camera imaging parameters and the initial spectral response function, using a curve fitting model to obtain nonlinear parameters in the digital camera full-parameter imaging model;

[0019] Step 44 : Calculate the color difference between the normalized original format response value and the predicted response value, use it as a loss function, and iteratively optimize to obtain the final camera spectral response function.

[0020] Furthermore, the specific implementation of step 42 is as follows:

[0021] The camera spectral response is decomposed into s using principal component analysis. k,n =σ k *E k ,k=R,G,B, where the coefficient σ k =[σ k,1 ,σ k,2 ] is a 1×2 vector, E k =[e k,1 ,e k,2 ] T is a 2×31 eigenvector matrix; based on this, the initial digital camera imaging model is expressed as:

[0022]

[0023] in is the predicted response of the camera, g k σ k E k is the camera spectral response function before normalization, l is the spectral power distribution of the scene light source, and R is the spectral reflectance of the training sample;

[0024] By solving The superscript '+' represents the pseudo-inverse, and the camera spectral response function is restored. Then the initial spectral response function S k Expressed as:

[0025]

[0026] Where r is the original digital response value of the training sample captured by the camera.

[0027] Furthermore, the specific implementation of step 43 is as follows:

[0028] When the initial camera spectral response function is known, the preliminary predicted camera response can be obtained:

[0029]

[0030] Among them, S k is the initial spectral response function, l is the spectral power distribution of the scene light source, and R is the spectral reflectance of the training sample;

[0031] The initial values of nonlinear parameters can be obtained through curve fitting model:

[0032]

[0033] Among them, r is the original digital response value of the training sample captured by the camera, f S is the sensitivity function, c1, c2, and β are parameters that characterize the non-equivalence between camera exposure time and ISO.

[0034] Furthermore, the specific implementation of step 44 is as follows:

[0035] First, convert the normalized response value of each channel from RGB color space to XYZ color space, then convert it to Lab color space, and then calculate its CIEDE2000 color difference ΔE 00 As a loss function:

[0036]

[0037] in, p represents the camera prediction response and the original format response of each sample, and μ is an adjustable parameter that sets the regularization constraint of the loss function by the maximum color difference in the sample.

[0038] Furthermore, the specific implementation of step 5 is as follows:

[0039] First, the radiation spectrum distribution l(λ) of the light source is irradiated on the surface of the object, and after absorption and reflection by the object, a radiation spectrum image is formed, as shown in formula (6):

[0040] L scene (x,y,λ)=∫r(x,y,λ)l(λ) (6)

[0041] Where λ represents the wavelength, r(x,y,λ) is the spectral reflectance of the (x,y) pixel in the image, l(λ) represents the relative spectral radiance of the light source, and Lscene (x, y, λ) represents the radiation spectrum at each pixel. Subsequently, the radiation spectrum image is focused by the camera lens and enters the sensor, forming an irradiance image before entering the sensor, as shown in formula (7):

[0042]

[0043] Where f / # is the effective aperture number of the lens, m is the lens magnification, T(λ) is the transmittance of the lens, L(x, y, λ) represents the radiation spectrum at each pixel, and I(x, y, λ) is the irradiance at each pixel. Furthermore, the irradiance image enters the camera sensor, undergoes photoelectric conversion and analog-to-digital conversion, and obtains a Bayer pattern mosaic image. The camera raw response value image of the scene is obtained through the interpolation algorithm. The process is shown in Equations (8) to (12):

[0044] Q(x,y)=∫ Ω I image (x,y,λ)S k (λ)dλ (8)

[0045] Where Ω is the spectral wavelength range, S k (λ) is the spectral response function of the camera, Q(x,y) is the number of photoelectric conversions per unit time at the pixel (x,y), and the total photoelectric conversion amount of the camera sensor within a given exposure time τ is defined as:

[0046] φ(x,y)=f s ∫ τ Q(x,y)dt (9)

[0047] Where, f s is the sensitivity function, τ is the exposure time; when the spectral irradiance and the quantum efficiency of the sensor remain unchanged during one shooting, φ(x) can be further simplified to the form shown in formula (10):

[0048] φ(x,y)=f s τQ(x,y) (10)

[0049] According to equations (8) and (10), the raw response value p(x, y) at any pixel (x, y) in the image can be expressed as:

[0050]

[0051] In addition, considering the nonlinear relationship between the irradiance of the camera sensor and the camera readout response, the model uses three coefficients c1, c2 and β to characterize the non-equivalence between exposure time and ISO, and the final raw response value p(x,y) of any pixel point can be obtained:

[0052]

[0053] Furthermore, the method for constructing the spectrum reconstruction model using the spectrum reconstruction algorithm in step 7 is as follows:

[0054] First, the third-order root polynomial is used to expand the normalized response values of the training and test samples. The expansion form is as follows:

[0055]

[0056] Where r, g, and b are the response values of the R channel, G channel, and B channel of all samples respectively, and d exp Represents a response value vector after sample expansion;

[0057] Then, using the pseudo-inverse algorithm, the spectrum reconstruction matrix Q is calculated from the training sample spectral reflectance and the corresponding extended response value as follows:

[0058] Q=R(D T D+λI) -1 D T

[0059] Where R is the training sample spectral data matrix, D is the matrix after the training sample response value is expanded, Q is the spectral reconstruction matrix, the superscript 'T' is the transpose symbol, the superscript '-1' indicates the inversion operation, λ is the regularization constraint coefficient, I is the identity matrix, and λI is used to counteract the noise information in the solution of the spectral reconstruction matrix to prevent model overfitting.

[0060] Furthermore, the spectrum reconstruction method in step 8 is as follows:

[0061]

[0062] Where d is the root polynomial expansion vector of the original format response value of the test sample, Q is the spectrum reconstruction matrix calculated by step 7, is the spectral data vector of the reconstructed test sample.

[0063] Furthermore, in step 1, the spectral reflectance of the training sample is measured using a spectrophotometer;

[0064] In step 2, a digital camera is used to obtain the original digital response values of the training samples under different imaging scenes; the camera imaging parameters include exposure time and ISO sensitivity;

[0065] In step 3, a color illuminance meter is used to synchronously measure the spectral power distribution of the light source during the shooting process.

[0066] The present invention also provides a spectrum reconstruction system based on optimizing the spectral response function of a digital camera, comprising:

[0067] one or more processors;

[0068] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement a spectral reconstruction method based on optimizing the spectral response function of a digital camera as described in the above solution.

[0069] This method addresses the problem that existing spectral reconstruction methods are unable to adapt to varying exposure levels or uneven illumination under the same illumination source. Based on a digital camera imaging model, this method predicts the raw response values of a training sample set under any imaging condition to perform spectral characterization modeling of the camera. It then performs multispectral reconstruction of the subject under the same imaging conditions. Theoretically, this method can achieve real-time spectral characterization modeling of the camera in different open scenes, thereby reconstructing multispectral images of the subject in the corresponding scene, supporting a variety of practical spectral-based applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 Flowchart of an embodiment of the present invention. DETAILED DESCRIPTION

[0071] When the technical solution of the present invention is specifically implemented, those skilled in the art can use computer software technology to run it.

[0072] like Figure 1 As shown, the present invention provides a spectrum reconstruction method based on optimizing the spectral response function of a digital camera, which specifically includes the following steps:

[0073] Step 1: Use a spectrophotometer to measure and obtain the spectral reflectance R of the training sample set;

[0074] Step 2: Use a digital camera to capture the original digital response values of the test samples under different imaging scenes, and record the camera imaging parameters (such as exposure time, ISO sensitivity, etc.);

[0075] Step 3, using a color illuminance meter to synchronously measure the spectral power distribution l of the light source during the shooting process;

[0076] Step 4: combining the above-mentioned training sample spectral reflectance, camera imaging parameters and light source spectral power distribution, using the spectral response function optimization method to estimate the digital camera spectral response function, obtain the camera spectral response function, and obtain the nonlinear parameters of the digital camera full-parameter imaging model;

[0077] The specific method for estimating the spectral response function and nonlinear parameters of the digital camera in step 4 is as follows:

[0078] Step 41: Collect existing public camera spectral response function databases, select data from the 400 to 700 nanometer band, and normalize the camera spectral response functions using the RGB channels.

[0079]

[0080] where s k,n is the normalized camera spectral response function, s k,n =[s k (400nm),s k (410nm),…,s k (700nm)], S k is the camera spectral response function before normalization, S k =g k *s k,n ,g k >0,k=R,G,B,g k are constants for the R, G, and B channels. R, G, and B are the R, G, and B channels of the sample, respectively. This method normalizes the existing camera spectral response function by channel, normalizing the peak values of all RGB channels to unity.

[0081] Step 42: combining the spectral reflectance of the training sample and the spectral power distribution of the light source, using principal component analysis (PCA) to calculate the principal components of the camera spectral response function database by channel, and using a pseudo-inverse algorithm to calculate the initial spectral response function;

[0082] In step 42, principal component analysis (PCA) is used to calculate the principal components of the camera spectral response function database by channel. The method of using the pseudo-inverse algorithm to calculate the initial spectral response function is as follows:

[0083] The camera spectral response function is decomposed into s using principal component analysis method. k,n =σ k *E k ,k=R,G,B, where the coefficient σ k =[σ k,1 ,σ k,2 ] is a 1×2 vector, E k =[e k,1 ,e k,2 ] T is a 2×31 eigenvector matrix. Based on this, the initial digital camera imaging model can be expressed as:

[0084]

[0085] in is the predicted response of the camera, g k σ k Ek is the camera sensitivity function before normalization, l is the spectral power distribution of the scene light source, and R is the spectral reflectance of the training sample.

[0086] By solving The superscript '+' represents the pseudo-inverse, which can restore the camera spectral response function. Then the initial spectral response function S k Expressed as:

[0087]

[0088] Where r is the original digital response value of the training sample captured by the camera.

[0089] Step 43, based on the initial digital camera imaging model, combined with the camera imaging parameters and the initial spectral response function, using a curve fitting model to obtain nonlinear parameters in the digital camera full-parameter imaging model;

[0090] In step 43, based on the initial digital camera imaging model, a method for obtaining nonlinear parameters in the digital camera full-parameter imaging model using a curve fitting model is as follows:

[0091] When the initial camera spectral response function is known, the preliminary predicted camera response can be obtained:

[0092]

[0093] The initial values of nonlinear parameters can be obtained through curve fitting model:

[0094]

[0095] Among them, r is the original digital response value of the training sample captured by the camera, f S It is a sensitivity function, usually obtained by dividing the ISO value by 100. c1, c2, and β are parameters that characterize the non-equivalence between camera exposure time and ISO.

[0096] Step 44 : Calculate the color difference between the normalized original format response value and the predicted response value, use it as a loss function, and use a sequential quadratic programming method to iteratively optimize to obtain the final camera spectral response function.

[0097] The loss function definition and optimization method in step 44 are as follows:

[0098] First, the original format response value after channel normalization is converted from RGB color space to XYZ color space, and then converted to Lab color space, and then the CIEDE2000 color difference (ΔE 00 ) as the loss function:

[0099]

[0100] in, p represents the camera's predicted response and the original format response for each sample. Since we selected the ColorChecker SG140 as training samples, the total number of samples N is 140. μ is an adjustable parameter that sets the regularization constraint of the loss function by the maximum color difference in the sample. This method empirically sets it to 0.1.

[0101] Our goal is to find the camera sensitivity function S that minimizes the loss function k (λ) and nonlinear parameters. Since the full-parameter imaging model of a digital camera is a power function, it is difficult to directly calculate its gradient during the optimization process. Therefore, we choose the sequential quadratic programming method to solve this constrained nonlinear optimization problem.

[0102] Step 5: Based on the digital camera full-parameter imaging model, combined with the camera spectral response function, camera imaging parameters and nonlinear parameters, predict the raw response value of the training sample set under any imaging conditions;

[0103] In step 5, based on the full-parameter imaging model of the digital camera, the method for predicting the raw response value of the training sample set under any imaging conditions is as follows:

[0104] First, the radiation spectrum distribution l(λ) of the light source is irradiated on the surface of the object, and after absorption and reflection by the object, a radiation spectrum image is formed, as shown in formula (6):

[0105] L scene (x,y,λ)=∫r(x,y,λ)l(λ) (6)

[0106] Where λ represents the wavelength, r(x,y,λ) is the spectral reflectance of the (x,y) pixel in the image, l(λ) represents the relative spectral radiance of the light source, and L scene (x, y, λ) represents the radiation spectrum at each pixel. Subsequently, the radiation spectrum image is focused by the camera lens and enters the sensor, forming an irradiance image before entering the sensor, as shown in formula (7):

[0107]

[0108] Where f / # is the effective aperture of the lens, m is the lens magnification, T(λ) is the transmittance of the lens, L(x, y, λ) represents the radiation spectrum at each pixel, and I(x, y, λ) is the irradiance at each pixel. Furthermore, the irradiance image enters the camera sensor, undergoes photoelectric conversion and analog-to-digital conversion, and obtains a Bayer pattern mosaic image. The interpolation algorithm then generates the camera's raw response value image of the scene. The process is shown in Equations (8) to (12):

[0109] Q(x,y)=∫ Ω I image (x,y,λ)S k (λ)dλ (8)

[0110] Where, Ω is the spectral wavelength range, which is 400-700 nm in this study, and S k (λ) is the spectral response function of the camera, Q(x,y) is the number of photoelectric conversions per unit time at the pixel (x,y), and the total photoelectric conversion amount of the camera sensor within a given exposure time τ is defined as:

[0111] φ(x,y)=f s ∫ τ Q(x,y)dt (9)

[0112] Where, f s is the sensitivity function, usually obtained by dividing the ISO value by 100, and τ is the exposure time. When the spectral irradiance and the quantum efficiency of the sensor remain unchanged during a single shot, φ(x) can be further simplified to the form shown in Equation (10):

[0113] φ(x,y)=f s τQ(x,y) (10)

[0114] According to equations (8) and (10), the raw response value p(x, y) at any pixel (x, y) in the image can be expressed as:

[0115]

[0116] In addition, considering the nonlinear relationship between the irradiance of the camera sensor and the camera readout response, the model uses three constants c1, c2 and β to characterize the non-equivalence between exposure time and ISO, and the final expression of the raw response value p(x,y) of any pixel point can be obtained.

[0117]

[0118] Step 6: Normalize the original format response value of the test sample and the response value of the predicted training sample set by RGB channels respectively;

[0119] The channel normalization method is shown in formula (13):

[0120]

[0121] In the formula, R, G, B are the R channel, G channel and B channel of the sample respectively. Represents the response value vector of a sample after channel normalization.

[0122] Step 7: Combine the spectral reflectance of the training samples and the predicted response values of the training samples to construct a spectral reconstruction model using a spectral reconstruction algorithm.

[0123] The method for constructing a spectral reconstruction model using a spectral reconstruction algorithm in step 7 is as follows:

[0124] First, the normalized response values of the training and test samples are expanded using the third-order root polynomial. The expanded form is shown in the following formula:

[0125]

[0126] Where r, g, and b are the response values of the R channel, G channel, and B channel of all samples respectively, and d exp Represents a vector of response values after sample expansion.

[0127] Then, using the pseudo-inverse algorithm, the spectrum reconstruction matrix Q is calculated from the training sample spectral reflectance and the corresponding extended response value as follows:

[0128] Q=R(D T D+λI) -1 D T

[0129] Where R is the training sample spectral data matrix, D is the matrix after the training sample original format response value is expanded, Q is the spectral reconstruction matrix, the superscript 'T' is the transpose symbol, the superscript '-1' indicates the inversion operation, λ is the regularization constraint coefficient, I is the identity matrix, and λI is used to counteract the noise information in the solution of the spectral reconstruction matrix to prevent model overfitting. The value of λ is usually 0.001.

[0130] Step 8: Reconstruct the test sample using the spectrum reconstruction model to obtain the test sample spectrum.

[0131] The spectrum reconstruction method in step 8 is as follows:

[0132]

[0133] Where d is the root polynomial expansion vector of the original format response value of the test sample, Q is the spectrum reconstruction matrix calculated by step 7, is the spectral data vector of the reconstructed test sample.

[0134] The following is an example to illustrate the present invention.

[0135] This example uses a Nikon D7200 digital camera, an EVERFINE SPIC-300AW spectral color illuminator, and an X-rite i1-Pro3 spectrophotometer as experimental instruments. Experiments were conducted in both a laboratory environment and an outdoor environment. The ColorChecker SG140 (CCSG) color chart was used as a training sample, and the ColorChecker 24 (CC) color chart was used as a test sample.

[0136] In this example, digital images of a CC color chart were captured using a Nikon D7200 digital camera in a laboratory environment, using a closed, uniformly illuminated light box. Raw response values were extracted from the captured images. During the capture process, the color chart was placed in the center of the field of view, with the plane of the chart approximately 1 meter from the camera sensor plane, and the camera focal length was 35 mm. The CC color chart was captured under five different exposure and ISO combinations (as shown in Table 1). All exposure combinations resulted in no overexposure, ensuring the validity of the experimental data.

[0137] Table 1 Five exposure settings for laboratory environment experiments

[0138]

[0139] In an open outdoor environment, a CC color chart was photographed using a Nikon D7200 digital camera, and the raw response value data for the color chart was extracted. In addition, five different exposure and ISO combinations were set for the open-air experiment, as shown in Table 2. The color chart images under the first exposure and ISO combination were slightly overexposed, while the images under the remaining exposure and ISO combinations were not overexposed. During the shooting process, the color chart was placed in the center of the shooting field of view, with the color chart plane approximately 3 meters from the camera sensor plane, and the camera focal length was 22 mm. All other experimental conditions and the color chart raw response value prediction method remained the same as those in the laboratory environment.

[0140] Table 2 Five exposure settings for the open environment experiment

[0141]

[0142] In the experiment, the relative spectral power distribution of the light source was measured using an EVERFINE SPIC-300AW spectral colorimeter. The spectral reflectance of the color chart was measured using an X-rite i1-Pro3 spectrophotometer. Given the spectral reflectance of the training sample, the spectral response function of the Nikon D7200 camera, the spectral power distribution of the light source, and imaging parameters, the response value prediction model can be used to predict the raw response data of the CCSG color chart under the corresponding imaging conditions. This is expressed as follows:

[0143]

[0144] Where, f s is the sensitivity function, which is usually obtained by dividing the ISO value by 100, τ is the exposure time, Ω is the spectral wavelength range, which is 400-700nm in this study, f / # is the effective aperture number of the lens, and m is the lens magnification. represents the radiation spectrum at each pixel, S k (λ) is the camera spectral response function of the camera.

[0145] Normalization is performed on the original format response values of the test samples and the response values of the predicted training sample set. The specific normalization methods are as follows:

[0146]

[0147] In the formula, R, G, B are the R channel, G channel and B channel of the sample respectively. Represents the response value vector of a sample after channel normalization.

[0148] After obtaining the normalized predicted training sample set response value and the training sample set spectral reflectance as well as the test sample original format response value, the existing spectral reconstruction algorithm can be used to construct a spectral reconstruction matrix and reconstruct the test sample to obtain a reconstructed spectrum.

[0149] In the embodiment, the spectral root mean square error (RMSE) and CIEDE2000 color difference (ΔE 00 ) is used as the evaluation index of spectral estimation, and the performance of the method of the present invention is evaluated from the perspectives of spectral error and chromaticity error. The calculation method of the spectral root mean square error RMSE is as follows:

[0150]

[0151] Wherein, r1 is the measured spectral reflectance, r2 is the reconstructed spectral reflectance, the superscript T is the transpose operator, and N is the number of sampling wavelengths, which is 31 in the embodiment of the present invention.

[0152] Table 3 Spectral reconstruction errors under five different imaging conditions in the laboratory environment

[0153]

[0154] Table 4 Spectral reconstruction errors under five different imaging conditions in open environment

[0155]

[0156] Tables 3 and 4 respectively give the comparison of the method of the present invention using four different spectral reconstruction algorithms in a laboratory environment and an open environment, where 'Liang', 'OLS', 'Cao', and 'Kernal' represent the spectral reconstruction algorithms proposed in references [1] to [4] respectively.

[0157] [1]Liang J,Wan X.Optics Express,2017,25.23:28273-28287.

[0158] [2]Connah DR,Hardeberg J Y.Color Imaging X:Processing,Hardcopy,andApplications.SPIE,2005,5667:65-75.

[0159] [3]Cao B, Liao N, Cheng H. Color Research & Application, 2017, 42.3: 327-332.

[0160] [4]Xu P,Xu H,Diao C.Applied optics,2017,56.3:8461-8470.

[0161] It can be seen intuitively from the evaluation index values in the table that the method of the present invention has achieved satisfactory reconstruction accuracy both in the laboratory environment and in the open environment.

[0162] On the other hand, an embodiment of the present invention further provides a spectral reconstruction system based on optimizing the spectral response function of a digital camera, comprising:

[0163] one or more processors;

[0164] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement a spectral reconstruction method based on optimizing the spectral response function of a digital camera as described in the above solution.

[0165] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

Claims

1. A spectrum reconstruction method based on optimizing the spectral response function of a digital camera, characterized in that: The steps include: Step 1: Measure and obtain the spectral reflectance of the training sample; Step 2: photograph and obtain the original digital response values of the test samples under different imaging scenes, and record the camera imaging parameters; Step 3, measuring the spectral power distribution of the light source during the shooting process; Step 4: Combine the spectral reflectance of the training sample, the camera imaging parameters and the spectral power distribution of the light source, use the spectral response function optimization method to estimate the spectral response function of the digital camera, obtain the camera spectral response function, and obtain the nonlinear parameters of the digital camera full-parameter imaging model; Step 5: Based on the full-parameter imaging model of the digital camera, combined with the camera spectral response function, camera imaging parameters and nonlinear parameters, predict the original format digital response value of the training sample under any imaging conditions; Step 6: Normalize the original format response value of the test sample and the predicted training sample response value in RGB channels respectively; Step 7, combining the spectral reflectance of the training sample and the two normalized response values in step 6, and constructing a spectral reconstruction model using a spectral reconstruction algorithm; Step 8: Reconstruct the test sample using the spectrum reconstruction model to obtain the test sample spectrum.

2. The spectrum reconstruction method based on optimizing the spectral response function of a digital camera according to claim 1, characterized in that: The specific implementation of step 4 is as follows: Step 41: Collect the existing public camera spectral response function database and normalize the camera spectral response function by three channels of RGB; Step 42: combining the spectral reflectance of the training sample and the spectral power distribution of the light source, using principal component analysis to calculate the principal components of the camera spectral response function database by channel, and using a pseudo-inverse algorithm to calculate the initial spectral response function; Step 43, based on the initial digital camera imaging model, combined with the camera imaging parameters and the initial spectral response function, using a curve fitting model to obtain nonlinear parameters in the digital camera full-parameter imaging model; Step 44 : Calculate the color difference between the normalized original format response value and the predicted response value, use it as a loss function, and iteratively optimize to obtain the final camera spectral response function.

3. The spectrum reconstruction method based on optimizing the spectral response function of a digital camera according to claim 2, characterized in that: The specific implementation of step 42 is as follows: The camera spectral response is decomposed into s using principal component analysis. k,n =σ k *E k ,k=R,G,B, where the coefficient σ k =[σ k,1 ,σ k,2 ] is a 1×2 vector, E k =[e k,1 ,e k,2 ] T is a 2×31 eigenvector matrix; based on this, the initial digital camera imaging model is expressed as: in is the predicted response of the camera, g k σ k E k is the camera spectral response function before normalization, l is the spectral power distribution of the scene light source, and R is the spectral reflectance of the training sample; By solving The superscript '+' represents the pseudo-inverse, and the camera spectral response function is restored. Then the initial spectral response function S k Expressed as: Where r is the original digital response value of the training sample captured by the camera.

4. The spectrum reconstruction method based on optimizing the spectral response function of a digital camera according to claim 2, characterized in that: The specific implementation of step 43 is as follows: When the initial camera spectral response function is known, the preliminary predicted camera response can be obtained: Among them, S k is the initial spectral response function, l is the spectral power distribution of the scene light source, and R is the spectral reflectance of the training sample; The initial values of nonlinear parameters can be obtained through curve fitting model: Among them, r is the original digital response value of the training sample captured by the camera, f S is the sensitivity function, c1, c2, and β are parameters that characterize the non-equivalence between camera exposure time and sensitivity ISO.

5. The spectrum reconstruction method based on optimizing the spectral response function of a digital camera according to claim 2, characterized in that: The specific implementation of step 44 is as follows: First, convert the normalized response value of each channel from RGB color space to XYZ color space, then convert it to Lab color space, and then calculate its CIEDE2000 color difference ΔE 00 As a loss function: in, Represent the camera prediction response and the original format response of each sample respectively, and μ is an adjustable parameter, which is the maximum color difference in the sample to set the regularization constraint of the loss function.

6. The spectrum reconstruction method based on optimizing the spectral response function of a digital camera according to claim 1, characterized in that: The specific implementation of step 5 is as follows: First, the radiation spectrum distribution l(λ) of the light source is irradiated on the surface of the object, and after absorption and reflection by the object, a radiation spectrum image is formed, as shown in formula (6): L scene (x,y,λ)=∫r(x,y,λ)l(λ) (6) Where λ represents the wavelength, r(x,y,λ) is the spectral reflectance of the (x,y) pixel in the image, l(λ) represents the relative spectral radiance of the light source, and L scene (x, y, λ) represents the radiation spectrum at each pixel. Subsequently, the radiation spectrum image is focused by the camera lens and enters the sensor, forming an irradiance image before entering the sensor, as shown in formula (7): Where f / # is the effective aperture of the lens, m is the magnification of the lens, T(λ) is the transmittance of the lens, and L is the scene Represents the radiation spectrum at each pixel, I image (x, y, λ) is the irradiance at each pixel. Furthermore, the irradiance image enters the camera sensor, undergoes photoelectric conversion and analog-to-digital conversion, and obtains a Bayer pattern mosaic image. The camera's original format digital response value image of the scene is obtained through an interpolation algorithm. The process is shown in Equations (8) to (12): Q(x,y)=∫ Ω I image (x,y,λ)S k (λ)dλ (8) Where Ω is the spectral wavelength range, S k (λ) is the spectral response function of the camera, Q(x,y) is the number of photoelectric conversions per unit time at the pixel (x,y), and the total photoelectric conversion amount of the camera sensor within a given exposure time τ is defined as: φ(x,y)=f s ∫ τ Q(x,y)dt (9) Where, f s is the sensitivity function, τ is the exposure time; when the spectral irradiance and the quantum efficiency of the sensor remain unchanged during one shooting, φ(x) can be further simplified to the form shown in formula (10): φ(x,y)=f s τQ(x,y) (10) According to equations (8) and (10), the original digital response value p(x, y) at any pixel point (x, y) in the image can be expressed as: In addition, considering the nonlinear relationship between the irradiance of the camera sensor and the camera readout response, the model uses three coefficients c1, c2 and β to characterize the non-equivalence between exposure time and sensitivity ISO, and finally obtains the original format digital response value p(x,y) of any pixel point:

7. The spectrum reconstruction method based on optimizing the spectral response function of a digital camera according to claim 1, characterized in that: The method for constructing a spectral reconstruction model using a spectral reconstruction algorithm in step 7 is as follows: First, the third-order root polynomial is used to expand the normalized response values of the training and test samples. The expansion form is as follows: Where r, g, and b are the response values of the R channel, G channel, and B channel of all samples respectively, and d exp Represents a response value vector after sample expansion; Then, using the pseudo-inverse algorithm, the spectrum reconstruction matrix Q is calculated from the training sample spectral reflectance and the corresponding extended response value as follows: Q=R(D T D+λI) -1 D T Where R is the training sample spectral data matrix, D is the matrix after the training sample response value is expanded, Q is the spectral reconstruction matrix, the superscript 'T' is the transpose symbol, the superscript '-1' indicates the inversion operation, λ is the regularization constraint coefficient, I is the identity matrix, and λI is used to counteract the noise information in the solution of the spectral reconstruction matrix to prevent model overfitting.

8. The spectrum reconstruction method based on optimizing the spectral response function of a digital camera according to claim 1, characterized in that: The spectrum reconstruction method in step 8 is as follows: Where d is the root polynomial expansion vector of the original format response value of the test sample, Q is the spectrum reconstruction matrix calculated by step 7, is the spectral data vector of the reconstructed test sample.

9. The spectrum reconstruction method based on optimizing the spectral response function of a digital camera according to claim 1, characterized in that: In step 1, the spectral reflectance of the training sample is measured using a spectrophotometer; In step 2, a digital camera is used to obtain the original digital response values of the training samples under different imaging scenes; the camera imaging parameters include exposure time and ISO sensitivity; In step 3, a color illuminance meter is used to synchronously measure the spectral power distribution of the light source during the shooting process.

10. A spectrum reconstruction system based on optimizing the spectral response function of a digital camera, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement a spectral reconstruction method based on optimizing the spectral response function of a digital camera as claimed in any one of claims 1 to 9.

Images