Spectral reconstruction method and system based on optimized digital camera spectral response function

Abstract

The invention provides a spectral reconstruction method and system based on an optimized digital camera spectral response function. The method comprises the following steps: measuring the spectral reflectivity of a training sample; shooting to obtain original format response values of the test sample in different imaging scenes, and recording camera imaging parameters; synchronously measuring light source spectral power distribution of the imaging scene; then, on the basis of a digital camera full-parameter imaging model, the raw response value of a training sample set under the same imaging condition is predicted, maximum and minimum normalization is carried out on the original format response value of a test sample and the predicted training sample set response value in RGB channels, spectral characterization modeling is carried out on the camera, and the spectral characteristic of the camera is obtained; and finally, performing multispectral reconstruction on the test sample under the same imaging condition to obtain reconstructed spectral data of the test sample.

Description

Technical Field

[0001] The invention belongs to the technical field of computer digital image processing, and in particular relates to a spectrum reconstruction method and system based on optimizing the spectrum response function of a digital camera. Background Art

[0002] Multispectral images refer to images with multiple spectral channel data. Multispectral images contain both spatial and spectral information of the scene. Unlike RGB images, where one pixel includes color information of three channels, one pixel of a multispectral image can include spectral reflectance information of dozens of channels. When the light source is irradiated onto the observed object, a reflection spectrum is generated after selective absorption by the object. The reflection spectrum is independent of the observer and the light source, and is not affected by the observer and the light source. It is an inherent physical property of the observed object. Therefore, the spectrum is the fingerprint of color information, and it is also an important feature that characterizes the physical and chemical properties of an object. It has important application value in color reproduction, cultural relic protection, medical diagnosis, computer vision, remote sensing detection and other fields.

[0003] Multispectral imaging technology is based on high spatial resolution monochrome or color imaging sensors. By configuring bandpass filters, multi-primary color filter arrays or multi-color light sources, different types of imaging systems are established to record digital images of object surfaces, and multispectral images of objects are obtained through sample set optimization and spectral reconstruction calculations. It can overcome the problem of isochromaticity in color imaging technology, accurately characterize and distinguish objects, and achieve high-fidelity acquisition and replication of color information through spectral color management technology. Compared with expensive hyperspectral cameras, it has more universal application advantages. It has developed rapidly in recent decades and has become one of the important frontier technologies in current scientific and technological development. Spectral reconstruction is the key to multispectral imaging technology. It refers to reconstructing the object response signal obtained by the imaging system into its corresponding accurate spectral information. Therefore, the selection and design of the spectral reconstruction method directly determines the accuracy of obtaining multispectral images, which in turn affects its application performance in the above-mentioned fields.

[0004] However, existing spectral reconstruction methods are sensitive to changes in imaging conditions (such as light source, illumination, imaging parameters, etc.). That is, the spectral reconstruction matrix established under a certain imaging condition cannot be directly applied to other imaging conditions for spectral reconstruction. Otherwise, the reconstructed spectral curve will be deformed and lose its due spectral feature information. In addition, when multi-spectral reconstruction is applied in an open environment, the uncontrollable and uneven illumination and changes in camera imaging parameters (such as exposure time, sensitivity ISO, etc.) will cause changes in imaging conditions, thereby increasing the spectral reconstruction error and causing the reconstructed spectrum to deviate from the true value, making it impossible to apply it in practice. In response to the challenges of multi-spectral reconstruction applications in open environments with changing imaging conditions, the academic and industrial communities at home and abroad have not yet proposed a good solution. Summary of the invention

[0005] The purpose of the present invention is to solve the problem described in the background technology and to propose a spectrum reconstruction method based on optimizing the spectral response function of a digital camera.

[0006] In view of the problems existing in the above-mentioned existing research, the present invention proposes a method for solving the problems, firstly measuring the spectral reflectance of the training samples, photographing and obtaining the original format response values ​​of the test samples under different imaging scenes and recording the camera imaging parameters, synchronously measuring the spectral power distribution of the light source of the imaging scene, and using the spectral response function optimization method to estimate the spectral response function of the digital camera, and then based on the full-parameter imaging model of the digital camera, predicting the raw response values ​​of the training sample set under the same imaging conditions, and performing maximum and minimum normalization on the original format response values ​​of the test samples and the predicted training sample set response values ​​for the RGB three channels, spectrally characterizing the camera, modeling the spectral characteristics, constructing a spectral reconstruction model using a spectral reconstruction algorithm, and finally performing multi-spectral reconstruction on the test samples under the same imaging conditions to obtain the reconstructed spectrum of the test samples. The technical solution of the present invention is a spectral reconstruction method based on optimizing the spectral response function of a digital camera, which specifically includes the following steps:

[0007] Step 1, measuring and obtaining the spectral reflectance of the training sample;

[0008] Step 2, photograph and obtain the original digital response values ​​of the test samples under different imaging scenes, and record the camera imaging parameters;

[0009] Step 3, measuring the spectral power distribution of the light source during the shooting process;

[0010] Step 4, combining the above-mentioned training sample spectral reflectance, camera imaging parameters and light source spectral power distribution, using the spectral response function optimization method to estimate the spectral response function of the digital camera, obtain the camera spectral response function, and obtain the nonlinear parameters of the digital camera full-parameter imaging model;

[0011] Step 5, based on the digital camera full parameter imaging model, combined with the camera spectral response function, camera imaging parameters and nonlinear parameters, predict the raw response value of the training sample under any imaging conditions;

[0012] Step 6, normalizing the original format response value of the test sample and the predicted training sample response value by RGB three channels respectively;

[0013] Step 7, combining the spectral reflectance of the training sample and the two normalized response values ​​in step 6, and using a spectral reconstruction algorithm to construct a spectral reconstruction model;

[0014] Step 8: Reconstruct the test sample using the spectrum reconstruction model to obtain the test sample spectrum.

[0015] Furthermore, the specific implementation of step 4 is as follows;

[0016] Step 41, collecting the existing public camera spectral response function database, and normalizing the camera spectral response function by three channels of RGB;

[0017] Step 42, combining the spectral reflectance of the training sample and the spectral power distribution of the light source, using principal component analysis to calculate the principal components of the camera spectral response function database by channel, and using a pseudo-inverse algorithm to calculate the initial spectral response function;

[0018] Step 43, based on the initial digital camera imaging model, combined with the camera imaging parameters and the initial spectral response function, a curve fitting model is used to obtain nonlinear parameters in the digital camera full parameter imaging model;

[0019] Step 44, calculate the color difference between the normalized original format response value and the predicted response value, use it as a loss function, and iteratively optimize to obtain the final camera spectral response function.

[0020] Furthermore, the specific implementation of step 42 is as follows:

[0021] The camera spectral response is decomposed into s using principal component analysis. k,n =σ k *E k ,k=R,G,B, where coefficient σ k =[σ k,1 ,σ k,2 ] is a 1×2 vector, E k =[e k,1 ,e k,2 ] T is a 2×31 eigenvector matrix; based on this, the initial digital camera imaging model is expressed as:

[0022]

[0023] in is the predicted response of the camera, g k σ k E k is the camera spectral response function before normalization, l is the spectral power distribution of the scene light source, and R is the spectral reflectance of the training sample;

[0024] By solving The superscript '+' represents the pseudo-inverse, and the camera spectral response function is restored. Then the initial spectral response function S k It is expressed as:

[0025]

[0026] Where r is the original digital response value of the training sample captured by the camera.

[0027] Furthermore, the specific implementation of step 43 is as follows:

[0028] When the initial camera spectral response function is known, the preliminary predicted camera response can be obtained:

[0029]

[0030] Among them, S k is the initial spectral response function, l is the spectral power distribution of the scene light source, and R is the spectral reflectance of the training sample;

[0031] The initial values ​​of nonlinear parameters can be obtained through the curve fitting model:

[0032]

[0033] Among them, r is the original digital response value of the training sample captured by the camera, and f S is the sensitivity function, c1, c2, and β are parameters that characterize the non-equivalence between camera exposure time and ISO.

[0034] Furthermore, the specific implementation of step 44 is as follows:

[0035] First, convert the normalized response values ​​of each channel from RGB color space to XYZ color space, then convert it to Lab color space, and then calculate its CIEDE2000 color difference ΔE 00 As a loss function:

[0036]

[0037] in, Represent the camera prediction response and the original format response of each sample respectively, and μ is an adjustable parameter, which is the maximum color difference in the sample to set the regularization constraint of the loss function.

[0038] Furthermore, the specific implementation of step 5 is as follows;

[0039] First, the radiation spectrum distribution l(λ) of the light source is irradiated on the surface of the object, and after absorption and reflection by the object, a radiation spectrum image is formed, as shown in formula (6):

[0040] L scene (x,y,λ)=∫r(x,y,λ)l(λ) (6)

[0041] Where λ represents the wavelength, r(x,y,λ) is the spectral reflectance of the (x,y) pixel in the image, l(λ) represents the relative spectral radiance of the light source, and Lscene (x, y, λ) represents the radiation spectrum at each pixel. Subsequently, the radiation spectrum image is focused by the camera lens and enters the sensor, forming an irradiance image before entering the sensor, as shown in formula (7):

[0042]

[0043] Where f / # is the effective aperture number of the lens, m is the lens magnification, T(λ) is the transmittance of the lens, L(x, y, λ) represents the radiation spectrum at each pixel, and I(x, y, λ) is the irradiance at each pixel. Furthermore, the irradiance image enters the camera sensor, undergoes photoelectric conversion and analog-to-digital conversion, and obtains a Bayer pattern mosaic image. The camera raw response value image of the scene is obtained through the interpolation algorithm. The process is shown in equations (8) to (12):

[0044] Q(x,y)=∫ Ω I image (x,y,λ)S k (λ)dλ (8)

[0045] Where Ω is the spectral wavelength range, S k (λ) is the spectral response function of the camera, Q(x,y) is the number of photoelectric conversions per unit time at the pixel (x,y), and within a given exposure time τ, the total photoelectric conversion amount of the camera sensor is defined as:

[0046] φ(x,y)=f s ∫ τ Q(x,y)dt (9)

[0047] In the formula, f s is the sensitivity function, τ is the exposure time; when the spectral irradiance and the quantum efficiency of the sensor remain unchanged during one shooting, φ(x) can be further simplified to the form shown in formula (10):

[0048] φ(x,y)=f s τQ(x,y) (10)

[0049] According to equations (8) and (10), the raw response value p(x, y) at any pixel (x, y) in the image can be expressed as:

[0050]

[0051] In addition, considering the nonlinear relationship between the irradiance of the camera sensor and the camera readout response, the model uses three coefficients c1, c2 and β to characterize the non-equivalence between exposure time and ISO, and the final raw response value p(x,y) expression of any pixel point can be obtained:

[0052]

[0053] Furthermore, the method of constructing a spectral reconstruction model using a spectral reconstruction algorithm in step 7 is as follows:

[0054] First, the third-order root polynomial is used to expand the normalized response values ​​of the training samples and test samples. The expansion form is as follows:

[0055]

[0056] In the formula, r, g and b are the response values ​​of the R channel, G channel and B channel of all samples respectively, d exp Represents a response value vector after sample expansion;

[0057] Then, using the pseudo-inverse algorithm, the method for calculating the spectral reconstruction matrix Q from the spectral reflectance of the training sample and the corresponding extended response value is as follows:

[0058] Q=R(D T D+λI) -1 D T

[0059] Where R is the training sample spectral data matrix, D is the matrix after the training sample response value is expanded, Q is the spectral reconstruction matrix, the superscript 'T' is the transpose symbol, the superscript '-1' indicates the inverse operation, λ is the regularization constraint coefficient, I is the unit matrix, and λI is used to counter the noise information in the solution of the spectral reconstruction matrix to prevent the model from overfitting.

[0060] Furthermore, the spectrum reconstruction method in step 8 is as follows:

[0061]

[0062] Where d is the root polynomial expansion vector of the original format response value of the test sample, Q is the spectrum reconstruction matrix calculated by step 7, is the spectral data vector of the reconstructed test sample.

[0063] Furthermore, in step 1, the spectral reflectance of the training sample is measured using a spectrophotometer;

[0064] In step 2, a digital camera is used to obtain the original digital response values ​​of the training samples in different imaging scenes; the camera imaging parameters include exposure time and ISO sensitivity;

[0065] In step 3, a color illuminance meter is used to synchronously measure the spectral power distribution of the light source during the shooting process.

[0066] The present invention also provides a spectrum reconstruction system based on optimizing the spectrum response function of a digital camera, comprising:

[0067] one or more processors;

[0068] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement a spectral reconstruction method based on optimizing the spectral response function of a digital camera as described in the above scheme.

[0069] In view of the problem that the existing spectral reconstruction method cannot adapt to the exposure level change or uneven illumination under the same illumination source, the present invention is based on the digital camera imaging model, and by predicting the raw response value of the training sample set under any imaging condition, the camera is spectrally characterized and modeled, and then the multi-spectral reconstruction of the photographed object under the same imaging condition is performed. The method can theoretically realize the real-time spectral characteristic modeling of the camera in different open scenes, thereby reconstructing the multi-spectral image of the photographed object in the corresponding scene, and supporting various practical applications based on spectroscopy. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0071] When the technical solution of the present invention is specifically implemented, it can be operated by those skilled in the art using computer software technology.

[0072] like Figure 1 As shown, the present invention provides a spectrum reconstruction method based on optimizing the spectral response function of a digital camera, which specifically includes the following steps:

[0073] Step 1: Use a spectrophotometer to measure and obtain the spectral reflectance R of the training sample set;

[0074] Step 2: Use a digital camera to obtain the original digital response value of the test sample under different imaging scenes, and record the camera imaging parameters (such as exposure time, ISO sensitivity, etc.);

[0075] Step 3, using a color illuminance meter to synchronously measure the spectral power distribution l of the light source during the shooting process;

[0076] Step 4, combining the above-mentioned training sample spectral reflectance, camera imaging parameters and light source spectral power distribution, using the spectral response function optimization method to estimate the spectral response function of the digital camera, obtain the camera spectral response function, and obtain the nonlinear parameters of the digital camera full-parameter imaging model;

[0077] The specific method for estimating the spectral response function and nonlinear parameters of the digital camera in step 4 is as follows:

[0078] Step 41, collecting existing public camera spectral response function databases, selecting data in the 400 to 700 nanometer band, and normalizing the camera spectral response function by three channels of RGB;

[0079]

[0080] where s k,n is the normalized camera spectral response function, s k,n =[s k (400nm),s k (410nm),…,s k (700nm)],S k is the camera spectral response function before normalization, S k =g k *s k,n ,g k >0,k=R,G,B,g k is a constant of the three channels R, G, and B. R, G, and B are the R channel, G channel, and B channel of the sample, respectively. In this method, the existing camera spectral response function is normalized by channel, and the peak values ​​of all RGB channels are normalized to one.

[0081] Step 42, combining the spectral reflectance of the training sample and the spectral power distribution of the light source, using principal component analysis (PCA) to calculate the principal components of the camera spectral response function database by channel, and using a pseudo-inverse algorithm to calculate the initial spectral response function;

[0082] In step 42, principal component analysis (PCA) is used to calculate the principal components of the camera spectral response function database by channel, and the method of using the pseudo-inverse algorithm to calculate the initial spectral response function is as follows:

[0083] The camera spectral response function is decomposed into s using principal component analysis. k,n =σ k *E k ,k=R,G,B, where coefficient σ k =[σ k,1 ,σ k,2 ] is a 1×2 vector, E k =[e k,1 ,e k,2 ] T is a 2×31 eigenvector matrix. Based on this, the initial digital camera imaging model can be expressed as:

[0084]

[0085] in is the predicted response of the camera, g k σ k Ek is the camera sensitivity function before normalization, l is the spectral power distribution of the scene light source, and R is the spectral reflectance of the training sample.

[0086] By solving The superscript '+' represents the pseudo-inverse, which can restore the camera spectral response function. Then the initial spectral response function S k It is expressed as:

[0087]

[0088] Where r is the original digital response value of the training sample captured by the camera.

[0089] Step 43, based on the initial digital camera imaging model, combined with the camera imaging parameters and the initial spectral response function, a curve fitting model is used to obtain nonlinear parameters in the digital camera full parameter imaging model;

[0090] In step 43, based on the initial digital camera imaging model, the method of using the curve fitting model to obtain the nonlinear parameters in the digital camera full parameter imaging model is as follows:

[0091] When the initial camera spectral response function is known, the preliminary predicted camera response can be obtained:

[0092]

[0093] The initial values ​​of nonlinear parameters can be obtained through the curve fitting model:

[0094]

[0095] Among them, r is the original digital response value of the training sample captured by the camera, and f S It is a sensitivity function, usually obtained by dividing the ISO value by 100. c1, c2, and β are parameters that characterize the non-equivalence between camera exposure time and ISO.

[0096] Step 44, calculate the color difference between the normalized original format response value and the predicted response value, use it as a loss function, and use a sequential quadratic programming method to iteratively optimize to obtain the final camera spectral response function.

[0097] The loss function definition and optimization method in step 44 are as follows:

[0098] First, the original format response value after channel normalization is converted from RGB color space to XYZ color space, and then converted to Lab color space, and then the CIEDE2000 color difference (ΔE 00 ) as the loss function:

[0099]

[0100] in, Represent the camera prediction response and the original format response of each sample. Since we selected ColorChecker SG140 as the training sample, the total number of samples N is 140. μ is an adjustable parameter, which is the maximum color difference in the sample to set the regularization constraint of the loss function. This method is empirically set to 0.1.

[0101] Our goal is to find the camera sensitivity function S that minimizes the loss function k (λ) and nonlinear parameters. Since the full-parameter imaging model of digital cameras is in the form of a power function, it is difficult to directly calculate its gradient during the optimization process. Therefore, we choose the sequential quadratic programming method to solve this constrained nonlinear optimization problem.

[0102] Step 5, based on the digital camera full parameter imaging model, combined with the camera spectral response function, camera imaging parameters and nonlinear parameters, predict the raw response value of the training sample set under any imaging conditions;

[0103] In step 5, based on the full-parameter imaging model of the digital camera, the method for predicting the raw response value of the training sample set under any imaging condition is as follows:

[0104] First, the radiation spectrum distribution l(λ) of the light source is irradiated on the surface of the object, and after absorption and reflection by the object, a radiation spectrum image is formed, as shown in formula (6):

[0105] L scene (x,y,λ)=∫r(x,y,λ)l(λ) (6)

[0106] Where λ represents the wavelength, r(x,y,λ) is the spectral reflectance of the (x,y) pixel in the image, l(λ) represents the relative spectral radiance of the light source, and L scene (x, y, λ) represents the radiation spectrum at each pixel. Subsequently, the radiation spectrum image is focused by the camera lens and enters the sensor, forming an irradiance image before entering the sensor, as shown in formula (7):

[0107]

[0108] In the formula, f / # is the effective aperture number of the lens, m is the lens magnification, T(λ) is the transmittance of the lens, L(x, y, λ) represents the radiation spectrum at each pixel, and I(x, y, λ) is the irradiance at each pixel. Further, the irradiance image enters the camera sensor, undergoes photoelectric conversion and analog-to-digital conversion, and obtains a Bayer pattern mosaic image. The camera raw response value image of the scene is obtained through the interpolation algorithm. The process is shown in equations (8) to (12):

[0109] Q(x,y)=∫ Ω I image (x,y,λ)S k (λ)dλ (8)

[0110] Where Ω is the spectral wavelength range, which is 400-700nm in this study, and S k (λ) is the spectral response function of the camera, Q(x,y) is the number of photoelectric conversions per unit time at the pixel (x,y), and within a given exposure time τ, the total photoelectric conversion amount of the camera sensor is defined as:

[0111] φ(x,y)=f s ∫ τ Q(x,y)dt (9)

[0112] In the formula, f s is the sensitivity function, usually obtained by dividing the ISO value by 100, and τ is the exposure time. When the spectral irradiance and the quantum efficiency of the sensor remain unchanged during a single shot, φ(x) can be further simplified to the form shown in equation (10):

[0113] φ(x,y)=f s τQ(x,y) (10)

[0114] According to equations (8) and (10), the raw response value p(x, y) at any pixel (x, y) in the image can be expressed as:

[0115]

[0116] In addition, considering the nonlinear relationship between the irradiance of the camera sensor and the camera readout response, the model uses three constants c1, c2 and β to characterize the non-equivalence between exposure time and ISO, and the final raw response value p(x,y) expression of any pixel point can be obtained.

[0117]

[0118] Step 6, normalizing the original format response value of the test sample and the predicted training sample set response value by RGB three channels respectively;

[0119] The channel normalization method is shown in formula (13):

[0120]

[0121] In the formula, R, G, and B are the R channel, G channel, and B channel of the sample respectively. Represents a sample channel-normalized response value vector.

[0122] Step 7: Combining the spectral reflectance of the training samples with the predicted response values ​​of the training samples, a spectral reconstruction model is constructed using a spectral reconstruction algorithm.

[0123] The method of constructing a spectral reconstruction model using a spectral reconstruction algorithm in step 7 is as follows:

[0124] First, the third-order root polynomial is used to expand the normalized response values ​​of the training samples and test samples. The expansion form is shown in the following formula:

[0125]

[0126] In the formula, r, g and b are the response values ​​of the R channel, G channel and B channel of all samples respectively, d exp Represents a vector of response values ​​after sample expansion.

[0127] Then, using the pseudo-inverse algorithm, the method for calculating the spectral reconstruction matrix Q from the spectral reflectance of the training sample and the corresponding extended response value is as follows:

[0128] Q=R(D T D+λI) -1 D T

[0129] Where R is the spectral data matrix of the training samples, D is the matrix after the original format response value of the training samples is expanded, Q is the spectral reconstruction matrix, the superscript 'T' is the transpose symbol, the superscript '-1' indicates the inverse operation, λ is the regularization constraint coefficient, I is the unit matrix, λI is used to combat the noise information in the solution of the spectral reconstruction matrix to prevent the model from overfitting, and the value of λ is usually 0.001.

[0130] Step 8: Reconstruct the test sample using the spectrum reconstruction model to obtain the test sample spectrum.

[0131] The spectrum reconstruction method in step 8 is as follows:

[0132]

[0133] Where d is the root polynomial expansion vector of the original format response value of the test sample, Q is the spectrum reconstruction matrix calculated by step 7, is the spectral data vector of the reconstructed test sample.

[0134] The following is a specific example to illustrate the embodiment of the present invention.

[0135] The embodiment uses a Nikon D7200 digital camera, an EVERFINE SPIC-300AW spectral color illuminance meter, and an X-rite i1-Pro3 spectrophotometer as experimental instruments, and conducts experiments in a laboratory environment and an outdoor open environment. In the specific implementation, the ColorChecker SG140 (CCSG) color card is used as a training sample, and the ColorChecker 24 (CC) color card is used as a test sample.

[0136] In the embodiment, in a laboratory environment, based on a closed uniform lighting light box, a Nikon D7200 digital camera is used to shoot a CC color card digital image, and the shooting raw response value of the color card is extracted. During the shooting process, the color card is placed in the center of the shooting field of view, where the color card plane is about 1 meter away from the camera sensor plane, and the camera focal length is 35 mm. The experiment shoots the CC color card under 5 different exposure and ISO combinations (as shown in Table 1), and the color card images under each exposure combination condition have no overexposure problem to ensure the validity of the experimental data.

[0137] Table 1 Five exposure settings for laboratory environment experiments

[0138]

[0139] In an open outdoor environment, the CC color card was also photographed with a Nikon D7200 digital camera, and the raw response value data of the color card was extracted. In addition, in the open environment experiment, 5 different exposure and ISO combinations were also set, as shown in Table 2. Among them, the color card image under the first exposure and ISO combination was slightly overexposed, and the color card images under the other exposure and ISO combinations were not overexposed. During the shooting process, the color card was placed in the center of the shooting field of view, where the color card plane was about 3 meters away from the camera sensor plane, and the camera focal length was 22 mm. The rest of the experimental conditions and the color card raw response value prediction method were consistent with the laboratory environment.

[0140] Table 2 Five exposure settings for the open environment experiment

[0141]

[0142] In the experiment, the relative spectral power distribution of the light source was measured using an EVERFINE SPIC-300AW spectral color illuminance meter. The spectral reflectance of the color card was measured using an X-rite i1-Pro3 spectrophotometer. Given the known spectral reflectance of the training sample, the spectral response function of the Nikon D7200 camera, the spectral power distribution of the light source, and the imaging parameters, the response value prediction model can be used to predict the raw response data of the CCSG color card under the corresponding imaging conditions, which can be expressed as follows:

[0143]

[0144] In the formula, f s is the sensitivity function, which is usually obtained by dividing the ISO value by 100, τ is the exposure time, Ω is the spectral wavelength range, which is 400-700nm in this study, f / # is the effective aperture number of the lens, and m is the lens magnification. represents the radiation spectrum at each pixel, S k (λ) is the camera spectral response function of the camera.

[0145] The original format response value of the test sample and the response value of the predicted training sample set are normalized respectively. The specific normalization method is as follows:

[0146]

[0147] In the formula, R, G, and B are the R channel, G channel, and B channel of the sample respectively. Represents a sample channel-normalized response value vector.

[0148] After obtaining the normalized predicted training sample set response value, the training sample set spectral reflectance and the test sample original format response value, the existing spectral reconstruction algorithm can be used to construct a spectral reconstruction matrix, and the test sample can be reconstructed to obtain a reconstructed spectrum.

[0149] In the embodiment, the spectral root mean square error (RMSE) and CIEDE2000 color difference (ΔE 00 ) is used as the evaluation index of spectral estimation, and the performance of the method of the present invention is evaluated from the perspectives of spectral error and chromaticity error. The calculation method of spectral root mean square error RMSE is as follows:

[0150]

[0151] Wherein, r1 is the spectral reflectance obtained by measurement, r2 is the spectral reflectance obtained by reconstruction, the superscript T is the transposition operator, and N is the number of sampling wavelengths, which is 31 in the embodiment of the present invention.

[0152] Table 3 Spectral reconstruction errors under five different imaging conditions in the laboratory environment

[0153]

[0154] Table 4 Spectral reconstruction errors under five different imaging conditions in an open environment

[0155]

[0156] Tables 3 and 4 respectively give the comparison of the method of the present invention using four different spectral reconstruction algorithms in a laboratory environment and an open environment, where 'Liang', 'OLS', 'Cao', and 'Kernal' represent the spectral reconstruction algorithms proposed in references [1] to [4] respectively.

[0157] [1]Liang J,Wan X.Optics Express,2017,25.23:28273-28287.

[0158] [2]Connah DR,Hardeberg J Y.Color Imaging X:Processing,Hardcopy,andApplications.SPIE,2005,5667:65-75.

[0159] [3]Cao B, Liao N, Cheng H. Color Research & Application, 2017, 42.3: 327-332.

[0160] [4]Xu P,Xu H,Diao C.Applied optics,2017,56.3:8461-8470.

[0161] It can be seen intuitively from the evaluation index values ​​in the table that the method of the present invention has achieved satisfactory reconstruction accuracy both in the laboratory environment and in the open environment.

[0162] On the other hand, an embodiment of the present invention further provides a spectrum reconstruction system based on optimizing the spectral response function of a digital camera, comprising:

[0163] one or more processors;

[0164] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement a spectral reconstruction method based on optimizing the spectral response function of a digital camera as described in the above scheme.

[0165] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A spectrum reconstruction method based on optimizing the spectral response function of a digital camera, characterized in that: The steps include: Step 1, measuring and obtaining the spectral reflectance of the training sample; Step 2, photograph and obtain the original digital response values ​​of the test samples under different imaging scenes, and record the camera imaging parameters; Step 3, measuring the spectral power distribution of the light source during the shooting process; Step 4, combining the above-mentioned training sample spectral reflectance, camera imaging parameters and light source spectral power distribution, using the spectral response function optimization method to estimate the spectral response function of the digital camera, obtain the camera spectral response function, and obtain the nonlinear parameters of the digital camera full-parameter imaging model; Step 5, based on the digital camera full parameter imaging model, combined with the camera spectral response function, camera imaging parameters and nonlinear parameters, predict the raw response value of the training sample under any imaging conditions; Step 6, normalizing the original format response value of the test sample and the predicted training sample response value by RGB three channels respectively; Step 7, combining the spectral reflectance of the training sample and the two normalized response values ​​in step 6, and using a spectral reconstruction algorithm to construct a spectral reconstruction model; Step 8: Reconstruct the test sample using the spectrum reconstruction model to obtain the test sample spectrum.

2. A spectrum reconstruction method based on optimizing the spectral response function of a digital camera as claimed in claim 1, characterized in that: The specific implementation of step 4 is as follows; Step 41, collecting the existing public camera spectral response function database, and normalizing the camera spectral response function by three channels of RGB; Step 42, combining the spectral reflectance of the training sample and the spectral power distribution of the light source, using principal component analysis to calculate the principal components of the camera spectral response function database by channel, and using a pseudo-inverse algorithm to calculate the initial spectral response function; Step 43, based on the initial digital camera imaging model, combined with the camera imaging parameters and the initial spectral response function, a curve fitting model is used to obtain nonlinear parameters in the digital camera full parameter imaging model; Step 44, calculate the color difference between the normalized original format response value and the predicted response value, use it as a loss function, and iteratively optimize to obtain the final camera spectral response function.

3. The spectrum reconstruction method based on optimizing the spectral response function of a digital camera according to claim 1, characterized in that: The specific implementation of step 42 is as follows; The camera spectral response is decomposed into s using principal component analysis. k，n =σ k *E k , k = R, G, B, where the coefficient σ k =[σ k，1 , σ k，2 ] is a 1×2 vector, E k =[e k，1 , e k，2 ] T is a 2×31 eigenvector matrix; based on this, the initial digital camera imaging model is expressed as: in is the predicted response of the camera, g k σ k E k is the camera spectral response function before normalization, l is the spectral power distribution of the scene light source, and R is the spectral reflectance of the training sample; By solving k=R,G,B,where the superscript '+' represents pseudo-inverse, and the camera spectral response function is restored. Then the initial spectral response function S k It is expressed as: Where r is the original digital response value of the training sample captured by the camera.

4. The spectrum reconstruction method based on optimizing the spectral response function of a digital camera according to claim 1, characterized in that: The specific implementation of step 43 is as follows; When the initial camera spectral response function is known, the preliminary predicted camera response can be obtained: Among them, S k is the initial spectral response function, l is the spectral power distribution of the scene light source, and R is the spectral reflectance of the training sample; The initial values ​​of nonlinear parameters can be obtained through the curve fitting model: Among them, r is the original digital response value of the training sample captured by the camera, and f S is the sensitivity function, c1, c2, and β are parameters that characterize the non-equivalence between camera exposure time and ISO.

5. The spectrum reconstruction method based on optimizing the spectral response function of a digital camera according to claim 1, characterized in that: The specific implementation of step 44 is as follows; First, convert the normalized response values ​​of each channel from RGB color space to XYZ color space, then convert it to Lab color space, and then calculate its CIEDE2000 color difference ΔE 00 As a loss function: in, p represents the camera prediction response and the original format response of each sample, and μ is an adjustable parameter, which is the maximum color difference in the sample to set the regularization constraint of the loss function.

6. The spectrum reconstruction method based on optimizing the spectral response function of a digital camera according to claim 1, characterized in that: The specific implementation of step 5 is as follows; First, the radiation spectrum distribution l(λ) of the light source is irradiated on the surface of the object, and after absorption and reflection by the object, a radiation spectrum image is formed, as shown in formula (6): L scene (x,y,λ)=∫r(x,y,λ)l(λ) (6) Where λ represents the wavelength, r(x, y, λ) is the spectral reflectance of the pixel (x, y) in the image, l(λ) represents the relative spectral radiance of the light source, and L scene (x, y, λ) represents the radiation spectrum at each pixel. Subsequently, the radiation spectrum image is focused by the camera lens and enters the sensor, and forms an irradiance image before entering the sensor, as shown in formula (7): Where f / # is the effective aperture number of the lens, m is the lens magnification, T(λ) is the transmittance of the lens, L(x, y, λ) represents the radiation spectrum at each pixel, and I(x, y, λ) is the irradiance at each pixel. Furthermore, the irradiance image enters the camera sensor, undergoes photoelectric conversion and analog-to-digital conversion, and obtains a Bayer pattern mosaic image. The camera raw response value image of the scene is obtained through the interpolation algorithm. The process is shown in equations (8) to (12): Q(x,y)=∫ Ω I image (x,y,λ)S k (λ)dλ (8) Where Ω is the spectral wavelength range, S k (λ) is the spectral response function of the camera, Q(x, y) is the number of photoelectric conversions per unit time at the pixel (x, y), and within a given exposure time τ, the total photoelectric conversion amount of the camera sensor is defined as: φ(x,y)=f s ∫ τ Q(x,y)dt (9) In the formula, f s is the sensitivity function, τ is the exposure time; when the spectral irradiance and the quantum efficiency of the sensor remain unchanged during one shooting, φ(x) can be further simplified to the form shown in formula (10): φ(x,y)=f s τQ(x,y) (10) According to equations (8) and (10), the raw response value p(x, y) at any pixel (x, y) in the image can be expressed as: In addition, considering the nonlinear relationship between the irradiance of the camera sensor and the camera readout response, the model uses three coefficients c1, c2 and β to characterize the non-equivalence between exposure time and ISO, and the final raw response value p(x, y) expression of any pixel point can be obtained:

7. The spectrum reconstruction method based on optimizing the spectral response function of a digital camera according to claim 1, characterized in that: The method of constructing a spectral reconstruction model using a spectral reconstruction algorithm in step 7 is as follows: First, the third-order root polynomial is used to expand the normalized response values ​​of the training samples and test samples. The expansion form is as follows: In the formula, r, g and b are the response values ​​of the R channel, G channel and B channel of all samples respectively, d exp Represents a response value vector after sample expansion; Then, using the pseudo-inverse algorithm, the method for calculating the spectral reconstruction matrix Q from the spectral reflectance of the training sample and the corresponding extended response value is as follows: Q=R(D T D+λI)-1D T Where R is the training sample spectral data matrix, D is the matrix after the training sample response value is expanded, Q is the spectral reconstruction matrix, the superscript 'T' is the transpose symbol, the superscript '-1' indicates the inverse operation, λ is the regularization constraint coefficient, I is the unit matrix, and λ is used to counter the noise information in the solution of the spectral reconstruction matrix to prevent the model from overfitting.

8. The spectrum reconstruction method based on optimizing the spectral response function of a digital camera according to claim 1, characterized in that: The spectrum reconstruction method in step 8 is as follows: Where d is the root polynomial expansion vector of the original format response value of the test sample, Q is the spectrum reconstruction matrix calculated by step 7, is the spectral data vector of the reconstructed test sample.

9. The spectrum reconstruction method based on optimizing the spectral response function of a digital camera according to claim 1, characterized in that: In step 1, the spectral reflectance of the training sample is measured using a spectrophotometer; In step 2, a digital camera is used to obtain the original digital response values ​​of the training samples in different imaging scenes; the camera imaging parameters include exposure time and ISO sensitivity; In step 3, a color illuminance meter is used to synchronously measure the spectral power distribution of the light source during the shooting process.

10. A spectrum reconstruction system based on optimizing the spectral response function of a digital camera, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement a spectral reconstruction method based on optimizing the spectral response function of a digital camera as described in any one of claims 1 to 9.

Images