Color correction method based on physical imaging process and spectral power distribution embedding

Abstract

The invention discloses a color constancy method based on physical imaging characteristics and light source spectral power embedding. According to the method, the consistency of image colors under different illumination conditions is ensured through a series of fine steps. The method comprises the following steps of: firstly, rendering spectral data by adopting a sunlight and ambient light dual-light-source model, converting the spectral data into an RGB space through a CIE standard observer function, and generating spectrums under different color temperatures and corresponding RGB data of the spectrums; then, a characteristic spectrum of a non-shadow area in the scene is extracted to serve as input of a variational auto-encoder, and hidden layer distribution of the auto-encoder guides a light source decoder to output light source spectrum power distribution matched with the label. In addition, spectral features and color and texture features of an RGB image are extracted through a convolutional neural network, and the features are fused to estimate the chromaticity of an image light source. And finally, according to the estimated chromaticity of the light source, performing color correction on the original RGB image by using a colorimetric principle, so that the image color adapts to a standard light source.

Description

Technical Field

[0001] The invention relates to a color correction method based on physical imaging process and spectral power distribution embedding, and belongs to the field of computer vision. Background Art

[0002] Color constancy is a fundamental phenomenon in human vision that allows us to perceive the color of objects as relatively unchanged despite extremely varying lighting conditions. This perceived stability is critical to our daily visual experience as it helps us identify objects and make informed judgments based on their color. Furthermore, color variations not only improve the quality of images for visual consumption, but also greatly impact all downstream vision tasks that rely on images, such as image segmentation and object recognition, which are very sensitive to the color temperature of the input images.

[0003] In the field of image processing and computer vision, replicating the human ability to maintain color constancy is critical for a variety of applications, including digital photography, filmmaking, and various forms of visual communication. These fields place great emphasis on color fidelity. By improving color correction techniques, we can better simulate and enhance people's natural perception of color, thereby improving the overall visual quality of images.

[0004] The current mainstream color correction algorithms for image white balance mainly rely on RGB images for analysis. RGB images can provide spatial information of the scene as well as color information of color cast. However, estimating the ambient light source from the three color values ​​of an RGB image is an ill-posed problem, so it needs to be based on some strict prior assumptions. Since these assumptions often oversimplify the actual situation, algorithms based on RGB images usually perform poorly in terms of generalization. Especially in complex outdoor environments with multiple light sources, the results of these algorithms are often unsatisfactory. Spectral images can provide more information than RGB images due to their higher sampling frequency domain, especially in multi-light source environments. Existing color correction methods still mainly use spectral data to generate multi-color temperature data sets. Based on the Retinex theory, the spectral reflectance and the spectrum of a single light source are multiplied to obtain data sets of different color temperatures. From the perspective of the imaging process, the data generated in this way will not conform to the physical laws of the real environment.

[0005] For complex environments with multiple light sources, existing light source estimation and color correction methods have the following problems:

[0006] (1) The dataset generated by the single light source assumption does not conform to the real scene: The existing light source simulation method does not consider the physical rationality of the image in the real imaging process such as multiple light sources and light source incident angle. The obtained image is inconsistent with the physical laws of the real scene and is not sufficient to reflect the diversity of the real complex environment, resulting in weak adaptability of the algorithm.

[0007] (2) Based on strict prior constraints or relying on large amounts of data training: Existing methods mainly analyze based on RGB data and ignore problems that arise in the real physical imaging process, such as multi-light source mixing, spectral stability, and light source incident angle. This results in insufficient generalization ability and makes it difficult to solve the problem of multi-light source scenes. Summary of the invention

[0008] In response to the problems existing in the existing white balance color correction algorithms in complex outdoor scenes with multiple light sources, the present invention proposes a complete algorithm framework for color constancy from data rendering to light source estimation, which includes: firstly, light source analysis and modeling are performed for the monitoring viewing angle environment, and a sunlight-ambient light dual light source model is established to generate multi-color temperature data that is more in line with physical properties. Then, the narrow band characteristics of the spectral data are fully utilized, and the mapping relationship between the spectral power distribution of the light source and the color temperature is used as a priori to estimate the light source power. Relying on the light source estimation results and RGB images, the spectral features and texture features are integrated to estimate the chromaticity of the light source in the complex outdoor environment, and the color of the image under the standard light source is restored in combination with the principles of colorimetry. Specifically including:

[0009] By modeling and analyzing the physical imaging process, the present invention proposes a sunlight-ambient light dual light source model to address the problem that the existing multi-color temperature data rendering method does not take complex light sources into consideration. Since the main light sources for outdoor scenes are sunlight and sky ambient light, the color temperature of sunlight is affected by the sun's altitude angle, and ambient light is affected by the weather, the color deviation spectral data and color deviation RGB data are combined according to the situations that may occur in a real open environment to form color deviation spectral data and color deviation RGB data that are more in line with physical characteristics.

[0010] In order to solve the color correction problem in complex multi-light source scenes, the present invention proposes a light source chromaticity estimation method based on spectral data. This method takes advantage of the high sampling frequency of spectral data and its high sensitivity to light source changes, and incorporates the strong correlation between spectral images and light source spectral power distribution into the light source estimation process. By using a variational autoencoder, the present method extracts latent representations from spectral data, and decodes part of the latent representations into light source spectral power distribution through iterative training. Then, the light source chromaticity is estimated by combining the light source spectral features and the color and texture features provided by the RGB image. Finally, the present invention realizes color correction by converting the image color from the estimated actual light source conditions to the standard D65 light source.

[0011] According to one aspect of the present invention, a color correction method based on physical imaging process and spectral power distribution embedding is provided, which is characterized by comprising the following steps:

[0012] A) Using a dual light source model of sunlight and ambient light based on a physical imaging process to render a spectral reflectance dataset, we obtain color shift spectral data of different scenes. Then, we convert these spectral data into RGB space using the CIE standard observer function, thereby obtaining color shift RGB data at different color temperatures.

[0013] B) extracting a characteristic spectrum of a non-shadow area in the color-shifted spectrum data, and using the characteristic spectrum as input data of a variational autoencoder, wherein the hidden layer distribution of the variational autoencoder is used to guide a light source decoder to output a light source spectral power distribution that matches the input spectrum;

[0014] C) Through Figure 6 The convolutional network extracts the spectral features obtained in step C and the color and texture features of the color-shifted RGB data obtained in step A, respectively, fuses the two features and estimates the chromaticity of the light source of the image;

[0015] D) using the light source chromaticity estimated in step C, correcting the color of the color-shifted RGB data so that its color adapts to the standard light source;

[0016] in:

[0017] Step A) comprises:

[0018] A1) First, define two main types of light sources: sunlight and ambient light. The color temperature of sunlight is about 2000K to 3000K in the morning and evening, and 5000K to 6500K at noon. The color temperature of ambient light is 6000K to 7500K on cloudy and overcast days, and 5000K on sunny days.

[0019] A2) Based on the color temperature of the selected light source, the corresponding light source spectrum data is rendered using a physical model. Outdoor natural light sources of different color temperatures are characterized by Planck's radiation law of blackbody radiation of different color temperatures:

[0020]

[0021] Where B(λ,T) is the radiation intensity at temperature T and wavelength λ, h is Planck's constant, c is the speed of light, k is the Boltzmann constant,

[0022] A3) Combine sunlight and ambient light in different proportions and render them onto the spectral reflectance dataset to obtain the color-shift spectrum data of the new scene. When rendering sunrise and / or sunset scenes, in order to avoid data that contradicts the visual effects in the generated and real environments, the color temperature range of sunlight is limited to between 2000K and 4000K, that is:

[0023] S new (λ)=αS s (λ,T)+βS e (λ,T)

[0024] α+β=1

[0025] I(λ)=S new (λ)·R(λ)

[0026] Where α and β are proportional factors, S s (λ,T) and S e (λ,T) are the power distribution of sunlight ambient light at wavelength λ and color temperature T, S new (λ) is the spectrum of the new light source, R(λ) is the reflectance in the scene, and I(λ) is the color shift spectrum data under the combined light source.

[0027] A4) The color deviation spectrum data obtained by rendering is converted into RGB color space, which is completed by CIE standard observer function and XYZ color space to RGB color space conversion. First, the spectrum data is converted into XYZ values ​​using CIE standard observer function, and then the XYZ to RGB conversion matrix is ​​used to map the XYZ values ​​to RGB space:

[0028]

[0029]

[0030] Where λ represents the wavelength. Here, we take the visible light band from 400nm to 720nm, with 33 bands at intervals of 10nm.

[0031] I(λ) is the spectrum data rendered in A3, z(λ) is the chromaticity matching function of the CIE standard observer, corresponding to the three coordinates of the XYZ color space,

[0032] X, Y, and Z represent the mapping of spectral data to XYZ color space values, and R, G, and B represent the mapping of XYZ space to RGB space values.

[0033] Step B) comprises:

[0034] B1) Extract the data of the spectral image in the visible light red, green, and blue bands and convert them into the hue, saturation, and brightness HSV space, and use the calculation of the normalized saturation value difference index NSVDI to extract the non-shadow area. The NSVDI calculation method is as follows:

[0035]

[0036] Where S and V represent the saturation and brightness in the HSV space. Through experiments, the threshold for extracting non-shadow areas using NSVDI is 0.2.

[0037] B2) Input the extracted non-shadow area spectrum into the variational autoencoder VAE, and use reconstruction iterative optimization to capture the potential representation of the spectral data.

[0038] B3) Input the potential representation captured by VAE into the light source spectrum decoder, and use the mean square error MSE as the loss function to guide the decoder to output the correct light source spectral power distribution.

[0039] Step D) comprises:

[0040] D1) Convert the light source chromaticity calculated in step C to XYZ space, and calculate the color correction factor for correcting the current light source to the standard D65 light source:

[0041]

[0042] Where S X , S Y , S Z are the color correction factors of X, Y, and Z respectively, X D65 , Y D65 , Z D65 are the XYZ values ​​of the standard D65 light source, X source , Y source , Z source are the light source chromaticity calculated in step C,

[0043] D2) Convert the color-shifted RGB data obtained in step A to XYZ space and apply a color correction factor, and finally convert it back to RGB space to obtain an image under a standard light source.

[0044]

[0045] Where XYZ represents the value of the color-biased image converted to the XYZ space. correct is the corrected value. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the modeling of the physical imaging process of a typical outdoor scene.

[0047] Figure 2 a-2l is a schematic diagram of the RGB space of the same scene rendered with different settings of the sunlight-ambient light dual light source model and the spectral curves of the same pixel under different settings.

[0048] Figure 3 This is a schematic diagram of the impact of light source and color temperature on the final image chromaticity.

[0049] Figure 4 a is the original image, and 4b and 4c are schematic diagrams of S (saturation) and V (value) of the original image converted to the HSV space.

[0050] Figure 5 It is a schematic diagram of the characteristic spectrum extraction and light source spectrum estimation process.

[0051] Figure 6 It is the structure diagram of the chromaticity estimation model.

[0052] Figure 7 a-7d is a comparison of the light source spectrum estimation results for different scenes. DETAILED DESCRIPTION

[0053] According to one aspect of the present invention, a color correction method based on physical imaging process and spectral power distribution embedding is provided, which is characterized by comprising the following steps:

[0054] A) Use a dual light source model of sunlight and ambient light based on the physical imaging process to render the spectral reflectance dataset and obtain the color deviation spectral data of different scenes. Then, apply the CIE standard observer function to convert these spectral data into RGB space to obtain the color deviation RGB data under different color temperatures;

[0055] B) extracting characteristic spectra of non-shadow areas in the color-shifted spectrum data, and using the characteristic spectrum as input data of a variational autoencoder, wherein the hidden layer distribution of the variational autoencoder is used to guide a light source decoder to output a light source spectral power distribution that matches the color-shifted spectrum data;

[0056] C) Through Figure 6 The convolutional neural network shown extracts the spectral features of the light source spectral power distribution obtained in step B and the color and texture features of the color-shifted RGB data obtained in step A, respectively, fuses the two features and estimates the light source chromaticity of the image;

[0057] D) using the light source chromaticity estimated in step C, correcting the color of the color-shifted RGB data so that its color adapts to the standard light source;

[0058] in:

[0059] Step A) comprises:

[0060] A1) Basis Figure 1 The sensor imaging process model shown in the figure first defines two main types of light sources: sunlight and ambient light, each with a specific color temperature range. The color temperature of sunlight varies with the sun's altitude angle, ranging from 2000K to 3000K at sunrise and sunset to 4800K to 5800K at noon. Ambient light ranges from 10000K to 15000K depending on different natural conditions such as cloudy, blue sky scattering, etc. The relationship between the light source, color temperature, and image chromaticity is shown in Figure 3 As shown,

[0061] A2) Based on the color temperature of the selected light source, the corresponding light source spectrum data is rendered using a physical model. Outdoor natural light sources of different color temperatures are characterized by Planck's radiation law of blackbody radiation of different color temperatures:

[0062]

[0063] Among them, B(λ,T) is the radiation intensity at temperature T and wavelength λ, h is Planck's constant, c is the speed of light, and k is the Boltzmann constant. According to the setting of the CIE standard observer function, the visible light band is 400nm to 720nm, with 33 bands at intervals of 10nm.

[0064] A3) Combine sunlight and ambient light in different proportions and render to the spectral reflectance dataset to obtain new color-shifted spectral data. When rendering sunrise / sunset scenes, in order to avoid data that contradicts the visual effects in the generated and real environments, the color temperature range of sunlight is limited to 2000K to 4000K. The specific method is as follows:

[0065] S new (λ)=αS s (λ,T)+βS e (λ,T)

[0066] α+β=1

[0067] I(λ)=S new (λ)·R(λ)

[0068] Where α and β are proportional factors, S s (λ,T) and S e (λ,T) are the power distribution of sunlight ambient light at wavelength λ and color temperature T, and (λ,T) are the power distribution of ambient light at wavelength S new (λ) is the spectrum of the new light source, R(λ) is the reflectance in the scene, and I(λ) is the color shift spectrum data under the combined light source.

[0069] A4) The color deviation spectrum data obtained by rendering is converted into RGB color space, which is completed by CIE standard observer function and XYZ color space to RGB color space conversion. First, the spectrum data is converted into XYZ values ​​using CIE standard observer function, and then the XYZ to RGB conversion matrix is ​​used to map the XYZ values ​​to RGB space:

[0070]

[0071]

[0072] Where λ represents the wavelength. Here, we take the visible light band from 400nm to 720nm, with 33 bands at intervals of 10nm.

[0073] I(λ) is the spectrum data rendered in A3, z(λ) is the chromaticity matching function of the CIE standard observer, corresponding to the three coordinates of the XYZ color space,

[0074] X, Y, and Z represent the mapping of spectral data to XYZ color space values, and R, G, and B represent the mapping of XYZ space to RGB space values.

[0075] Figure 2 (a)-2(c), 2(e)-2(g), and 2(i)-2(k) are the color shift RGB visualization images of the same scene under different light source settings. Figure 2 (d), 2(h), and 2(l) are schematic diagrams of the spectral curves corresponding to the same pixel under different settings above. The curves have marked the corresponding scene settings.

[0076] Step B) comprises:

[0077] B1) Extract the data of the spectral image in the visible light red, green, and blue bands and convert them into the HSV color space, and use the calculation of the normalized saturation value difference index (NSVDI) to extract the non-shadow area. Its principle is that in the non-shadow area, the difference between S and V is usually small, such as Figure 4 a shows the original image. In the shadow area of ​​the same material, Figure 4 c shows that the brightness V will be significantly reduced, and Figure 4 The decrease in saturation S shown in b is relatively small, resulting in a larger SV. The NSVDI is calculated as follows:

[0078]

[0079] Where S and V represent the saturation and brightness in the HSV space. Through experiments, the threshold for extracting non-shadow areas using NSVDI is 0.2.

[0080] B2) First, the spectral data of the non-shadow area is extracted from the color-shifted spectral data. These data are then input into a variational autoencoder VAE for processing. VAE is a powerful deep learning model that is specifically designed to learn the potential representation of data. In specific operations, the encoder part of VAE first compresses the high-dimensional spectral data into a low-dimensional latent space. Next, through an iterative reconstruction process, VAE optimizes these latent representations, trying to minimize the difference between the input data and the reconstructed data while ensuring the continuity and integrity of the latent space.

[0081] When the model converges, it can be considered that the captured potential representation reflects the core characteristics of the original data. Figure 2As shown in Figure 1, there is a strong correlation between the spectral data and the light source spectrum, which indicates that the potential representation extracted by VAE can clearly show the reflectance characteristics of the light source in different bands. This process not only reveals the inherent structure of the data, but also helps to understand and explain the physical properties behind the spectral data.

[0082] B3) After extracting and optimizing the latent representations, some of the latent representations are input into the light source spectrum decoder, and the mean square error (MSE) loss function is used as part of supervised learning to optimize these latent representations to ensure that they can accurately capture the core reflectance characteristics of the input data. Thanks to the fact that these representations have effectively expressed the characteristics of the spectral data, the decoder can efficiently and accurately reconstruct the light source spectrum. The algorithm flow is as follows: Figure 5 As shown,

[0083] Step D) comprises:

[0084] D1) Convert the light source chromaticity calculated in step C to XYZ space, and calculate the color correction factor for correcting the current light source to the standard D65 light source:

[0085]

[0086] Where S X , S Y , S Z are the color correction factors of X, Y, and Z respectively, X D65 , Y D65 , Z D65 are the XYZ values ​​of the standard D65 light source, X source , Y source , Z source are the chromaticity values ​​of the light source calculated in step C,

[0087] D2) Convert the color-shifted RGB data obtained in step A to the XYZ color space and apply a color correction factor, and finally convert it back to the RGB space to obtain an image under a standard light source.

[0088]

[0089] Where XYZ represents the value of the color-biased image converted to the XYZ space. correct is the corrected value, S X , S Y , S Z is the color correction factor obtained in step D1.

[0090] Advantages and beneficial effects of the present invention

[0091] The present invention improves the color correction technology by simulating the multi-light source environment of the real world and embedding the light source spectrum information, thereby improving the generalization ability and applicability of the color estimation technology, including:

[0092] (1) Improved physical authenticity and generalization ability of color correction technology: This paper deeply analyzes the physical imaging process of the real environment and optimizes the method of generating color deviation datasets to make them more consistent with the lighting and light source characteristics of the real world. By improving the physical authenticity of the training data, the generalization ability of color correction technology in the face of different light sources and environmental conditions is enhanced, thereby achieving more extensive and stable applications.

[0093] (2) Reduced reliance on strict prior constraints and large amounts of training data: The present invention introduces spectral data for light source estimation, and uses the characteristic of spectral data being sensitive to light source changes to achieve more accurate chromaticity estimation in a multi-light source environment. Since spectral data is directly affected by light source changes, this method can effectively learn light source change characteristics from limited spectral samples, reducing reliance on large amounts of training data and improving technical applicability. Figure 7 (a) and 7(b) are the color shift RGB images of two real data sets and the corresponding scene light source spectral power estimation results. Figure 7 (c) and 7(d) are two color-shifted RGB images rendered using this method and the corresponding light source spectral power estimation results. In the light source estimation result, the horizontal axis represents the wavelength, the vertical axis represents the normalized relative power intensity, the solid line represents the real light source spectral power value, and the dotted line represents the estimated value of this method. The results show that this method can accurately estimate the real light source spectral information.

Claims

1. A color correction method based on physical imaging process and spectral power distribution embedding, characterized in that The following steps are involved: A) Using the dual light source model of sunlight and ambient light established based on the physical imaging process to render the spectral reflectance dataset, the color shift spectral data of different scenes are obtained, and then the CIE standard observer function is applied to convert these spectral data into RGB space, thereby obtaining the color shift RGB data under different color temperatures; B) extracting a characteristic spectrum of a non-shadow area in the color-shifted spectrum data, and using the characteristic spectrum as input data of a variational autoencoder, wherein the hidden layer distribution of the variational autoencoder is used to guide a light source decoder to output a light source spectral power distribution that matches the input spectrum; C) extracting the spectral features of the light source spectral power distribution obtained in step B and the color and texture features of the color-shifted RGB data obtained in step A through a stacked fully connected layer module and a 3×3 convolution kernel module, respectively, fusing the two features and estimating the light source chromaticity of the image; D) using the light source chromaticity estimated in step C, correcting the color of the color-shifted RGB data so that its color adapts to the standard light source; in: Step A) comprises: A1) First, define two main types of light sources: sunlight and ambient light. The color temperature of sunlight is 2000K to 3000K in the morning and / or evening, and 5000K to 6500K at noon. The color temperature of ambient light is 6000K to 7500K when it is cloudy and / or overcast, and 5000K when it is sunny. A2) According to the color temperature of the selected light source, the corresponding light source spectrum data is rendered using the physical model. Outdoor natural light sources with different color temperatures are characterized by the Planck radiation law of black body radiation with different color temperatures: Where B(λ,T) is the radiation intensity at temperature T and wavelength λ, h is Planck's constant, c is the speed of light, k is the Boltzmann constant, A3) Combining sunlight and ambient light in different proportions and rendering them onto the spectral reflectance dataset to obtain color deviation spectrum data of the new scene, wherein when rendering sunrise and / or sunset scenes, in order to avoid data that is inconsistent with the visual effects in the generated and real environments, the color temperature range of sunlight is limited to between 2000K and 4000K, that is: S new (λ)=αS s (λ,T)+βS e (λ,T) α+β＝1 I(λ)=S new (λ)·R(λ) Where α and β are proportional factors, S s (λ) and S e (λ) are the power distribution of sunlight and ambient light at wavelength λ and color temperature T, respectively. enw (λ) is the spectrum of the new light source, R(λ) is the reflectance in the scene, and I(λ) is the color shift spectrum data under the combined light source. A4) Convert the rendered spectral data to RGB color space, which is accomplished through the CIE standard observer function and the conversion from XYZ color space to RGB color space. First, convert the spectral data to XYZ values ​​using the CIE standard observer function, and then map the XYZ values ​​to RGB space using the XYZ to RGB conversion matrix: in: λ represents the wavelength, which ranges from 400nm to 720nm in the visible light band, with 33 bands at intervals of 10nm. I(λ) is the color deviation spectrum data obtained by rendering in step A3, z(λ) is the chromaticity matching function of the CIE standard observer, corresponding to the three coordinates of the XYZ color space, X, Y, and Z represent the mapping of spectral data to XYZ color space values, and R, G, and B represent the mapping of XYZ space to RGB space values. Step B) comprises: B1) Extract the color deviation spectrum data in the red, green and blue bands of visible light and convert them into the hue, saturation and brightness HSV space, and extract the non-shadow area by calculating the normalized saturation value difference index NSVDI. The NSVDI calculation method is as follows: Where S and V represent the saturation and brightness in the HSV space, respectively. Through experiments, NSVDI is used to extract the threshold of the non-shadow area. B2) Input the extracted non-shadow region spectrum into the variational autoencoder VAE, and use reconstruction iterative optimization to capture the potential representation of the spectral data. B3) Input the potential representation captured by VAE into the light source spectrum decoder, and use the mean square error MSE as the loss function to guide the decoder to output the correct light source spectral power distribution. Step D) comprises: D1) Convert the light source chromaticity calculated in step C to XYZ space, and calculate the color correction factor for correcting the current light source to the standard D65 light source: Where S X , S Y , S Z are the color correction factors of X, Y, and Z respectively, X D65 , Y D65 , ZD 65 are the XYZ values ​​of the standard D65 light source, X source , T soure , Z source are the light source chromaticity calculated in step C, D2) converting the color-shifted RGB data obtained in step A into XYZ space and applying a color correction factor, and finally converting back to RGB space to obtain an image under a standard light source, The XYZ vector represents the value of the color-shifted image converted to the XYZ space. correct is the corrected value, S X , S Y , S Z is the color correction factor obtained in step D1.

2. The color correction method based on physical imaging process and spectral power distribution embedding according to claim 1, characterized in that: The threshold for non-shadow areas is 0.2.

Images