Color constancy implementation method for neutral color pixel extraction assisted by ambient light sensor

The ambient light sensor assists in extracting neutral color pixels and using the color space mapping model to estimate light source, solving the problems of poor performance and dependence on data sets in the existing color constancy algorithm in the hypothetical scenario, achieving higher Accuracy and stability, especially improvements in low color complexity scenarios.

CN116033280BActive Publication Date: 2024-11-26ZHEJIANG UNIV
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202310052076.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-02-02
Publication Date
2024-11-26
Estimated Expiration
2043-02-02

AI Technical Summary

Technical Problem

The existing color constancy algorithms perform poorly in scenarios that do not meet the assumptions, and the learning-based algorithms have strong dependence on the data set and have poor generalization capabilities, especially in scenarios with low color complexity, which are more likely to make mistakes.

Method used

Through the ambient light sensor assistance, using the relationship between pixels and ambient light sensor light source estimation results and bold trajectory, neutral color pixels are extracted, light source estimation and image white balance correction are carried out, color space mapping model is established, and dependence on large-scale data sets is reduced. , improve the cross-device generalization capability of algorithms.

Benefits of technology

A device-independent color constancy method is realized, which improves the accuracy and stability of the algorithm, especially in low color complexity scenarios, reduces the dependence of light source estimation algorithm on data sets, and improves cross-device applicability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116033280B_ABST
    Figure CN116033280B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for realizing color constancy by using an ambient light sensor to assist in extracting neutral color pixels, and belongs to the field of image processing. First, a color space mapping model is established to map the sensor response value to a device-related RGB space as the sensor light source estimation result. On this basis, the weighted geometric distances between the pixel and the black body trajectory, and the pixel and the sensor light source estimation result in the image to be processed are calculated in the RB chromaticity space as the neutral index, and the neutral color pixels are screened accordingly, and their mean is used as the final light source estimation result. The present invention applies an ambient light sensor to a color constancy algorithm, enriches the basis for light source estimation, improves the accuracy of the algorithm and its cross-device generalization ability, and improves the performance of the algorithm in low color complexity scenes with single color and content.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular relates to a method for realizing color constancy by using an ambient light sensor to assist in neutral color pixel extraction. Background Art

[0002] Color constancy is the property of the human eye to eliminate the influence of light source color and perceive the color of objects in a consistent manner under different light sources. The color constancy algorithm estimates the light source based on information such as image content and eliminates the image color cast caused by the light source color through white balance correction. It is the basis of computer vision tasks such as object recognition and target tracking.

[0003] Most of the existing color constancy algorithms are for single images and can be divided into two categories: hypothesis-based and learning-based. Hypothesis-based algorithms make assumptions based on the statistical laws of image color distribution and perform light source estimation. This type of algorithm is simple to calculate, but performs poorly in scenes that do not meet the assumptions. Learning-based algorithms use large-scale data sets to train light source estimation models. Their accuracy and stability are generally better than hypothesis-based algorithms, but they have a strong dependence on data sets, poor generalization ability, and are more likely to make mistakes in difficult scenes such as low color complexity.

[0004] With the optimization and upgrading of the hardware of smart terminal photography systems, the information that can be used for light source estimation is gradually increasing. In recent years, the ambient light sensors equipped in terminal devices such as smartphones have been continuously updated and have the potential to be used for light source estimation. Summary of the invention

[0005] The purpose of the present invention is to improve the accuracy and stability of the color constancy algorithm and provide a method for realizing color constancy by using an ambient light sensor to assist in neutral color pixel extraction. The method is based on color space mapping and uses the relationship between pixels, ambient light sensor light source estimation results, and black body trajectory to extract neutral color pixels and perform light source estimation, which is mainly used for white balance correction of images.

[0006] The specific technical solutions adopted by the present invention are as follows:

[0007] The present invention provides a method for realizing color constancy by using an ambient light sensor to assist in neutral color pixel extraction, which is characterized by the following steps:

[0008] S1: Establish a mapping model between sensor response value and camera RGB value;

[0009] S2: Obtaining the image to be processed and the response value of the ambient light sensor arranged next to the camera lens;

[0010] S3: Map the sensor response value of S2 to the camera RGB space using the model of S1 as the sensor light source estimation result;

[0011] S4: down-sample the image to be processed to H×W, and calculate the weighted geometric distance between the pixel and the sensor light source estimation result described in S3, and the pixel and the black body trajectory on the rb chromaticity plane as a neutral index;

[0012] S5: Arrange the neutral indexes in S4 in ascending order, use the first n% of pixels as the neutral color pixel screening result, and use their average as the light source estimation result.

[0013] Preferably, the step S1 is specifically as follows:

[0014] S101: Use a camera to capture a white balance card image and simultaneously obtain a sensor response value;

[0015] S102: using the channel with the largest sensor bandwidth to normalize its response value as an independent variable; extracting the RGB value of the white balance card, and normalizing it using the G channel as a dependent variable;

[0016] S103: Repeat S101 and S102 under various lighting sources to establish a training set;

[0017] S104: Taking the mean angle error of the training samples in the training set as the loss function L, the mapping model is optimized:

[0018]

[0019] In the formula, n is the total number of training samples, e k and are the RGB reference value and estimated value of the kth training sample respectively.

[0020] Preferably, the step S4 is specifically as follows:

[0021] S401: Map the sensor light source estimation result and the downsampled image to be processed to the rb chromaticity plane:

[0022]

[0023] Where R, G, B are the camera RGB space response values; r, b are the chromaticity components;

[0024] S402: On the rb chromaticity plane, the black body locus (r BBL ,b BBL ) is characterized by:

[0025]

[0026] In the formula, A, B, and C are the unknown parameters that can be fitted based on the training samples;

[0027] S403: Calculate the distance D between the pixel and the blackbody trajectoryBBL :

[0028]

[0029] In the formula, r i ,b i is the pixel chromaticity value;

[0030] S404: Calculate the distance D between the pixel and the sensor light source estimation result S :

[0031]

[0032] In the formula, r S ,b S Estimate the resulting chromaticity value for the sensor illuminant;

[0033] S405: Calculate the neutral index I G :

[0034] I G =aD S +(1-a)D BBL ,

[0035] Where a is the geometric distance D between the pixel and the sensor light source estimation result S The weight of .

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] This paper uses the ambient light sensor light source estimation results to assist in the extraction of neutral color pixels in the image to be processed, and proposes a device-independent color constancy method. This method fully utilizes the potential of the photography system hardware, reduces the dependence of the light source estimation algorithm on large-scale data sets, improves the algorithm's cross-device generalization ability, and has more advantages in difficult scenes such as low color complexity where it is difficult to estimate the light source color based on image content. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is the chromaticity distribution of the color space mapping model training samples in the embodiment of the present invention.

[0039] Figure 2 This is a rendering of the color constancy realization method provided by the present invention. DETAILED DESCRIPTION

[0040] The present invention is further described and illustrated below in conjunction with the accompanying drawings and specific embodiments. The technical features of each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0041] Example

[0042] This embodiment takes a smart phone as an example to illustrate the color constancy realization method of ambient light sensor-assisted neutral color pixel extraction. The front camera of the smart phone has a resolution of 3072×4096 and is equipped with an 8-channel ambient light sensor.

[0043] The method for realizing color constancy by using an ambient light sensor to assist in neutral color pixel extraction in this embodiment specifically includes the following steps:

[0044] 1) Obtain color space mapping model training samples under multiple light sources. Use a multi-channel LED platform to simulate the light source near the black body trajectory for lighting, place a white balance card in the scene, and shoot images with a smartphone perpendicular to the horizontal plane, while obtaining the ambient light sensor response value.

[0045] like Figure 1 As shown, it is the distribution of the simulated light source on the CIE1976UCS chromaticity diagram. The dotted line in the figure is the black body trajectory. The CCT of the training sample light source is selected between 2500K and 10000K with an interval of 500K, and Duv is selected between -0.01 and 0.01 with an interval of 0.005.

[0046] 2) Select the root polynomial as the mapping model, use the channel with the largest sensor bandwidth to normalize the response value, calculate the square root cross term of adjacent channels, and connect it with the response value in series as the independent variable; extract the uniformly illuminated part of the white balance card in the raw format image, calculate the mean of its R, G, and B channels respectively, and use the G channel for normalization as the dependent variable. Use the mean of the angle error of the training sample as the loss function, set the initial value to the least squares fitting result, and train the mapping model.

[0047]

[0048] In the formula, f 1 ~f 8 is the normalized sensor channel response value, M is the mapping matrix, is the RGB estimation value, n is the total number of training samples, e k and are the RGB reference value and estimated value of the kth training sample respectively.

[0049] 3) Use a smartphone to capture the image to be processed and obtain the sensor response value at the same time, and use the model M in 2) to map it to the camera RGB space as the sensor light source estimation result.

[0050] 4) Use bilinear interpolation to downsample the image to be processed to H×W. In this embodiment, H=48, W=64.

[0051] 5) Map the sensor light source estimation result and the downsampled image to be processed to the rb chromaticity plane:

[0052]

[0053] Where R, G, B are the camera RGB space response values, and r and b are the chromaticity components.

[0054] 6) Select the RGB value of the white balance card under the illumination source of Duv=0 in 1) as the training sample, map it to the rb chromaticity plane according to 5), and fit the black body trajectory (r BBL ,b BBL ):

[0055]

[0056] Wherein, A, B, C are parameters to be determined. In this embodiment, A=0.023, B=0.287, C=0.031.

[0057] 7) For each pixel in the downsampled image to be processed, calculate its geometric distance D to the blackbody locus on the rb chromaticity plane BBL :

[0058]

[0059] In the formula, r 0 ,b 0 is the chrominance value of the pixel to be processed.

[0060] 8) For each pixel of the downsampled image to be processed, calculate its geometric distance D with the sensor light source estimation result on the rb chromaticity plane S :

[0061]

[0062] In the formula, r S ,b S Resulting chromaticity value for the sensor illuminant estimate.

[0063] 9) According to D BBL With D S Calculate the neutral index I G :

[0064] I G =aD S +(1-a)D BBL ,

[0065] Where a is the distance D between the pixel and the sensor light source estimation result S In this embodiment, a=0.4.

[0066] 10) The neutral indexes are arranged in ascending order, and the first n% of pixels are used as the neutral color pixel screening result. In this embodiment, n=2.5, and the average value of the neutral color pixels is used as the light source estimation result.

[0067] like Figure 2 The effect diagram of the color constancy realization method provided by the present invention is shown, wherein (a) is the image to be processed under an unknown light source, (b) is the reference image under a standard light source, and (c) is the image after the light source is estimated and color corrected by the method of the present invention. It can be seen from the figure that the color constancy realization method proposed by the present invention effectively eliminates the influence of the light source color on the object being photographed, making it present the color under the standard light source.

[0068] It can be seen that the present invention applies ambient light sensors to the color constancy algorithm, enriches the basis for light source estimation, improves the accuracy and cross-device generalization capability of the algorithm, and improves the performance of the algorithm in low color complexity scenes with single color and content.

[0069] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A method for realizing color constancy by ambient light sensor-assisted neutral color pixel extraction, characterized in that: Here are the steps: S1: Establish a mapping model between sensor response value and camera RGB value; S2: Obtaining the image to be processed and the response value of the ambient light sensor arranged next to the camera lens; S3: Map the sensor response value of S2 to the camera RGB space using the model of S1 as the sensor light source estimation result; S4: down-sample the image to be processed to H×W, and calculate the weighted geometric distance between the pixel and the sensor light source estimation result described in S3, and the pixel and the black body trajectory on the rb chromaticity plane as a neutral index; S5: Arrange the neutral indexes in S4 in ascending order, use the first n% of pixels as the neutral color pixel screening result, and use the average value as the light source estimation result; The step S4 is specifically as follows: S401: Map the sensor light source estimation result and the downsampled image to be processed to the rb chromaticity plane: Where R, G, B are the camera RGB space response values; r, b are the chromaticity components; S402: On the rb chromaticity plane, the black body locus (r BBL ,b BBL ) is characterized by: In the formula, A, B, and C are the unknown parameters that can be fitted based on the training samples; S403: Calculate the distance D between the pixel and the blackbody trajectory BBL : In the formula, r i ,b i is the pixel chromaticity value; S404: Calculate the distance D between the pixel and the sensor light source estimation result S : In the formula, r S ,b S Estimate the resulting chromaticity value for the sensor illuminant; S405: Calculate the neutral index I G : I G =aD S +(1-a)D BBL , Where a is the geometric distance D between the pixel and the sensor light source estimation result S The weight of .

2. The method for realizing color constancy by ambient light sensor-assisted neutral color pixel extraction according to claim 1, wherein step S1 is specifically as follows: S101: Use a camera to capture a white balance card image and simultaneously obtain a sensor response value; S102: using the channel with the largest sensor bandwidth to normalize its response value as an independent variable; extracting the RGB value of the white balance card, and normalizing it using the G channel as a dependent variable; S103: Repeat S101 and S102 under various lighting sources to establish a training set; S104: Taking the mean angle error of the training samples in the training set as the loss function L, the mapping model is optimized: In the formula, n is the total number of training samples, e k and are the RGB reference value and estimated value of the kth training sample respectively.

Citation Information

Patent Citations

  • Color constancy method based on grey pixel statistics

    CN112308791A