Light source estimation method based on planckian locus prior and texture collaborative soft weighting

Abstract

This invention discloses a light source estimation method based on Planck trajectory prior and texture co-weighting, comprising: an offline calibration stage, establishing a Planck trajectory fitting model corresponding to the camera; an online estimation stage, dividing the input image into multiple image blocks after removing black-level pixels and saturated pixels, and calculating their local mean color and local texture gradient; mapping the local mean color to a logarithmic chromaticity space, and calculating the shortest Euclidean distance from the chromaticity point of the image block to the Planck trajectory fitting curve; constructing Gaussian prior weights based on this distance, and constructing log-smooth texture weights based on the local texture gradient; multiplicatively fusing the two weights to obtain the total weight of each image block; obtaining a global light source estimation vector based on the total weight of all valid image blocks, and performing L2 normalization on the vector to obtain the final unit light source direction. Using this invention, white balance failure can be avoided in low-texture pure color areas, high-frequency noise areas, and complex scenes.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, computational photography, and camera image signal processing, and in particular to a light source estimation method based on Planck trajectory prior and texture co-weighting. Background Technology

[0002] Automatic white balance is a crucial step in the camera image signal processing pipeline. Its purpose is to estimate the color of the light source in the shooting scene based on the input image, and then perform gain correction on each color channel of the image based on the estimated light source, thereby reducing the influence of light color on the imaging result and keeping the object color relatively stable under different lighting conditions. However, since the camera imaging result is simultaneously affected by factors such as the spectral power distribution of the light source, the reflectivity of the object surface, the spectral response of the camera sensor, and the geometric conditions of the scene, estimating the light source color based on only a single image is a typical underdetermined problem.

[0003] In existing technologies, the closest technical solutions mainly fall into the following categories. The first category is light source estimation methods based on deep learning, such as CCC, FFCC, FC4, and DS-Net. These methods typically learn the mapping relationship between image features and light source colors in a logarithmic color space using convolutional neural networks or discriminative learning models. While these methods can achieve high accuracy on specific training datasets, they suffer from the following drawbacks: First, they are highly dependent on the training scene, sensor model, and data distribution, and lack generalization ability across camera sensors; second, the model has a large number of parameters, and the inference process involves a large number of convolution operations or feature mapping calculations, making it difficult to meet the low power consumption and high real-time performance requirements of mobile image signal processors; third, the interpretability of the model decision process is weak, which is not conducive to stable debugging and engineering deployment in embedded image processing pipelines.

[0004] The second category comprises light source estimation methods based on statistical priors, such as Gray World, White-Patch, Gray-Edge, and Shades-of-Gray. These methods typically rely on priors such as "the average reflectance of the scene is neutral gray," "the brightest area of ​​the image represents the color of the light source," or "image edge statistics satisfy the grayscale assumption," performing statistical calculations on the entire image or image edges to estimate the light source. While these methods have low computational complexity, they suffer from significant drawbacks in complex real-world scenarios: when a large area of ​​solid-color background exists in the image, this solid-color area can easily dominate the global statistical results, causing the algorithm to misjudge the inherent color of objects as the color of the light source; when the image contains high-frequency noise, texture artifacts, or locally overexposed areas, edge or brightness statistics are easily affected by abnormal pixels, leading to a shift in light source estimation.

[0005] The third category comprises light source estimation methods based on grayscale pixel detection, such as those based on duochromatic reflectance models, local contrast, or the Grayness Index. These methods attempt to filter pixels or image patches from the image that are more likely to belong to neutral gray reflective surfaces and estimate the light source based on the selected regions. Compared to global statistical methods, these methods are more targeted, but they typically rely only on local contrast, grayscale index, or a single statistical feature, failing to effectively integrate the physical distribution patterns of the light source, especially the Planck trajectory constraint of blackbody radiation or daylight sources in the camera's color space, with the local texture statistical features of the image. Therefore, in scenes where low-texture solid color areas and high-frequency noise areas coexist, problems such as false grayscale pixel detection or unreasonable weight allocation still easily occur.

[0006] In summary, existing technologies have at least the following technical problems: First, against a background of large areas of solid color with low texture, solid color reflection areas are easily misidentified as light source colors, leading to white balance correction failure; Second, in areas with high-frequency noise or local texture artifacts, abnormal gradients are easily given excessive weights, interfering with global light source estimation; Third, existing statistical methods lack an effective mechanism for synergistically fusing the physical distribution constraints of light sources with local statistical features of images; Fourth, although deep learning methods can improve accuracy on some datasets, they suffer from strong training dependence, high model complexity, high power consumption, and insufficient cross-sensor generalization ability. Summary of the Invention

[0007] This invention provides a light source estimation method based on Planck trajectory prior and texture co-weighting, which can avoid white balance failure in low-texture solid color areas, high-frequency noise areas and complex scenes, and reduce the dependence of deep learning light source estimation methods on training data, model parameters and computing resources.

[0008] A light source estimation method based on Planck trajectory prior and texture co-weighting includes: (1) Collect several calibration samples of known real light source colors and map each calibration sample to the logarithmic color space; use the coordinates of the calibration samples in the logarithmic color space to obtain the Planck trajectory curve corresponding to the camera through polynomial regression fitting; (2) For images captured by the camera, black level pixels and saturation pixels are removed, and then the image is divided into... Identify non-overlapping image blocks and filter out the valid ones; (3) For each valid image patch, calculate the local mean color and local texture gradient; (4) For each valid image block, the local mean color is mapped to the logarithmic chromaticity space, and the shortest Euclidean distance from the chromaticity point of the image block to the Planck trajectory curve is calculated; Gaussian prior weights are constructed based on this distance, and logarithmic smooth texture weights are constructed based on the local texture gradient; the two weights are then multiplicatively fused to obtain the total weight of each valid image block. (5) The mean color of each valid image block is weighted and averaged to obtain the initial estimation vector of the global light source. The initial estimation vector of the global light source is then L2 normalized to obtain the final unit light source direction.

[0009] In step (1), the color of the real light source is denoted as Mapping each calibrated sample to the logarithmic chromaticity space, the expression is: ; ; in, This represents the first coordinate of the k-th calibration sample in the logarithmic chromaticity space. This represents the second coordinate of the kth calibration sample in the logarithmic chromaticity space; These represent the normalized linear responses of the kth calibration sample in the R, G, and B channels, respectively. It is used to prevent division by zero and logarithmic odd numbers.

[0010] In step (2), after removing black-level pixels and saturated pixels, the effective pixel set is obtained, expressed as: ; in, Represents the set of valid pixels; Indicates pixel position, Represents the black level threshold; This represents the linear response of pixel x on color channel c; R represents the saturation threshold, and R, G, and B represent the color channels.

[0011] In step (3), the local mean color is calculated, and the expression is: ; in, Indicates the first Local mean color vectors of valid image patches; , , They represent the first The mean response of each valid image patch in the R, G, and B channels; Representing the The number of valid pixels in each image block; , , Representing pixels Linear response on R, G, and B channels.

[0012] In step (3), the local texture gradient is calculated, and the expression is: ; in, Indicates the first Normalized local texture gradients of a valid image patch; This represents the maximum effective linear response value of the G channel, used to normalize the gradient magnitude. Represents pixels Linear response on the G channel; , These represent the first-order gradient operators in the horizontal and vertical directions, respectively; This represents the convolution operation; This represents absolute value operations.

[0013] In step (4), log-smooth texture weights are constructed based on local texture gradients, and the expression is: ; in, Indicates the first Log-smooth texture weights for each valid image patch; Indicates the first Normalized local texture gradients of a valid image patch.

[0014] In step (5), the mean color of each valid image patch is weighted and averaged to obtain the initial estimation vector of the global light source, expressed as: ; in, Represents the initial estimation vector of the global light source; This represents the set of valid image patch indices used in light source estimation; Indicates the first The total weight of each valid image block; Indicates the first The local mean color vector of each valid image patch.

[0015] In step (5), the initial estimation vector of the global light source is L2 normalized to obtain the final unit light source direction, which is expressed as: ; in, This represents the final output unit light source direction vector; Represents the initial estimation vector of the global light source; Represents the initial estimation vector of the global light source. The 2-norm.

[0016] Compared with the prior art, the present invention has the following beneficial effects: This invention establishes a Planck trajectory fitting model corresponding to a specific camera sensor during the offline calibration stage. In the light source estimation stage, it simultaneously calculates the distance to the Planck trajectory and the local texture gradient for each image patch, constructing Planck trajectory prior weights and log-smooth texture weights respectively. These are then multiplicatively fused to obtain the total weight of the image patch. Finally, the global light source direction is estimated based on the weighted average of the mean colors of each image patch. This invention addresses the problem of white balance failure in existing light source estimation methods in low-texture solid color regions, high-frequency noise regions, and complex scenes, and reduces the dependence of deep learning light source estimation methods on training data, model parameters, and computational resources. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the online estimation stage of the method of the present invention.

[0019] Figure 2 This is a comparison of the effects of the method of the present invention on estimating the light source and performing white balance correction on two sets of RAW images of real scenes. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0022] The linear response of a camera can be expressed as the result of the combined effects of the light source, the reflectivity of the object's surface, and the camera's spectral response, as shown in the following formula: ; in, This represents the linear response value of pixel position x on color channel c; This represents the spectral power distribution of the light source at wavelength λ; It is the reflectance of the object's surface at pixel position x to wavelength λ; The spectral response function of the camera's color channel c; Wavelength is indicated, and the unit is micrometer. Indicates the range of visible light wavelengths.

[0023] As shown in the above formula, the color response in a single image is affected by both the light source and the surface reflectivity. Directly separating the light source color from the image response is an underdetermined problem. For an approximately neutral gray reflective region, its three-channel reflectivity is approximately equal. In this case, the average color of the image patch can be approximately expressed as the product of the light source color and the brightness scaling factor, as shown in the following formula: ; in, Represents the mean color vector of the i-th image patch; These represent the normalized linear mean responses of the i-th image patch in the R, G, and B channels, respectively. Represents the brightness scaling factor of the i-th image block; This represents the color vector of the scene light source to be estimated; These represent the components of the light source color vector in the R, G, and B channels, respectively.

[0024] As shown in the above formula, in the neutral gray reflective region, the luminance scaling factor can be eliminated by calculating the ratio between color channels. Therefore, this invention maps the average color of image patches to a logarithmic chromaticity space to reduce the impact of brightness variations on light source estimation.

[0025] This embodiment provides a light source estimation method based on Planck trajectory prior and texture co-weighting, including an offline calibration stage and an online estimation stage.

[0026] The offline calibration stage is used to establish a Planck trajectory fitting model corresponding to a specific camera sensor. Several calibration samples with known real light source colors are collected, the real light source colors are mapped to a logarithmic chromaticity space, and the Planck trajectory curve corresponding to the camera is obtained through multinomial regression fitting.

[0027] The online estimation stage is used to estimate the light source of the input linear RAW image. For example... Figure 1As shown, the input image is first divided into multiple image blocks by removing black-level and saturated pixels. For each valid image block, its local mean color and local texture gradient are calculated. Then, the local mean color is mapped to the logarithmic chromaticity space, and the shortest Euclidean distance from the chromaticity point of the image block to the Planck trajectory fitting curve is calculated. Gaussian prior weights are constructed based on this distance, and logarithmically smoothed texture weights are constructed based on the local texture gradient. The two weights are then multiplicatively fused to obtain the total weight of each image block. Finally, the mean color of each image block is weighted and averaged to obtain the global light source estimation vector, and this vector is L2 normalized to obtain the final unit light source direction, which is used for white balance correction.

[0028] The following section details the processes of the offline calibration phase and the online estimation phase.

[0029] During the offline calibration phase, N calibration samples with known real light source colors are collected, and the real light source colors are denoted as . It can be obtained through calibration using a neutral gray patch on a color chart, a standard lighting box, or a spectroradiometer.

[0030] Map each real light source sample to the logarithmic chromaticity space using the following formula: ; ; in, This represents the first coordinate of the k-th calibration sample in the logarithmic chromaticity space. This represents the second coordinate of the kth calibration sample in the logarithmic chromaticity space; These represent the normalized linear responses of the k-th real light source color in the R, G, and B channels, respectively. It is used to prevent division by zero and logarithmic odd numbers.

[0031] Using the coordinates of the above-mentioned calibrated samples in the logarithmic chromaticity space, a camera-specific Planck trajectory polynomial fitting model is constructed, as shown in the following formula: ; in, Let represent the first and second coordinates of the Planck trajectory fitting curve in the logarithmic chromaticity space, respectively; It is the Planck trajectory fitting function, where n represents the polynomial order. Indicates the first coefficients of the order polynomial.

[0032] The polynomial coefficients were obtained through least squares regression: ; in, This represents the polynomial coefficient vector obtained from optimization; N represents the number of calibration samples, and n represents the order of the polynomial. This represents the parameter value that minimizes the objective function.

[0033] The coefficients obtained from the above fitting Related to the spectral response of the camera sensor, it can be calibrated separately for different camera models and stored in the image signal processor, firmware parameter table or camera configuration file.

[0034] The input for the online estimation stage is a linear RAW image output from the camera sensor. For Bayer format RAW images, statistics can be directly performed at the sampling positions of each color channel, or the same-size RGB plane can be obtained through linear interpolation before the method of this invention is executed.

[0035] First, invalid pixels are removed based on black level and saturation thresholds. The set of valid pixels is represented as: ; in, Represents the set of valid pixels; Indicates pixel position, Represents the black level threshold; This represents the linear response of pixel x on color channel c; This represents the saturation threshold.

[0036] The preprocessed image is divided into Non-overlapping image blocks ,in For the first A number of image patches, whose set of valid pixels is denoted as . .when If the number of valid pixels is less than the preset minimum, the image block is determined to be an invalid image block and will not be included in subsequent light source estimation.

[0037] No. The local mean color of a valid image patch is calculated using the following formula: ; in, Indicates the first Local mean color vectors of valid image patches; , , They represent the first The mean response of each valid image patch in the R, G, and B channels; Representing the The number of valid pixels in each image block; , , Representing pixels Linear response on R, G, and B channels.

[0038] To suppress the interference of large areas of low-texture solid color regions on light source estimation, this invention calculates the local texture gradient for each valid image block. The G channel is used as the texture calculation channel because it has a high sampling density in the Bayer array and a high signal-to-noise ratio in most camera sensors. The formula for calculating the local texture gradient is as follows: ; in, Indicates the first Normalized local texture gradients of a valid image patch; This represents the maximum effective linear response value of the G channel, used to normalize the gradient magnitude. Represents pixels Linear response on the G channel; , The first-order gradient operators in the horizontal and vertical directions, respectively, can be represented by the Sobel operator; This represents the convolution operation; This represents absolute value operations.

[0039] When an image patch is a large area of ​​solid color, its local gradient is small. Approaching 0; when the image patch contains valid texture or edges, Increase; when image patches contain high-frequency noise. It may also increase, therefore, the present invention further employs a logarithmic smoothing function to perform soft limiting processing on it.

[0040] Based on normalized local texture gradient Construct the logarithmically smoothed texture weights, as shown in the following expression: ; in, Indicates the first Texture weights of each valid image patch; Indicates the first Normalized local texture gradients of a valid image patch.

[0041] The input image is a linear RAW response; black level pixels and saturation pixels have been removed from the image patches; local gradients... It has been normalized to a dimensionless value using the maximum effective response value or a preset calibration factor. With As the gradient increases, the marginal gain of the texture weight gradually decreases. Therefore, low-texture solid color regions can be given lower weights; for high-frequency noise regions, even if the local gradient is large, the weight will not be linearly amplified indefinitely, thus achieving soft limiting of high-frequency noise.

[0042] The Planck trajectory prior weights and log-smooth texture weights are multiplicatively fused to obtain the first... The total weight of each valid image patch is calculated using the following formula: ; in, Indicates the first The total weight of each valid image block; Indicates the first Prior weights of Planck trajectories for each valid image patch; Indicates the first Texture weights of each valid image patch.

[0043] The technical effect of multiplicative fusion is that if an image patch is far from the Planck trajectory, even if its texture is strong, it will still be affected by the multiplicative fusion technique. Smaller areas are suppressed; if an image patch is close to the Planck trajectory but lacks texture, it will be suppressed due to... Smaller images are suppressed; only image patches that simultaneously satisfy the physical trajectory of the light source and have effective local texture will make a higher contribution to global light source estimation.

[0044] Based on the total weight of all valid image patches, the initial estimation vector of the global light source is calculated using the following formula: ; in, Represents the initial estimation vector of the global light source; This represents the set of valid image patch indices used in light source estimation; Indicates the first The total weight of each valid image block; Indicates the first The local mean color vector of each valid image patch.

[0045] like If the image size is less than the preset minimum weight threshold, alternative strategies can be adopted, such as increasing the image block size, reducing the minimum effective pixel count requirement, or degrading to global mean color estimation, in order to ensure stable system output.

[0046] Perform L2 normalization on the initial global light source estimation vector to obtain the final unit light source direction, as shown in the following formula: ; in, This represents the final output unit light source direction vector; Represents the initial estimation vector of the global light source; Representing vectors The 2-norm.

[0047] To verify the effectiveness of this invention, two sets of linear RAW images acquired by the camera under test were selected for testing. Before the test, a Planck trajectory fitting model corresponding to the camera was established according to the aforementioned offline calibration steps. During the test, the RAW images were input into the online estimation process of this invention, and the following steps were performed sequentially: black level pixel and saturation pixel removal, image block division, local mean color and local texture gradient calculation, Planck trajectory prior weight and logarithmic smoothing texture weight calculation and multiplicative fusion. The global light source estimation vector was obtained according to the total weight of each effective image block. The vector was L2 normalized to obtain the unit light source direction, and then the color channel gain was set according to the estimated light source direction to perform white balance correction on the RAW images.

[0048] Test results are as follows Figure 2 As shown, the left column is the visualization result of the RAW input image, and the right column is the algorithm processing result. Figure 2 The upper part is an outdoor scene containing vegetation, tree trunks, gray facilities, color cards, and locally highlighted areas. After the algorithm processing, the different colors of vegetation, tree trunks, and color cards can be clearly distinguished. No obvious large-scale single color shift appears in the neutral areas of the scene, indicating that the locally highlighted areas and complex textures do not prevent the algorithm from outputting stable light source estimation results. Figure 2 The lower part is a scene containing a large area of ​​low-texture wall, bicycles, and color swatches; after processing, the wall maintains a near-neutral overall tone, and the colors and structures of the bicycles, tires, and color swatches are still discernible, indicating that the large area of ​​low-texture background does not dominate the global light source estimation.

[0049] The two sets of tests above demonstrate that, following the aforementioned offline calibration and online estimation steps, this invention can complete image patch selection, collaborative soft weighting, and global light source direction calculation for actually acquired linear RAW images. Furthermore, it can utilize the obtained light source estimation results to perform white balance correction and output a recognizable color image. Therefore, the technical solution described in this invention has a complete input, processing, and output chain and can be implemented in practical image processing systems.

[0050] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A light source estimation method based on Planck trajectory prior and texture co-soft weighting, characterized in that, include: (1) Collect several calibration samples of known real light source colors and map each calibration sample to the logarithmic color space; Using the coordinates of the calibrated samples in the logarithmic chromaticity space, the Planck trajectory curve corresponding to the camera is obtained by fitting a polynomial regression. (2) For images captured by the camera, black level pixels and saturation pixels are removed, and then the image is divided into... Identify non-overlapping image blocks and filter out the valid ones; (3) For each valid image patch, calculate the local mean color and local texture gradient; (4) For each valid image patch, the local mean color is mapped to the logarithmic chromaticity space, and the shortest Euclidean distance from the chromaticity point of the image patch to the Planck trajectory curve is calculated; Gaussian prior weights are constructed based on this distance, and logarithmically smoothed texture weights are constructed based on the local texture gradient. The two weights are then multiplicatively fused to obtain the total weight for each valid image block; (5) The mean color of each valid image block is weighted and averaged to obtain the initial estimation vector of the global light source. The initial estimation vector of the global light source is then L2 normalized to obtain the final unit light source direction.

2. The light source estimation method based on Planck trajectory prior and texture co-weighting as described in claim 1, characterized in that, In step (1), the color of the real light source is denoted as Mapping each calibrated sample to the logarithmic chromaticity space, the expression is: ； ； in, This represents the first coordinate of the k-th calibration sample in the logarithmic chromaticity space. This represents the second coordinate of the kth calibration sample in the logarithmic chromaticity space; These represent the normalized linear responses of the kth calibration sample in the R, G, and B channels, respectively. It is used to prevent division by zero and logarithmic odd numbers.

3. The light source estimation method based on Planck trajectory prior and texture co-weighting as described in claim 1, characterized in that, In step (2), after removing black-level pixels and saturated pixels, the effective pixel set is obtained, expressed as: ； in, Represents the set of valid pixels; Indicates pixel position, Represents the black level threshold; This represents the linear response of pixel x on color channel c; R represents the saturation threshold, and R, G, and B represent the color channels.

4. The light source estimation method based on Planck trajectory prior and texture co-weighting as described in claim 1, characterized in that, In step (3), the local mean color is calculated, and the expression is: ； in, Indicates the first Local mean color vectors of valid image patches; , , They represent the first The mean response of each valid image patch in the R, G, and B channels; Representing the The number of valid pixels in each image block; , , Representing pixels Linear response on R, G, and B channels.

5. The light source estimation method based on Planck trajectory prior and texture co-weighting as described in claim 1, characterized in that, In step (3), the local texture gradient is calculated, and the expression is: ； in, Indicates the first Normalized local texture gradients of a valid image patch; This represents the maximum effective linear response value of the G channel, used to normalize the gradient magnitude. Represents pixels Linear response on the G channel; , These represent the first-order gradient operators in the horizontal and vertical directions, respectively; This represents the convolution operation; This represents absolute value operations.

6. The light source estimation method based on Planck trajectory prior and texture co-weighting as described in claim 1, characterized in that, In step (4), log-smooth texture weights are constructed based on local texture gradients, and the expression is: ； in, Indicates the first Log-smooth texture weights for each valid image patch; Indicates the first Normalized local texture gradients of a valid image patch.

7. The light source estimation method based on Planck trajectory prior and texture co-weighting as described in claim 1, characterized in that, In step (5), the mean color of each valid image patch is weighted and averaged to obtain the initial estimation vector of the global light source, expressed as: ； in, Represents the initial estimation vector of the global light source; This represents the set of valid image patch indices used in light source estimation; Indicates the first The total weight of each valid image block; Indicates the first The local mean color vector of each valid image patch.

8. The light source estimation method based on Planck trajectory prior and texture co-weighting as described in claim 1, characterized in that, In step (5), the initial estimation vector of the global light source is L2 normalized to obtain the final unit light source direction, which is expressed as: ； in, This represents the final output unit light source direction vector; Represents the initial estimation vector of the global light source; Represents the initial estimation vector of the global light source. The 2-norm.

Images