Auto white balance processing method and apparatus, imaging apparatus, and electronic device

Abstract

To provide an automatic white balance processing method with high accuracy by a simple and comparatively simple method.SOLUTION: The automatic white balance processing method includes the following content. An image to be processed is divided into a plurality of image regions. Statistics is performed on brightness information of pixels included in each image region. The luminance information includes at least a luminance level at which a pixel included in each image region is located. Density estimation is performed on the pixels at the plurality of luminance levels using a non-parametric approach to obtain at least a density estimate of the pixels at the bright levels. A first white balance gain corresponding to each color is calculated based on at least the density estimation value of the bright level pixels. In addition, the present disclosure further provides an automatic white balance processing device, an imaging device and an electronic device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to the technical field of image processing, and more particularly to an auto white balance processing method and apparatus, an imaging device, and an electronic device. [Background technology]

[0002] During image capture, white balance processing is performed on the pixels output from the image sensor to compensate for differences in sensitivity of the image sensor and spectral differences between the light sources used. The purpose of white balance processing is to prevent color tinge from being added to what should be reproduced as white, and to eliminate color shifts, thereby correcting the colors of the image to be more realistic and natural. However, existing white balance processing methods are complicated, require a great deal of time and labor, or are ineffective. Summary of the Invention

[0003] In order to solve the above problems, the present application provides an auto white balance processing method and apparatus, an imaging device, and an electronic device.

[0004] In a first aspect, there is provided an auto white balance processing method for performing white balance processing on an image. The auto white balance processing method includes the following steps: dividing an image to be processed into a plurality of image regions; calculating luminance information of pixels included in each image region; the luminance information including at least the luminance level at which the pixels included in each image region are located; performing density estimation on pixels at a plurality of luminance levels using a non-parametric method to obtain at least density estimates for pixels at bright levels; and calculating a first white balance gain corresponding to each color based at least on the density estimates for pixels at bright levels.

[0005] In a second aspect, an auto white balance processing device is further provided. The auto white balance processing device includes a statistical analysis unit, a density estimation unit, and a white balance gain determination unit. The statistical analysis unit is configured to divide an image to be processed into a plurality of image regions and collect statistics on luminance information of pixels included in each image region, the luminance information including at least the luminance level at which the pixels included in each image region are located. The density estimation unit is configured to perform density estimation on pixels of a plurality of luminance levels using a nonparametric method to obtain at least density estimates for pixels of a bright level. The white balance gain determination unit is configured to calculate a first white balance gain corresponding to each color based at least on the density estimates for pixels of a bright level.

[0006] In a third aspect, an imaging device is further provided. The imaging device includes an auto white balance processing device. The auto white balance processing device includes a statistical analysis unit, a density estimation unit, and a white balance gain determination unit. The statistical analysis unit is configured to divide an image to be processed into a plurality of image regions and perform statistical analysis of luminance information of pixels included in each image region, the luminance information including at least the luminance level at which the pixels included in each image region are located. The density estimation unit is configured to perform density estimation on pixels of a plurality of luminance levels using a nonparametric method to obtain at least density estimates for pixels of a bright level. The white balance gain determination unit is configured to calculate a first white balance gain corresponding to each color based at least on the density estimates for pixels of a bright level.

[0007] In a fourth aspect, an electronic device is further provided. The electronic device includes a memory and a processor. A computer program is stored in the memory, and the processor is configured to call the computer program to execute an auto white balance processing method. The auto white balance processing method includes the following steps: dividing an image to be processed into a plurality of image regions; calculating luminance information of pixels included in each image region; the luminance information including at least a luminance level at which the pixels included in each image region are located; performing density estimation on pixels at a plurality of luminance levels using a non-parametric method to obtain at least a density estimate value for pixels at a bright level; and calculating a first white balance gain corresponding to each color based at least on the density estimate value for pixels at the bright level.

[0008] In a fifth aspect, a computer-readable storage medium is further provided. A computer program is stored in the computer-readable storage medium, and when the computer program is called by a computer, an auto white balance processing method is executed. The auto white balance processing method includes the following steps: dividing an image to be processed into a plurality of image regions; calculating luminance information of pixels included in each image region; the luminance information including at least the luminance level at which the pixels included in each image region are located; performing density estimation on pixels at a plurality of luminance levels using a non-parametric method to obtain at least density estimates for pixels at bright levels; and calculating first white balance gains corresponding to each color based at least on the density estimates for pixels at bright levels.

[0009] In the auto white balance processing method and apparatus, imaging device, and electronic device of the present application, an image is divided into a plurality of image regions, luminance information of pixels contained in each image region is statistically analyzed, and density estimation is performed on pixels of a plurality of luminance levels using a non-parametric method to obtain density estimates for pixels of at least bright levels, and first white balance gains corresponding to each color are calculated based at least on the density estimates for pixels of bright levels, thereby making it possible to obtain accurate white balance gains in a relatively simple manner. [Brief explanation of the drawings]

[0010] In order to more clearly describe the technical solutions of the embodiments of the present application or the technical solutions of the background art, the following describes the drawings necessary for describing the embodiments of the present application or the background art. [Figure 1] FIG. 1 is a flowchart illustrating an auto white balance processing method in some embodiments of the present application. [Figure 2] FIG. 2 is a schematic diagram illustrating division of an image into a plurality of image regions A0 to A(n-1) in some embodiments of the present application. [Figure 3] FIG. 3 illustrates a sub-process of step 12 of FIG. 1 of the present application in some embodiments. [Figure 4] FIG. 4 is a schematic diagram illustrating the correspondence between luminance levels and luminance intervals in some embodiments. [Figure 5] FIG. 5 is a schematic diagram illustrating a storage structure for total pixel values ​​for each luminance level in each image region in some embodiments of the present application. [Figure 6] FIG. 6 is another flowchart illustrating an auto white balance processing method in some embodiments of the present application. [Figure 7] FIG. 7 is a schematic diagram showing a storage structure of R / G values ​​for each luminance level in each image region in some embodiments of the present application. [Figure 8] FIG. 8 illustrates a sub-process of step 13 of FIG. 1 of the present application in some embodiments. [Figure 9] FIG. 9 is a diagram illustrating pixel density estimates and data points for bright levels in some embodiments of the present application. [Figure 10] FIG. 10 is a diagram illustrating density estimates and data points for dark level pixels in some embodiments of the present application. [Figure 11] FIG. 11 is a schematic diagram of the integration of bright level density estimates and dark level data point scatter in some embodiments of the present application. [Figure 12] FIG. 12 illustrates a sub-process of step 14 of FIG. 1 of the present application in some embodiments. [Figure 13] FIG. 13 is another flowchart illustrating an auto white balance processing method in some embodiments of the present application. [Figure 14] FIG. 14 is a schematic diagram illustrating the relationship between the gain mixing ratio and the cross-correlation coefficient in some embodiments of the present application. [Figure 15] FIG. 15 is a schematic diagram showing a comparison of images obtained after white balancing by several methods. [Figure 16] FIG. 16 is a block diagram illustrating the structure of an auto white balance processing device in some embodiments of the present application. [Figure 17] FIG. 17 is a block diagram illustrating the structure of an imaging device in some embodiments of the present application. [Figure 18] FIG. 18 is a block diagram illustrating the structure of an electronic device in some embodiments of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0011] Terms such as "first" and "second" in the specification, claims, and drawings of this application are used to distinguish between different objects, not to describe a particular sequence. Terms such as "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or the number of technical features being referenced. Thus, features defined by terms such as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present embodiments, unless otherwise specified, "multiple" refers to two or more. Furthermore, terms such as "comprises," "includes," and any other variants are intended to cover and not exclude the inclusion of other elements. For example, a process, method, system, product, or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally further include other steps or units not listed, or may optionally further include other steps or units inherent to the process, method, system, product, or device.

[0012] As can be understood, in this application, "plurality" refers to two or more. The term "and / or" describes the relationship between related objects and indicates that three types of relationships exist. For example, A and / or B refers to three situations: the presence of only A, the presence of only B, and the simultaneous presence of A and B. A and B may be singular or plural. The symbol " / " generally indicates that the related objects before and after it are in an "or" relationship. Both "when" and "if" refer to the carrying out of appropriate processing under certain objective circumstances, and do not imply a time limit, nor do they require any judgmental actions to be taken when realizing the situation, nor do they imply any other limitations.

[0013] In the embodiments of the present application, terms such as "exemplary" or "for example" are used to indicate an example, illustration, or explanation. Any embodiment or solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as preferred or advantageous over other embodiments or solutions. To be clear, the use of terms such as "exemplary" or "for example" is intended to present relevant concepts in a concrete manner for ease of understanding.

[0014] Auto white balance functions generally use one method: assuming the average color of the scene is achromatic, and calculating the white balance correction coefficient from the integrated value of all pixels; or calculating the correction coefficient from the integrated value of pixels included in the achromatic area. Calibration techniques are often used in conjunction with this method, which measure the color sensitivity of the sensor in advance for a standard light source to improve the accuracy of calculating the correction coefficient. Of the aforementioned methods, the method of calculating the correction coefficient from the integrated value of pixels included in the achromatic area often requires the user to specify the achromatic area using parameters that can be adjusted, and the adjustment results can have a significant impact on performance, sometimes resulting in a significant degradation of the original performance. Furthermore, the adjustment process requires a significant amount of time.

[0015] Therefore, how to provide a more efficient and accurate white balance processing method is a problem to be solved.

[0016] 1, which is a flowchart illustrating an auto white balance processing method according to some embodiments of the present application, which is used to perform white balance processing on an image, and includes the following steps: 11: Divide the image to be processed into multiple image regions. 12: Calculate the luminance information of the pixels included in each image region. The luminance information includes at least the luminance level at which the pixels included in each image region are located. 13: Using a non-parametric method, density estimation is performed for pixels at multiple brightness levels to obtain density estimates for at least the bright level pixels. 14: Calculate a first white balance gain corresponding to each color based at least on the density estimate of pixels at the light level.

[0017] In the present application, an image is divided into a plurality of image regions, luminance information of pixels contained in each image region is collected, and density estimation is performed on pixels of a plurality of luminance levels using a non-parametric method to obtain density estimates for pixels of at least bright levels, and first white balance gains corresponding to each color are calculated based at least on the density estimates for pixels of bright levels, thereby making it possible to obtain accurate white balance gains in a relatively simple manner.

[0018] Referring to FIG. 2, FIG. 2 is a schematic diagram illustrating dividing an image P1 into a plurality of image regions A0 to A(n-1) in some embodiments of the present application.

[0019] That is, in some embodiments, image P1 can be divided into n image regions A0 to A(n-1). In some embodiments, image P1 can be divided into any number of image regions, for example, n can be an integer greater than or equal to 2, and image P1 can be divided into any number of image regions greater than or equal to 2. As another example, as shown in FIG. 2, n can be 30, that is, image P1 can be divided into 30 image regions, for example, 30 image regions with 5 rows and 6 columns.

[0020] In some embodiments, the image regions are approximately equal in size, ie, image P1 is evenly divided into image regions of equal size.

[0021] Each image area has its own identifier, for example, A0 to A(n-1) in the drawing. A0 to A(n-1) each correspond to one image area and are the identifiers of the corresponding image areas.

[0022] 2 is merely an example, and image P1 may be divided into other numbers of image regions, and the arrangement mode of the multiple image regions may be other arrangement modes. For example, if image P1 can be divided into 30 image regions, the 30 image regions may be arranged in 2 rows and 15 columns, 3 rows and 10 columns, 6 rows and 5 columns, etc. In some embodiments, the identifiers of the image regions may be other identifiers. For example, if image P1 is divided into n image regions, the identifiers of the n image regions may each be a number from 0 to "n-1."

[0023] Referring to Figure 3, Figure 3 is a diagram illustrating a sub-process of step 12 of Figure 1 of the present application in some embodiments. In some embodiments, as shown in Figure 3, step 12 of "compute statistics on luminance information of pixels included in each image region" may include the following content: 121: Divide into a plurality of brightness levels from the darkest brightness to the brightest brightness according to a preset rule. Each brightness level corresponds to a brightness section. 123: Quantize the pixel values ​​of the pixels of each color in each image region into corresponding luminance values, and determine the luminance levels corresponding to the pixels of each color based on the luminance interval in which the luminance values ​​corresponding to the pixels of each color are located.

[0024] That is, in some embodiments, since the pixels include pixels of different colors, calculating the luminance information of the pixels included in each image region includes calculating the luminance information of the pixels of each color included in each image region. Specifically, the image is first divided into a plurality of luminance levels from the darkest to the brightest, and then the pixel values ​​of the pixels of each color in each image region are quantized to corresponding luminance values. The luminance levels corresponding to the pixels of each color can be determined based on the luminance ranges in which the luminance values ​​corresponding to the pixels of each color are located. This allows the luminance levels at which the pixels of each color in each image region are located to be determined, which is equivalent to obtaining the pixels and their colors located at each luminance level in each image region.

[0025] Referring to FIG. 4, FIG. 4 is a schematic diagram illustrating the correspondence between luminance levels and luminance intervals in some embodiments. In some embodiments, the darkest luminance to the brightest luminance may correspond to black and white, respectively. In other words, for example, if a pixel value (gray value) is divided into m gray values ​​from black to white, 0 may be the darkest level and (m-1) may be the brightest level, and the luminance value may be considered to be a plurality of gray values ​​from 0 to (m-1). The preset rule may include a preset number of levels. Based on the number of levels included in the preset rule, the image may be divided into a corresponding number of luminance levels. The luminance interval corresponding to each luminance level may be obtained by dividing the total number of gray values ​​by the number of luminance levels. For example, as shown in FIG. 4, if m is equal to 256, i.e., if the total number of gray values ​​is 256 and the number of luminance levels is 16, the luminance interval corresponding to each luminance level corresponds to 256 / 16=16 gray values. The higher the luminance level, the brighter it is. As shown in FIG. 4, when the brightness level is 1, the corresponding brightness interval is a gray value range of 0-15, and when the brightness level is 2, the corresponding brightness interval is a gray value range of 16-31.

[0026] After obtaining the brightness values ​​corresponding to the pixels of each color in each image region through quantization, the brightness level at which each pixel of each color is located can be obtained according to the range of gray values ​​in which the brightness values ​​are located.

[0027] In some embodiments, the pixels include first, second, and third color pixels according to the type of color. In some embodiments, for example, the first, second, and third color pixels are R (Red), G (Green), and B (Blue), respectively, and the pixels in the image are arranged in a Bayer array. "Quantizing the pixel values ​​of pixels of each color in each image region to a corresponding luminance value" may include the following: Average the pixel values ​​of two pixels of the same color in a pixel unit in which each pixel of each color is located to obtain an average pixel value, and use the average pixel value as the luminance value of the pixel.

[0028] The Bayer array is a type of color pattern arrangement for image sensor color filters. In this color pattern arrangement, a color image is captured by arranging red, green, and blue filters on the image sensor, allowing each pixel to recognize only one color of light. Color filters are attached to the photodiodes (corresponding to pixels) on the surface of the image sensor in a Bayer pattern. Image light shining on the image sensor surface passes through each color filter and reaches the photodiode, which generates a voltage according to the amount of light received that passes through the attached filter color. The voltage level is converted into a digital value and output from the image sensor as Bayer array data.

[0029] In the Bayer array, each pixel is located in one pixel unit, and the pixel unit includes 2x2 pixels, which include two pixels of the same color located diagonally. The luminance value of a pixel is an average pixel value obtained by averaging the pixel values ​​of the two pixels of the same color in the pixel unit in which the pixel is located. In this application, pixel value, gray value, and luminance value can be understood as the same value.

[0030] Referring back to FIG. 3, in some embodiments, step 12 of "compute statistics on the luminance information of the pixels included in each image region" further includes the following content. 125: Obtain pixel values ​​for pixels of each color located at each luminance level in each image region. 127: Calculate the sum of pixel values ​​of all pixels of each color located at each luminance level in each image region to obtain the sum of pixel values ​​of each color located at each luminance level in each image region.

[0031] That is, in some embodiments, after determining the luminance level corresponding to each pixel of each color, the luminance level corresponding to each pixel of each color is clear, and further determining the pixels of each color located at each luminance level in each image region, obtaining pixel values ​​for the pixels of each color located at each luminance level in each image region, and then summing the pixel values ​​of all pixels of each color located at each luminance level for each image region to obtain a total pixel value for each color located at each luminance level in each image region.

[0032] In some embodiments, the method may further include storing in a memory the total pixel values ​​of each color located at each luminance level within each image region.

[0033] 5, which is a schematic diagram showing a storage structure for storing the total pixel values ​​of each luminance level in each image region in some embodiments of the present application, i.e., the storage structure for storing the total pixel values ​​of each luminance level in each image region of image P1 in memory.

[0034] As described above, a pixel includes a first color pixel, a second color pixel, and a third color pixel according to the color type. An example of a storage structure for the total pixel value of one color pixel is shown in FIG. 5. For example, FIG. 5 shows an example of a storage structure for the total pixel value of an R pixel in an RGB color system.

[0035] FIG. 5 shows an example in which an image P1 is divided into 30 image regions A0 to A29, and the brightness levels are divided into 16 brightness levels Lv0 to Lv15.

[0036] By calculating the sum of the pixel values ​​of R pixels located at brightness level Lv0 in image area A0, a total pixel value R0_0 of R pixels located at brightness level Lv0 in image area A0 is obtained; by calculating the sum of the pixel values ​​of R pixels located at brightness level Lv1 in image area A0, a total pixel value R1_0 of R pixels located at brightness level Lv1 in image area A0 is obtained; ...by calculating the sum of the pixel values ​​of R pixels located at brightness level Lv15 in image area A0, a total pixel value R15_0 of R pixels located at brightness level Lv15 in image area A0 is obtained. Similarly, by tallying the sum of the pixel values ​​of R pixels located at brightness level Lv0 in image region A1, a total pixel value R0_1 of R pixels located at brightness level Lv0 in image region A1 is obtained... By tallying the sum of the pixel values ​​of R pixels located at brightness level Lv15 in image region A1, a total pixel value R15_1 of R pixels located at brightness level Lv15 in image region A1 is obtained... Furthermore, by tallying the sum of the pixel values ​​of R pixels located at brightness level Lv0 in image region A29, a total pixel value R0_29 of R pixels located at brightness level Lv0 in image region A29 is obtained... By tallying the sum of the pixel values ​​of R pixels located at brightness level Lv15 in image region A29, a total pixel value R15_29 of R pixels located at brightness level Lv15 in image region A29 is obtained. In this way, one total pixel value of R pixels located at each brightness level in each image region is obtained. That is, in this way, the memory structure shown in FIG. 5 can be obtained.

[0037] In an RGB color system, the storage structure for the total pixel values ​​of G pixels located at each luminance level in each image region and the storage structure for the total pixel values ​​of B pixels located at each luminance level in each image region can both be the structure shown in Figure 5, and will not be described in detail here.

[0038] The term "storage structure" in this application can refer to the format in which data is stored in memory, for example, the format of a table as shown in FIG.

[0039] Referring to FIG. 6, FIG. 6 is another flowchart illustrating an auto white balance processing method in some embodiments of the present application. FIG. 6 shows some processes of the auto white balance processing method. In some embodiments, the pixels include first color pixels, second color pixels, and third color pixels according to color types. As shown in FIG. 6, after step 127 of "calculating the sum of pixel values ​​of all pixels of each color located at each brightness level in each image region to obtain the sum of pixel values ​​of each color located at each brightness level in each image region," the auto white balance processing method further includes the following content: 61: Perform color space conversion to convert the total pixel values ​​of each luminance level corresponding to the first, second and third color pixels into first and second chromaticity values ​​of each luminance level.

[0040] That is, in some embodiments, the present application performs color space conversion to convert the total pixel values ​​of each luminance level corresponding to the first color pixel, the second color pixel, and the third color pixel into first chromaticity values ​​and second chromaticity values ​​of each luminance level for subsequent calculation of white balance gains.

[0041] In some embodiments, the first color pixel, the second color pixel, and the third color pixel are R pixels, G pixels, and B pixels, respectively. Performing color space conversion to convert the total pixel values ​​of the first color pixel, the second color pixel, and the third color pixel at each luminance level into first and second chromaticity values ​​at each luminance level includes converting the total pixel values ​​of the RGB pixel at each luminance level into R / G values ​​and B / G values ​​at each luminance level.

[0042] The R / G value of each brightness level is the ratio of the total pixel value of the R pixels of the brightness level to the total pixel value of the G pixels of the brightness level, and the B / G value of each brightness level is the ratio of the total pixel value of the B pixels of the brightness level to the total pixel value of the G pixels of the brightness level.

[0043] That is, in some embodiments, taking the RGB color system as an example, if the first color pixel, the second color pixel, and the third color pixel are R pixels, G pixels, and B pixels, respectively, the R / G value for each brightness level can be calculated by dividing the sum of the R pixels at each brightness level by the sum of the G pixels at that brightness level, and the B / G value for each brightness level can be calculated by dividing the sum of the B pixels at each brightness level by the sum of the G pixels at that brightness level. The sum of the pixel values ​​at each brightness level corresponding to the RGB three-color pixels is converted into the R / G values ​​and B / G values ​​at each brightness level to achieve color space conversion, i.e., converting the ternary color space into a binary color space.

[0044] In some embodiments, the method may further include storing R / G and B / G values ​​for each luminance level within each image region in a memory.

[0045] 7, which is a schematic diagram showing a storage structure of the R / G values ​​of each luminance level in each image region in some embodiments of the present application, that is, FIG. 7 shows a storage structure when the R / G values ​​of each luminance level in each image region of image P1 are stored in memory.

[0046] FIG. 7 also shows an example in which an image P1 is divided into 30 image regions A0 to A29, and the brightness levels are divided into 16 brightness levels Lv0 to Lv15.

[0047] The R / G value corresponding to brightness level Lv0 in image area A0 (i.e., R0_0 / G0_0) is obtained by dividing the total pixel value of R pixels located at brightness level Lv0 in image area A0 by the total pixel value of G pixels located at brightness level Lv0 in image area A0. The R / G value corresponding to brightness level Lv1 in image area A0 (i.e., R1_0 / G1_0) is obtained by dividing the total pixel value of R pixels located at brightness level Lv1 in image area A0 by the total pixel value of G pixels located at brightness level Lv1 in image area A0. ...The R / G value corresponding to brightness level Lv15 in image area A0 (i.e., R15_0 / G15_0) is obtained by dividing the total pixel value of R pixels located at brightness level Lv15 in image area A0 by the total pixel value of G pixels located at brightness level Lv15 in image area A0. In this way, the R / G values ​​for each brightness level in image area A0 can be obtained. Similarly, it is possible to obtain the R / G values ​​of each luminance level in the image areas A1 to A15, etc. In this way, the storage structure shown in FIG.

[0048] The storage structure of the B / G values ​​for each luminance level in each image region can also be the structure shown in FIG. 7, and will not be described in detail here.

[0049] In some embodiments, the image to be processed may not be in the RGB color space / color system, but may be in other color spaces such as the YUV color space or the Lab color space, before the color space conversion is performed.

[0050] In the YUV color space, the first color pixel, the second color pixel, and the third color pixel can be Y pixel, U pixel, and V pixel, respectively, where Y represents luminance, and U and V represent chrominance. The Y component represents the luminance information of the image, and the U and V components represent the color information. During color space conversion, the Y component is removed, leaving the U and V components, thereby achieving color space conversion, i.e., conversion from the YUV color space to the UV color space.

[0051] A Y pixel, a U pixel, and a V pixel may form one pixel unit. The Y component is removed, leaving the U and V components, meaning that only the value corresponding to the U pixel and the value corresponding to the V pixel are left in each pixel unit.

[0052] Similarly, in the Lab color space, the first color pixel, the second color pixel, and the third color pixel can be an L pixel, an a pixel, and a b pixel, respectively, where L represents luminance, a represents a range from red to green, and b represents a range from yellow to blue. During color space conversion, the L component is removed, leaving the a and b components, thereby achieving color space conversion, i.e., conversion from the Lab color space to the ab color space.

[0053] The L pixel, the a pixel, and the b pixel may form one pixel unit. Removing the L component and leaving the a component and the b component means that only the value corresponding to the a pixel and the value corresponding to the b pixel (e.g., pixel value) are left in each pixel unit.

[0054] In some embodiments, in the case of a YUV color space, the total pixel value of each color located at each luminance level in each image region may include the total pixel value of U pixels located at each luminance level in each image region and the total pixel value of V pixels located at each luminance level in each image region. In some embodiments, in the case of a Lab color space, the total pixel value of each color located at each luminance level in each image region may include the total pixel value of a pixels located at each luminance level in each image region and the total pixel value of b pixels located at each luminance level in each image region.

[0055] Therefore, the auto white balance processing method of the present application can perform white balance processing on images in various color spaces, not just the RGB color space, such as the YUV color space and Lab color space. In this application, white balance processing on images in the RGB color space will be mainly described as an example.

[0056] Referring to Figure 8, Figure 8 is a diagram illustrating a sub-process of step 13 of Figure 1 of the present application in some embodiments. In some embodiments, as shown in Figure 8, step 13 of "performing density estimation for pixels of multiple luminance levels using a non-parametric method to obtain density estimates for at least bright level pixels" may include the following content: 81: Obtain R / G and B / G values ​​corresponding to the bright level and R / G and B / G values ​​corresponding to the dark level. 83: Based on the R / G values ​​and B / G values ​​corresponding to the bright level and the R / G values ​​and B / G values ​​corresponding to the dark level, a non-parametric method is used to estimate the density of the data-free areas in the color space, thereby obtaining density estimates for bright level pixels and dark level pixels.

[0057] That is, in some embodiments, taking the RGB color space as an example, the total pixel values ​​of each brightness level corresponding to RGB color pixels are converted into R / G values ​​and B / G values ​​of each brightness level. After the color space conversion is completed, R / G values ​​and B / G values ​​corresponding to the brightness levels at the bright levels and R / G values ​​and B / G values ​​corresponding to the brightness levels at the dark levels can be obtained. Next, based on the R / G values ​​and B / G values ​​corresponding to the brightness levels and the R / G values ​​and B / G values ​​corresponding to the dark levels, a non-parametric method is used to estimate the density of the data-free parts in the color space, thereby obtaining density estimates for two types of pixels, i.e., density estimates for brightness-level pixels and density estimates for dark-level pixels.

[0058] Using a nonparametric method to estimate the density of areas with no data in a color space can refer to using a nonparametric method to estimate the density of areas with no data in a transformed color space.

[0059] In some embodiments, obtaining R / G values ​​and B / G values ​​corresponding to a bright level and R / G values ​​and B / G values ​​corresponding to a dark level includes the following: A secondary bright level is set as the bright level, and R / G values ​​and B / G values ​​corresponding to the bright level are obtained. The secondary bright level is a brightness level that is one step lower than the highest brightness level among all brightness levels. A brightness level close to the target brightness value is set as the dark level, and R / G values ​​and B / G values ​​corresponding to the dark level are obtained. The target brightness value is obtained by multiplying the highest brightness value by a preset ratio.

[0060] That is, in some embodiments, the bright level can be a secondary bright level that is one step lower than the highest brightness level among all brightness levels, and the dark level can be a brightness level that is close to the target brightness value.

[0061] Referring again to FIG. 4, for example, when the brightness levels are divided into a total of 16 levels, Lv0 to Lv15, as shown in FIG. 4, the brightest brightness level is Lv15, and the next brightest level is Lv14.

[0062] The reason why the highest brightness level (Lv15) is not selected as the bright level is to prevent an increase in error due to a saturated region. Obviously, in some embodiments, the bright level may be a brightness level that is a predetermined number of steps lower than the highest brightness level, and may not be a secondary bright level. For example, the bright level may be a brightness level Lv13 that is two steps lower than the highest brightness level Lv15, or may be a brightness level Lv12 that is three steps lower than the highest brightness level Lv15.

[0063] In some embodiments, the luminance levels close to the target luminance value include a target luminance level at which the target luminance value is located and a luminance level adjacent to the target luminance level, the difference between the upper limit or lower limit and the target luminance value being smaller than a predetermined luminance value. The target luminance value is obtained by multiplying the maximum luminance value by a predetermined ratio. Therefore, in some embodiments, when the target luminance value is obtained by multiplying the maximum luminance value by a predetermined ratio, the target luminance level corresponding to the luminance section to which the target luminance value falls can be determined to be a dark level, and a luminance level adjacent to the target luminance level, the difference between the upper limit or lower limit and the target luminance value being smaller than a predetermined luminance value, can also be determined to be a dark level.

[0064] In some embodiments, the preset percentage may be a value such as 18%. In some embodiments, the preset brightness value is less than half the number of gray values ​​included in the brightness range corresponding to each brightness level. For example, as described above, if there are a total of 16 brightness levels, the darkest brightness value / darkest gray value is 0, and the highest brightness value / highest gray value is 255, i.e., if a total of 256 gray values ​​are included, the brightness range corresponding to each brightness level will correspond to 256 / 16=16 gray values, i.e., the number of corresponding gray values ​​is 16. Therefore, the preset brightness value is less than 16 / 2=8. In some embodiments, the preset brightness value may be a value such as 5 or 6.

[0065] Taking FIG. 4 as an example, the highest brightness value, i.e., the highest gray value, is 255. When the preset percentage is 18%, the target brightness value is 255×18%=45.9, which falls within the brightness range (32 to 47) corresponding to brightness level Lv2, so brightness level Lv2 is a dark level. Furthermore, the lower limit of Lv3 is 48, and the difference between the lower limit of Lv3 and 45.9 is 2.1, which is relatively small and smaller than the preset brightness value, so brightness level Lv3 is also a dark level. Therefore, in some embodiments, the dark level may include two brightness levels, Lv2 and Lv3.

[0066] Generally, automatic exposure control works to make the brightness of the subject about 18% of its maximum brightness. Therefore, if 18% of the maximum brightness value is selected as the target brightness value and a brightness level close to the target brightness value is determined as the dark level, an appropriate white balance gain can be obtained by automatic exposure control.

[0067] Obviously, in some embodiments, the preset percentage may be other values, such as 16%, 20%, etc.

[0068] Referring to FIG. 9, FIG. 9 is a diagram illustrating pixel density estimates and data points for brightness levels in some embodiments of the present application.

[0069] 9, the abscissa is the R / G value, the ordinate is the B / G value, and the brightness level is shown as Lv 14. In FIG. 9, a contour line L1 of a plurality of density estimates is shown, and points on the contour line of the density estimates correspond to the same density estimate.

[0070] 9 may represent one image region, and the abscissa of the data point D1 corresponding to the image region is the R / G value corresponding to the brightness level Lv14 in the image region, and the ordinate is the B / G value corresponding to the brightness level Lv14 in the image region. Because only some image regions may have B / G values ​​corresponding to the brightness level Lv14 and R / G values ​​corresponding to the brightness level Lv14, if an image is divided into 30 image regions, the number of data points D1 shown in FIG. 9 may be less than 30. For example, as shown in FIG. 9, the number of data points D1 is less than 30.

[0071] As shown in FIG. 9, the density estimates of bright level pixels are mainly concentrated within the contours where the density estimate is 0.8 or 0.6.

[0072] Referring to FIG. 10, FIG. 10 is a diagram illustrating density estimates and data points for dark level pixels in some embodiments of the present application.

[0073] 10, the abscissa is also the R / G value, the ordinate is also the B / G value, and the dark levels are Lv2 and Lv3 as an example. FIG. 10 also shows a contour line L1 of a plurality of density estimates, and points on the contour line of the density estimates correspond to the same density estimate.

[0074] 10 may represent one image region, and the abscissa of the data point D1 corresponding to the image region is the R / G value corresponding to the brightness levels Lv2 and Lv3 in the image region, and the ordinate is the B / G value corresponding to the brightness levels Lv2 and Lv3 in the image region. Since only some image regions may have R / G values ​​and B / G values ​​corresponding to the brightness levels Lv2 and Lv3, if an image is divided into 30 image regions, the number of data points D1 shown in FIG. 10 may be less than 60 because the maximum number is the number of image regions multiplied by the number of brightness levels, i.e., 30×2=60.

[0075] As shown in FIG. 10, the density estimates of the dark level pixels are mainly concentrated within the contours where the density estimate is 0.8 or 0.6.

[0076] 9 and 10, the distribution of the concentrated areas of the density estimates of bright level pixels and the distribution of the concentrated areas of the density estimates of dark level pixels are located in similar positions. Therefore, the density estimates of bright level pixels can be used as weights to calculate the first white balance gains corresponding to each color based on the pixel values ​​corresponding to the dark level pixels of each color.

[0077] 11, which is a schematic diagram integrating the density estimates of the bright levels with the scatter of data points of the dark levels in some embodiments of the present application. In FIG. 11, the density estimates of the bright levels are represented by a contour line L1 of the density estimates, and the scatter points, i.e., data points D1 in the figure, represent the coordinate positions of the dark levels. Areas where the density estimates of the bright levels are high (e.g., the area indicated by A in the figure) are considered to be close to the light source color and therefore can be weighted heavily, while areas where the density estimates of the bright levels are low (e.g., the area indicated by B in the figure) are considered to be far from the light source color and therefore can be weighted lightly.

[0078] If the distribution of dark level pixels in some coordinate regions in the color space is close to the distribution of bright level pixels, the distribution under diffuse reflection in those coordinate regions will be approximately equal to the distribution of the light source color, that is, the object color in those regions will be considered as achromatic. As can be seen from the above, it is reasonable to use the density estimate of the bright level pixels as a weight to calculate the first white balance gain corresponding to each color based on the pixel values ​​corresponding to the dark level pixels of each color.

[0079] In some embodiments, the non-parametric technique may be a kernel density estimation technique. Estimating density in data-free portions of the color space using a non-parametric technique to obtain density estimates for light level pixels and dark level pixels may include estimating density in data-free portions of the color space using a kernel density estimation technique to obtain density estimates for light level pixels and dark level pixels.

[0080] In some embodiments, the data-free portion can be a data-free portion other than the portion with data points shown in FIGS. 9 and 10 above.

[0081] Kernel density estimation is a commonly used non-parametric method. In kernel density estimation, a probability density function for estimating a random variable is used to estimate the density of a data-free area to obtain density estimates for light-level pixels and dark-level pixels. Therefore, kernel density estimation can be used to estimate the density of a data-free area in a color space to obtain density estimates for light-level pixels and dark-level pixels.

[0082] In some embodiments, the kernel function used in the kernel density estimation technique may be a Gaussian function.

[0083] In some embodiments, the density estimate for bright level pixels or the density estimate for dark level pixels is:

number

[0084]

number

[0085] The kernel density estimation technique described above can provide density estimates for bright level pixels and dark level pixels.

[0086] In some embodiments, the bandwidth is 0.1. Obviously, in other embodiments, the bandwidth may be any other suitable value, such as 0.08.

[0087] Kernel density estimation techniques are commonly used non-parametric techniques and will not be described in detail here.

[0088] 12, which is a diagram illustrating a sub-process of step 14 of FIG. 1 of the present application in some embodiments. In some embodiments, step 14 of "calculating a first white balance gain corresponding to each color based at least on density estimates of pixels of light levels" includes the following content: 141: Obtain the R / G and B / G values ​​corresponding to the dark level. 143: Calculate a first white balance gain corresponding to each color based on the total pixel value of the dark level of each color and the density estimate value of the light level of the corresponding coordinate.

[0089] That is, in some embodiments, the total pixel value of each luminance level of each color is obtained, and color space conversion is performed to obtain a binary color space with R / G values ​​as the abscissa and B / G values ​​as the ordinate, as shown in Figures 9 and 10, and obtain density estimates of the bright levels as shown in Figure 9. Next, a first white balance gain corresponding to each color is calculated based on the total pixel value of each coordinate in the binary color space for the dark levels of each color and the density estimates of the bright levels of the corresponding coordinates.

[0090] As described above and shown in FIG. 5, after obtaining the total pixel value of each color at each brightness level in each image region, the total pixel value of each color in each image region has a corresponding brightness level. As described above and shown in FIG. 7, after performing color space conversion to obtain R / G values ​​and B / G values ​​corresponding to each brightness level in each image region, the R / G values ​​and B / G values ​​of each image region also have corresponding brightness levels. Therefore, the total pixel value of each color in each image region and the R / G values ​​and B / G values ​​of the corresponding image region can be related by brightness level. Therefore, the coordinates in the binary color space, i.e., the abscissa and ordinate, of the total pixel value of the dark level of each color in each image region are the R / G value and B / G value corresponding to the dark level in the image region, respectively.

[0091] In some embodiments, step 143 of "calculating a first white balance gain corresponding to each color based on the total pixel value of the dark level of each color and the density estimate of the light level of the corresponding coordinate" includes the following: Obtaining a target value for each color based on the total pixel value of the dark level of each color and the density estimate of the light level of the corresponding coordinate; Obtaining a first white balance gain corresponding to each color based on the target value for each color.

[0092] That is, in some embodiments, calculating the first white balance gain corresponding to each color based on the total pixel value of the dark level of each color and the density estimate of the light level of the corresponding coordinate can be by first obtaining a target value for each color based on the total pixel value of the dark level of each color and the density estimate of the light level of the corresponding coordinate, and then obtaining the first white balance gain corresponding to each color based on the target value of each color.

[0093] In some embodiments, obtaining a target value for each color based on a sum of pixel values ​​of the dark levels of each color and a density estimate of the light levels of the corresponding coordinates includes obtaining a target value for each color based at least on a product of a sum of pixel values ​​of the dark levels of each color and a density estimate of the light levels of the corresponding coordinates.

[0094] That is, in some embodiments, the target value for each color is derived based at least on the product of the total pixel value of the dark level of each color and the density estimate of the light level of the corresponding coordinate.

[0095] In some embodiments, obtaining the target value for each color based at least on the product of the total pixel value of the dark level of each color and the density estimate of the light level of the corresponding coordinate includes calculating the target value for each color based on the following formula:

number

[0096] R1 is the target value for red, G1 is the target value for green, B1 is the target value for blue, and R LV2Lv3 is the total pixel value of the red dark level, and G Lv2Lv3 is the total pixel value of the green dark level, and B Lv2Lv3 is the total pixel value of the blue dark level, and KDE Lv14 is the density estimate of the light level, and (x,y) are the coordinates in color space, i.e., the coordinates in the binary color space after the transformation shown in FIGS.

[0097] That is, in some embodiments, specifically, the red target value R1, the green target value G1, and the blue target value B1 are calculated based on the above formulas.

[0098] In some embodiments, obtaining a first white balance gain corresponding to each color based on the target value of each color includes: determining that the first white balance gain corresponding to red is G1 / R1; determining that the first white balance gain corresponding to blue is G1 / B1; and determining that the first white balance gain corresponding to green is 1.

[0099] That is, in some embodiments, in an RGB color system, after obtaining target values ​​R1, G1, and B1 of RGB colors, it determines that the first white balance gain corresponding to red is G1 / R1, the first white balance gain corresponding to blue is G1 / B1, and the first white balance gain corresponding to green is 1.

[0100] In white balance processing, the green white balance gain is typically set to 1. This is because the human eye is more sensitive to green than red or blue. Therefore, in image processing, the green channel is often used as a reference to ensure that the overall brightness and color balance of an image conforms to the visual habits of the human eye. Setting the green white balance gain to 1 also simplifies the calculation process for white balance processing. Furthermore, many image sensors (such as Bayer array lenses) are designed with twice the number of G pixels. Furthermore, the green channel is generally more sensitive and has less noise in terms of S / N. Therefore, fixing the green gain allows the sensor to be more effectively utilized to capture original information and ensure image quality. Furthermore, fixing the green gain also makes it easier to balance colors by adjusting the red and blue gains, ensuring that images maintain natural color representation even under different lighting conditions.

[0101] 13, which is another flowchart illustrating an auto white balance processing method in some embodiments of the present application. As shown in FIG. 13, the auto white balance processing method includes the following contents: 131: Divide the image to be processed into multiple image regions. 132: Calculate the luminance information of the pixels included in each image region. The luminance information includes at least the luminance level at which the pixels included in each image region are located. 133: Using a non-parametric method, density estimation is performed on pixels at multiple brightness levels to obtain density estimates for at least the bright level pixels. 134: Calculate a first white balance gain corresponding to each color based at least on the density estimate of the bright level pixels. 135: The image waiting to be processed is processed using a different method to obtain a second white balance gain for each color. 136: The first white balance gain for each color and the second white balance gain for each color are mixed according to the gain mixing ratio to obtain the final white balance gain for each color.

[0102] Therefore, in some embodiments of the present application, a to-be-processed image is processed in a different manner to obtain a second white balance gain for each color, and then a final white balance gain is obtained based on the first white balance gain and the second white balance gain. Therefore, since multiple image processing methods are comprehensively taken into consideration, it is possible to adapt to the requirements of white balance processing for multiple different imaging scenes, and images in different imaging scenes can have better effects through white balance processing.

[0103] Steps 131 to 134 correspond to steps 11 to 14 in FIG. 1, respectively, and for more specific details, please refer to the above description.

[0104] In some embodiments, step 136 of "mixing the first white balance gain for each color and the second white balance gain for each color according to a gain mix ratio to obtain a final white balance gain for each color" includes calculating the final white balance gain for each color based on the formula MixRatio * Pgain0 + (1 - MixRatio) * Pgain1, where Pgain0 is the first white balance gain for any one color, MixRatio is the gain mix ratio, and Pgain1 is the second white balance gain for any one color.

[0105] In some embodiments, the range of values ​​for the gain mix ratio is 0≦MixRatio≦1.

[0106] That is, in some embodiments, a gain mix ratio with a value ranging from 0 to 1 can be obtained, and the gain mix ratio is used as a weight for the first white balance gain, and 1-MixRatio is used as a weight for the second white balance gain, thereby obtaining a weighted sum of the first white balance gain and the second white balance gain to obtain the final white balance gain.

[0107] In the auto white balance processing method shown in FIG. 1, obtaining a first white balance gain by processing using a nonparametric method is generally suitable for scenes in which dark-level pixels are concentrated near areas where bright-level pixels are concentrated. For example, consider the density distribution situations shown in FIGS. 9 to 11. In such scenes, the first white balance gain obtained by processing using the auto white balance processing method shown in FIG. 1 can be used as the final white balance gain. Therefore, the MixRatio may be relatively large, even 1, and in this case, there is no need to obtain a second white balance gain using a different method. In other scenes, for example, scenes other than those in which dark-level pixels are concentrated near areas where bright-level pixels are concentrated, a different method may be more suitable. Therefore, in this case, a second white balance gain is further obtained using a different method, and then the first white balance gain for each color and the second white balance gain for each color are mixed according to the gain mixing ratio to obtain the final white balance gain for each color. The final white balance gain is more suitable for the current scene.

[0108] In some embodiments, the gain blending ratio is obtained based on a cross-correlation coefficient between a density estimate of a first pixel and a density estimate of a second pixel, where the first pixel is a pixel at least one luminance level that is a predetermined number of steps lower than the highest luminance level, and the second pixel is all pixels at luminance levels close to a target luminance value, where the target luminance value is obtained by multiplying the highest luminance value by a predetermined ratio.

[0109] In some embodiments, the at least one brightness level that is a predetermined number of steps lower than the highest brightness level includes a secondary bright level. The secondary bright level is a brightness level that is one step lower than the highest brightness level among all brightness levels. The brightness levels close to the target brightness value include a target brightness level at which the target brightness value is located and a brightness level that is adjacent to the target brightness level and whose difference between the upper limit or lower limit and the target brightness value is smaller than a predetermined brightness value.

[0110] That is, in some embodiments, the first pixel can be the bright level pixel, and the second pixel can be the dark level pixel. For the bright level pixel and the dark level pixel, please refer to the related description above, and will not be described in detail here.

[0111] In some embodiments, there is a positive correlation between the gain blending ratio and the cross-correlation coefficient between the density estimates of the first and second pixels, i.e., the larger the cross-correlation coefficient between the density estimates of the first and second pixels, the larger the gain blending ratio will generally be.

[0112] As shown in Figures 9 and 10, in some scenes, for example, when dark-level pixels are concentrated near a concentration of bright-level pixels, the distribution of the kernel density estimates for bright level Lv14 is similar to the distribution of the kernel density estimates for dark levels Lv2 and Lv3, resulting in a relatively large cross-correlation coefficient and a relatively large gain mixing ratio, i.e., a relatively large weight for the first white balance gain. Therefore, as described above, in this case, it is more appropriate to use the first white balance gain obtained primarily by processing using the auto white balance processing method shown in Figure 1 as the final white balance gain. Conversely, in some scenes, when the kernel density estimates for bright level Lv14 differ significantly from the kernel density estimates for dark levels Lv2 and Lv3, the maximum cross-correlation coefficient is small and the gain mixing ratio is relatively small. In this case, it is more appropriate to obtain the final white balance gain using the second white balance gain.

[0113] Referring to FIG. 14, FIG. 14 is a schematic diagram showing the relationship between the gain mixing ratio and the cross-correlation coefficient in some embodiments of the present application. In some embodiments, two thresholds, namely, a lower threshold Low_Thr and an upper threshold High_Thr, are set for the cross-correlation coefficient so that the application / non-application of the nonparametric method shown in FIG. 1 and another method is not abruptly switched. As shown in FIG. 14, the relationship between the gain mixing ratio MixRatio and the cross-correlation coefficient Cc is as follows: MixRatio=k1*(Cc-Low_Thr), where Low_Thr≦Cc≦High_Thr and k1=1 / (High_Thr-Low_Thr).

[0114] The above formula can determine a gain mixing ratio MixRatio that is more suitable for the current scene, and further, can obtain a final white balance gain that is more suitable for the current scene. In some embodiments, the lower threshold Low_Thr may be 0.75, and the upper threshold High_Thr may be 0.8. Of course, in other embodiments, the lower threshold Low_Thr and the upper threshold High_Thr may be other appropriate values.

[0115] In some embodiments, the alternative approach may be one that uses a dichromatic reflection model, or another approach.

[0116] If the other method is a method using a dichromatic reflection model, the specific scheme of the method using the dichromatic reflection model can be referred to the contents described in Patent Application No. 2023-195971, and will not be described in detail here.

[0117] In some embodiments, the auto white balance processing method may further include: performing gain processing on pixel values ​​of pixels of each color using the first white balance gain or the final white balance gain of each color, obtaining and outputting pixel values ​​of pixels of each color after gain processing, and displaying the white balance processed image.

[0118] In some embodiments, after obtaining the first white balance gain or the final white balance gain for each color, the auto white balance processing method may further include correcting the first white balance gain or the final white balance gain for each color to obtain a corrected first white balance gain or the final white balance gain.

[0119] In some embodiments, performing gain processing on pixel values ​​of pixels of each color using the first white balance gain or final white balance gain for each color may include performing gain processing on pixel values ​​of pixels of each color using the corrected first white balance gain or final white balance gain for each color.

[0120] Referring to FIG. 15, FIG. 15 is a schematic diagram showing a comparison of images obtained after white balancing by several methods.

[0121] Figure 15 shows multiple developed images obtained after performing white balance processing on images primarily consisting of red, blue, and green using each method. Here, the images primarily consisting of red, blue, and green were obtained by photographing a subject primarily consisting of red, blue, and green under a standard light source.

[0122] Specifically, Figure 15 shows multiple developed images a1, a2, a3 obtained after performing white balance processing on images primarily consisting of red, blue, and green using the method shown in Figure 1, multiple developed images b1, b2, b3 obtained after performing white balance processing on images primarily consisting of red, blue, and green using a method using a dichromatic reflection model, and multiple developed images c1, c2, c3 obtained by performing white balance processing using a white balance gain calculated from bright level RGB values.

[0123] As can be clearly seen from Fig. 15, the multiple developed images a1, a2, and a3 obtained after performing white balance processing on the images primarily consisting of red, blue, and green using the method shown in Fig. 1 have better contrast than developed images obtained using other methods or schemes. Therefore, the method shown in Fig. 1 provides better image quality than other methods.

[0124] The auto white balance processing method of the present application divides an image into multiple image regions, collects statistics on the luminance information of pixels contained in each image region, performs density estimation on pixels of multiple luminance levels using a nonparametric method to obtain at least density estimates for pixels of bright levels, and calculates first white balance gains corresponding to each color based at least on the density estimates for pixels of bright levels, thereby achieving accurate white balance gains in a relatively simple manner. In particular, the present application further mixes the first white balance gain and a second white balance gain obtained by a separate method according to a gain mixing ratio to obtain a final white balance gain. This enables better adaptation to the white balance processing requirements of different imaging scenes, and images of different imaging scenes can achieve better results from white balance processing.

[0125] 16, which is a block diagram showing the structure of an auto white balance processing device in some embodiments of the present application. As shown in FIG. 16, the present application further provides an auto white balance processing device 100. The auto white balance processing device 100 includes a statistical analysis unit 101, a density estimation unit 102, and a white balance gain determination unit 103.

[0126] The statistical analysis unit 101 is configured to divide an image awaiting processing into multiple image regions and collect statistics on luminance information of pixels included in each image region, the luminance information including at least the luminance level at which the pixels included in each image region are located. The density estimation unit 102 is configured to perform density estimation on pixels of multiple luminance levels using a nonparametric method to obtain at least density estimates for pixels of bright levels. The white balance gain determination unit 103 is configured to calculate first white balance gains corresponding to each color based at least on the density estimates for pixels of bright levels.

[0127] The statistical analysis unit 101 performs functional operations corresponding to steps 11 and 12 shown in Fig. 1, the density estimation unit 102 performs functional operations corresponding to step 13 shown in Fig. 1, and the white balance gain determination unit 103 performs functional operations corresponding to step 14 shown in Fig. 1. For details of the functional operations performed by each unit in the auto white balance processing device 100, please refer to the above-mentioned auto white balance processing method.

[0128] In some embodiments, the white balance gain determination unit 103 is further configured to perform color space conversion to convert the total pixel values ​​of each luminance level corresponding to the first color pixel, the second color pixel, and the third color pixel into the first chromaticity value and the second chromaticity value of each luminance level, i.e., in some embodiments, the white balance gain determination unit 103 further performs a functional operation corresponding to step 61 in FIG. 6 above.

[0129] 16, the auto white balance processing device 100 may further include a mixing processing unit 104. The mixing processing unit 104 is configured to process the image to be processed in a different manner to obtain a second white balance gain for each color, and to mix the first white balance gain for each color and the second white balance gain for each color according to a gain mixing ratio to obtain a final white balance gain for each color.

[0130] The blending processing unit 104 can perform functional operations corresponding to steps 135 and 136 in Fig. 13. For further functional operations performed by the blending processing unit 104, reference can be made to the relevant contents of steps 135 and 136 in the above-described auto white balance processing method.

[0131] The auto white balance processing device 100 can be a chip such as an image processing chip, a central processing chip, etc. The statistical analysis unit 101, the density estimation unit 102, the white balance gain determination unit 103, etc. can be hardware units within the image processing chip.

[0132] 17, which is a block diagram showing a structure of an imaging device according to some embodiments of the present application. As shown in FIG. 17, the present application further provides an imaging device 200. The imaging device 200 includes the auto white balance processing device 100. As shown in FIG. 17, the imaging device 200 further includes an image acquisition unit 201 and an image generation unit 202.

[0133] The image acquisition unit 201 is configured to acquire images waiting to be processed.

[0134] The image generation unit 202 is configured to generate a processed image, i.e., generate a white balance processed image, based on the white balance gain obtained by the auto white balance processing device 100. For example, the image generation unit 202 performs gain processing on the pixel values ​​of pixels of each color based on the white balance gain obtained by the auto white balance processing device 100, to generate a white balance processed image.

[0135] The image acquisition unit 201 can be a camera module including a camera and an image sensor, etc. The image generation unit 202 can be a hardware unit within an image processing chip, a central processing chip, etc.

[0136] 18, which is a block diagram showing a structure of an electronic device in some embodiments of the present application. As shown in FIG. 18, the present application further provides an electronic device 300. The electronic device 300 includes a memory 301 and a processor 302. A computer program is stored in the memory 301, and the processor 302 is configured to call the computer program and execute the auto white balance processing method in any one of the above embodiments.

[0137] For example, the processor 302 invokes a computer program to execute an auto white balance processing method including the following steps: Divide an image to be processed into a plurality of image regions; Collect luminance information of pixels included in each image region; The luminance information includes at least the luminance level at which the pixels included in each image region are located; Use a non-parametric method to perform density estimation on pixels at a plurality of luminance levels to obtain at least density estimates for pixels at bright levels; Calculate a first white balance gain corresponding to each color based at least on the density estimates for pixels at bright levels.

[0138] The processor 302 calls a computer program to execute the above-mentioned auto white balance processing method. For more specific steps included in the above-mentioned auto white balance processing method, please refer to the relevant content of the auto white balance processing method in any one of the above embodiments, and detailed description will not be given here.

[0139] In some embodiments, a computer-readable storage medium is further provided, which stores a computer program, and when the computer program is invoked by a computer, executes the auto white balance processing method of any one of the above embodiments.

[0140] For example, when the computer program is invoked by a computer, an auto white balance processing method is executed, which includes the following steps: Divide an image to be processed into a plurality of image regions; Collect luminance information of pixels included in each image region; The luminance information includes at least the luminance level at which the pixels included in each image region are located; Use a non-parametric method to perform density estimation on pixels at a plurality of luminance levels to obtain at least density estimates for pixels at bright levels; Calculate a first white balance gain corresponding to each color based at least on the density estimates for pixels at bright levels.

[0141] When the computer program is executed by the computer, the above-mentioned auto white balance processing method is executed. For more specific steps included in the above-mentioned auto white balance processing method, please refer to the relevant content of the auto white balance processing method in any one of the above embodiments, and detailed description will not be given here.

[0142] An embodiment of the present application further provides a chip, which can perform steps related to the above-mentioned auto white balance processing method, and which includes a processor and a communication interface, and the communication interface is used to receive or transmit data.

[0143] In one embodiment, the chip performs the relevant steps in the method embodiments above.

[0144] For example, the processor may be configured to cause the chip to perform the following operations: divide an image to be processed into a plurality of image regions; calculate luminance information of pixels included in each image region, where the luminance information includes at least the luminance level at which the pixels included in each image region are located; perform density estimation on pixels at a plurality of luminance levels using a non-parametric method to obtain at least density estimates for pixels at a bright level; and calculate a first white balance gain corresponding to each color based at least on the density estimates for pixels at the bright level.

[0145] For other specific operations performed by the chip, please refer to the relevant content of the above method embodiments, and will not be described in detail here.

[0146] For each device or product applied to or integrated into a chip, each module included therein may be implemented by hardware such as a circuit, or at least a portion of the module may be implemented by a software program executed by a processor integrated within the chip, with the remaining portion of the module (if any) being implemented by hardware such as a circuit.

[0147] The embodiments of the present application further provide a computer program product, which, when executed by a processor, realizes the method flows of the above method embodiments.

[0148] The auto white balance processing method, auto white balance processing device 100, imaging device 200, and electronic device 300 of the present application divide an image into multiple image regions, collect statistics on the luminance information of pixels contained in each image region, perform density estimation on pixels of multiple luminance levels using a nonparametric method to obtain at least density estimates for pixels of bright levels, and calculate first white balance gains corresponding to each color based at least on the density estimates for pixels of bright levels, thereby achieving accurate white balance gains in a relatively simple manner. In particular, the present application further mixes the first white balance gain with a second white balance gain obtained by a separate method according to a gain mixing ratio to obtain a final white balance gain. This enables better adaptation to the white balance processing requirements of different imaging scenes, and images of different imaging scenes can achieve better white balance processing results.

[0149] Each module / unit included in each device and product described in the above embodiments may be a software module / unit or a hardware module / unit, or may be partly software modules / units and partly hardware modules / units. For example, for each device or product applied to or integrated into a chip, each module / unit included therein may be implemented by hardware such as a circuit, or at least part of the module / unit may be implemented by a software program executed by a processor integrated in the chip, and the remaining part of the module / unit (if any) may be implemented by hardware such as a circuit. For each device or product applied to or integrated into a chip module, each module / unit included therein may be implemented by hardware such as a circuit, and different modules / units may be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least part of the module / unit may be implemented by a software program executed by a processor integrated in the chip module, and the remaining part of the module / unit (if any) may be implemented by hardware such as a circuit. For each device or product applied to or integrated into a terminal, the modules / units included therein may be implemented by hardware such as circuits, and different modules / units may be located in the same component (e.g., chip, circuit module, etc.) or different components within the terminal, or at least a part of the modules / units may be implemented by a software program executed on a processor integrated within the terminal, and the remaining part of the modules / units (if any) may be implemented by hardware such as circuits.

[0150] It should be noted that for simplicity, the above method embodiments are expressed as a combination of a series of operations. However, it should be understood by those skilled in the art that the present application is not limited to the order of operations described, and that some operations may be performed in other orders or simultaneously based on the present application. It should also be understood by those skilled in the art that the embodiments described in the specification are preferred embodiments, and that such operations and modules are not necessarily required for the present application.

[0151] The descriptions of the embodiments of the present application may be mutually referenced. Each embodiment has its own focus. For parts of an embodiment that are not described in detail, reference may be made to the relevant descriptions of other embodiments. For convenience and conciseness of description, for example, the functions and operations performed by each apparatus and device according to the embodiments of the present application may be referred to the relevant descriptions of the method embodiments of the present application, and each method embodiment and each apparatus embodiment may be mutually referenced, combined, or cited.

[0152] Finally, the above embodiments are only used to explain the technical solutions of the present application, and do not limit the technical solutions of the present application. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand the following: Those skilled in the art may still amend the technical solutions described in the above embodiments, and may make equivalent substitutions for some or all of these technical features, and these amendments or substitutions will not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An auto white balance processing method for performing white balance processing on an image, comprising: Dividing an image to be processed into a plurality of image regions; Statistical analysis of luminance information of pixels included in each image region, the luminance information including at least luminance levels at which the pixels included in each image region are located; performing density estimation for pixels at multiple luminance levels using a non-parametric technique to obtain density estimates for at least the bright level pixels; calculating a first white balance gain corresponding to each color based at least on the density estimate of the bright level pixels; 1. An auto white balance processing method comprising:

2. The auto white balance processing method includes: processing the pending image in a different manner to obtain a second white balance gain for each color; mixing the first white balance gains of each color and the second white balance gains of each color according to a gain mixing ratio to obtain final white balance gains of each color; further comprising:

2. The auto white balance processing method according to claim 1, wherein:

3. The first white balance gain of each color and the second white balance gain of each color are mixed in accordance with a gain mixing ratio to obtain a final white balance gain of each color, This involves calculating the final white balance gain for each color based on the formula MixRatio * Pgain0 + (1-MixRatio) * Pgain1. Pgain0 is a first white balance gain for one of the colors, MixRatio is the gain mixing ratio, and Pgain1 is a second white balance gain for one of the colors.

3. The auto white balance processing method according to claim 2.

4. The value range of the gain mix ratio is 0≦MixRatio≦1; 4. The auto white balance processing method according to claim 3.

5. The gain mixing ratio is obtained based on a cross-correlation coefficient between a density estimation value of a first pixel and a density estimation value of a second pixel, the first pixel being a pixel having at least one luminance level a preset number of steps lower than a maximum luminance level, the second pixel being all pixels having luminance levels close to a target luminance value, and the target luminance value being obtained by multiplying the maximum luminance value by a preset ratio.

3. The auto white balance processing method according to claim 2.

6. the at least one luminance level that is a preset number of steps lower than the highest luminance level includes a secondary bright level, the secondary bright level being a luminance level that is one step lower than the highest luminance level among all the luminance levels, and the luminance level close to the target luminance value includes a target luminance level at which the target luminance value is located and a luminance level that is adjacent to the target luminance level and has a difference between an upper limit value or a lower limit value and the target luminance value that is smaller than a preset luminance value; 6. The auto white balance processing method according to claim 5,

7. The step of collecting luminance information of pixels included in each image region includes: Dividing the image into a plurality of brightness levels from the darkest brightness to the brightest brightness according to a predetermined rule, each brightness level corresponding to a brightness section; quantizing pixel values ​​of pixels of each color in each image region into corresponding luminance values, and determining luminance levels corresponding to the pixels of each color based on luminance intervals in which the luminance values ​​corresponding to the pixels of each color are located; Including, 2. The auto white balance processing method according to claim 1, wherein:

8. The step of collecting luminance information of pixels included in each image region includes: obtaining pixel values ​​for pixels of each color located at each luminance level within each image region; calculating a total pixel value of all pixels of each color located at each luminance level in each image region to obtain a total pixel value of each color located at each luminance level in each image region; further comprising:

8. The auto white balance processing method according to claim 7,

9. The pixels include first color pixels, second color pixels, and third color pixels according to the type of color, and the auto white balance processing method includes: performing color space conversion to convert the total pixel values ​​of each luminance level corresponding to the first color pixel, the second color pixel, and the third color pixel into first chromaticity values ​​and second chromaticity values ​​of each luminance level; further comprising:

9. The auto white balance processing method according to claim 8.

10. the first color pixel, the second color pixel, and the third color pixel are R pixel, G pixel, and B pixel, respectively, and performing the color space conversion to convert the total pixel values ​​of each luminance level corresponding to the first color pixel, the second color pixel, and the third color pixel into first chromaticity values ​​and second chromaticity values ​​of each luminance level, converting the total pixel values ​​of each luminance level corresponding to the RGB pixels into R / G values ​​and B / G values ​​of each luminance level; 10. The auto white balance processing method according to claim 9.

11. The R / G value of each luminance level is a ratio of the total pixel value of R pixels of the luminance level to the total pixel value of G pixels of the luminance level, and the B / G value of each luminance level is a ratio of the total pixel value of B pixels of the luminance level to the total pixel value of G pixels of the luminance level.

11. The auto white balance processing method according to claim 10.

12. Performing density estimation for pixels of multiple luminance levels using the non-parametric technique to obtain density estimates for at least bright level pixels includes: obtaining R / G and B / G values ​​corresponding to a light level and R / G and B / G values ​​corresponding to a dark level; estimating the density of a data-free portion in the color space using a non-parametric method based on the R / G values ​​and B / G values ​​corresponding to the bright level and the R / G values ​​and B / G values ​​corresponding to the dark level, thereby obtaining a density estimate of a pixel at the bright level and a density estimate of a pixel at the dark level; Including, 11. The auto white balance processing method according to claim 10.

13. Obtaining R / G values ​​and B / G values ​​corresponding to the bright level and R / G values ​​and B / G values ​​corresponding to the dark level includes: A secondary bright level is set as the brightness level, and an R / G value and a B / G value corresponding to the brightness level are obtained, the secondary bright level being a brightness level one step lower than the highest brightness level among all brightness levels; A luminance level close to a target luminance value is set as the dark level, and an R / G value and a B / G value corresponding to the dark level are obtained, and the target luminance value is obtained by multiplying the maximum luminance value by a preset ratio; Including, 13. The auto white balance processing method according to claim 12.

14. The non-parametric method is a kernel density estimation method.

2. The auto white balance processing method according to claim 1, wherein:

15. Calculating a first white balance gain corresponding to each color based at least on the density estimate of the bright level pixels includes: obtaining R / G and B / G values ​​corresponding to a dark level; Calculating a first white balance gain corresponding to each color based on a total pixel value of the dark level of each color and an estimated density value of the light level of the corresponding coordinate; Including, 13. The auto white balance processing method according to claim 12.

16. Calculating a first white balance gain corresponding to each color based on a total pixel value of a dark level of each color and an estimated density value of a light level of a corresponding coordinate, obtaining a target value for each color based on the total pixel value of the dark level of each color and the density estimate of the light level of the corresponding coordinate; obtaining a first white balance gain corresponding to each color based on the target value of each color; 16. The auto white balance processing method according to claim 15.

17. Obtaining a target value for each color based on a total pixel value of the dark level of each color and a density estimate value of the light level of the corresponding coordinate, obtaining a target value for each color based at least on a product of a total pixel value of the dark level of each color and a density estimate of the light level of the corresponding coordinate; 17. The auto white balance processing method according to claim 16.

18. Obtaining a target value for each color based at least on a product of a total pixel value of a dark level of each color and a density estimate of a light level of a corresponding coordinate includes: calculating a target value for each color based on the following formula: [Equation 1] R1 is the target value for red, G1 is the target value for green, B1 is the target value for blue, and R LV2Lv3 is the total pixel value of the red dark level, and G Lv2Lv3 is the total pixel value of the green dark level, and B Lv2Lv3 is the total pixel value of the blue dark level, and KDE Lv14 is the density estimate of the light level, and (x,y) are the coordinates in color space.

18. The auto white balance processing method according to claim 17.

19. obtaining a first white balance gain corresponding to each color based on the target value of each color, determining a first white balance gain corresponding to red to be G1 / R1; determining a first white balance gain corresponding to blue to be G1 / B1; determining a first white balance gain corresponding to green to be 1; 19. The auto white balance processing method according to claim 18.

20. The other method is a method using a dichromatic reflection model.

3. The auto white balance processing method according to claim 2.

21. An auto white balance processing device, The apparatus includes a statistical analysis unit, a density estimation unit, and a white balance gain determination unit, the statistical analysis unit is configured to divide the image to be processed into a plurality of image regions and to perform statistical analysis of luminance information of pixels included in each image region, the luminance information including at least a luminance level at which the pixels included in each image region are located; the density estimation unit is configured to perform density estimation on pixels of a plurality of luminance levels using a non-parametric technique to obtain density estimates for at least pixels of a bright level; the white balance gain determination unit is configured to calculate a first white balance gain corresponding to each color based at least on the density estimate value of the bright level pixels. An auto white balance processing device comprising:

22. The auto white balance processing device further includes a mixing processing unit, the mixing processing unit is configured to process the waiting image using another method to obtain a second white balance gain for each color, and to mix the first white balance gain for each color and the second white balance gain for each color according to a gain mixing ratio to obtain a final white balance gain for each color.

22. The automatic white balance processing device according to claim 21.

23. An imaging device, An automatic white balance processing device according to claim 21 or 22, An imaging device characterized by:

24. the imaging device further includes an image acquisition unit and an image generation unit; the image acquisition unit is configured to acquire the image waiting to be processed; the image generation unit is configured to generate a processed image based on the white balance gain obtained by the auto white balance processing device.

24. The imaging device according to claim 23.

25. An electronic device comprising a memory and a processor, a computer program stored in the memory, and the processor configured to call the computer program and execute the auto white balance processing method according to any one of claims 1 to 20; An electronic device characterized by:

26. 1. A computer-readable storage medium, comprising: A computer program is stored in the computer-readable storage medium, and when the computer program is called by a computer, the auto white balance processing method according to any one of claims 1 to 20 is executed. A computer-readable storage medium comprising:

Images