Auto white balance processing method and apparatus, imaging apparatus and electronic equipment

Abstract

To provide a simple and relatively easy auto white balance processing method with high accuracy. [Solution] The auto white balance processing method includes the following steps: Dividing an image to be processed into a plurality of image regions; Compiling statistics on luminance information of pixels included in each image region; The luminance information includes at least the luminance level at which the pixels included in each image region are located; Using a non-parametric method, performing density estimation on pixels at a plurality of luminance levels to obtain at least density estimates for pixels at bright levels; Calculating a first white balance gain corresponding to each color based at least on the density estimates for pixels at bright levels. The present application also provides an auto white balance processing device, an imaging device, and an electronic device.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to an auto white balance processing method and apparatus, an imaging device, and an electronic device.

Background Art

[0002] In imaging processing, in order to compensate for the sensitivity difference of an image sensor and the spectral difference of a used light source, white balance processing is performed on the pixels output from the image sensor. The purpose of white balance processing is to prevent the color cast on what should be originally reproduced white and eliminate color deviation, thereby correcting the color of the image more realistically and naturally. However, existing white balance processing methods are complex, resulting in a large amount of time and human costs, or having poor effects.

Summary of the Invention

[0003] In order to solve the above problems, the present application provides an auto white balance processing method and apparatus, an imaging device, and an electronic device.

[0004] In a first aspect, an auto white balance processing method for performing white balance processing on an image is provided. The auto white balance processing method includes the following. Divide an image waiting for processing into a plurality of image regions. Statistically analyze the luminance information of the pixels included in each image region. The luminance information includes at least the luminance level at which the pixels included in each image region are located. Using a non-parametric method, perform density estimation on the pixels at a plurality of luminance levels to obtain at least the density estimation value of the pixels at a bright level. Calculate a first white balance gain corresponding to each color based at least on the density estimation value of the pixels at a bright level.

[0005] In a second embodiment, an auto white balance processing device is further provided. This auto white balance processing device comprises a statistical analysis unit, a density estimation unit, and a white balance gain determination unit. The statistical analysis unit is configured to divide an image awaiting processing into a plurality of image regions and statistically analyze the luminance information of pixels contained in each image region, the luminance information including at least the luminance level to which the pixels contained in each image region are located. The density estimation unit is configured to perform density estimation for pixels of multiple luminance levels using a nonparametric method and to obtain at least a density estimate of the brightest level pixels. The white balance gain determination unit is configured to calculate a first white balance gain corresponding to each color based at least on the density estimate of the brightest level pixels.

[0006] In a third embodiment, an imaging device is further provided. The imaging device includes an auto white balance processing device. The auto white balance processing device includes a statistical analysis unit, a density estimation unit, and a white balance gain determination unit. The statistical analysis unit is configured to divide an image awaiting processing into a plurality of image regions and statistically analyze the brightness information of pixels included in each image region, the brightness information including at least the brightness level to which the pixels included in each image region are located. The density estimation unit is configured to perform density estimation for pixels at multiple brightness levels using a nonparametric method and to obtain at least a density estimate of the brightest level pixels. The white balance gain determination unit is configured to calculate a first white balance gain corresponding to each color based at least on the density estimate of the brightest level pixels.

[0007] In a fourth embodiment, an electronic device is further provided, comprising memory and a processor. A computer program is stored in the memory, and the processor is configured to call the computer program to execute an auto white balance processing method. The auto white balance processing method includes the following: dividing an image awaiting processing into multiple image regions; statistically calculating the luminance information of pixels contained in each image region; the luminance information including at least the luminance level to which the pixels contained in each image region are located; performing density estimation for pixels at multiple luminance levels using a nonparametric method to obtain at least a density estimate of brighter pixels; and calculating a first white balance gain corresponding to each color, at least based on the density estimate of brighter pixels.

[0008] In the fifth embodiment, a computer-readable storage medium is further provided. A computer program is stored in the computer-readable storage medium, and when the computer program is invoked by the computer, an auto white balance processing method is executed. The auto white balance processing method includes the following: dividing an image awaiting processing into multiple image regions; statistically calculating the luminance information of pixels contained in each image region; the luminance information including at least the luminance level to which the pixels contained in each image region are located; performing density estimation for pixels of multiple luminance levels using a nonparametric method to obtain at least a density estimate of the brightest level pixels; and calculating a first white balance gain corresponding to each color based at least on the density estimate of the brightest level pixels.

[0009] In the auto white balance processing method and apparatus, imaging apparatus and electronic device of this application, an image is divided into multiple image regions, the brightness information of pixels contained in each image region is statistically analyzed, and density estimation is performed for pixels of multiple brightness levels using a non-parametric method to obtain at least a density estimate of brighter pixels. Based at least on the density estimate of brighter pixels, a first white balance gain corresponding to each color is calculated, thereby enabling the acquisition of a highly accurate white balance gain in a relatively simple manner. [Brief explanation of the drawing]

[0010] In order to more clearly explain the technical proposal of the embodiments of this application or the technical proposal of the background art, the drawings necessary for explaining the embodiments of this application or the background art will be described below. [Figure 1] Figure 1 is a flowchart showing an auto white balance processing method in some embodiments of this application. [Figure 2] Figure 2 is a schematic diagram showing how an image is divided into multiple image regions A0 to A(n-1) in some embodiments of this application. [Figure 3] Figure 3 shows a subprocess of step 12 of Figure 1 of this application in several embodiments. [Figure 4] Figure 4 is a schematic diagram showing the correspondence between luminance levels and luminance intervals in several embodiments. [Figure 5] Figure 5 is a schematic diagram showing a storage structure for the total pixel values ​​of each brightness level within each image region in some embodiments of this application. [Figure 6] Figure 6 is another flowchart illustrating an auto white balance processing method in some embodiments of this application. [Figure 7] Figure 7 is a schematic diagram showing the storage structure of R / G values ​​for each brightness level within each image region in some embodiments of this application. [Figure 8] Figure 8 shows a subprocess of step 13 of Figure 1 of this application in several embodiments. [Figure 9] Figure 9 shows the density estimates and data points of bright-level pixels in some embodiments of the present application. [Figure 10] Figure 10 shows density estimates and data points for dark levels of pixels in some embodiments of this application. [Figure 11] Figure 11 is a schematic diagram integrating density estimates of bright levels and scatter of data points of dark levels in several embodiments of this application. [Figure 12] Figure 12 shows a subprocess of step 14 of Figure 1 of this application in several embodiments. [Figure 13] Figure 13 is another flowchart illustrating an auto white balance processing method in some embodiments of this application. [Figure 14] Figure 14 is a schematic diagram showing the relationship between the gain mixing ratio and the cross-correlation coefficient in several embodiments of this application. [Figure 15] Figure 15 is a schematic diagram showing a comparison of images obtained after white balance processing using multiple methods. [Figure 16] Figure 16 is a block diagram showing the structure of an auto white balance processing device in several embodiments of this application. [Figure 17] Figure 17 is a block diagram showing the structure of an imaging device in several embodiments of this application. [Figure 18] Figure 18 is a block diagram showing the structure of an electronic device in several embodiments of this application. [Modes for carrying out the invention]

[0011] In the description, claims, and drawings of this application, terms such as "first" and "second" are not used for the purpose of describing a specific sequence, but are used to distinguish different objects. It should not be understood that terms such as "first" and "second" are only used for the purpose of description, indicating or implying relative importance, or implying the number of technical features mentioned. Therefore, features limited by terms such as "first" and "second" can explicitly or implicitly include one or more of the said features. In the description of this embodiment, unless otherwise specified, "a plurality" refers to two or more. Also, terms such as "include", "comprise", or any other variant do not exclude including other components and are intended to cover. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, and can optionally further include other steps or units not listed, or can optionally further include other steps or units specific to these processes, methods, systems, products, or devices.

[0012] As can be understood, in this application, "a plurality" refers to two or more. The term "and / or" describes the relationship between related objects and indicates that there are three types of relationships. For example, in the case of A and / or B, it indicates three situations: only A exists, only B exists, and A and B exist simultaneously. A and B can be singular or plural. The symbol " / " generally indicates that the related objects before and after are in an "or" relationship. Both "when..." and "case" mean performing corresponding processing under a certain objective situation, do not limit time, do not require an operation of judgment when realized, and do not mean that there are other restrictions.

[0013] In the embodiments of this application, terms such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or solution described as "exemplary" or "for example" in the embodiments of this application should not be construed as more preferable or advantageous than other embodiments or solutions. To put it plainly, the use of terms such as "exemplary" or "for example" is intended to present related concepts in a specific way for easier understanding.

[0014] The automatic white balance function generally has methods for obtaining the white balance correction coefficient from the integrated values of all pixels assuming that the average color of the scene is achromatic, and methods for obtaining the correction coefficient from the integrated values of pixels included in the achromatic region. Also, a calibration method for measuring the color sensitivity of the sensor in advance for a standard light source and improving the calculation accuracy of the correction coefficient is often used in combination. Among the above-mentioned methods, in the method of obtaining the correction coefficient from the integrated values of pixels included in the achromatic region, the achromatic region is often specified by a parameter that the user can adjust, and the adjustment result has a great influence on the performance and may deteriorate significantly more than the original performance. Also, it requires a lot of adjustment work time.

[0015] Therefore, the problem to be solved is how to provide a more efficient and accurate white balance processing method.

[0016] Referring to FIG. 1, FIG. 1 is a flowchart showing an automatic white balance processing method in some embodiments of this application. The automatic white balance processing method is used to perform white balance processing on an image. The automatic white balance processing method includes the following steps. 11: Divide the image waiting for processing into a plurality of image regions. 12: Statistically analyze the luminance information of the pixels included in each image region. The luminance information includes at least the luminance level at which the pixels included in each image region are located. 13: Use a non-parametric method to perform density estimation on the pixels at multiple luminance levels to obtain at least the density estimation value of the pixels at a bright level. 14. Calculate a first white balance gain corresponding to each color, based at least on the pixel density estimates for the brighter levels.

[0017] In this application, an image is divided into multiple image regions, the brightness information of pixels contained in each image region is statistically analyzed, and density estimation is performed for pixels of multiple brightness levels using a non-parametric method to obtain at least a density estimate of the brightest pixels. Based at least on the density estimate of the brightest pixels, a first white balance gain corresponding to each color is calculated, thereby enabling the acquisition of a highly accurate white balance gain in a relatively simple manner.

[0018] Referring to Figure 2, Figure 2 is a schematic diagram showing how image P1 is divided into multiple image regions A0 to A(n-1) in some embodiments of this application.

[0019] That is, in some embodiments, image P1 can be divided into n image regions A0 to A(n-1). In some embodiments, image P1 may be divided into any number of image regions, for example, n may be an integer of 2 or more, and image P1 may be divided into any number of image regions of 2 or more. As another example, as shown in Figure 2, n may be 30, that is, image P1 can be divided into 30 image regions, for example, 30 image regions of 5 rows and 6 columns.

[0020] In some embodiments, the sizes of multiple image regions are approximately equal; that is, image P1 is evenly divided into multiple image regions of equal size.

[0021] Each image region has its own identifier, for example, A0 to A(n-1) in the figure. A0 to A(n-1) each correspond to one image region, and are identifiers for that corresponding image region.

[0022] Clearly, Figure 2 is merely illustrative, and image P1 may be divided into a different number of image regions, and the arrangement mode of the multiple image regions may also be a different arrangement mode. For example, if image P1 can be divided into 30 image regions, the 30 image regions may be arranged in a 2x15, 3x10, 6x5, etc. In some embodiments, the identifiers of the image regions may be other identifiers; for example, if image P1 is divided into n image regions, the identifiers of the n image regions may each be numbers from 0 to "n-1".

[0023] Referring to Figure 3, Figure 3 shows a subprocess of step 12 of Figure 1 of this application in several embodiments. In some embodiments, as shown in Figure 3, step 12, which "statistically calculates the brightness information of pixels contained in each image region," may include the following: 121: Divide the brightness into multiple levels, from the darkest to the brightest, according to pre-set rules. Each brightness level corresponds to a brightness interval. 123: The pixel values ​​of each color pixel within each image region are quantized to their corresponding luminance values, and the luminance level corresponding to each color pixel is determined based on the luminance interval in which the luminance values ​​corresponding to each color pixel are located.

[0024] In other words, in some embodiments, since pixels include pixels of different colors, statistically compiling the luminance information of pixels contained in each image region includes statistically compiling the luminance information of pixels of each color contained in each image region. Specifically, first, the image is divided into multiple luminance levels from the darkest to the brightest. Next, the pixel values ​​of pixels of each color in each image region are quantized to corresponding luminance values. Based on the luminance interval in which the luminance values ​​corresponding to pixels of each color are located, the luminance level corresponding to each pixel of each color can be determined. This allows determining the luminance level in which each pixel of each color in each image region is located, which is equivalent to obtaining the pixels and their colors located at each luminance level within each image region.

[0025] Referring to Figure 4, which is a schematic diagram showing the correspondence between luminance levels and luminance intervals in several embodiments. In some embodiments, the darkest luminance to the brightest luminance can correspond to black and white, respectively. In other words, for example, if the pixel value (gray value) is divided into m gray values ​​from black to white, then 0 can be the darkest level and (m-1) can be the brightest level, and the luminance value can be considered as multiple gray values ​​from 0 to "m-1". A predefined rule can include a predefined number of levels. Based on the number of levels included in the predefined rule, it can be divided into a corresponding number of luminance levels. The luminance interval corresponding to each luminance level can be obtained by dividing the total number of gray values ​​by the number of luminance levels. For example, as shown in Figure 4, if m is equal to 256, i.e., the total number of gray values ​​is 256 and the number of luminance levels is 16, then the luminance interval corresponding to each luminance level will correspond to 256 / 16 = 16 gray values. The higher the luminance level, the brighter it is. As shown in Figure 4, when the luminance level is 1, the corresponding luminance interval is in the gray range of 0 to 15, and when the luminance level is 2, the corresponding luminance interval is in the gray range of 16 to 31.

[0026] After obtaining the luminance value corresponding to each color pixel within each image region through quantization, the luminance level where each color pixel is located can be obtained according to the gray range in which that luminance value lies.

[0027] In some embodiments, a pixel includes a first color pixel, a second color pixel, and a third color pixel, depending on the type of color. In some embodiments, for example, the first color pixel, the second color pixel, and the third color pixel are R (Red) pixels, G (Green) pixels, and B (Blue) pixels, respectively, and the pixels in the image are arranged in a Bayer array. "Quantizing the pixel value of each color pixel in each image region to the corresponding luminance value" may include the following: Average the pixel values ​​of two pixels of the same color in the pixel unit in which each pixel of each color is located to obtain an average pixel value, and the average pixel value is taken as the luminance value of the pixel.

[0028] The Bayer array is a type of color pattern arrangement method for color filters in an image sensor. In this color pattern arrangement method, a color image is captured by arranging red, green, and blue filters on the image sensor, and each pixel can recognize only one color of light. The photodiodes (corresponding to pixels) on the surface of the image sensor are fitted with color filters in a Bayer pattern. The image light shining on the surface of the image sensor passes through each color filter and reaches the photodiode. The photodiode generates a voltage according to the amount of light received that has passed through the fitted filter color, and the voltage level is converted into a digital value and output from the image sensor as Bayer array data.

[0029] In a Bayer array, each pixel is located in one pixel unit, which contains 2x2 pixels, and each 2x2 pixel unit contains two pixels of the same color located diagonally opposite each other. The luminance value of a pixel is the average pixel value obtained by averaging the pixel values ​​of two pixels of the same color within the pixel unit in which the pixel is located. In this application, the pixel value, gray value, and luminance value can be understood as the same value.

[0030] Referring again to Figure 3, in some embodiments, step 12, which is "statistically calculating the brightness information of pixels contained in each image region," further includes the following: 125: Obtain the pixel values ​​of each color pixel located at each brightness level within each image region. 127: The sum of the pixel values ​​of all pixels of each color located at each brightness level within each image region is statistically calculated to obtain the total pixel value of each color located at each brightness level within each image region.

[0031] In other words, after determining the luminance level corresponding to each pixel of each color in several methods, the luminance level corresponding to each pixel of each color becomes clear. Furthermore, the pixels of each color located at each luminance level within each image region are determined, the pixel values ​​of the pixels of each color located at each luminance level within each image region are obtained, and then, for each image region, the pixel values ​​of all pixels of each color located at each luminance level are summed up to obtain the total pixel value of each color located at each luminance level within each image region.

[0032] In some embodiments, the method may further include storing in memory the total pixel values ​​of each color located at each brightness level within each image region.

[0033] Referring to Figure 5, Figure 5 is a schematic diagram showing a storage structure for the total pixel values ​​of each brightness level within each image region in some embodiments of the present application. Specifically, Figure 5 shows a storage structure in which the total pixel values ​​of each brightness level within each image region of image P1 are stored in memory.

[0034] As described above, a pixel includes a first color pixel, a second color pixel, and a third color pixel, depending on the type of color. Figure 5 shows an example of the storage structure for the total pixel value of a single color pixel. For example, Figure 5 shows an example of the storage structure for the total pixel value of an R pixel in an RGB color system.

[0035] Figure 5 shows an example where image P1 is divided into 30 image regions A0 to A29, and the brightness levels are divided into 16 brightness levels Lv0 to Lv15.

[0036] By statistically summing the pixel values ​​of R pixels located at brightness level Lv0 within image region A0, the total pixel value R0_0 for R pixels located at brightness level Lv0 within image region A0 is obtained. By statistically summing the pixel values ​​of R pixels located at brightness level Lv1 within image region A0, the total pixel value R1_0 for R pixels located at brightness level Lv1 within image region A0 is obtained. ... (omitted) ... By statistically summing the pixel values ​​of R pixels located at brightness level Lv15 within image region A0, the total pixel value R15_0 for R pixels located at brightness level Lv15 within image region A0 is obtained. Similarly, by statistically summing the pixel values ​​of the R pixels located at brightness level Lv0 in image region A1, the total pixel value R0_1 of the R pixels located at brightness level Lv0 in image region A1 is obtained, ... (omitted) ... By statistically summing the pixel values ​​of the R pixels located at brightness level Lv15 in image region A1, the total pixel value R15_1 of the R pixels located at brightness level Lv15 in image region A1 is obtained, ... (omitted) ... Furthermore, by statistically summing the pixel values ​​of the R pixels located at brightness level Lv0 in image region A29, the total pixel value R0_29 of the R pixels located at brightness level Lv0 in image region A29 is obtained, ... (omitted) ... By statistically summing the pixel values ​​of the R pixels located at brightness level Lv15 in image region A29, the total pixel value R15_29 of the R pixels located at brightness level Lv15 in image region A29 is obtained. In this way, one total pixel value of the R pixels located at each brightness level in each image region is obtained. That is, in this way, the memory structure shown in Figure 5 can be obtained.

[0037] In an RGB color system, the storage structure for the total pixel value of G pixels located at each brightness level within each image region, and the storage structure for the total pixel value of B pixels located at each brightness level within each image region, can both be the structures shown in Figure 5, and are not described in detail here.

[0038] In this application, the memory structure can refer to the memory format stored in memory, for example, the table format shown in Figure 5.

[0039] Referring to Figure 6, Figure 6 is another flowchart illustrating an auto white balance processing method in some embodiments of the present application. Figure 6 shows some of the processes of the auto white balance processing method. In some embodiments, the pixels include a first color pixel, a second color pixel, and a third color pixel, depending on the type of color. As shown in Figure 6, after step 127, which is to "statistically obtain the total pixel value of each color located at each brightness level in each image area by summing the pixel values ​​of all pixels of each color located at each brightness level in each image area," the auto white balance processing method further includes the following: 61: Perform a color space conversion to convert the sum of the pixel values ​​for each luminance level corresponding to the first, second, and third color pixels into the first and second chromaticity values ​​for each luminance level.

[0040] That is, in some embodiments, the present application performs a color space conversion for the subsequent calculation of the white balance gain, converting the sum of the pixel values ​​of the luminance levels corresponding to the first, second, and third color pixels into the first and second chromaticity values ​​of each luminance level.

[0041] In some embodiments, the first color pixel, the second color pixel, and the third color pixel are R pixel, G pixel, and B pixel, respectively. Performing a color space conversion to convert the sum of the pixel values ​​of each luminance level corresponding to the first color pixel, the second color pixel, and the third color pixel to the first and second chromaticity values ​​of each luminance level includes converting the sum of the pixel values ​​of each luminance level corresponding to the RGB pixels to the R / G and B / G values ​​of each luminance level.

[0042] The R / G value for each luminance level is the ratio of the total pixel value of the R pixels for that luminance level to the total pixel value of the G pixels for that luminance level, and the B / G value for each luminance level is the ratio of the total pixel value of the B pixels for that luminance level to the total pixel value of the G pixels for that luminance level.

[0043] That is, in some embodiments, taking an RGB color system as an example, if the first color pixel, second color pixel, and third color pixel are R pixels, G pixels, and B pixels, respectively, the R / G value for each luminance level can be calculated by dividing the total pixel value of the R pixels for each luminance level by the total pixel value of the G pixels for that luminance level, and the B / G value for each luminance level can be calculated by dividing the total pixel value of the B pixels for each luminance level by the total pixel value of the G pixels for that luminance level. The total pixel values ​​for each luminance level corresponding to the pixels of the three RGB colors can be converted into R / G and B / G values ​​for each luminance level, thereby realizing color space conversion, that is, converting a ternary color space to a binary color space.

[0044] In some embodiments, the method may further include storing the R / G and B / G values ​​of each brightness level within each image region in memory.

[0045] Referring to Figure 7, Figure 7 is a schematic diagram showing the storage structure for the R / G values ​​of each brightness level within each image region in some embodiments of this application. Specifically, Figure 7 shows a storage structure when the R / G values ​​of each brightness level within each image region of image P1 are stored in memory.

[0046] Figure 7 also shows an example where image P1 is divided into 30 image regions A0 to A29, and the brightness levels are divided into 16 brightness levels Lv0 to Lv15.

[0047] By dividing the total pixel value of the R pixels located at brightness level Lv0 in image region A0 by the total pixel value of the G pixels located at brightness level Lv0 in image region A0, the R / G value corresponding to brightness level Lv0 in image region A0 (i.e., R0_0 / G0_0) is obtained. By dividing the total pixel value of the R pixels located at brightness level Lv1 in image region A0 by the total pixel value of the G pixels located at brightness level Lv1 in image region A0, the R / G value corresponding to brightness level Lv1 in image region A0 (i.e., R1_0 / G1_0) is obtained. ... (omitted) ... By dividing the total pixel value of the R pixels located at brightness level Lv15 in image region A0 by the total pixel value of the G pixels located at brightness level Lv15 in image region A0, the R / G value corresponding to brightness level Lv15 in image region A0 (i.e., R15_0 / G15_0) is obtained. In this way, the R / G values ​​for each brightness level in image region A0 can be obtained. Similarly, the R / G values ​​of each brightness level within image regions such as image regions A1 to A15 can be obtained. In this way, the memory structure shown in Figure 7 can be obtained.

[0048] The storage structure for the B / G values ​​of each brightness level within each image region can also be the structure shown in Figure 7, which will not be described in detail here.

[0049] In some embodiments, the image awaiting processing may not be in the RGB color space / color system before the color space conversion is performed, but may be in another color space such as the YUV color space or Lab color space.

[0050] In the YUV color space, the first, second, and third color pixels can be Y, U, and V pixels, respectively, where Y represents luminance and U and V represent chrominance. The Y component represents the luminance information of the image, while the U and V components represent color information. During color space conversion, the Y component is removed, leaving the U and V components, thereby achieving color space conversion, i.e., conversion from the YUV color space to the UV color space.

[0051] The Y, U, and V pixels can form a single pixel unit. When the Y component is removed and the U and V components remain, it means that each pixel unit is left with only the values ​​corresponding to the U and V pixels.

[0052] Similarly, in the Lab color space, the first, second, and third color pixels can be L pixels, a pixels, and b pixels, respectively, where L represents luminance, a represents the range from red to green, and b represents the range from yellow to blue. During color space conversion, the L component is removed, leaving the a and b components, thereby achieving color space conversion, i.e., conversion from Lab color space to ab color space.

[0053] The L pixel, a pixel, and b pixel can form a single pixel unit. When the L component is removed and the a and b components remain, it means that each pixel unit is left with only the value corresponding to the a pixel and the value corresponding to the b pixel (e.g., the pixel value).

[0054] In some embodiments, when the color space is YUV, the total pixel value of each color located at each brightness level within each image region may include the total pixel value of U pixels located at each brightness level within each image region and the total pixel value of V pixels located at each brightness level within each image region. In some embodiments, when the color space is Lab, the total pixel value of each color located at each brightness level within each image region may include the total pixel value of a pixels located at each brightness level within each image region and the total pixel value of b pixels located at each brightness level within each image region.

[0055] Therefore, the auto white balance processing method of this application can perform white balance processing on images in various color spaces, including not only the RGB color space but also the YUV color space and Lab color space. In this application, we will mainly explain the process of performing white balance processing on images in the RGB color space as an example.

[0056] Referring to Figure 8, Figure 8 shows a subprocess of step 13 of Figure 1 of this application in several embodiments. In some embodiments, as shown in Figure 8, step 13, which "performs density estimation for pixels of multiple brightness levels using a nonparametric method to obtain at least a density estimate for the brightest level pixels," may include the following: 81: Obtain R / G and B / G values ​​corresponding to the bright level and R / G and B / G values ​​corresponding to the dark level. 83: Based on the R / G and B / G values ​​corresponding to the bright levels and the R / G and B / G values ​​corresponding to the dark levels, a non-parametric method is used to estimate the density of areas without data in the color space, thereby obtaining estimated pixel density values ​​for the bright levels and the dark levels.

[0057] In other words, in some embodiments, using the RGB color space as an example, the sum of the pixel values ​​of each luminance level corresponding to the pixels of RGB colors is converted to the R / G and B / G values ​​of each luminance level. After completing the color space conversion, the R / G and B / G values ​​corresponding to levels where the luminance level is a bright level and the R / G and B / G values ​​corresponding to levels where the luminance level is a dark level can be obtained. Next, based on the R / G and B / G values ​​corresponding to the bright levels and the R / G and B / G values ​​corresponding to the dark levels, the density of the dataless portion in the color space is estimated using a non-parametric method to obtain two types of pixel density estimates: density estimates for bright levels and density estimates for dark levels.

[0058] Estimating the density of dataless areas in a color space using a nonparametric method can be interpreted as estimating the density of dataless areas in the transformed color space using a nonparametric method.

[0059] In some embodiments, obtaining R / G and B / G values ​​corresponding to bright levels and R / G and B / G values ​​corresponding to dark levels includes the following: A secondary bright level is designated as a bright level, and R / G and B / G values ​​corresponding to the bright level are obtained. The secondary bright level is a luminance level that is one step lower than the highest luminance level among all luminance levels. A luminance level close to the target luminance value is designated as a dark level, and R / G and B / G values ​​corresponding to the dark level are obtained. The target luminance value is obtained by multiplying the highest luminance value by a predetermined ratio.

[0060] That is, in some embodiments, the bright level can be a sub-bright level that is one step lower than the highest luminance level among all luminance levels, and the dark level can be a luminance level close to the target luminance value.

[0061] Referring again to Figure 4, for example, if we divide the light into a total of 16 luminance levels from Lv0 to Lv15, as shown in Figure 4, the brightest luminance level is Lv15, and the second brightest level is Lv14.

[0062] The reason for not selecting the highest luminance level (Lv15) as the brightest level is to prevent an increase in errors due to the saturation region. Clearly, in some embodiments, the brightest level may be a predetermined number of steps lower than the highest luminance level, and may not be a secondary brightest level. For example, the brightest level may be a luminance level Lv13, which is two steps lower than the highest luminance level Lv15, or a luminance level Lv12, which is three steps lower than the highest luminance level Lv15.

[0063] In some embodiments, a luminance level close to the target luminance value includes the target luminance level where the target luminance value is located, and luminance levels adjacent to the target luminance level, where the difference between the upper or lower limit and the target luminance value is smaller than a preset luminance value. The target luminance value is obtained by multiplying the maximum luminance value by a preset percentage. Therefore, in some embodiments, when the target luminance value is obtained by multiplying the maximum luminance value by a preset percentage, it can be determined that the target luminance level corresponding to the luminance interval to which the target luminance value is located is a dark level, and it can also be determined that luminance levels adjacent to the target luminance level, where the difference between the upper or lower limit and the target luminance value is smaller than a preset luminance value, are also dark levels.

[0064] In some embodiments, the preset percentage may be a value such as 18%. In some embodiments, the preset luminance value is less than half the number of gray values ​​contained in the luminance interval corresponding to each luminance level. For example, as described above, if the system is divided into a total of 16 luminance levels, and the darkest luminance value / darkest gray value is 0, and the highest luminance value / highest gray value is 255, i.e., if a total of 256 gray values ​​are included, then the luminance interval corresponding to each luminance level will correspond to 256 / 16 = 16 gray values, i.e., the number of corresponding gray values ​​is 16. Therefore, the preset luminance value is less than 16 / 2 = 8. In some embodiments, the preset luminance value may be a value such as 5 or 6.

[0065] Using Figure 4 as an example, the maximum luminance value, i.e., the maximum gray value, is 255. If the preset percentage is 18%, the target luminance value becomes 255 × 18% = 45.9, which falls within the luminance interval (32 to 47) corresponding to luminance level Lv2, so luminance level Lv2 is a dark level. Also, the lower limit of Lv3 is 48, and the difference between the lower limit of Lv3 and 45.9 is 2.1, which is relatively small and smaller than the preset luminance value, so luminance level Lv3 is also a dark level. Therefore, in some embodiments, a dark level can include two luminance levels, Lv2 and Lv3.

[0066] Generally, automatic exposure control works to adjust the brightness of the subject to approximately 18% of its maximum brightness. Therefore, by selecting 18% of the maximum brightness as the target brightness value and determining brightness levels close to the target brightness value as the dark level, an adapted white balance gain can be obtained through automatic exposure control.

[0067] Clearly, in some embodiments, the preset percentage may be other values, such as 16%, 20%, etc.

[0068] Referring to Figure 9, Figure 9 shows the density estimates and data points of bright levels of pixels in some embodiments of the present application.

[0069] In Figure 9, the horizontal coordinate is the R / G value mentioned above, and the vertical coordinate is the B / G value mentioned above, illustrating that the brightness level is Lv14 as an example. Figure 9 shows contour lines L1 for multiple density estimates, and points on the contour lines of density estimates correspond to the same density estimate.

[0070] Each data point D1 shown in Figure 9 can represent one image region. The horizontal coordinate of the data point D1 corresponding to that image region is the R / G value corresponding to the luminance level Lv14 within that image region, and the vertical coordinate is the B / G value corresponding to the luminance level Lv14 within that image region. Since only some image regions may have both a B / G value and an R / G value corresponding to the luminance level Lv14, if the image is divided into 30 image regions, the number of data points D1 shown in Figure 9 may be less than 30. For example, as shown in Figure 9, the number of data points D1 is less than 30.

[0071] As shown in Figure 9, the density estimates for bright pixels are mainly concentrated within contour lines where the density estimate is 0.8 or 0.6.

[0072] Referring to Figure 10, Figure 10 shows the density estimates and data points of dark levels of pixels in some embodiments of the present application.

[0073] In Figure 10, the horizontal coordinate is the R / G value mentioned above, and the vertical coordinate is the B / G value mentioned above, illustrating that the dark levels are Lv2 and Lv3 as an example. Figure 10 also shows contour lines L1 for multiple density estimates, and the points on the contour lines for density estimates correspond to the same density estimate.

[0074] Each data point D1 shown in Figure 10 can represent one image region. The horizontal coordinate of the data point D1 corresponding to that image region is the R / G value corresponding to the luminance levels Lv2 and Lv3 within that image region, and the vertical coordinate is the B / G value corresponding to the luminance levels Lv2 and Lv3 within that image region. Since only some image regions may have R / G values ​​and B / G values ​​corresponding to luminance levels Lv2 and Lv3, if the image is divided into 30 image regions, the maximum number of data points D1 shown in Figure 10 will be the number of image regions multiplied by the number of luminance levels, i.e., 30 × 2 = 60. Therefore, the number of data points D1 shown in Figure 10 may be less than 60.

[0075] As shown in Figure 10, the density estimates for dark-level pixels are mainly concentrated within contour lines where the density estimate is 0.8 or 0.6.

[0076] As can be seen from Figures 9 and 10, the distribution of the concentration regions of the density estimates for bright-level pixels and the distribution of the concentration regions of the density estimates for dark-level pixels are located in similar positions. Therefore, by using the density estimates for bright-level pixels as weights, it is possible to calculate the first white balance gain corresponding to each color based on the pixel values ​​corresponding to the dark-level pixels of each color.

[0077] Referring to Figure 11, which is a schematic diagram integrating the density estimates of bright levels and the scatter of data points of dark levels in several embodiments of this application. In Figure 11, the density estimates of bright levels are represented by the contour line L1 of the density estimates, and the scatter points, i.e., the data points D1 in the figure, represent the coordinate positions of the dark levels. Areas with high density estimates of bright levels (for example, the area indicated by A in the figure) are considered to be close to the light source color and can therefore be given a larger weight, while areas with low density estimates of bright levels (for example, the area indicated by B in the figure) are considered to be far from the light source color and can therefore be given a smaller weight.

[0078] If the distribution of dark-level pixels in several coordinate regions in color space is close to the distribution of bright-level pixels, then the distribution under diffuse reflection in those coordinate regions will be approximately equal to the distribution of the light source color, meaning that the object color in those regions is considered colorless. As can be seen from the above, it is reasonable to use the density estimate of bright-level pixels as a weight to calculate the first white balance gain corresponding to each color based on the pixel values ​​corresponding to the dark-level pixels of each color.

[0079] In some embodiments, the nonparametric method may be a kernel density estimation method. Estimating the density of undata-filled areas in the color space using a nonparametric method to obtain density estimates for bright and dark pixels may include estimating the density of undata-filled areas in the color space using a kernel density estimation method to obtain density estimates for bright and dark pixels.

[0080] In some embodiments, the dataless portion may be a dataless portion other than the portion with data points shown in Figures 9 and 10.

[0081] Kernel density estimation is a commonly used nonparametric method. In kernel density estimation, a probability density function is used to estimate the density of areas without data, thereby obtaining density estimates for both bright and dark pixels. Therefore, kernel density estimation can be used to estimate the density of areas without data in the color space, thereby obtaining density estimates for both bright and dark pixels.

[0082] In some embodiments, the kernel function used in the kernel density estimation method may be a Gaussian function.

[0083] In some embodiments, the estimated pixel density for bright levels or for dark levels is as follows:

number

[0084]

number

[0085] The kernel density estimation method described above allows us to obtain density estimates for both bright and dark pixels.

[0086] In some embodiments, the bandwidth is 0.1. Obviously, in other embodiments, the bandwidth may be any other suitable value, such as 0.08.

[0087] Kernel density estimation methods are commonly used nonparametric methods and will not be discussed in detail here.

[0088] Referring to Figure 12, Figure 12 shows a subprocess of step 14 of Figure 1 of the present application in several embodiments. In some embodiments, step 14, which "calculates a first white balance gain corresponding to each color based at least on an estimate of the pixel density of the brightest levels," includes the following: 141: Obtain R / G and B / G values ​​corresponding to the dark level. 143: Calculate a first white balance gain for each color based on the total pixel value of the dark levels for each color and the density estimate of the bright levels at the corresponding coordinates.

[0089] Specifically, in some embodiments, the total pixel values ​​of each luminance level for each color are obtained, a color space conversion is performed to obtain a binary color space where the R / G value is the horizontal coordinate and the B / G value is the vertical coordinate, as shown in Figures 9 and 10, for example, and a density estimate of the bright level is obtained as shown in Figure 9. Next, a first white balance gain corresponding to each color is calculated based on the total pixel values ​​of each coordinate in the binary color space for the dark level of each color and the density estimate of the bright level at the corresponding coordinate.

[0090] As described above and shown in Figure 5, after obtaining the total pixel values ​​for each color at each luminance level within each image region, the total pixel values ​​for each color within each image region have corresponding luminance levels. As described above and shown in Figure 7, after performing a color space conversion to obtain the R / G and B / G values ​​corresponding to each luminance level within each image region, the R / G and B / G values ​​of each image region also have corresponding luminance levels. Therefore, the total pixel values ​​for each color within each image region and the R / G and B / G values ​​of the corresponding image region can be associated with luminance levels. Consequently, the coordinates in the binary color space of the total pixel values ​​for the dark levels of each color within each image region, i.e., the horizontal and vertical coordinates, are the R / G and B / G values ​​corresponding to the dark levels within that image region, respectively.

[0091] In some embodiments, step 143, which "calculates a first white balance gain corresponding to each color based on the total pixel value of the dark levels of each color and the density estimate of the bright levels at the corresponding coordinates," includes the following: Obtain a target value for each color based on the total pixel value of the dark levels of each color and the density estimate of the bright levels at the corresponding coordinates. Obtain a first white balance gain corresponding to each color based on the target value for each color.

[0092] That is, in some embodiments, calculating a first white balance gain corresponding to each color based on the total pixel value of the dark levels of each color and the density estimate of the bright levels at the corresponding coordinates can be done by first obtaining a target value for each color based on the total pixel value of the dark levels of each color and the density estimate of the bright levels at the corresponding coordinates, and then obtaining a first white balance gain corresponding to each color based on the target value for each color.

[0093] In some embodiments, obtaining a target value for each color based on the total pixel value of the dark levels of each color and the density estimate of the bright levels at the corresponding coordinates includes obtaining a target value for each color based at least on the product of the total pixel value of the dark levels of each color and the density estimate of the bright levels at the corresponding coordinates.

[0094] That is, in some embodiments, the target value for each color is obtained based at least on the product of the total pixel value of the dark levels of each color and the density estimate of the bright levels at the corresponding coordinates.

[0095] In some embodiments, obtaining a target value for each color based at least on the product of the total pixel value of the dark levels of each color and the density estimate of the bright levels at the corresponding coordinates includes calculating the target value for each color based on the following formula.

number

[0096] R1 is the target value in red, G1 is the target value in green, B1 is the target value in blue, R LV2Lv3 This is the total pixel value of the dark red level, G Lv2Lv3 This is the total pixel value of the dark green level, B Lv2Lv3 This is the total pixel value of the dark blue level, and KDE Lv14 is an estimated density value of the brightness level, and (x,y) is the coordinate in the color space, i.e., the coordinate in the converted binary color space shown in Figures 9 and 10.

[0097] Specifically, in some embodiments, the red target value R1, the green target value G1, and the blue target value B1 are calculated based on the above formula.

[0098] In some embodiments, obtaining a first white balance gain corresponding to each color based on a target value for each color includes the following: determining that the first white balance gain corresponding to red is G1 / R1; determining that the first white balance gain corresponding to blue is G1 / B1; and determining that the first white balance gain corresponding to green is 1.

[0099] In other words, in some embodiments, in an RGB color system, after obtaining the target values ​​R1, G1, and B1 of the RGB colors, it is determined that the first white balance gain corresponding to red is G1 / R1, the first white balance gain corresponding to blue is G1 / B1, and the first white balance gain corresponding to green is determined to be 1.

[0100] In white balance processing, the green white balance gain is typically set to 1. This is because the human eye perceives green more sensitively than red or blue, and in image processing, the green channel is often used as a reference to ensure that the overall brightness and color balance of the image match the habits of human vision. Setting the green white balance gain to 1 also simplifies the calculation process of white balance processing. Furthermore, many image sensors (such as Bayer array lenses) are designed to double the number of green pixels, and the green channel is generally highly sensitive and has low noise in terms of signal-to-noise ratio in typical sensors. By fixing the green gain, the sensor can be utilized more effectively to capture original information and ensure image quality. In addition, by fixing the green gain, it becomes easier to balance colors by adjusting the red and blue gains, and the image can maintain natural color representation even under different lighting conditions.

[0101] Referring to Figure 13, Figure 13 is another flowchart illustrating an auto white balance processing method in some embodiments of this application. As shown in Figure 13, the auto white balance processing method includes the following: 131: Divide the image awaiting processing into multiple image regions. 132: Statistically calculate the brightness information of pixels contained in each image region. The brightness information includes at least the brightness level to which the pixels contained in each image region are located. 133: Using a nonparametric method, density estimation is performed for pixels at multiple brightness levels to obtain at least a density estimate for the brightest pixels. 134: Calculate a first white balance gain corresponding to each color, based at least on an estimate of the pixel density at the brightest levels. 135: Process the images awaiting processing using a different method to obtain a second white balance gain for each color. 136: The first white balance gain for each color and the second white balance gain for each color are mixed according to the gain mixing ratio to obtain the final white balance gain for each color.

[0102] Accordingly, in some embodiments of this application, images awaiting processing can be processed in a different manner to obtain a second white balance gain for each color, and further, a final white balance gain can be obtained based on the first white balance gain and the second white balance gain. Thus, since multiple image processing methods are comprehensively considered, it becomes possible to adapt to the white balance processing requirements of multiple different imaging scenes, and images in different imaging scenes can have a better effect through white balance processing.

[0103] Steps 131-134 correspond to steps 11-14 in Figure 1, and for more specific details, please refer to the explanation above.

[0104] In some embodiments, step 136, which involves "mixing the first white balance gain for each color and the second white balance gain for each color according to the gain mixing ratio to obtain the final white balance gain for each color," includes calculating the final white balance gain for each color based on the formula MixRatio * Pgain0 + (1-MixRatio) * Pgain1, where Pgain0 is the first white balance gain for any one color, MixRatio is the gain mixing ratio, and Pgain1 is the second white balance gain for any one color.

[0105] In some embodiments, the range of the gain mixing ratio value is 0 ≤ MixRatio ≤ l.

[0106] That is, in some embodiments, a gain mixing ratio with a value in the range of 0 to 1 can be obtained, and by using the gain mixing ratio as the weight of the first white balance gain and 1-MixRatio as the weight of the second white balance gain, a weighted sum of the first white balance gain and the second white balance gain is obtained, and the final white balance gain is obtained.

[0107] In the auto white balance processing method shown in Figure 1, obtaining the first white balance gain by processing using a non-parametric method is generally suitable for scenes where dark-level pixels are concentrated near areas with a concentration of bright-level pixels. For example, consider the density distribution shown in Figures 9 to 11. In such scenes, the first white balance gain obtained by processing with the auto white balance processing method shown in Figure 1 can be used as the final white balance gain. Therefore, the MixRatio can be relatively large, and may even be 1, in which case there is no need to obtain a second white balance gain by another method. In other scenes, for example, in scenes different from those where dark-level pixels are concentrated near areas with a concentration of bright-level pixels, another method may be more suitable. Therefore, in this case, a second white balance gain can be obtained by another method, and then the first white balance gain and the second white balance gain for each color can be mixed according to the gain mixing ratio to obtain the final white balance gain for each color. The final white balance gain is more suitable for the current scene.

[0108] In some embodiments, the gain mixing ratio is obtained based on the cross-correlation coefficient between the density estimate of the first pixel and the density estimate of the second pixel. The first pixel is a pixel with at least one brightness level that is a preset number of steps lower than the highest brightness level. The second pixel is all pixels with brightness levels close to a target brightness value, which is obtained by multiplying the highest brightness value by a preset ratio.

[0109] In some embodiments, at least one luminance level that is a preset number of steps lower than the highest luminance level includes a secondary bright level. The secondary bright level is a luminance level that is one step lower than the highest luminance level among all luminance levels. The luminance levels close to the target luminance value include the target luminance level where the target luminance value is located, and luminance levels adjacent to the target luminance level, where the difference between the upper or lower limit and the target luminance value is smaller than a preset luminance value.

[0110] That is, in some embodiments, the first pixel may be a pixel of the bright level described above, and the second pixel may be a pixel of the dark level described above. The bright level pixels and dark level pixels can be described in the related explanation above and will not be elaborated here.

[0111] In some embodiments, there is a positive correlation between the gain mixing ratio and the cross-correlation coefficient between the density estimate of the first pixel and the density estimate of the second pixel. That is, as the cross-correlation coefficient between the density estimate of the first pixel and the density estimate of the second pixel increases, the gain mixing ratio generally increases as well.

[0112] As shown in Figures 9 and 10, in some scenes, for example, in scenes where dark-level pixels are concentrated near areas with a concentration of bright-level pixels, the distribution of kernel density estimates for bright-level Lv14 and dark-level Lv2 and Lv3 are similar. Therefore, the cross-correlation coefficient is relatively large, and the gain mixing ratio in this case is also relatively large, meaning the weight of the first white balance gain is relatively large. Consequently, as mentioned above, in this case, it is more appropriate to use the first white balance gain obtained by processing mainly with the auto white balance processing method shown in Figure 1 as the final white balance gain. Conversely, in some scenes, when the kernel density estimates for bright-level Lv14 and dark-level Lv2 and Lv3 differ significantly, the maximum value of the cross-correlation coefficient becomes small, and the gain mixing ratio is also relatively small. In this case, it is more appropriate to use a second white balance gain to obtain the final white balance gain.

[0113] Referring to Figure 14, Figure 14 is a schematic diagram showing the relationship between the gain mixing ratio and the cross-correlation coefficient in some embodiments of this application. In some embodiments, two thresholds are provided for the cross-phase function, namely a lower threshold Low_Thr and an upper threshold High_Thr, so that the application and non-application of the non-parametric method shown in Figure 1 does not switch discontinuously. As shown in Figure 14, the relationship between the gain mixing ratio MixRatio and the cross-phase function Cc is as follows: MixRatio = k1 * (Cc - Low_Thr). Low_Thr ≤ Cc ≤ High_Thr, and k1 = 1 / (High_Thr - Low_Thr).

[0114] The above formula allows for the determination of a gain mixing ratio (MixRatio) that is more suitable for the current scene, and furthermore, a final white balance gain that is more suitable for the current scene can be obtained. In some embodiments, the lower threshold Low_Thr may be 0.75 and the upper threshold High_Thr may be 0.8. Of course, in other embodiments, the lower threshold Low_Thr and the upper threshold High_Thr may be other appropriate values.

[0115] In some embodiments, the alternative method may be a method using a dichroic reflection model or another method.

[0116] If the alternative method uses a dichroic reflection model, specific details of the method using the dichroic reflection model can be found in Japanese Patent Application No. 2023-195971 and will not be described in detail here.

[0117] In some embodiments, the auto white balance processing method may further include the following: Gain processing is performed on the pixel values ​​of each color's pixels using a first white balance gain or a final white balance gain for each color, the pixel values ​​of each color's pixels after gain processing are obtained and output, and the image after white balance processing is displayed.

[0118] In some embodiments, after obtaining a first white balance gain or final white balance gain for each color, the auto white balance processing method may further include correcting the first white balance gain or final white balance gain for each color to obtain a corrected first white balance gain or final white balance gain.

[0119] In some embodiments, performing gain processing on the pixel values ​​of each color's pixels using a first white balance gain or final white balance gain for each color may include performing gain processing on the pixel values ​​of each color's pixels using a corrected first white balance gain or final white balance gain for each color.

[0120] Referring to Figure 15, Figure 15 is a schematic diagram showing a comparison of images obtained after white balance processing using multiple methods.

[0121] Figure 15 shows multiple developed images obtained after applying white balance processing to images predominantly in red, blue, and green using each method. Here, images predominantly in red, blue, and green are images obtained by photographing subjects predominantly in red, blue, and green under a standard light source.

[0122] Specifically, Figure 15 shows multiple developed images a1, a2, and a3 obtained after performing white balance processing on images predominantly in red, blue, and green using the method shown in Figure 1; multiple developed images b1, b2, and b3 obtained after performing white balance processing on images predominantly in red, blue, and green using a dichroic reflection model; and multiple developed images c1, c2, and c3 obtained by performing white balance processing using a white balance gain determined from the RGB values ​​of the brightness levels.

[0123] Clearly, as can be seen from Figure 15, the multiple developed images a1, a2, and a3 obtained after applying white balance processing to images primarily composed of red, blue, and green, respectively, using the method shown in Figure 1, have better contrast and other characteristics compared to developed images obtained using other methods or techniques. Therefore, the method shown in Figure 1 improves image quality compared to other methods.

[0124] In the auto white balance processing method of this application, an image is divided into multiple image regions, the brightness information of pixels contained in each image region is statistically analyzed, and density estimation is performed for pixels of multiple brightness levels using a non-parametric method to obtain at least a density estimate of bright-level pixels. Based at least on the density estimate of bright-level pixels, a first white balance gain corresponding to each color is calculated, thereby enabling the acquisition of a highly accurate white balance gain in a relatively simple manner. In particular, in this application, the first white balance gain and a second white balance gain obtained by a different method are further mixed according to the gain mixing ratio to obtain a final white balance gain. This makes it possible to better adapt to the white balance processing requirements of different imaging scenes, and images in different imaging scenes can have better effects through white balance processing.

[0125] Referring to Figure 16, Figure 16 is a block diagram showing the structure of an auto white balance processing device in some embodiments of the present application. As shown in Figure 16, the present application further provides an auto white balance processing device 100, which comprises a statistical analysis unit 101, a density estimation unit 102, and a white balance gain determination unit 103.

[0126] The statistical analysis unit 101 is configured to divide an image awaiting processing into multiple image regions and statistically analyze the brightness information of pixels contained in each image region, with the brightness information including at least the brightness level at which the pixels contained in each image region are located. The density estimation unit 102 is configured to perform density estimation for pixels at multiple brightness levels using a non-parametric method and to obtain at least the density estimates of brighter pixels. The white balance gain determination unit 103 is configured to calculate a first white balance gain corresponding to each color, based at least on the density estimates of brighter pixels.

[0127] The statistical analysis unit 101 performs the functional operations corresponding to steps 11 and 12 shown in Figure 1, the density estimation unit 102 performs the functional operation corresponding to step 13 shown in Figure 1, and the white balance gain determination unit 103 performs the functional operation corresponding to step 14 shown in Figure 1. For specific details of the functional operations performed by each part of the auto white balance processing device 100, refer to the auto white balance processing method described above.

[0128] In some embodiments, the white balance gain determination unit 103 is further configured to perform a color space conversion to convert the sum of the pixel values ​​of the luminance levels corresponding to the first, second, and third color pixels into the first and second chromaticity values ​​of each luminance level. That is, in some embodiments, the white balance gain determination unit 103 further performs the functional operation corresponding to step 61 in Figure 6.

[0129] In some embodiments, as shown in Figure 16, the auto white balance processing unit 100 may further include a mixing processing unit 104. The mixing processing unit 104 is configured to process the images awaiting processing in a different manner to obtain a second white balance gain for each color, and to mix the first white balance gain for each color and the second white balance gain for each color according to the gain mixing ratio to obtain a final white balance gain for each color.

[0130] The mixing unit 104 can perform functional operations corresponding to steps 135 and 136 in Figure 13. Further functional operations performed by the mixing unit 104 can be found by referring to the relevant content of steps 135 and 136 in the auto white balance processing method described above.

[0131] The auto white balance processing unit 100 can be a chip such as an image processing chip or a central processing chip. The statistical analysis unit 101, density estimation unit 102, white balance gain determination unit 103, etc., can be hardware units within the image processing chip.

[0132] Referring to Figure 17, Figure 17 is a block diagram showing the structure of an imaging device in several embodiments of the present application. As shown in Figure 17, the present application further provides an imaging device 200, which comprises the auto white balance processing device 100. As shown in Figure 17, the imaging device 200 further comprises an image acquisition unit 201 and an image generation unit 202.

[0133] The image acquisition unit 201 is configured to acquire images that are awaiting processing.

[0134] The image generation unit 202 is configured to generate a processed image, that is, an image after white balance processing, based on the white balance gain obtained by the auto white balance processing device 100. For example, the image generation unit 202 generates a white balance processed image by performing gain processing on the pixel values ​​of each color pixel based on the white balance gain obtained by the auto white balance processing device 100.

[0135] The image acquisition unit 201 may be a camera module including a camera and an image sensor. The image generation unit 202 may be a hardware unit within an image processing chip, a central processing chip, etc.

[0136] Referring to Figure 18, which is a block diagram showing the structure of an electronic device in several embodiments of the present application, the present application further provides an electronic device 300, which comprises a memory 301 and a processor 302. A computer program is stored in the memory 301, and the processor 302 is configured to call the computer program to execute the auto white balance processing method in any one of the embodiments described above.

[0137] For example, processor 302 calls a computer program to execute an auto white balance processing method which includes the following: Divide the image awaiting processing into multiple image regions. Statistically collect brightness information for pixels contained in each image region. The brightness information includes at least the brightness level at which the pixels contained in each image region are located. Using a nonparametric method, perform density estimation for pixels at multiple brightness levels to obtain at least a density estimate of the brightest pixels. Based at least on the density estimate of the brightest pixels, calculate a first white balance gain corresponding to each color.

[0138] The processor 302 invokes a computer program to execute the auto white balance processing method described above. More specific steps included in the auto white balance processing method can be found by referring to the relevant content of the auto white balance processing method in any one of the embodiments described above, and are not described in detail here.

[0139] In some embodiments, a computer-readable storage medium is further provided. A computer program is stored in the computer-readable storage medium, and when the computer program is invoked by the computer, the auto white balance processing method in any one of the above embodiments is executed.

[0140] For example, when a computer program is called by a computer, an auto white balance processing method is executed that includes the following: The image awaiting processing is divided into multiple image regions. The brightness information of the pixels contained in each image region is statistically calculated. The brightness information includes at least the brightness level to which the pixels contained in each image region are located. Density estimation is performed on pixels at multiple brightness levels using a nonparametric method to obtain at least a density estimate of the brightest level pixels. Based at least on the density estimate of the brightest level pixels, a first white balance gain corresponding to each color is calculated.

[0141] When the computer program is invoked by the computer, the above-described auto white balance processing method is executed. More specific steps included in the above-described auto white balance processing method can be found by referring to the relevant content of the auto white balance processing method in any one of the embodiments described above, and are not described in detail here.

[0142] Embodiments of this application further provide a chip capable of performing steps related to the auto white balance processing method described above. The chip comprises a processor and a communication interface, which is used to receive or transmit data.

[0143] In one embodiment, the chip performs the relevant steps in the above-described method embodiment.

[0144] For example, the processor is configured to have the chip perform the following operations: Divide the image awaiting processing into multiple image regions. Statistically calculate the brightness information of the pixels contained in each image region. The brightness information includes at least the brightness level to which the pixels contained in each image region are located. Using a nonparametric method, perform density estimation for pixels at multiple brightness levels to obtain at least a density estimate of the brightest pixels. Based at least on the density estimate of the brightest pixels, calculate a first white balance gain corresponding to each color.

[0145] Other specific operations performed by the chip can be found in the relevant details of the above-described embodiment and are not described in detail here.

[0146] For each device or product applied to or integrated on a chip, each module contained therein may be implemented by hardware such as circuits, or at least a portion of a module may be implemented by a software program executed on a processor integrated within the chip, and the rest of the module (if any) may be implemented by hardware such as circuits.

[0147] Embodiments of this application further provide a computer program product. When the computer program product is executed on a processor, the method flow of the above-described embodiment is realized.

[0148] In the auto white balance processing method, auto white balance processing device 100, imaging device 200, and electronic device 300 of this application, an image is divided into multiple image regions, the brightness information of pixels contained in each image region is statistically analyzed, and density estimation is performed for pixels of multiple brightness levels using a non-parametric method to obtain at least a density estimate of bright-level pixels. Based at least on the density estimate of bright-level pixels, a first white balance gain corresponding to each color is calculated, thereby enabling the acquisition of a highly accurate white balance gain in a relatively simple manner. In particular, in this application, the first white balance gain and a second white balance gain obtained by a different method are further mixed according to the gain mixing ratio to obtain a final white balance gain. This makes it possible to better adapt to the white balance processing requirements of different imaging scenes, and images in different imaging scenes can have better effects through white balance processing.

[0149] Each module / unit included in each device and product described in the above embodiments may be a software module / unit, a hardware module / unit, or partly a software module / unit and partly a hardware module / unit. For example, for each device or product applied to or integrated on a chip, each module / unit included therein may be implemented by hardware such as circuits, or at least part of a module / unit may be implemented by a software program executed on a processor integrated within the chip, and the rest of the module / unit (if any) may be implemented by hardware such as circuits. For each device or product applied to or integrated on a chip module, each module / unit included therein may be implemented by hardware such as circuits, different modules / units may be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least part of a module / unit may be implemented by a software program executed on a processor integrated within the chip module, and the rest of the module / unit (if any) may be implemented by hardware such as circuits. For each device or product applied to or integrated into a terminal, the modules / units contained therein may be implemented by hardware such as circuits, different modules / units may be located in the same component (e.g., chip, circuit module, etc.) or different components within the terminal, or at least a portion of a module / unit may be implemented by a software program executed on a processor integrated into the terminal, while the remaining portion of the module / unit (if any) may be implemented by hardware such as circuits.

[0150] For the sake of simplification, it should be noted that each of the above method embodiments is expressed as a combination of a series of operations. However, it should be understood by those skilled in the art that this application is not limited to the order of operations described, and that based on this application, some operations can be performed in other orders or simultaneously. Furthermore, it should be understood by those skilled in the art that all embodiments described in the specification are preferred embodiments, and such operations and modules are not necessarily required for this application.

[0151] The descriptions of each embodiment relating to this application may refer to one another. Each description of an embodiment has its own focus. For aspects not described in detail in one embodiment, the relevant descriptions of other embodiments may be referenced. For convenience and brevity of explanation, for example, the functions and operations performed by each apparatus and device according to the embodiments of this application may refer to the relevant descriptions of the method embodiments of this application, which may be referenced, combined, or cited between the method embodiments and between the apparatus embodiments.

[0152] Finally, the embodiments described above are used solely to illustrate the technical proposal of this application and are not intended to limit the technical proposal. Although this application has been described in detail with reference to the embodiments described above, those skilled in the art should understand the following: Those skilled in the art may still amend the technical proposals described in the embodiments above, or make equivalent substitutions for some or all of these technical features, and such amendments or substitutions will not cause the essence of the corresponding technical proposal to deviate from the scope of the technical proposal of each embodiment of this application.

Claims

1. An automatic white balance processing method for performing white balance processing on an image, Dividing an image awaiting processing into multiple image regions, The statistical calculation involves statistically determining the brightness information of pixels contained in each image region, wherein the brightness information includes at least the brightness level to which the pixels contained in each image region are located. Using a nonparametric method, density estimation is performed for pixels with multiple luminance levels, including bright levels, to obtain at least a density estimate for the bright level pixels. Based at least on the estimated pixel density of the aforementioned brightness levels, a first white balance gain corresponding to each color is calculated, Includes, The aforementioned pixels include R pixels, G pixels, and B pixels, and the auto white balance processing method further includes performing a color space conversion to convert the sum of the pixel values ​​of the luminance levels corresponding to the R pixels, G pixels, and B pixels into the R / G and B / G values ​​of each luminance level. Using the aforementioned nonparametric method, density estimation is performed for pixels with multiple luminance levels, including bright levels, to obtain at least a density estimate for the bright level pixels. To obtain at least R / G and B / G values ​​corresponding to the brightness level, Based at least on the R / G and B / G values ​​corresponding to the brightness levels, the density of the dataless portion in the color space is estimated using a non-parametric method to obtain at least an estimated density value of the pixels at the brightness levels, wherein the estimated density value is an estimated value of the density distribution in the color space. Includes, The aforementioned auto white balance processing method is: The process involves processing the images awaiting processing using a different method to obtain a second white balance gain for each color, wherein the different method is different from the nonparametric method. The first white balance gain for each color and the second white balance gain for each color are mixed according to the gain mixing ratio to obtain the final white balance gain for each color. Further including, An automatic white balance processing method characterized by the following.

2. The process of obtaining the final white balance gain for each color by mixing the first white balance gain for each color and the second white balance gain for each color according to the gain mixing ratio is as follows: This includes calculating the final white balance gain for each color based on the formula MixRatio * Pgain0 + (1-MixRatio) * Pgain1. Pgain0 is the first white balance gain for any one color, MixRatio is the gain mixing ratio, and Pgain1 is the second white balance gain for any one color. The auto white balance processing method according to feature 1.

3. The range of the aforementioned gain mixing ratio is 0 ≤ MixRatio ≤ 1. The auto white balance processing method according to feature 2.

4. The gain mixing ratio is obtained based on the cross-correlation coefficient between the density estimate of the first pixel and the density estimate of the second pixel, wherein the first pixel is a pixel with at least one brightness level that is a preset level lower than the highest brightness level, the second pixel is all pixels with a brightness level close to the target brightness value, and the target brightness value is obtained by multiplying the highest brightness value by a preset ratio. The auto white balance processing method according to feature 1.

5. At least one luminance level that is a preset level lower than the highest luminance level includes a secondary bright level, the secondary bright level being one level lower than the highest luminance level among all luminance levels, and the luminance levels close to the target luminance value include the target luminance level where the target luminance value is located, and luminance levels adjacent to the target luminance level, where the difference between the upper or lower limit and the target luminance value is smaller than a preset luminance value. The auto white balance processing method according to feature 4.

6. The luminance information of pixels included in each of the aforementioned image regions is statistically calculated as follows: The process involves dividing the brightness into multiple levels, from the darkest to the brightest, according to pre-set rules, with each brightness level corresponding to a brightness interval. The pixel values ​​of each color's pixels within each image region are quantized to their corresponding luminance values, and the luminance level corresponding to each color's pixel is determined based on the luminance interval in which the corresponding luminance values ​​are located. including, The auto white balance processing method according to feature 1.

7. The luminance information of pixels included in each of the aforementioned image regions is statistically calculated as follows: Obtain the pixel value of each color pixel located at each brightness level within each image region, The sum of the pixel values ​​of all pixels of each color located at each brightness level within each image region is statistically calculated to obtain the total pixel value of each color located at each brightness level within each image region. Further including, The auto white balance processing method according to feature 6.

8. The R / G value for each brightness level is the ratio of the total pixel value of the R pixels for that brightness level to the total pixel value of the G pixels for that brightness level, and the B / G value for each brightness level is the ratio of the total pixel value of the B pixels for that brightness level to the total pixel value of the G pixels for that brightness level. The auto white balance processing method according to feature 1.

9. Using the aforementioned nonparametric method, density estimation is performed for pixels with multiple luminance levels, including bright levels, to obtain at least a density estimate for the bright level pixels. To obtain R / G and B / G values ​​corresponding to dark levels, Based on the R / G and B / G values ​​corresponding to the bright levels and the R / G and B / G values ​​corresponding to the dark levels, a non-parametric method is used to estimate the density of dataless areas in the color space, thereby obtaining estimated pixel density values ​​for the bright levels and the dark levels. Further including, The auto white balance processing method according to feature 1.

10. Obtaining the R / G and B / G values ​​corresponding to the brightness level and the R / G and B / G values ​​corresponding to the darkness level is: The method involves defining a secondary brightness level as the brightness level, and obtaining the R / G value and B / G value corresponding to the brightness level, wherein the secondary brightness level is a brightness level one level lower than the highest brightness level among all brightness levels. The method involves setting a luminance level close to the target luminance value as the dark level, and obtaining the R / G value and B / G value corresponding to the dark level, wherein the target luminance value is obtained by multiplying the maximum luminance value by a predetermined ratio. including, The auto white balance processing method according to feature 9.

11. The aforementioned nonparametric method is a kernel density estimation method. The auto white balance processing method according to feature 1.

12. Calculating a first white balance gain corresponding to each color, based at least on the estimated pixel density of the aforementioned brightness levels, To obtain R / G and B / G values ​​corresponding to dark levels, Based on the total pixel value of the dark levels of each color and the density estimate of the bright levels at the corresponding coordinates, a first white balance gain corresponding to each color is calculated. including, The auto white balance processing method according to feature 9.

13. Calculating a first white balance gain corresponding to each color based on the total pixel value of the dark levels of each color and the density estimate of the bright levels at the corresponding coordinates is: Based on the total pixel value of the dark levels of each color and the density estimate of the bright levels at the corresponding coordinates, the target value for each color is obtained. This includes obtaining a first white balance gain corresponding to each color based on the target value for each color. The auto white balance processing method according to feature 12.

14. Based on the total pixel value of the dark levels of each color and the density estimate of the bright levels at the corresponding coordinates, the target value for each color can be obtained. This includes obtaining a target value for each color based at least on the product of the total pixel value of the dark levels for each color and the density estimate of the bright levels at the corresponding coordinates. The auto white balance processing method according to feature 13.

15. Obtaining the target value for each color is at least based on the product of the total pixel value of the dark levels for each color and the density estimate of the bright levels at the corresponding coordinates. This includes calculating the target value for each color based on the following formula: [Math 1] R1 is the target value in red, G1 is the target value in green, B1 is the target value in blue, R LV2Lv3 This is the total pixel value of the dark red level, G Lv2Lv3 This is the total pixel value of the dark green level, B Lv2Lv3 This is the total pixel value of the dark blue level, and KDE Lv14 is an estimate of the brightness level density, and (x,y) are the coordinates in color space. The auto white balance processing method according to feature 14.

16. Based on the target values ​​for each color, obtaining a first white balance gain corresponding to each color is: It is determined that the first white balance gain corresponding to red is G1 / R1, It is determined that the first white balance gain corresponding to blue is G1 / B1, This includes determining that the first white balance gain corresponding to green is 1, The auto white balance processing method according to feature 15.

17. The aforementioned alternative method is a method that uses a dichroic reflection model. The auto white balance processing method according to feature 1.

18. An automatic white balance processing device, It comprises a statistical analysis unit, a density estimation unit, and a white balance gain determination unit. The statistical analysis unit is configured to divide an image awaiting processing into multiple image regions and to statistically analyze the brightness information of pixels contained in each image region, wherein the brightness information includes at least the brightness level to which the pixels contained in each image region are located. The density estimation unit is configured to perform density estimation for pixels at multiple brightness levels, including bright levels, using a nonparametric method, and to obtain at least a density estimate for the bright level pixels. The white balance gain determination unit is configured to calculate a first white balance gain corresponding to each color, based at least on the estimated pixel density of the brightest levels. The aforementioned pixels include R pixels, G pixels, and B pixels, and the white balance gain determination unit is further configured to obtain R / G values ​​and B / G values ​​corresponding to the brightness levels by performing a color space conversion on the sum of the pixel values ​​of the brightness levels corresponding to the R pixels, G pixels, and B pixels. The density estimation unit is configured to obtain at least R / G and B / G values ​​corresponding to the brightness levels, and to estimate the density of the dataless portion in the color space using a non-parametric method based at least on the R / G and B / G values ​​corresponding to the brightness levels, thereby obtaining at least an estimated density value of the pixels at the brightness levels, wherein the estimated density value is an estimated value of the density distribution in the color space. The aforementioned auto white balance processing device further comprises a mixing processing unit, The mixing unit is configured to process the images awaiting processing using a different method to obtain a second white balance gain for each color, and then mix the first white balance gain for each color with the second white balance gain for each color according to the gain mixing ratio to obtain a final white balance gain for each color. The different method is different from the non-parametric method. An automatic white balance processing device characterized by the following.

19. An imaging device, The auto white balance processing apparatus according to claim 18 is provided, An imaging device characterized by the following features.

20. The imaging device further comprises an image acquisition unit and an image generation unit. The image acquisition unit is configured to acquire the image awaiting processing. The image generation unit is configured to generate a processed image based on the white balance gain obtained by the auto white balance processing device. The imaging apparatus according to feature 19.

21. An electronic device comprising memory and a processor, A computer program is stored in the memory, and the processor is configured to call the computer program and execute the auto white balance processing method described in any one of claims 1 to 17. An electronic device characterized by the following features.

22. A computer-readable storage medium, A computer program is stored in the computer-readable storage medium, and when the computer program is called by a computer, the auto white balance processing method according to any one of claims 1 to 17 is executed. A computer-readable storage medium characterized by the following features.

Images