Auto white balance processing method and device, photographing device, and storage medium

By dividing images into blocks and classifying regions based on brightness, the method improves light source estimation and white balance accuracy in digital imaging devices, addressing the limitations of existing auto white balance techniques.

JP7583898B1Active Publication Date: 2024-11-14ACUTELOGIC
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2023195971
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-11-17
Publication Date
2024-11-14
Estimated Expiration
2043-11-17

AI Technical Summary

Technical Problem

Existing auto white balance methods in digital imaging devices struggle with inaccurate light source estimation due to biased saturation and hue in scenes without a white area, requiring costly calibration and manual adjustments, which can lead to performance deterioration.

Method used

A method that divides images into blocks, calculates brightness information, classifies blocks by object color, and extracts specular reflection components to determine a white balance gain, improving light source estimation accuracy.

Benefits of technology

This approach enables highly accurate light source estimation and enhances the performance of auto white balance processing by classifying regions with similar object colors, allowing for precise white balance correction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007583898000001_ABST
    Figure 0007583898000001_ABST
Patent Text Reader

Abstract

An auto white balance processing method, device, photographing device, and non-volatile computer-readable storage medium are provided that can improve the performance of auto white balance processing. [Solution] The auto white balance processing method includes a statistical analysis step of dividing an image into blocks, calculating brightness information of the pixels contained in each image block, and performing statistical processing of the brightness information for each image block, a classification processing step of classifying each of the image blocks into groups that are regarded as having the same object color based on the results of the statistical processing of each of the image blocks, an extraction processing step of extracting image blocks for calculating white balance gain based on the results of the classification, and a white balance gain calculation step of calculating white balance gain from the specular reflection component in the extracted image block.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] The present disclosure relates to the field of digital imaging, and in particular to an auto white balance processing method and apparatus, an imaging device, and a non-volatile computer-readable storage medium. [Background technology]

[0002] Many devices equipped with digital cameras or video cameras, such as mobile phones, are equipped with an auto white balance function that automatically adjusts the white balance of images output from the sensor.

[0003] Generally, auto white balance functions use one method of calculating the white balance correction coefficient from the integrated value of all pixels, assuming that the average color of the scene is achromatic, or one of calculating the correction coefficient from the integrated value of pixels included in the white area. Furthermore, a calibration method is often used in combination, in which the RGB sensitivity of the sensor is measured in advance for a standard light source, thereby improving the accuracy of calculating the correction coefficient.

[0004] The above-mentioned methods will not produce accurate results if the saturation and hue in the scene are uneven or if there is no white area. Furthermore, the white area is often specified using parameters that can be adjusted by the user, and the adjustment results can have a significant impact on performance, sometimes resulting in a significant degradation of the original performance. Calibration requires properly calibrated equipment, which can be costly. Furthermore, the measurements required for adjustment and calibration require a significant amount of work.

[0005] An effective method for solving the above problems is to estimate the light source from the information contained in the scene and calculate the white balance correction coefficient so that the light source color becomes white.To estimate the light source, a method has been developed that separates reflected light into a specular reflection component and a diffuse reflection component based on a dichromatic reflection model and extracts the specular reflection component as the light source.

[0006] Conventional methods use two sufficiently bright regions, remove the diffuse reflection components contained in both regions, and extract only the specular reflection component. This process assumes that the diffuse reflection components in both regions, i.e., the object color, are the same, and that the specular reflection components must be from the same light source, differing only in intensity. However, the method for selecting both regions that meet these conditions is not clear. Methods have been proposed that select both regions from the entire image and use average values, or select both regions from parts of the image based on the assumption that reflected light from the same object with the same light source will likely be similar in brightness, but these methods do not necessarily produce accurate light source estimation results. Summary of the Invention

[0007] In view of the above-described circumstances, the present disclosure proposes an auto white balance processing method and apparatus, an imaging device, and a non-volatile computer-readable storage medium that can improve the accuracy of light source estimation and the performance of auto white balance processing.

[0008] According to one aspect of the present disclosure, a statistical analysis step of dividing an image into blocks, calculating brightness information of pixels included in each image block, and performing statistical processing of the brightness information for each image block; The present invention provides an auto white balance processing method including a classification processing step of classifying each of the image blocks into groups that are considered to have the same object color based on the results of the statistical processing of each of the image blocks, an extraction processing step of extracting image blocks for calculating white balance gains based on the results of the classification, and a white balance gain calculation step of calculating white balance gains from specular reflection components in the extracted image blocks.

[0009] According to another aspect of the present disclosure, there is provided an auto white balance processing device including: a statistical analysis unit that divides an image into blocks, calculates brightness information of pixels included in each image block, and performs statistical processing of the brightness information for each of the image blocks; a classification processing unit that classifies each of the image blocks into groups that are considered to have the same object color based on the results of the statistical processing of each of the image blocks; an extraction processing unit that extracts image blocks for calculating white balance gains based on the results of the classification; and a white balance gain calculation unit that calculates white balance gains from specular reflection components in the extracted image blocks.

[0010] According to another aspect of the present disclosure, there is provided an imaging device including a processor and a memory unit for storing instructions executable by the processor, wherein the processor is configured to perform the above-mentioned auto white balance processing method when it executes the instructions stored in the memory unit.

[0011] According to another aspect of the present disclosure, there is provided a non-volatile computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the non-volatile computer-readable storage medium to implement the above-described auto white balance processing method.

[0012] According to another aspect of the present disclosure, there is provided a computer program product including computer-readable code or a non-volatile computer-readable storage medium storing computer-readable code, the computer-readable code, when executed in a processor of an electronic device, causing the processor of the electronic device to perform the above-described auto white balance processing method.

[0013] According to the auto white balance processing method and apparatus, imaging device, and computer-readable storage medium of each aspect of the present disclosure, by referencing brightness information of an image signal, regions containing specular reflection components and diffuse reflection components are extracted and classified, and regions classified into the same group are considered to be reflected light from the same object color, thereby making it possible to obtain an appropriate light source color and determine appropriate white balance correction coefficients.This enables highly accurate light source estimation and improves the performance of auto white balance processing.

[0014] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments taken in conjunction with the drawings. [Brief explanation of the drawings]

[0015] The drawings that form a part of the specification illustrate exemplary embodiments, features, and aspects of the present disclosure and together with the specification serve to explain the principles of the present disclosure. [Figure 1] FIG. 1 shows a flowchart of an auto white balance processing method according to an embodiment of the present disclosure. [Figure 2] FIG. 2 shows a flowchart of an auto white balance processing method according to another embodiment of the present disclosure. [Figure 3] FIG. 3 shows an example of dividing an image output from a sensor into blocks. [Figure 4] FIG. 4 shows an example of level classification for each pixel in one image block. [Figure 5] FIG. 5 shows an example of a table storing the average pixel values ​​and the number of pixels of the pixel group at the brightest level and the pixel group at the next brightest level. [Figure 6] FIG. 6 shows an example of a table storing BlockIdx, R / G, B / G, and useIdx. [Figure 7] FIG. 7 shows an example of block removal processing using brightness information. [Figure 8] FIG. 8 shows an example of a table storing classifications for image blocks. [Figure 9]FIG. 9 shows an example of a table that combines sorted useIdx1 and useIdx2, R1, G1, B1, Cnt1, R2, G2, B2, and Cnt2 of FIG. 5, and Group1 and Group2 of FIG. [Figure 10] FIG. 10 shows an example of an updated table in which useIdx1=24 has been removed from the table of FIG. [Figure 11] Figure 11 (A), (B), and (C) show examples of photographs that have been white balance processed by applying the white balance gain stored in DNG, the white balance gain calculated using a conventional method, and the white balance gain calculated using the method proposed in this disclosure, respectively. [Figure 12] FIG. 12 is a block diagram of an auto white balance processing device according to an exemplary embodiment. [Figure 13] FIG. 13 shows an example of the configuration of an image capturing device to which the auto white balance processing device according to the embodiment is applied. DETAILED DESCRIPTION OF THE INVENTION

[0016] Various exemplary embodiments, features, and aspects of the present disclosure are described in detail below with reference to the drawings, in which like reference numerals represent functionally identical or similar elements. While various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0017] The word "exemplary" as used herein means "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0018] In addition, in order to better explain the present disclosure, many specific details are shown in the following specific embodiments. Those skilled in the art should understand that the present disclosure can be similarly implemented without some specific details. In some examples, methods, means, elements, and circuits that are well known to those skilled in the art are not described in detail in order to emphasize the gist of the present disclosure.

[0019] Related art has proposed a method for auto white balance processing that separates specular reflection components and diffuse reflection components based on pixel brightness based on a dichromatic reflection model, but the white balance processing performance is not necessarily sufficient. To solve this problem, the present disclosure was achieved by considering improving the processing accuracy by selecting areas of similar color when extracting two types of sufficiently bright areas. This disclosure solves the above problem by improving the accuracy of light source estimation by extracting areas of reflected light that are considered to have the same object color.

[0020] This disclosure proposes an auto white balance processing method that is executed in a group of products that require white balance correction, such as mobile phones, digital still cameras / video cameras, camera modules, image signal processors, and image processing software, as shown in FIG.

[0021] In the embodiment shown in FIG. 1, the auto white balance processing method may include the following steps.

[0022] Step S110 (statistical analysis step): The image is divided into blocks, brightness information of the pixels contained in each image block is calculated, and statistical processing of the brightness information is performed for each image block.

[0023] In this embodiment, a raw image output from a sensor is divided into blocks, brightness information of pixels contained in each image block is calculated, and statistical processing of the brightness information is performed for each image block.

[0024] In one possible implementation, based on the brightness information of the pixels in each image block, the brightest and second brightest groups of pixels in each image block with different brightness levels are identified, and the average pixel values ​​of the identified brightest and second brightest groups of pixels are calculated.

[0025] Furthermore, if the pixels in the brightest level pixel group are saturated pixels, it is possible to set the pixel group that is one level darker from the brightest level pixel group as the brightest level pixel group, and the pixel group that is one level darker from the next brightest level pixel group as the next brightest level pixel group.

[0026] Step S120 (classification processing step): Based on the results of the statistical processing of each of the image blocks, each of the image blocks is classified into groups that are regarded as having the same object color.

[0027] In this embodiment, the results of statistical processing of the brightness information of each image block are converted into an arbitrary space to generate pixel value information for each image block, and based on the pixel value information for each image block, each image block is classified into groups that are considered to have the same object color. That is, the results of statistical processing of the brightness information of each image block are converted into an arbitrary space to generate pixel value information for each image block, and based on the pixel value information for each image block, each image block is classified into different groups, and the image blocks in each group are considered to have reflections from the same object color.

[0028] In one possible implementation, classification is performed for each image block based on the average pixel value of the pixel group of the brightest level and the average pixel value of the pixel group of the next brightest level calculated in the statistical analysis step. For example, grouping processing is performed in an arbitrary space for the average pixel values ​​of the pixel group of the brightest level and the pixel group of the next brightest level for each block calculated in the statistical analysis step. Furthermore, classification may be performed by eliminating image blocks in which the average brightness information of each pixel calculated in the statistical analysis step is lower than a predetermined value, and image blocks in which the average pixel value information of each pixel deviates by more than a predetermined value from the average pixel value information based on all of the image blocks.

[0029] Step S130 (extraction processing step): Based on the classification result, image blocks for calculating white balance gain are extracted.

[0030] In this embodiment, based on the classification results, image blocks containing specular reflection components and diffuse reflection components for use in calculating white balance gain are extracted.

[0031] In one possible implementation form, the image blocks are sorted by brightness information of the pixel group with the brightest level in each image block calculated in the statistical analysis step, and a predetermined number of image blocks are extracted as first image blocks based on the sorting, and the image blocks are sorted by brightness information of the pixel group with the next brightest level in each image block calculated in the statistical analysis step, and the predetermined number of image blocks are extracted as second image blocks based on the sorting. For example, the brightness information used for sorting is the average pixel value of the pixel group with the brightest level or the pixel group with the next brightest level.

[0032] Furthermore, based on the classification results, the first image blocks and the second image blocks may be extracted as image blocks for calculating white balance gains so that the types of groups into which the predetermined number of first image blocks are classified are the same as the types of groups into which the predetermined number of second image blocks are classified. For example, by utilizing the fact that image blocks considered to have the same object color have approximately the same saturation and hue when they contain specular reflection components and diffuse reflection components, the first image blocks and the second image blocks may be extracted as image blocks for calculating white balance gains so that the types of group numbers into which the first image blocks are classified are the same as the types of group numbers into which the second image blocks are classified.

[0033] Alternatively, image blocks having pixels at the next brightest level whose saturation or hue range deviates by more than a predetermined value from that of pixels at the brightest level may be removed. For example, since the saturation and hue of pixels at the brightest level and pixels at the next brightest level are distributed in approximately the same range, image blocks having pixels at the next brightest level whose saturation and hue range deviates by more than a predetermined value from that of pixels at the brightest level may be removed.

[0034] Furthermore, image blocks may be removed in which the number of pixels in the brightest level pixel group and the next brightest level pixel group is smaller than the average number of pixels in the brightest level pixel group and the next brightest level pixel group of the first image block and the second image block by a predetermined value or more. For example, if the number of pixels in the brightest level pixel group and the next brightest level pixel group of a block is smaller than the average number of pixels in the brightest level pixel group and the next brightest level pixel group of the first image block and the second image block by a predetermined value or more, the block may be removed as noise.

[0035] Step S140 (white balance gain calculation step): Calculate a white balance gain from the specular reflection component in the extracted image block.

[0036] In one possible implementation, a specular reflection component is obtained based on the average pixel values ​​of the brightest and next brightest pixel groups in the image block for calculating the white balance gain extracted in the extraction processing step, and the white balance gain is calculated based on the specular reflection component. For example, the specular reflection component and the diffuse reflection component may be separated by taking the difference between pixel value statistics of the image block, and the white balance gain may be calculated from the specular reflection component and the pixel value statistics.

[0037] Furthermore, if the standard deviation of the average pixel values ​​of each image block calculated in the statistical analysis step is smaller than a predetermined value, the white balance gain may be calculated based on the average pixel value of the brightest pixel group in the image block for calculating the white balance gain extracted in the extraction processing step. For example, in the case of an image that is composed almost entirely of diffuse reflection components of the light source color, such as a white chart or a white patch on a color chart, it is not necessary to separate the specular reflection component from the diffuse reflection component. In such cases, the white balance is calculated using only the diffuse reflection component, which is the white component, rather than the specular reflection component.

[0038] According to this embodiment, by referencing the brightness information of the image signal, regions containing specular reflection components and diffuse reflection components are extracted and classified, and regions classified into the same group are considered to be reflected light from the same object color, making it possible to obtain an appropriate light source color and determine appropriate white balance correction coefficients.This enables highly accurate light source estimation and improves the performance of auto white balance processing.

[0039] In calculating the white balance gain using the dichromatic reflection model, the brightest area and the areas slightly darker and next brightest are considered to contain specular reflection components and diffuse reflection components, and can be described as follows: I1 = f1*Spec + Diff I2 = f2*Spec + Diff … (1) I1 indicates the intensity of the brightest area, and I2 indicates the intensity of the next brightest area. Spec is the intensity of the specular reflection component, and Diff is the intensity of the diffuse reflection component. f1 indicates the proportion of the specular reflection component in the brightest area, and f2 indicates the proportion of the specular reflection component in the next brightest area. This formula for intensity can be applied to each of the tristimulus values ​​(in the case of RGB, it can be applied to each of R, G, and B).

[0040] Since the diffuse reflection component is common, subtracting I1 and I2 will give us an equation that removes the diffuse reflection component. I1-I2 = (f1-f2)*Spec … (2)

[0041] When applied to each of the R, G, and B stimulus values, it can be written as follows: R1-R2 = (f1-f2)*Spec_r G1-G2 = (f1-f2)*Spec_g B1-B2 = (f1-f2)*Spec_b … (3) R1, G1, and B1 are the values ​​of the stimulus values ​​of I1, R2, G2, and B2 are the values ​​of the stimulus values ​​of I2, and Spec_r, Spec_g, and Spec_b are the values ​​of the stimulus values ​​of Spec.

[0042] It is clear that the RGB ratios of the specular reflection components, i.e., Spec_r / Spec_g and Spec_b / Spec_g, are equivalent to (R1-R2) / (G1-G2) and (B1-B2) / (G1-G2) in (3). In other words, by taking the difference between the intensity of the brightest area and the intensity of the next brightest area, we can obtain the RGB ratio of the specular reflection component, which is the light source, and use it to calculate the white balance gain.

[0043] As a prerequisite for the above calculation, the Spec and Diff of I1 and I2 must be the same. The present invention provides a method for extracting I1 and I2 that have the closest Spec and Diff as possible.

[0044] 2 shows a flowchart of an auto white balance processing method according to another embodiment of the present disclosure. As shown in FIG. 2, the auto white balance processing method may include the following steps.

[0045] Step S201: An image output from a sensor is divided into blocks to obtain a plurality of image blocks.

[0046] The image output from the sensor may be divided into any number of blocks. For example, as shown in Figure 3, the image may be divided into 30 blocks of 5 rows and 6 columns, and each divided image block may be assigned an ID number.

[0047] Step S202: The pixel values ​​of each image block are classified according to brightness level.

[0048] For example, the pixel values ​​of each image block are classified into brightness levels divided into an arbitrary number of levels. Specifically, the pixels of each image block are divided into n levels based on the brightness of the pixel value (e.g., n=16). For example, in the case of a Bayer array such as RGGB, a 2x2 unit is treated as one pixel, and the average of the Gr and Gb pixel values ​​is taken as the brightness.

[0049] Figure 4 shows an example of level classification for each pixel in one image block. Levels 0 and 1 indicate dark areas, and levels 14 and 15 indicate bright areas.

[0050] Step S203: Pixel values ​​are totaled for each brightness level.

[0051] For example, the integrated value of the pixel values ​​and the number of pixels of each channel (e.g., RGB) are calculated for each divided brightness level and stored in memory.

[0052] Step S204: The average pixel values ​​of the pixel group with the brightest level and the pixel group with the next brightest level are calculated.

[0053] Among the pixels classified into levels in step S203, the average values ​​of the pixels in the top m1 levels (e.g., m1=3) are calculated. These are designated as R1, G1, and B1. Furthermore, the number of target pixels Cnt1 is also calculated.

[0054] Next, calculate the average pixel values ​​for the m2 level (e.g., m2=3) from the level immediately below the top m1 level. These are designated as R2, G2, and B2. Then, calculate the number of target pixels, Cnt2.

[0055] It is preferable to perform saturated region determination processing before calculating the average pixel values ​​of the pixel group at the brightest level and the pixel group at the next brightest level.

[0056] Specifically, if a pixel value belonging to the highest brightness level is recognized as a saturated pixel (if the pixel values ​​of all RGB channels are the maximum values ​​that can be represented by the bit width of the pixel value, or if the image output from the sensor has a threshold for saturated pixel values ​​and the pixel values ​​of all RGB channels exceed that threshold), if the number of saturated pixels exceeds a predetermined number and the number of image blocks containing saturated pixels exceeds a predetermined number, the image output from the sensor is determined to have a saturated area.

[0057] If the image output from the sensor is determined to have saturated areas, the highest level (e.g., level 15) is not included in the selection of the upper m1 level, and instead, the next lower level, level 14, is shifted downward to avoid inaccuracies caused by saturated pixels. As a result of this processing, the m2 level is also shifted downward.

[0058] The above calculation is performed for each block. The calculated average value and number of pixels may be stored in the memory area shown in the table of Fig. 5. BlockIdx1 and BlockIdx2 in the table of Fig. 5 each indicate the block numbers of the image blocks divided in step S201.

[0059] Step S205: Color space conversion is performed on the calculated average pixel values ​​of the pixel group with the brightest level and the pixel group with the next brightest level.

[0060] Specifically, for R1, G1, and B1 calculated in step S204, the ratio is calculated using G1 as the denominator. The same calculation is performed for R2, G2, and B2. The calculated R1 / G1 and R2 / G2 are treated as a single column of data. B1 / G1 and B2 / G2 are treated similarly. BlockIdx is calculated by concatenating BlockIdx1 and an array obtained by adding the number of elements in BlockIdx1 to BlockIdx2.

[0061] Also, BlockIdx is copied to a separate variable and managed. Here, the name of this variable is set to useIdx.

[0062] For example, BlockIdx, R / G, B / G, and useIdx are stored in the memory areas shown in the table of FIG.

[0063] Step S206: The image blocks are classified, that is, grouped.

[0064] The memory area shown in the table in Figure 6 is referenced, the R / G and B / G of the image blocks included in useIdx are extracted, and the extracted data string is used as input to obtain a group number by applying various grouping methods. Common clustering methods such as the kmeans method and hierarchical clustering method can be used as the grouping method. With these clustering methods, the number of clusters and various options can be specified arbitrarily (e.g., 3 clusters).

[0065] The results of the grouping process are stored in the memory area shown in the table in Figure 8. useIdx is converted to values ​​corresponding to BlockIdx1 and BlockIdx2, which are set as useIdx1 and useIdx2, and the corresponding group numbers are stored in Group1 and Group2.

[0066] Preferably, block removal processing based on brightness information may be performed during classification processing. For example, by referencing the memory area shown in the table of Figure 6, image blocks where R / G = B / G = 0 and image blocks where the G value is lower than a certain value (e.g., image blocks below 1% of the maximum value) are removed from useIdx. The table of Figure 7 shows an example of useIdx after processing.

[0067] Preferably, block removal processing based on the mean and standard deviation may be performed during classification processing. For example, the memory area shown in the table of FIG. 6 is referenced to find the mean value and standard deviation of R / G for the image blocks included in useIdx, and block numbers of image blocks that fall outside the range of n times the standard deviation (e.g., n = 2 or 3 are appropriate) are removed from useIdx. The same processing is performed for B / G. After processing, useIdx is updated in the same way as the example shown in FIG. 7.

[0068] The values ​​listed for useIdx1 and useIdx2 in the table of FIG. 8 are examples, and will change depending on the results of FIGS. 6 and 7 when performing block removal processing based on brightness information or the mean and standard deviation.

[0069] Step S207: The top n image blocks are extracted from the image blocks sorted by brightness.

[0070] The image blocks included in useIdx1 and useIdx2 are sorted in descending order by brightness (e.g., pixel value of G) with reference to the table in Figure 5, and the top n image blocks are extracted (n is any natural number).

[0071] For example, the result of sorting in descending order is useIdx1: 5, 15, 24, 23, 4… useIdx2: 6, 5, 23, 22, 7, … If n=3, the extracted block numbers for useIdx1 and useIdx2 are [5, 15, 24] and [6, 5, 23], respectively. Let these block number arrays be Idx1_sel and Idx2_sel. In other words, Idx1_sel = [5, 15, 24] Idx2_sel = [6, 5, 23] This can be expressed as:

[0072] Fig. 9 shows a table that combines the sorted useIdx1 and useIdx2, R1, G1, B1, Cnt1, R2, G2, B2, and Cnt2 in the table of Fig. 5, and Group1 and Group2 in the table of Fig. 8. However, the numbers for Group1 and Group2 are just an example.

[0073] Preferably, the extraction process may involve removing blocks based on the number of pixels. For example, the Cnt counts of the blocks included in Idx1_sel are extracted by referring to the table in FIG. 9, the average of the extracted Cnt counts is calculated (denoted as Cnt_mean), and the Cnt counts of all blocks included in useIdx1 are compared with Cnt_mean. If an image block with an extremely low Cnt count, e.g., less than 1% of Cnt_mean, is found, that image block is removed from useIdx1, and the image block extraction process is performed again to extract the top n image blocks, thereby updating Idx1_sel. Similar processing is performed for Idx2_sel and useIdx2. FIG. 10 shows the updated table in FIG. 9, where useIdx1=24 is removed.

[0074] Preferably, the extraction process may include block removal processing based on the saturation and hue range. For example, if the saturation and hue of an image block included in useIdx2 deviates by a predetermined value or more from the distribution range of the image blocks included in Idx1_sel, the block number of that image block is removed from useIdx2, the image block extraction process is performed again, and Idx2_sel is updated by extracting the top n image blocks. Any color space for saturation and hue can be used, such as an R / G vs. B / G space.

[0075] Preferably, in the extraction process, the top n image blocks may be updated by referring to the classification results. For example, if there is a group number that exists in Group 1 but not in Group 2, the image block corresponding to that group number is removed from useIdx1, and the image block extraction process is performed again, and Idx1_sel is updated.

[0076] For example, referring to the table in Fig. 8, if the type of group number corresponding to Idx1_sel and the type of group number corresponding to Idx2_sel do not match, that is, if the group category corresponding to Idx1_sel and the group category corresponding to Idx2_sel do not match, the type of group number is made to match by updating the block of Idx2_sel. For example, in the table in Fig. 10, the group numbers corresponding to Idx1_sel and Idx2_sel are as follows: Idx1_sel = 5, 15, 23 Group1[Idx1_sel]= 0, 2, 2 Idx2_sel = 6, 5, 23 Group2[Idx2_sel]= 0, 0, 0

[0077] Since the group numbers corresponding to Idx1_sel are 0 and 2, and the group number corresponding to Idx2_sel is only 0, the lowest image block (23) in Idx2_sel is removed, and instead the highest image block in group number 2 from the top n+1 image blocks and below is added to Idx2_sel.

[0078] In this example, the block number to be added is 22, so Idx2_sel is updated as follows: Idx2_sel = 6, 5, 22 Group2[Idx2_sel] = 0, 0, 2

[0079] Step S208: It is determined whether the principal component of the image block is a diffuse reflection component of the light source color.

[0080] The memory area shown in the table in Figure 6 is referenced, and the image blocks included in useIdx are extracted. If the standard deviation of the R / G and B / G distribution of the extracted image blocks is smaller than a certain value, the chromaticity in the normalized color space of the brightest pixel group and the next brightest pixel group can be considered to be approximately the same, and therefore the principal component of the extracted image blocks is considered to be a diffuse reflection component. Furthermore, if the standard deviation of the R / G and B / G distribution of the extracted blocks is equal to or greater than the certain value, it is determined that the principal component of the extracted image blocks includes a specular reflection component and a diffuse reflection component. If it is determined that the principal component of the extracted image blocks includes a specular reflection component and a diffuse reflection component of the light source color, proceed to step S209. If it is determined that the principal component of the extracted image blocks is a diffuse reflection component of the light source color, proceed to step S211.

[0081] Note that useIdx may be updated to reflect useIdx1 and useIdx2 updated in block removal processing based on the number of pixels, block removal processing based on the saturation and hue range, and block update processing based on the results of classification processing.

[0082] Step S209: Specular reflection components and diffuse reflection components are separated and calculated.

[0083] In order to remove the diffuse reflection component according to the above-mentioned formula (3), the table shown in Fig. 5 is referenced to find the average values ​​of the RGB intensities corresponding to the image blocks included in Idx1_sel and Idx2_sel, and the difference between these averages is calculated. The formula is shown below. F*Spec_r = mean(R1[Idx1_sel])-mean(R2[Idx2_sel]) F*Spec_g = mean(G1[Idx1_sel])-mean(G2[Idx2_sel]) F*Spec_b = mean(B1[Idx1_sel])-mean(B2[Idx2_sel]) However, F=f1-f2.

[0084] Step S210: Calculate the white balance gain using the specular reflection component.

[0085] The white balance gain is calculated using the specular reflection component obtained in step S209, as shown in the following equation. White balance gain R = Spec_g / Spec_r White balance gain G = 1.0 White balance gain B = Spec_g / Spec_b

[0086] Step S211: The pixel group with the brightest level is regarded as the light source color, and the white balance gain is calculated.

[0087] If it is determined that the principal component of the extracted image block is the diffuse reflection component of the light source color, the brightest pixel group is treated as a white area, and the white balance gain is calculated from the average intensity of the white area using the following formula. White_r = mean(R1[Idx1_sel]) White _g = mean(G1[Idx1_sel]) White _b = mean(B1[Idx1_sel]) White balance gain R = White_g / White_r White balance gain G = 1.0 White balance gain B = White_g / White_b

[0088] The photos shown in Figure 11 (A), (B), and (C) are examples of images developed using a DNG file published in the MIT-Adobe FiveK Dataset (https: / / data.csail.mit.edu / graphics / fivek / ), applying the white balance gain stored in the DNG, the white balance gain calculated using the conventional method, and the white balance gain calculated using the proposed method. The photo shown in Figure 11 (A) is the correct image using the ideal white balance gain stored in the DNG. The photo shown in Figure 11 (B) differs from the photo shown in Figure 11 (A) in that the color tone is warmer (reddish). The photo shown in Figure 11 (C) is closer to the photo shown in Figure 11 (A), which has desirable color tones, compared to the photo shown in Figure 11 (B). Therefore, it can be seen that the proposed method produces better image quality than the conventional method.

[0089] According to this embodiment, by referencing the brightness information of the image signal, regions containing specular reflection components and diffuse reflection components are extracted and classified, and regions classified into the same group are considered to be reflected light from the same object color, making it possible to obtain an appropriate light source color and determine appropriate white balance correction coefficients.This enables highly accurate light source estimation and improves the performance of auto white balance processing.

[0090] 12 is a block diagram of an auto white balance processing device according to an exemplary embodiment. As shown in Fig. 12, this auto white balance processing device may include a statistical analysis unit 1201 that divides an image into blocks, calculates brightness information for pixels included in each image block, and performs statistical processing of the brightness information for each image block, a classification processing unit 1202 that classifies each image block into groups that are considered to have the same object color based on the results of the statistical processing of each image block, an extraction processing unit 1203 that extracts image blocks for calculating white balance gains based on the classification results, and a white balance gain calculation unit 1204 that calculates white balance gains from the specular reflection components in the extracted image blocks.

[0091] In one possible implementation, the statistical analysis unit 1201 identifies the brightest and second brightest pixel groups among multiple pixel groups with different brightness levels in each image block based on the brightness information of the pixels in each image block, and calculates the average pixel values ​​of the identified brightest and second brightest pixel groups.

[0092] In one possible implementation, if the pixels in the brightest level pixel group are saturated pixels, the statistical analysis unit 1201 determines the pixel group that is one level darker from the brightest level pixel group as the brightest level pixel group, and the pixel group that is one level darker from the next brightest level pixel group as the next brightest level pixel group.

[0093] In one possible implementation, the classification processing unit 1202 performs classification for each image block based on the average pixel value of the pixel group at the brightest level calculated by the statistical analysis unit 1201 and the average pixel value of the pixel group at the next brightest level.

[0094] In one possible implementation, the classification processing unit 1202 performs classification by removing image blocks in which the average brightness information of each pixel calculated by the statistical analysis unit 1201 is lower than a predetermined value, and image blocks in which the average pixel value information of each pixel deviates by more than a predetermined value from the average pixel value information based on all of the image blocks.

[0095] In one possible implementation, the extraction processing unit 1203 sorts the image blocks by brightness information of the pixel group with the brightest level in each image block calculated by the statistical analysis unit 1201, extracts a predetermined number of image blocks as first image blocks based on the sorting, sorts the image blocks by brightness information of the pixel group with the next brightest level in each image block calculated by the statistical analysis unit 1201, and extracts the predetermined number of image blocks as second image blocks based on the sorting.

[0096] In one possible implementation form, the extraction processing unit 1203 extracts the first image blocks and the second image blocks as image blocks for calculating white balance gain based on the classification results, so that the classification of the classified groups of the predetermined number of first image blocks is the same as the classification of the classified groups of the predetermined number of second image blocks.

[0097] In one possible implementation, the extraction processing unit 1203 removes image blocks having pixels at the next lightest level whose saturation or hue range deviates from the pixels at the lightest level by more than a predetermined value.

[0098] In one possible implementation, the extraction processing unit 1203 removes image blocks in which the number of pixels in the brightest level pixel group and the next brightest level pixel group is smaller than the average number of pixels in the brightest level pixel group and the next brightest level pixel group of the first image block and the second image block by more than a predetermined value.

[0099] In one possible implementation, the white balance gain calculation unit 1204 determines the specular reflection component based on the average pixel value of the pixel group at the brightest level and the average pixel value of the pixel group at the next brightest level in the image block for white balance gain calculation extracted by the extraction processing unit 1203, and calculates the white balance gain based on the specular reflection component.

[0100] In one possible implementation, if the standard deviation of the average pixel values ​​of each image block used to calculate the white balance gain calculated by the statistical analysis unit 1201 is smaller than a predetermined value, the white balance gain calculation unit 1204 calculates the white balance gain based on the average pixel values ​​of the brightest level pixel group of the image block extracted by the extraction processing unit 1203.

[0101] Regarding the apparatus in the above-described embodiment, the specific method by which each unit performs an operation is described in detail in the embodiment relating to the method, and therefore a detailed description thereof will be omitted here.

[0102] 13 shows an example of the configuration of an image capturing apparatus to which an auto white balance processing apparatus according to an embodiment of the present invention is applied. This image capturing apparatus includes an optical lens unit 1301, an image sensor unit 1302, a shading correction unit 1303, an auto white balance processing apparatus 1304, and a white balance correction unit 1305. As described above, the auto white balance processing apparatus 1304 includes the statistical analysis unit 1201, the classification processing unit 1202, the extraction processing unit 1203, and the white balance gain calculation unit 1204.

[0103] The auto white balance processing device may be realized by hardware, by software, or by a combination of hardware and software.

[0104] According to this embodiment, by referencing the brightness information of the image signal, regions containing specular reflection components and diffuse reflection components are extracted and classified, and regions classified into the same group are considered to be reflected light from the same object color, making it possible to obtain an appropriate light source color and determine appropriate white balance correction coefficients.This enables highly accurate light source estimation and improves the performance of auto white balance processing.

[0105] An embodiment of the present disclosure further provides a computer-readable storage medium having computer program instructions stored thereon, the computer program instructions, when executed by a processor, causing the above-described auto white balance processing method to be realized. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0106] An embodiment of the present disclosure further provides an imaging device including a processor and a memory unit for storing instructions executable by the processor, wherein the processor is configured to perform the above-mentioned auto white balance processing method when it executes the instructions stored in the memory unit.

[0107] An embodiment of the present disclosure further provides a computer program product including computer-readable code or a non-volatile computer-readable storage medium storing computer-readable code, the computer-readable code, when executed in a processor of an electronic device, causing the processor of the electronic device to perform the above-described auto white balance processing method.

[0108] Although the embodiments of the present disclosure have been described above, the above description is merely illustrative and not exhaustive, and is not limited to the disclosed embodiments. Various modifications and alterations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms chosen in this specification are intended to appropriately interpret the principles, practical applications, or improvements to existing technology of the embodiments, or to allow others skilled in the art to understand the disclosed embodiments.

Claims

1. a statistical analysis step of dividing the image into blocks, calculating brightness information of pixels contained in each image block, and performing statistical processing of the brightness information for each image block; a classification processing step of classifying each of the image blocks into groups that are regarded as having the same object color based on a result of the statistical processing of each of the image blocks; an extraction processing step of extracting image blocks for use in calculating a white balance gain based on a result of the classification; A white balance gain calculation step of calculating a white balance gain from the specular reflection component in the extracted image block, In the statistical analysis step, a pixel group having the brightest level and a pixel group having the second brightest level are identified from among a plurality of pixel groups having different brightness levels in each image block based on brightness information of pixels in each image block, and an average pixel value of the identified pixel group having the brightest level and pixel group having the second brightest level is calculated; In the classification processing step, the classification is performed for each image block based on the average pixel value of a pixel group with the brightest level calculated in the statistical analysis step and the average pixel value of a pixel group with the next brightest level calculated in the statistical analysis step.

13. An auto white balance processing method comprising:

2. 2. The auto white balance processing method according to claim 1, wherein in the classification processing step, the classification is performed by removing image blocks in which the average of brightness information of each pixel calculated in the statistical analysis step is lower than a predetermined value, and image blocks in which the average of pixel value information of each pixel deviates by a predetermined value or more from the average of pixel value information based on all of the image blocks.

3. In the extraction processing step, sorting the image blocks according to brightness information of the brightest level pixel group of each image block calculated in the statistical analysis step, and extracting a predetermined number of image blocks as first image blocks based on the sorting; 2. The auto white balance processing method according to claim 1, further comprising the steps of: sorting the image blocks according to brightness information of the pixel group of the next brightest level in each image block calculated in the statistical analysis step; and extracting the predetermined number of image blocks as a second image block based on the sorting.

4. 4. The auto white balance processing method according to claim 3, wherein in the extraction processing step, the first image blocks and the second image blocks are extracted as image blocks for calculating a white balance gain, based on a result of the classification, so that a category of the group into which the predetermined number of the first image blocks are classified is the same as a category of the group into which the predetermined number of the second image blocks are classified.

5. 4. The auto white balance processing method according to claim 3, wherein in the extraction processing step, an image block having a pixel group at the next brightest level whose saturation or hue range deviates from that of the pixel group at the brightest level by a predetermined value or more is removed.

6. 4. The auto white balance processing method according to claim 3, wherein in the extraction processing step, an image block in which the number of pixels in the brightest level pixel group and the second brightest level pixel group is smaller than an average number of pixels in the brightest level pixel group and the second brightest level pixel group of the first image block and the second image block by a predetermined value or more is eliminated.

7. 4. The auto white balance processing method according to claim 3, wherein in the white balance gain calculation step, a specular reflection component is determined based on an average pixel value of a pixel group having the brightest level and an average pixel value of a pixel group having the next brightest level in the image block for calculating the white balance gain extracted in the extraction processing step, and the white balance gain is calculated based on the specular reflection component.

8. 4. The auto white balance processing method according to claim 3, wherein the white balance gain calculation step includes a step of calculating a white balance gain based on an average pixel value of the brightest level pixel group of the image block for white balance gain calculation extracted in the extraction processing step when a standard deviation of the average pixel values ​​of each image block calculated in the statistical analysis step is smaller than a predetermined value.

9. a statistical analysis unit that divides an image into blocks, calculates brightness information of pixels included in each image block, and performs statistical processing of the brightness information for each image block; a classification processing unit that classifies each of the image blocks into groups that are regarded as having the same object color based on a result of the statistical processing of each of the image blocks; an extraction processing unit that extracts an image block for calculating a white balance gain based on a result of the classification; a white balance gain calculation unit that calculates a white balance gain from the specular reflection component in the extracted image block, the statistical analysis unit, based on brightness information of pixels in each image block, identifies a pixel group having the brightest level and a pixel group having the second brightest level among a plurality of pixel groups having different brightness levels in each image block, and calculates an average pixel value of the identified pixel group having the brightest level and pixel group having the second brightest level; the classification processing unit performs the classification for each image block based on the average pixel value of a pixel group with the brightest level and the average pixel value of a pixel group with the next brightest level, both calculated by the statistical analysis unit.

1. An auto white balance processing device comprising:

10. A processor; a memory unit for storing instructions executable by a processor; 9. An imaging device, wherein the processor is configured to execute the auto white balance processing method according to claim 1 when executing instructions stored in the storage unit.

11. A non-volatile computer-readable storage medium having computer program instructions stored thereon, the non-volatile computer-readable storage medium realizing the auto white balance processing method according to any one of claims 1 to 8, when the computer program instructions are executed by a processor.

Citation Information

Patent Citations

  • Image processor and image processing method

    JP2007013415A

  • Image processing device with auto white balance

    JP2009506700A

  • Image processor and control method thereof

    JP2014006745A

  • Image processing apparatus and control method thereof

    JP2014007528A

  • Image processing apparatus and control method therefor

    JP2014007611A