Method and device for stabilizing white point information, computer program product and storage medium

By generating confidence values ​​and locking/unlocking WP update status, combined with α-clipping mean filter, the time instability problem of AWB algorithm in difficult scenarios is solved, the smooth and stable output of WP information is achieved, and the calculation cost is reduced. Hardware dependencies.

CN114202468BActive Publication Date: 2024-11-29BEIJING XIAOMI MOBILE SOFTWARE CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202110649533.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-10
Publication Date
2024-11-29
Estimated Expiration
2041-06-10

AI Technical Summary

Technical Problem

Existing automatic white balance (AWB) algorithms are prone to time instability in difficult scenarios, resulting in delays or errors in white point (WP) output, and rely on high-frequency image feature detection, high computing costs or require dedicated hardware.

Method used

By receiving illuminance level information and AWB statistics, a confidence value is generated, a threshold is determined based on the main color count and illuminance level, a WP update status is locked or unlocked, and the WP information is smoothed using an α-cropped mean filter to avoid frame-to-frame fluctuations and jump.

Benefits of technology

Locking WP when the AWB algorithm fails, ensuring stability, avoiding error output, reducing calculation costs, independent of hardware requirements, and improving user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114202468B_ABST
    Figure CN114202468B_ABST
Patent Text Reader

Abstract

A method and apparatus, a computer program product and a storage medium for temporally stabilizing white point information. The method includes: receiving an illumination level, AWB statistics, and an input WP; generating a histogram of the AWB statistics; generating a confidence value of the input WP from the histogram; determining a dominant color count; determining a threshold based on the dominant color count and the illumination level; when the confidence value is below the threshold, setting the update state to locked, otherwise setting the update state to unlocked; when the update state is locked, obtaining final WP information from a previous frame, otherwise obtaining the final WP information from the input WP information. During the locked state, the WP remains unchanged as long as the AWB algorithm is in a difficult scene. Once the difficult scene has passed, the update parameter is set to unlocked so that the WP can be changed again.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a computer-implemented method for temporally stabilizing white point (WP) information during an automatic white balance (AWB) process. The present disclosure also relates to a data processing device for implementing the method, a computer program product including instructions for the method, and a computer-readable storage medium including instructions for the method. Background Art

[0002] In photography and image processing, auto white balance (AWB) is the process of globally adjusting the colors in an image to remove any undesirable color casts emanating from the illumination source on the imaged scene. Auto white balance plays a key role in the end-user experience of a camera by determining color image quality to a large extent. Prior art AWB algorithms generally perform well on many scenes, but still fail in certain situations. Pointing a camera at a scene close to the failure limit of AWB can produce temporal instabilities that significantly degrade the end-user experience.

[0003] A traditional approach to compensate for the instability of the AWB algorithm is to smooth the AWB output white point (WP) by a temporal filter. Typically, this can be a weighted and / or clipped mean filter acting on the WP over the past convergence time interval T. This is widely used in the industry by defining adjustment parameters for T depending on the acquisition mode (still, video, viewfinder).

[0004] The temporal AWB stabilization method in Niloufar Pourian and Rastislav Lukac, “Autowhite balance stabilization in digital video”, Electronic Imaging, 2019(9), pp. 373-1 to 373-5, adaptively selects a smoothing filter convergence time T from a set of pre-adjusted values. The decision on the convergence time selection is made by detecting changes in the illuminant and the image scene. There is a separate value of T for each of the following scenarios: (i) the scene changes but the illuminant does not change, (ii) the illuminant changes but the scene does not change, and (iii) the scene changes and the illuminant changes occur simultaneously. The scene and illuminant changes are detected by analyzing the temporal changes of several characteristic signals calculated for each frame acquired from the imaging sensor. The characteristic signals follow the correlated color temperature (CCT), color statistics, and high-frequency image information. The CCT is calculated within the image signal processing (ISP) pipeline by the AWB algorithm. Color change information is based on frame-to-frame distances of color statistics, which are individual color channel histograms measured locally and globally within an image frame. The final cue is based on overall motion detection in a sequence of frames, which is measured by matching high-frequency features (e.g., corners and edges) extracted in consecutive frames.

[0005] The main disadvantage of this method is that it uses only a smoothing filter to stabilize the AWB output for all three cases. Now, imagine a situation where the end user pans the camera to a difficult scene that can mislead the AWB to decide on a wrong WP, and the end user keeps the camera on that scene. Once the convergence time T is over, the smoothing filter delays the wrong WP output. However, no matter how large T is chosen, the user may decide to keep the camera on the difficult scene longer, and the output stabilized WP will eventually be wrong.

[0006] Furthermore, the CCT values ​​used to measure temporal color and / or illuminant variations are obtained from the same ISP chain as the WP, so they are closely correlated. Therefore, an erroneous estimate of the WP by the AWB will lead to an erroneous estimate of the CCT, which will be misleading in conclusions about environmental variations in the AWB stability.

[0007] Furthermore, this approach relies on high-frequency image feature detection and matching, which is either computationally expensive or requires specialized hardware (HW).

[0008] The prior art described in Fast Fourier Color Constancy, 2017, CVPR (Conference on Computer Vision and Pattern Recognition), by Jonathan Barron, Yun-Ta Tsai, pp. 6950-6958, relates to a low computational cost AWB method and provides temporal stability of the AWB result as a complement to that obtained directly from the AWB estimation process. WP is estimated in color chromaticity space by a two-step procedure. In the first step, the chromaticity spatial distribution of the color content of the image frame is analyzed by convolution with a set of pre-trained convolution kernels. The kernel is learned by training on a large dataset of images so that the convolution result reaches its peak at the annotated ground truth WP. During run-time operation, the second step fits a Gaussian-like probability density function on the convolution result and estimates the WP as the mean of the fitted distribution. The variance of the fitted distribution is used to evaluate the goodness of peak detection, because a large variance indicates that the convolution result is not too peaky and the WP is unlikely to be accurate. Therefore, after scale normalization, the variance is used to directly measure the confidence of the WP estimate. Complementary temporal WP stabilization is achieved by temporally smoothing the WP values ​​via a Kalman-type filter. The degree of smoothing in the filter and the speed of the WP transitions are controlled by the WP confidence estimate.

[0009] The main disadvantages of this AWB stabilization are similar to those of the method described in Niloufar Prian, Rastislav Lukac, "Auto white balance stabilization in digital video", Electronic Imaging, vol. 2019(9), pp. 373-1 to 373-5, 2019. Imagine the same situation: the end user pans the camera to a difficult scene and keeps the camera on the scene for as long as the end user likes. Difficult scenes will lead to widely scattered convolution results, which will have large fitting variance and low confidence. Low confidence will cause the Kalman filter to over-smooth the output WP, but will eventually reach the wrong WP sooner or later depending on how difficult the scene becomes.

[0010] More generally, each AWB algorithm uses different assumptions and therefore has a limited range of validity. Scenes that occur near the limits of validity of the inherent AWB algorithm can produce misleading WP results. As a result, the colors in the processed image may acquire rather unnatural color casts. In addition to this, small changes in camera framing near the boundaries where the AWB assumptions are valid can result in different color casts in consecutive frames of the recorded video, or different color casts as observed in the device viewfinder. This can lead to color oscillations in the recorded video, which depend primarily on the amplitude and frequency of the oscillations and can cause a negative experience for the user. Another phenomenon that reduces the user experience is changing the white balance (WB) gain when the scene changes, but the illuminant is the same. Typically, this phenomenon occurs when a correct AWB result becomes an erroneous result due to a scene change. Although the oscillation can be controlled with conventional filters (e.g., clipped mean), the latter situation cannot be solved with such a device. It is important to stabilize the perceived frame-to-frame color cast within the acquired frame sequence. By stabilizing WP at the output of the AWB algorithm, the stability of the color gain can be maintained. Summary of the invention

[0011] An object of the present disclosure is to provide an improved method of smoothing the WP information (ie, white point or white balance gain) output from an AWB process.

[0012] According to the present disclosure, the object is achieved by a computer-implemented method for temporally stabilizing white point (WP) information in an automatic white balance (AWB) process, the method comprising: a) receiving illumination level information, AWB statistics, and input WP information, the AWB statistics comprising image pixel data from frames of a sequence of frames representing a scene, the input WP information being obtained from the AWB process; b) generating a confidence value for the input WP information; c) determining a threshold based on the dominant color count and the illumination level information; d) setting a binary WP update state to a locked state when the confidence value is below the threshold, and setting the binary WP update state to an unlocked state when the confidence value is above the threshold; and, e) generating final WP information based on WP information obtained from previous frames of the sequence when the binary WP update state is locked, and generating final WP information from the input WP information when the binary WP update state is unlocked.

[0013] The method according to the present disclosure allows to lock or unlock the WP information. In other words, during the locked state, as long as the AWB algorithm is in a difficult scene, the WP remains unchanged, making the calculation of a new white point very unreliable. Once the difficult scene has passed, the update parameters are set to unlock so that the WP can be changed again. The decision about locking / unlocking is based on a confidence estimate to determine when a potential AWB algorithm may invalidate the WP estimate.

[0014] In an embodiment of the present disclosure, step b) includes: generating a histogram of the image pixel data, the confidence value is generated based on the histogram, and the histogram is optionally a two-dimensional histogram generated in a logarithmic chromaticity space.

[0015] Using a histogram to compute the confidence values ​​is convenient because the histogram can be easily computed from the image pixel data, which means that no external input is required for the confidence values. Using a logarithmic color space is optional because this allows for easier color separation.

[0016] In an embodiment of the present disclosure, step b) includes: determining a first confidence metric, and determining a second confidence metric. Determining the first confidence metric includes: counting the number of histogram intervals having at least two points to determine the number of intrinsic non-zero intervals; multiplying the number of intrinsic non-zero intervals by the interval area to determine the histogram area; and dividing the histogram area by the maximum area to determine the first confidence metric, the maximum area being the product of the total number of intervals and the interval area. Determining the second confidence metric includes: dividing the count of the histogram intervals by the total number of unsaturated blocks in the AWB statistics to determine a discrete probability distribution; and calculating the entropy of the discrete probability distribution to determine the second confidence metric.

[0017] The histogram area provides an estimate of the overall spread of colors in the scene, without providing an estimate of the importance of which colors are more dominant than others. This is a relevant WP confidence measure particularly for AWB algorithms, which are not affected by the dominance of a few colors, but provide importance for all colors. On the other hand, the entropy of the histogram is useful for AWB algorithms that are prone to errors when the color distribution is not uniform, i.e. when there are dominant colors.

[0018] In an embodiment of the present disclosure, step b) includes: determining a confidence value by calculating a weighted average of a first confidence metric and a second confidence metric; or determining one or more additional confidence metrics, in particular confidence metrics generated by the AWB process, and determining the confidence value by calculating a weighted average of the first confidence metric, the second confidence metric, and one or more additional confidence metrics.

[0019] Combining multiple confidence sub-metrics improves the robustness of the method and makes it applicable to various known AWB algorithms.

[0020] In an embodiment of the present disclosure, step c) comprises determining a dominant color count of the frame, in particular determining the dominant color count of the frame by counting the number of peaks in a histogram.

[0021] Using a histogram, in particular a logarithmic chromaticity histogram, is advantageous because this has different color separations. Furthermore, the dominant color count (ie the total number of dominant colors in a frame) is usually quite stable over time.

[0022] In an embodiment of the present disclosure, step c) includes: converting the illumination level information into a first probability value between 0 and 1, optionally including: for illumination level information with an illumination level lower than 3000 lux, setting the first probability value to 0; for illumination level information with an illumination level higher than 5000 lux, setting the first probability value to 1; and linearly mapping the illumination level information with an illumination level between 3000 and 5000 lux, and setting the first probability value to a value between 0 and 1; and based on the primary color count, optionally calculating a second probability value D using the following equation 颜色 :

[0023]

[0024] Wherein, the normalized level and the basic level are predetermined natural numbers, the dominant color count is the dominant color count of the scene at frame f; and the time interval is a predetermined time interval.

[0025] WP estimation in outdoor scenes captured in bright daylight is very reliable, because most practical AWB algorithms narrow the possible CCT range of bright daylight. This means that for illumination levels above 5000 lux, the adaptive threshold can take a smaller value, or alternatively, the first probability value can be set to 1 to represent an easy scene. Illumination levels below 3000 lux can be generated by outdoor scenes or indoor scenes under forecast weather and / or dim conditions, that is, practically any light source. Therefore, the CCT range cannot be restricted, and the WP confidence threshold must be set high, or the first probability value can be set to 0. Increasing the illumination level between 3000 lux and 5000 lux, the threshold should be gradually reduced, or the first probability value should be gradually changed from 0 to 1. In addition to the illumination level, the number of colors is also related to the WP confidence. The color content in the scene can be measured by the number of dominant colors present in the scene. Since this can still produce frame-to-frame jumps, the second probability value is optionally based on the trend of the color content rather than on the dominant color count itself. The trend of the color content is measured by the total number of colors within a predetermined time interval. The basic level may be selected to represent a plurality of dominant colors that represent a scene that is neither rich nor poor in color content. The number of basic level colors totaled within the time period should produce a value of 0 in the second probability value. If the scene consists of its minimum, i.e., only 1 dominant color, for the period, the second probability value should be close to -1.

[0026] In an embodiment of the present disclosure, step c) includes: determining a threshold drive signal by combining a first probability value and a second probability value, and determining the threshold based on the threshold drive signal; or, determining one or more additional probability values, in particular, probability values ​​determined based on the presence or absence of a face in the scene, determining the threshold drive signal by combining the first probability value, the second probability value and one or more additional probability values, and determining the threshold based on the threshold drive signal.

[0027] In this way, the threshold drive signal is determined by the lighting conditions and the color count. Thus, in the case of uncertain illumination levels caused by outdoor or indoor scenes, poor color content can still pull down the threshold drive signal.

[0028] In an embodiment of the present disclosure, the method further comprises: smoothing the input WP information using an alpha-clipped mean filter to determine smoothed input WP information, and optionally using the smoothed input WP information for step e).

[0029] Smoothing WP means following the trend of WP variation without any frame-to-frame fluctuations and jumps. By temporally averaging each WP component separately, the smooth trend can be followed quite effectively. Depending on the WP format, the WP components can be, for example, R, G and B channels, chroma ratios R / G and B / G, etc. Unfortunately, sudden jumps in the input signal can mislead the temporal averaging. This can happen when shooting scenes close to the effectiveness limit of the inherent AWB algorithm. The alpha clipping mean filter temporally smoothes each component of the WP by discarding the alpha part of the minimum and maximum values ​​before averaging to mitigate sudden jumps.

[0030] In an embodiment of the present disclosure, when the binary WP update state is unlocked, step e) includes: e1) setting the final WP information equal to the input WP information or equal to the smoothed input WP information when the binary WP update parameter is unlocked during a predetermined number of frames before the frame in the sequence; or, e2) generating the final WP information as a weighted average of the input WP information or the smoothed input WP information and the WP information obtained from the previous frame when the binary WP update parameter changes from locked to unlocked during a frame among the predetermined number of frames before the frame in the sequence.

[0031] This allows the locked WP to be converted back to the actual WP during a predetermined number of frames, thereby avoiding a sudden jump in the WP that may occur if the locked WP and the actual WP are substantially different and the WP update parameters become unlocked.

[0032] In an embodiment of the present disclosure, step e2) includes: estimating the derivative of the input WP information or the smoothed input WP information on a further predetermined number of frames before the frame in the sequence; determining whether the input WP information or the smoothed input WP information does not deviate from the WP information obtained from the previous frame based on the derivative, the input WP information or the smoothed input WP information, and the WP information obtained from the previous frame; if the input WP information or the smoothed input WP information does not deviate from the WP information obtained from the previous frame, setting the final WP information equal to the WP information obtained from the previous frame; and if the input WP information or the smoothed input WP information deviates from the WP information obtained from the previous frame, generating the final WP information as a weighted average of the input WP information or the smoothed input WP information and the WP information obtained from the previous frame.

[0033] In this embodiment, the WP derivatives are analyzed to identify WP trends, which in some cases help determine the final WP information.

[0034] In an embodiment of the present disclosure, step e2) comprises generating the final WP information as a weighted average by smoothly changing the final WP information from the WP information obtained from the previous frame to the input WP information or the smoothed input WP information during a predetermined switching period of a frame following the frame in the sequence.

[0035] This allows the locked WP to be gradually changed back to the actual WP over a predetermined number of frames, thereby avoiding a sudden jump in WP that may occur if the locked WP and actual WP are substantially different and the WP update parameter becomes unlocked.

[0036] In an embodiment of the present disclosure, the WP information includes a white point WP or a white balance WB gain.

[0037] This allows the method to work on white point values ​​or white balance values ​​commonly used in AWB algorithms.

[0038] According to the present disclosure, the object is achieved by means of a data processing device, which includes means for executing the above method.

[0039] According to the present disclosure, the object is achieved by a computer program product, which includes instructions that, when the program is executed by a computer, cause the computer to perform the above method.

[0040] According to the present disclosure, the object is achieved by using a computer-readable storage medium, which includes instructions, which, when executed by a computer, cause the computer to perform the above method.

[0041] It can be easily understood that one or more of the above-described embodiments can be easily combined with each other.

[0042] In summary, the above method allows to lock or unlock the WP information. In other words, during the locked state, as long as the AWB algorithm is in a difficult scene, the WP remains unchanged, making the calculation of a new white point very unreliable. Once the difficult scene has passed, the update parameters are set to unlock so that the WP can be changed again. The decision about locking / unlocking is based on a confidence estimate to determine when a potential AWB algorithm may invalidate the WP estimate. In particular, the confidence is based on a combination of multiple confidence sub-metrics to be able to operate with any modern AWB algorithm selected.

[0043] According to the present disclosure, in addition to the conventionally used WP smoothing, whenever the AWB cannot possibly produce a correct WP estimate, the WP is locked. Therefore, as long as the scene and / or lighting conditions indicate unsafe, the correctly locked WP is maintained instead of converging to the wrong WP after the pre-adjusted convergence time T has expired. The AWB stabilization algorithm according to the present disclosure does not degrade the end-user experience in three aspects. The first aspect is when the AWB stabilization keeps the WP unlocked and the AWB correctly estimates the WP. The second aspect is when the AWB stabilization unnecessarily locks the WP to the correct WP. The third aspect is when the AWB stabilization does not remain locked to the wrong WP for a long time after the AWB returns to the correct WP, in rare cases, when the AWB is misled by the input AWB WP and then locked to the wrong WP. The algorithm is independent of any potential AWB algorithm. In addition, if the AWB algorithm provides a reliable WP confidence estimate, the reliable WP confidence estimate can be used to improve the AWB stabilization quality. Any other available WP confidence estimates, such as those provided by the AWB process, can also be used to improve the AWB stabilization quality. The algorithm is able to utilize multiple different clues representing the reliability of the estimated WP. Here, illumination levels are shown to indicate the presence of daylight in an outdoor scene. Other cues, such as the presence of faces, may be used similarly if they prove to have a meaningful relationship with the AWB WP estimation accuracy. The adaptive threshold may be adjusted to achieve more or less aggressive AWB stabilization behavior with a higher or lower probability of WP lock. The algorithm has lightweight computation and memory tracking, which makes it attractive for software or firmware implementation. Stabilization does not rely on dedicated hardware for pixel processing such as motion estimation that consumes power. The algorithm is easy to implement software integration because it does not require time-consuming adjustment procedures and has minimal intra-component dependencies. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The present disclosure will be further explained through the following description and accompanying drawings.

[0045] Figure 1A diagram showing the location of the AWB stabilization algorithm relative to other related components from the SW stack and ISP pipeline.

[0046] Figure 2 A diagram showing the AWB stabilization algorithm.

[0047] Figure 3 A diagram showing the AWB stabilization algorithm.

[0048] Figure 4 Shows a video frame (top left), AWB statistics of a video frame with saturated blocks marked in red (top right), and a histogram of AWB statistics in chrominance space (lower left), and examples of AWB statistics histograms in logarithmic chroma space (log(R)-log(G), log(B)-log(G)) (lower right).

[0049] Figure 5 An input-output mapping between illumination levels and threshold drive signals based on the illumination levels is shown.

[0050] Figure 6 An input-output mapping between a threshold drive signal and a threshold value is shown.

[0051] Figure 7 A test scenario is shown where the AWB results consistently fail with input AWB WP, its smoothed version, and its stabilized version. DETAILED DESCRIPTION

[0052] The present disclosure will be described with respect to specific embodiments and with reference to specific drawings, but the present disclosure is not limited thereto but only by the claims. The drawings described are only schematic and non-restrictive. In the drawings, for illustrative purposes, the size of some elements may be exaggerated and not drawn to scale. The sizes and relative sizes do not necessarily correspond to the actual implementation of the present disclosure.

[0053] In addition, the terms "first", "second", "third", etc. in the specification and claims are used to distinguish between similar elements, and are not necessarily used to describe a sequential or chronological order. These terms are interchangeable under appropriate circumstances, and the embodiments of the present disclosure can be operated in other sequences than those described or illustrated herein.

[0054] Additionally, the terms "top," "bottom," "above," "below," etc., in the description and claims are used for descriptive purposes. The terms so used are interchangeable under appropriate circumstances, and the embodiments of the disclosure described herein can operate in other orientations than described or illustrated herein.

[0055] In addition, various embodiments, although referred to as “preferred” or “optional”, should be interpreted as exemplary ways to implement the present disclosure rather than limiting the scope of the present disclosure.

[0056] Figure 1 A diagram showing the location of an AWB stabilization algorithm 100 relative to other related components from a software (SW) stack 102 and an image signal processor (ISP) pipeline 104 is shown.

[0057] A sensor (e.g., camera) 106 provides input to the SW layer 102 and the ISP pipeline 104. The ISP pipeline 104 prepares AWB statistics 108. The AWB statistics used for the AWB stabilization algorithm 100 have exactly the same requirements as any advanced AWB algorithm. That is, the image pixel data used to prepare the local R, G, and B channel averages or sums must be corrected for black level, linearity, and lens shading. Otherwise, the distribution of data points in the chromaticity / logarithmic chromaticity space will show color information that is not present in the captured scene, and / or the distribution may be distorted. This can mislead the AWB and AWB stabilization algorithms. AWB stabilization assumes that the AWB statistics format defined for the AWB algorithm can also be applied to the AWB stabilization algorithm, and the same SW architecture can be reused.

[0058] There is a connection between the AWB algorithm 110 and the AWB stabilization 100. In particular, the AWB results need to be fed to the AWB stabilization algorithm 100. More specifically, the AWB WP 112 is sent together with the AWB WP confidence 114 (if available).

[0059] The AWB stabilization 100 also requires an illumination level input 116. The illumination level input 116 is provided by the camera sensor SW adaptation layer 102. The illumination level input 116 may be obtained from measurements or estimates based on average image brightness and camera sensor settings for exposure, analog gain and sensitivity. Most advanced AWB algorithms utilize such illumination level estimates, so some sensor adaptation SW layers are likely already available. The same illumination level value must also be fed to the AWB stabilization algorithm.

[0060] The AWB stabilization result, i.e., the WP information 118 (white point or white balance WB gain), needs to be fed to the white balance block 120 in the ISP pipeline 104, rather than the AWB result from the AWB algorithm 110. Example conversions between WP and WB gains are found in Jarno Nikkanen, Color constancy by characterization of illumination chromaticity, Optical Engineering, Vol. 50(5), 2011, p. 057204-1-14, and can be done in the SW layer 102.

[0061] For overall synchronization, the AWB stabilization algorithm must be fed with AWB statistics that are the same as the AWB statistics used to generate the input WP of the AWB stabilization algorithm by AWB stabilization. The AWB stabilization according to the present disclosure is an algorithm at low computational cost and does not introduce significant delays to those algorithms already existing in the system, so there is no synchronization problem.

[0062] Figure 2 The inputs and outputs of the AWB stabilization algorithm 100 are shown. The first input is the AWB statistics 108, which are also fed into the AWB process 110. The AWB statistics 108 are the local summation or average of n×n image pixels. The AWB statistics are used to construct a histogram in the logarithmic chrominance domain, which is needed to analyze the color content of the current frame in a variety of ways. A larger statistics block size n improves the estimated color accuracy because noise is filtered out. However, larger blocks are more likely to contain edges between two objects of different colors, and the averaging of pixels within the block results in the generation of artificial "transition" colors that do not exist in the scene. Therefore, a larger block size n will make the entire transition color portion larger.

[0063] The stabilization algorithm 100 expects to obtain the WP estimate (i.e., input WP) from the conventional AWB algorithm 110. No smoothing or filtering of the output of the AWB algorithm 110 is required. Ideally, the AWB stabilization 100 should be the only component responsible for smoothing the WB gain. The accuracy of the AWB WP also largely determines the accuracy of the AWB stabilization algorithm. If the AWB is incorrect from the beginning, the AWB cannot be fixed by the AWB stabilization 100.

[0064] The illumination level is well correlated to the performance of all modern AWB algorithms. The illumination level 116 is used to determine whether the scene is imaged during daylight, during which the confidence in the WP estimate can be greater since the CCT search range may be narrower. In addition, significant illumination level changes indicate light source changes that lead to WP unlocking. Illuminance can also be used to detect ultra-low light (ULL) scenes, where AWB stabilization can be disabled. Ideally, the illumination is the average illumination level measured at the imaged scene. Since accurate measurement of HW is not available for mobile imaging, a calculated estimate is also sufficient. An example method for estimating the illumination level from a scene can be found in "Color constancy by characterization of illumination chromaticity" by Jarno Nikanen, Optical Engineering, volume 50(5), page 057204-1-14, 2011.

[0065] Some AWB algorithms 110 may generate a confidence metric for the estimated WP. The AWB stabilization algorithm 100 does not necessarily require a WP confidence estimate, however, if available and relatively accurate, a WP confidence estimate will improve algorithm performance.

[0066] Optionally, face detection information 138 may also be provided to the AWB stabilization 100. Some AWB algorithms 110 greatly improve the accuracy of WP with the help of skin color information of detected faces from the acquired images. Therefore, the confidence in the WP estimate 112 will only be increased to a certain extent if the AWB algorithm 110 is known to be very reliable in the presence of faces.

[0067] In addition to the standard input list, the AWB stabilization algorithm 100 requires a set of fixed parameters and a set of tuning / configuration parameters 120. The adjustable parameters define whether the algorithm locks onto the final reliable WP more easily and more often, or the adjustable parameters give higher trust to the AWB algorithm and lock onto the WP more occasionally. The tuning parameters are mainly related to the analysis of the scene color content and the reliability of the WP. Another set of parameters is related to the sensor configuration such as the frame rate and resolution of the incoming AWB statistics.

[0068] The format of the output of the AWB stabilization 100 is the same as the output of the AWB algorithm 110 itself, for example WP in terms of WB gain or as a WB gain ratio, and depends on the existing SW / HW architecture.

[0069] refer to Figure 4100 is described in more detail. From a top-level perspective, the AWB statistics 108 are used to construct a histogram 122 in a logarithmic chromaticity space by a histogram calculation module 124. Two WP confidence metrics (i.e., histogram region confidence 128 and histogram entropy confidence 130) are derived from the histogram 122, and these confidence metrics can be combined with externally available and internally calculated AWB confidence metrics. Some or all of the available metrics are used by a confidence estimation block 134 to create an overall confidence 132. The histogram is also used to evaluate the (recent history) scene color content in a color detection block 136. A threshold generation block 142 uses the scene color content together with the illumination level estimate 116 to determine a value 140 for thresholding the overall confidence 132 within an adjustable range. Optional face detection information can also be used by the threshold generation block 142. The total confidence 132 passes through an adaptive threshold 144 to generate a flag 146 when the WP should be locked / unlocked to the current smoothed WP. Ideally, the AWB WP is also smoothed by a smoothing block 150 to provide a smoothed WP 152. Based on the flag 146 and the recent trend of the smoothed WP, ​​a WP decision block 148 decides whether to output the smoothed WP, ​​the last locked value of the WP, or a weighted average between the smoothed WP and the locked WP. When the above process results in a change in the locked / unlocked state, the transition between the locked WP and the smoothed WP occurs in a flexible and gradual manner, avoiding any jumps in the final stabilized WP 154. The various blocks of the AWB stabilization method 100 will be described in more detail below.

[0070] Typically, the AWB statistics 108 include a local summation and / or average of the frame pixel data in each nxn pixel block for each of the R, Gr, Gb, and B color channels. Figure 4 The top row of shows video frames (left) and their AWB statistics (right). The values ​​in the local sum / average depend strongly on the frame brightness, i.e. on the sensor exposure and gain settings. A simple way to remove the dependency on image brightness is to obtain and These two ratios, where G = 0.5 (Gr + Gb). Figure 4The lower left portion of shows a 2D histogram of AWB statistics in the chromaticity space (R / G, B / G). The histogram is formed by counting the number of (R / G, B / G) points from the AWB statistics 108 that fall into each bin from a set of predetermined bins. Each (R / G, B / G) point is obtained from each AWB statistics grid block, where only grid blocks containing saturated pixels are excluded. It should be noted that when the (R / G, B / G) values ​​of objects of different colors are below 1, the peaks of objects of different colors (such as plastic bags, T-shirts, jackets, and floors) are quite close to each other. This makes separation between these colors quite difficult, complicating the evaluation of color content. Reducing the size of the histogram bins may show better separation, however, this makes the stability of the frame-to-frame histogram binning worse and increases the total bin count, i.e., the computational load. This problem can be solved using a logarithmic function, which is known to expand the separation of values ​​less than 1. Therefore, the AWB stabilization algorithm 100 optionally uses a logarithmic chromaticity space Figure 4 The lower right portion of shows a histogram in logarithmic chromaticity space, where a clear separation can be seen between the colors of the bag, T-shirt, jacket, and floor. This logarithmic chromaticity space histogram is constructed in the same way as the chromaticity histogram. Each AWB statistics grid block is used to calculate a point by taking (log(R)-log(G),log(B)-log(G)) and placing all points into a predetermined set of intervals.

[0071] It should be noted that the histograms have the same overall resolution in color space, i.e. the histograms have the same histogram bin counts over nearly the same range. The range limits in logarithmic chromaticity space are obtained by taking the logarithm of the range limits in chromaticity space. The range and bin counts of the histograms depend on the range of the AWB statistics 108, i.e. on the camera sensor and ISP settings. The bin counts are chosen such that a high enough resolution is provided to allow separation between close colors, and any small frame-to-frame variations in the acquired frame sequence do not create instabilities in the histogram.

[0072] The total confidence metric 132 is a weighted average of some or all possible WP confidence metrics available. The weights for the averages should be positive, and the sum of these weights should ideally be 1. The possible AWB stabilization algorithm 100 uses equal weights for all WP confidence metrics. However, it should be noted that if one metric is significantly more reliable than the others, the weights for the disproportionate metrics are optional, but the sum of the weights should ideally remain at 1. In this case, extensive experimentation should confirm that one metric is much more reliable than the rest, and this is true in most practical cases where it is important.

[0073] When implementing the two basic WP confidence metrics 128, 130, a logarithmic chromaticity histogram 112 has been used. The histogram 122 shows the actual color content of the acquired frame and its distribution in the logarithmic chromaticity space. For example, according to the gray world (GW) hypothesis (for more information, see Edwin land, John McCann, "Lightness and Retinex Theory" published in Journal of the Optical Society of America, Vol. 61(1), pp. 1-11, 1971), if more kinds of colors constitute the scene, the light source estimation is more reliable. Many modern practical AWB algorithms are based on the gray search method described in "Color constancy by characterization of illumination chromaticity" by Jarno Nikanen, Optical Engineering, Vol. 50(5), pp. 057204-1-14, 2011, which gives higher weights to points below the characterization white map. This phenomenon, which is related to the distribution of scene colors in chromaticity space, or equivalently in logarithmic chromaticity space, can be measured by analyzing histograms.

[0074] The first confidence metric is the logarithmic chromaticity histogram area 128. This is the intrinsically non-zero area under the histogram bins. Essentially non-zero bins are those with bin counts greater than 2 points per bin, so that transition colors and noise do not mislead a large number of valid bins. The number of intrinsically non-zero bins is then multiplied by the area of ​​a single bin and divided by the maximum possible area. The maximum possible area is the area of ​​the range over which the histogram is constructed. The area may be further multiplied by a small normalization factor that depends on the current statistical configuration. The histogram area provides an estimate of the total spread of colors in the scene without providing an indication of which colors are more dominant than others. This is a relevant WP confidence measure, in particular for AWB algorithms that are not influenced by the dominance of a few colors but instead provide importance to all colors110, for example, the gray-edge assumption described in Joost van de Weijer, Theo Gevers, "Color constant based on the grey-edge deposition", IEEE International Conference on Image Processing 2005, Genoa, vol. 2, p. II-722.

[0075] The second confidence metric is the logarithmic chromaticity histogram entropy 130. A WP confidence metric is also needed for the AWB algorithm 110, which is prone to errors when the color distribution is not uniform, that is, when there is a dominant color. If the histogram bin count is divided by the total number of unsaturated blocks in the AWB statistical grid, the histogram acquires the meaning of a discrete probability distribution. Taking the entropy of this distribution yields another WP confidence metric:

[0076]

[0077] Here h i is the normalized bin count for the ith histogram bin, and the histogram size is N × N bins. Division by the normalization factor 2logN ensures that the entropy confidence remains in the interval [0,1].

[0078] Ideally, the minimum of the metrics used in AWB stabilization should be a combination of histogram area and log chrominance histogram entropy. However, other confidence metrics may be used. It will be readily appreciated that any of the following additional confidence metrics are purely exemplary and should not be construed as limiting the scope of protection of the appended claims.

[0079] One such confidence metric may be the internally calculated AWB confidence 126. Specifically, the computationally least expensive and reliable WP metric may be internally used in the AWB confidence.

[0080] Another such confidence metric may be the AWB algorithm confidence 114. In the event that the AWB algorithm 110 provides such a confidence metric, it is highly recommended to use this information since the accuracy of the AWB algorithm 110 confidence has a significant impact on the AWB stabilization parameter adjustments and metric weightings.

[0081] In addition to the WP confidence metrics 128, 130, the logarithmic chromaticity histogram 112 is used to extract and separate the dominant colors in the scene. This information is used to determine the current value of the adaptive threshold 140 for the total confidence 132 that determines the lock / unlock of the WP. The dominant color in the collected scene is a large spatial area with the same chromaticity value, or equivalently, a large area with very similar values ​​in the AWB statistical grid. Therefore, the dominant color forms a peak in the chromaticity / logarithmic chromaticity space histogram. In one embodiment, the dominant color in the scene is determined by detecting a large and well-separable peak in the logarithmic chromaticity histogram. In addition, the detected histogram peaks should be well separated because a pair of close peaks can indicate different shadows on the same object, or two different objects with similar colors. In either case, this can change significantly from one frame to another, which will produce unnatural uncertainty in color content in the scene.

[0082] In one embodiment, peaks are detected by running a 5×5 moving window through the histogram bins. A peak is detected if the bin count of the center bin of the 5×5 window has a larger bin count than all other bins falling within the window. A peak is not detected unless the cumulative bin count of all bins exceeds a certain minimum amount of chroma points within the limits of the 5×5 window that falls within the bin. The purpose of this threshold conversion is to ignore small peaks caused by object disappearance, object reflection, object occlusion, transitions between objects, and other types of distortion. This minimum amount of chroma points depends on the resolution of the AWB statistics grid. For larger resolutions, the number needs to increase proportionally because there will be more chroma points per square area of ​​the input image, and therefore more chroma points will participate in transition phenomena that cause instability. In addition, a 5×5 moving window ensures a minimum peak-to-peak distance of 3 bin sizes.

[0083] Furthermore, this method for dominant color detection is computationally inexpensive. A typical size of N=100 for the logarithmic chromaticity histogram size will result in a total of N×N=100×100=10,000 histogram bins. However, the count of non-zero bins is typically much less than 10,000, typically on the order of a few hundred, depending on the color content of the scene. Therefore, an efficient implementation of dominant color detection keeps track of the locations of the non-zero bins for each frame, and only tracks these locations as a 5×5 window slides across the histogram.

[0084] The following is a pseudo code describing a dominant color detection algorithm according to an embodiment of the present disclosure.

[0085]

[0086] The AWB stabilization algorithm 100 obtains the signals for locking and unlocking WP based on thresholding the total WP confidence 132. The confidence measure can sometimes be wrong because it is a result of the estimation process itself. For example, a difficult scene consisting of a single color may give a high WP confidence if most of the chromaticity points fall within some gray search area. However, this does not mean that the object needs to be achromatic, and both the WP confidence and the WP estimate can be fooled by the scene. In this case, it is better to increase the threshold level for the lock / unlock decision. There are other situations where the scene is captured in bright daylight and / or the color changes are large, and almost any AWB algorithm will succeed even if the confidence is incorrectly estimated to be low for some reason. In this case, it is better to lower the threshold for the lock / unlock decision.

[0087] In one embodiment, the threshold is determined based on a threshold drive signal. The drive signal optionally includes all available and known clues for the reliability of the scene. The clues used are the illumination level 116 and the (short history of) scene color changes 136.

[0088] The WP estimation in outdoor scenes captured in bright daylight is very reliable because most practical AWB algorithms 110 narrow the possible CCT range for bright daylight. This means that for illumination levels above 5000 lux, the adaptive threshold may take a smaller value, or alternatively, the drive signal for the adaptive threshold may be set to 1 to represent an easy scene. The available illumination levels are the result of an estimate, not a measurement, and are therefore not very precise. Illumination levels below 3000 lux may result from outdoor scenes or indoor scenes in forecasted weather and / or dim conditions, i.e., from virtually any light source. Therefore, the CCT range cannot be restricted and the WP confidence threshold must be set high, or the drive signal for the WP confidence threshold may be set to 0. Increasing illumination levels between 3000 lux and 5000 lux, the threshold should be gradually lowered, or the drive signal for the threshold should be gradually changed from 0 to 1. This behavior is shown in Figure 1. Figure 5 Shown in.

[0089] The threshold drive signal is determined more by scene color variations than by illumination levels. Therefore, it makes sense to define the color content drive signal in the range [-1,1] rather than in the range [0,1]. This way, when illumination levels are uncertain in situations caused by outdoor or indoor scenes, poor color content can still pull down the threshold drive signal.

[0090] The color content in a scene can be measured by the number of dominant colors present in the scene. Since this can still produce frame-to-frame jumps, the drive signal is ideally based on the trend of the color content rather than on the dominant color count itself. The trend of the color content is measured by the total number of colors over the past 1 second. In one embodiment, a base level of 4 dominant colors is selected, which represents that the color content of the scene is neither rich nor poor. Therefore, these 4 colors totaled over the past 1 second produce a value of 0 in the threshold drive signal. If the scene consists of its minimum value, that is, only 1 dominant color for the past 1 second, the drive signal value should be close to -1. If the scene contains more than BaseLevel=4 colors on average over the past 1 second, the drive signal can grow without an upper limit, thus always masking low light conditions. This is actually achieved by the following equation:

[0091]

[0092] The normalization factor of normalization level = 100 is chosen so that for 30 frames in the past 1 second, the minimum value of one color may result in a value close to -1. It should be noted that when the frame rate changes, the normalization level parameter, ie, normalization level, needs to be reset accordingly.

[0093] The final threshold drive signal is the sum of the color content and the luminance level components D = D 颜色 +D 照度水平 . use Figure 6 The mapping depicted, the drive signal D is used to smoothly set the threshold within the desired limits. That is, drive signal levels below -0.6 indicate that the scene is neither trusted by color counts nor by illumination levels, and the WP confidence threshold should be set at its maximum level. D values ​​above 1 mean that the scene is rich in color content, or the illumination level corresponds to a bright outdoor scene, or both, and the threshold is reduced to its minimum value. Every value of D between -0.6 and 1 is linearly mapped to a value between the threshold minimum and maximum values ​​of the threshold. The minimum and maximum values ​​of the mapping depend on the WP confidence and are set based on a large number of experiments. Therefore, repeated experiments are needed to change the overall WP confidence and / or the formation of the threshold drive signal.

[0094] Smoothing the WP means following the trend of the WP variation without any frame-to-frame fluctuations and jumps. The smooth trend can be followed quite effectively by temporally averaging each WP component separately. Depending on the WP format, the WP components can be, for example, R channel, G channel and B channel, chrominance ratios R / G and B / G, etc. Unfortunately, sudden jumps in the input signal can mislead the temporal averaging. In one embodiment, an alpha-trimmed mean filter (J. Bee Bednar, Terry Watt, "Alpha-trimmed means and their relationship to median filter", IEEE Transactions on Acoustics, Speech, and Signal Processing, February 1984, 32(1), pp. 145-153) is used to temporally smooth each component of the WP, which discards the alpha part of the minimum and maximum values ​​before averaging. This means that for each WP component, a buffer is formed of all WP component values ​​from the past 1 second, and 11% of these values ​​- the smallest 5.5% values ​​and the largest 5.5% values ​​- are discarded. The remaining 89% of the WP component values ​​are averaged to obtain a smoothed WP component value for the current frame. 5.5% of the buffered WP component values ​​may result in a fraction, depending on the frames per second (FPS) rate. This number is then rounded to the nearest integer, i.e., the largest round (FPS×5.5 / 100) values ​​and the smallest round (FPS×5.5 / 100) values ​​are discarded before averaging. The value of 5.5% was chosen so that at least one maximum value and one minimum value are discarded from each WP component buffer, even for the minimum rate of 10 to 15 FPS, where there are some frame rate fluctuations and / or occasional frame drops.

[0095] When the camera is pointed at a difficult scene for the AWB algorithm 110, the WP value moves in the wrong direction. No matter how the smoothing filter advances, the smoothed WP 152 will converge to the wrong direction sooner or later. Instead of simple smoothing, the present disclosure locks the WP 154 before the smoothing filter 150 starts to converge towards the wrong WP, and releases the WP 154 when the scene becomes reliable again.

[0096] When the scene is reliable and the AWB 110 is confident of this, the output of the stabilized WP 154 follows the smoothed WP 152. In each frame, the total WP confidence 132 is compared with the adaptive threshold 140. Once the confidence 132 drops below the threshold 140, a flag 146 is generated to indicate that the WP should be locked. This means that the output of the stabilized WP 154 no longer follows the value of the smoothed WP 152, but keeps outputting the same value until the decision to unlock is made.

[0097] The decision to unlock depends on the actual situation of the smoothed WP 152. Once the confidence 132 returns to above the threshold 140, a flag 146 is generated that the output stabilized WP 154 is ready to unlock. In addition, if the smoothed WP 152 and the last output WP are almost equal, the output WP 154 starts to follow the smoothed WP again. However, this situation is very rare, and when the smoothed WP 152 and the output WP 154 are not close to each other, switching the output WP directly to the smoothed WP will result in a jump in the WP value. The jump in WP results in a jump in color deviation, which is a very poor user experience. Therefore, in this case, within a conversion period of 1 second, the output WP 154 changes smoothly towards the smoothed WP 152. This occurs frame by frame within the conversion period, where for each frame f, the output WP is a convex combination (i.e., a weighted average) between the output WP and the smoothed WP of the past frame f-1. The weight of the convex combination depends on how long ago the flag for unlocking the WP was generated:

[0098] Stable WP f =γ×smooth WP f +(1-γ)×Stable WP f-1 ,

[0099] γ = (timestamp (f) - unlock mark timestamp) / (1 second).

[0100] In addition to the smooth transition from the locked WP to the smoothed WP 152, there is another problematic situation. Assume that the light conditions of the scene remain unchanged and the user slowly pans away from the problematic scene. Further imagine that the panning occurs slowly enough so that the smoothed WP will converge back to the correct WP of the light source in a longer time than the 1 second transition period. In this case, the output WP will move in the wrong direction toward the smoothed WP, ​​and after the transition is over, the output WP will move back toward the true WP of the light source together with the smoothed WP. This bump in the value of the output WP will be considered an unnecessary and erroneous color cast change in the image frame sequence, even if the change occurs smoothly. Therefore, in the case where the user pans away from the difficult scene and the WP unlock flag is generated, the output WP can remain the same, "soft locked", as long as the smoothed WP moves toward the locked value of the output WP. In the case where the smoothed WP value begins to move away from the current value of the output WP, a 1 second smooth transition can be used. In this case, the transition will begin when the smoothed WP and the output WP are as close to each other as possible. The entire process is called the fallback WP expectation method.

[0101] The trend of the smoothed WP can be monitored by analyzing its derivative. Small values ​​of the derivative correspond to small temporal fluctuations of the smoothed WP or a very slow trend. In either of these cases, the switch towards a smoothed WP is optional. The derivative is estimated from the last few frame values ​​of the smoothed WP by standard convolution with a causal filter, with temporal linear filtering:

[0102]

[0103] In one embodiment, a 5-tap (K=4) derivative filter formed by a finite difference approximation of the first-order derivative based on a truncated Taylor series expansion is used. If AWB stabilization 100 has recently been (re)started and the smoothed WP has less than 5 samples, a shorter derivative approximation is used. The impulse response of the filter may be:

[0104] Derivative filter = [25, -48, 36, -16, 3] / 12, for K = 4,

[0105] Derivative filter = [11, -18, 9, -2] / 6, for K = 3,

[0106] Derivative filter = [3, -4, 1] / 2, for K = 2,

[0107] Derivative filter = [1, -1], for K = 1.

[0108] The output WP decision algorithm 148 is illustrated by the following pseudo code.

[0109]

[0110] It should be noted that after the unlock flag 146 is generated, one component of the smoothed WP 152 may converge toward the corresponding component of the output WP, while the other component may be diverging. Therefore, it is best to implement the fallback anticipation and smooth transition decision for each WP component separately. Otherwise, for example, if the smooth transition decision is triggered by the first WP component that starts to diverge, the remaining components of the output WP will move back and forth in the wrong WP direction. This will cause unnecessary bulges.

[0111] It is easy to understand that the above-mentioned parameters and time periods (e.g., normalization level, base level, 1 second time period, weight, etc.) can be set to different values ​​and / or can be determined by testing the method on various data sets. In addition, these values ​​can also be different for different AWB processes.

[0112] Figure 7 Test results of the above method and system for temporally stabilizing WP information during AWB are shown. A typical test runs the AWB stabilization algorithm 100 offline on a sequence of image frames acquired by a mobile phone in a real-world test scenario. The data associated with each frame are AWB statistics, AWB WP, illumination level estimates, a distribution map with indices of AWB statistics grid blocks containing saturated pixels and timestamps corresponding to the time when AWB was performed. The test equipment is based on a mobile platform including an ISP statistics pipeline and an integrated AWB algorithm.

[0113] Figure 7 Shown is a sequence of test frames recorded while capturing a video of a scene in which overall AWB fails consistently. This scenario occurs under office environment conditions, where a person is sitting on a chair in front of a solid orange curtain as a background. First, the camera is pointed far away, the camera field of view (FoV) includes multiple objects of various colors, and the WP is estimated correctly. Next, the camera moves closer to the person, focusing on the face, the WP starts to move slightly in the wrong direction, and then the camera quickly pans away from the face and onto the orange curtains, where the estimated WP is completely wrong. Finally, the camera moves back to the person's face and continues to move further back to get a colorful scene within the FoV. As a result, the estimated WP moves from the incorrectly estimated WP for the curtains back to the correct value. This behavior is shown in Figure 1. Figure 7 The thin red and blue lines for the ratio R / G and the ratio B / G for WP are depicted in Fig. 2 . The dashed line shows the smoothed version of the integrated AWBWP obtained by the alpha clipped mean filter. The thick line shows the WP after the output is stabilized, and its labels are located at the positions of WP locked and WP unlocked. It should be noted that the R / G channel and the B / G channel unlock the WP at different frame numbers according to the component-by-component return expected method described above.

[0114] Although aspects of the disclosure have been described with reference to specific embodiments, it will be readily appreciated that these aspects may be embodied in other forms within the scope of the disclosure as defined by the claims.

Claims

1. A method for stabilizing white point WP information in time, the method comprising: a) receiving illumination level information, automatic white balance (AWB) statistics, and input WP information, wherein the AWB statistics include image pixel data from a frame of a sequence of frames representing a scene, and the input WP information is obtained from an AWB process; b) generating a confidence value for the input WP information; c) determining a threshold based on the dominant color count of the frame and the illumination level information; d) when the confidence value is lower than the threshold, setting the binary WP update state to a locked state; when the confidence value is higher than the threshold, setting the binary WP update state to an unlocked state; and e) when the binary WP update state is locked, generating final WP information according to WP information obtained from a previous frame of the frame sequence; when the binary WP update state is unlocked, generating final WP information according to the input WP information.

2. The method according to claim 1, wherein: Step b) comprises: generating a histogram of the image pixel data, the confidence value is generated according to the histogram, and the histogram is a two-dimensional histogram generated in a logarithmic chromaticity space.

3. The method according to claim 2, wherein: Step b) comprises: - determining a first confidence measure, comprising: - counting the number of histogram bins having at least two points to determine the number of non-zero bins; - multiplying the number of non-zero bins by the bin area to determine the histogram area; and - dividing the histogram area by a maximum area to determine the first confidence measure, wherein the maximum area is the product of the total number of bins and the area of ​​the bins; and - determining a second confidence measure, comprising: - dividing the count of the histogram bins by the total number of unsaturated blocks in the AWB statistics to determine a discrete probability distribution; and - calculating the entropy of said discrete probability distribution to determine said second confidence measure.

4. The method according to claim 3, wherein: Step b) comprises: - determining the confidence value by calculating a weighted average of the first confidence measure and the second confidence measure; or - determining one or more additional confidence metrics and determining the confidence value by calculating a weighted average of the first confidence metric, the second confidence metric and the one or more additional confidence metrics.

5. A method according to any one of the preceding claims, wherein: Step c) comprises determining the dominant color count of the frame by counting the number of peaks in a histogram generated from the image pixel data.

6. A method according to any one of the preceding claims, wherein: Step c) comprises: - Converting the illumination level information into a first probability value between 0 and 1, comprising: - for illumination level information of which the illumination level is lower than 3000 lux, setting the first probability value to 0; - for illumination level information with an illumination level higher than 5000 lux, setting the first probability value to 1; and - setting the first probability value to a value between 0 and 1 by performing linear mapping on illumination level information between illuminance levels of 3000 and 5000 lux; and Based on the main color count, the second probability value D is calculated using the following equation 颜色 : Wherein, the normalized level and the basic level are predetermined natural numbers, the dominant color count is the dominant color count of the scene at frame f; and the time interval is a predetermined time interval.

7. The method according to claim 6, wherein: Step c) comprises: - determining a threshold drive signal by combining the first probability value and the second probability value, and determining the threshold value based on the threshold drive signal; or -determining one or more additional probability values ​​based on the presence or absence of a face in the scene, determining a threshold drive signal by combining the first probability value, the second probability value and the one or more additional probability values, and determining the threshold based on the threshold drive signal.

8. A method according to any one of the preceding claims, wherein: The method further comprises: smoothing the input WP information using an alpha clipped mean filter to determine smoothed input WP information, and using the smoothed input WP information in step e).

9. A method according to any one of the preceding claims, wherein: When the binary WP update status is unlocked, step e) comprises: e1) when the binary WP update state is unlocked during a predetermined number of frames before the frame in the frame sequence, setting the final WP information equal to the input WP information or equal to the smoothed input WP information; or e2) generating the final WP information as a weighted average of the input WP information or the smoothed input WP information and the WP information obtained from the previous frame, when the binary WP update state changes from locked to unlocked during a frame among the predetermined number of frames before the frame in the frame sequence.

10. The method according to claim 9, wherein: Step e2) comprises: - estimating a derivative of the input WP information or the smoothed input WP information over a predetermined number of frames preceding the frame in the sequence of frames; - determining whether the input WP information or the smoothed input WP information does not deviate from the WP information obtained from the previous frame based on the derivative, the input WP information or the smoothed input WP information, and the WP information obtained from the previous frame; - if the input WP information or the smoothed input WP information does not deviate from the WP information obtained from the previous frame, setting the final WP information equal to the WP information obtained from the previous frame; and - if the input WP information or the smoothed input WP information deviates from the WP information obtained from the previous frame, generating the final WP information as a weighted average of the input WP information or the smoothed input WP information and the WP information obtained from the previous frame.

11. The method according to claim 9 or 10, wherein: Step e2) comprises generating the final WP information as a weighted average by smoothly changing the final WP information from the WP information obtained from the previous frame to the input WP information or the smoothed input WP information during a predetermined switching period of a frame following the frame in the frame sequence.

12. A method according to any one of the preceding claims, wherein: The WP information includes a white point WP or a white balance WB gain.

13. A device for temporally stabilizing white point information, comprising means for performing the method according to any one of claims 1 to 12.

14. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to perform the method according to any one of claims 1 to 12.

15. A storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 12.

Citation Information

Patent Citations

  • Dynamic frame skip for auto white balance

    CN107710755A

  • White balance gain correction method, device thereof and system and computer device

    CN111641819A

  • Auto white balance control algorithm based upon flicker frequency detection

    US20180070068A1

  • Estimation of illumination chromaticity in automatic white balancing

    US20190045161A1

  • Systems and methods for automatic white balance

    US20190253682A1