Automatic white balance (AWB) for camera systems

Abstract

This document describes techniques and apparatus for automatic white balancing for a camera system. The techniques and apparatus utilize a precursor image to detect one or more detected faces and determine color shades. The camera system retrieves color shade data based on a group of images determined to contain the same face as the detected face. Based on the color shade data, a white balance difference is determined based on the color shade difference of the detected faces in the precursor image and the associated color shade data. Camera settings are adjusted based on the white balance difference to enable capture of an image with improved color shades.

Description

Background Art

[0001] Background Camera systems generally have many functions that can be used to adjust the characteristics of an image or video (e.g., color, sharpness, magnification). One such function is white balance (WB), a camera setting that adjusts the coloration of colors in an image. WB is typically used to color white objects that appear white in the image. To assist the user, many cameras include an automatic white balance function (AWB) that automatically adjusts WB according to a color-correction algorithm. However, conventional camera systems may not be able to achieve the desired accuracy, especially in the most important parts of an image, such as a face. Additionally, when multiple parts of an image are identified as important, additional challenges arise. As a result, the image may present WB problems, especially in important parts of the image.

Summary of the Invention

Means for Solving the Problems

[0002] Summary This document describes techniques and apparatus for automatic white balance for a camera system. The techniques and apparatus utilize a precursor image to detect one or more detected faces and determine the coloration. The camera system extracts coloration data based on a group of images determined to include the same face as the detected face. Based on this coloration data, a white balance difference is determined based on the coloration differences of the detected faces within the precursor image and the associated coloration data. To enable the capture of images with improved coloration, the camera settings are adjusted based on the white balance difference. It is possible to improve the white balance adjustment and image coloration, especially in the most important parts of the image, such as a face.

[0003] Furthermore, additional faces can be detected within the preceding image, and these can be used through the disclosed steps to determine adjustments that enable the capture of an image with improved color saturation. These adjustments can be weighted based on priority values ​​to determine which adjustments improve the most important parts of the image. Alternatively, it is also possible to capture multiple images using individual adjustments to the camera settings. These images can be provided to an image merging module that combines parts of the individual images to produce a single image with improved color saturation.

[0004] The above summary is provided to introduce a simplified concept of the technology and apparatus for AWB for camera systems, which will be further explained in the detailed description and drawings below. The above summary is not intended to be used to identify the essential features of the claimed subject matter or to determine the scope of the claimed subject matter.

[0005] The following describes in detail one or more embodiments of AWB for camera systems. The use of the same reference numbers in different examples in the description and figures indicates similar elements. [Brief explanation of the drawing]

[0006] [Figure 1] This figure shows an exemplary operating environment for AWB for a camera system. [Figure 2] This figure shows an exemplary embodiment of an AWB module that works in conjunction with the Frequent Face Module. [Figure 3] This figure shows a detailed example of the exemplary operating environment shown in Figure 1. [Figure 4] This figure shows an exemplary embodiment of capturing a precursor image within the context of AWB for a camera system. [Figure 5] Figure 4 shows an exemplary embodiment for detecting a face in a precursor image. [Figure 6]This figure shows an exemplary embodiment of color determination by a color determination module. [Figure 7] This figure shows an example of a set of matching images that were determined to contain the same face as the first detected face in Figure 5. [Figure 8] This figure shows an example of a set of matching images that were determined to contain the same face as the additionally detected face in Figure 5. [Figure 9] This figure shows an exemplary embodiment for determining color tone from a group of matching images in Figure 7. [Figure 10] This figure shows an example of color data from a group of matching images in Figure 7. [Figure 11] This figure shows an example of another set of matching images that were determined to contain the same face as the first detected face in Figure 5. [Figure 12] This figure shows an example of color data from a group of matching images in Figure 11. [Figure 13] This figure shows an exemplary embodiment of AWB for a camera system in an electronic device. [Figure 14] This figure shows another exemplary embodiment of AWB for a camera system in an electronic device. [Figure 15] Figure 14 shows an exemplary embodiment of an image merging module for an exemplary embodiment of AWB. [Figure 16] This figure shows an exemplary method of AWB for a camera system. [Figure 17] This figure shows another exemplary method of AWB for a camera system. [Figure 18] This figure shows another exemplary method of AWB for a camera system. [Modes for carrying out the invention]

[0007] Detailed explanation Overview This document describes a technique and apparatus for automatic white balance (AWB) for camera systems. This technique uses a preceding image of the scene to detect a human face. Next, the hue of the detected face is determined and compared to hue data associated with the detected face. The hue data is determined from a set of previously captured images, which are determined to have been captured under preceding ambient conditions and to contain the same face as the detected face. The hue data can describe a preceding captured image, a group of preceding captured images, or a portion of a preceding captured image. By comparing the hue data with the hue of the detected face, AWB differences are determined, and these differences are used to adjust camera settings so that, for example, a camera setting is adjusted to allow the capture of an image with improved hue when compared to an equivalent image captured without adjusting camera settings. The improved hue may match the first hue data more closely than the hue of the detected face. The hue improvement may correspond, among other things, to the most important parts of the image, such as the face.

[0008] Additional faces can be detected in the preceding image, and these can be used via the above technique to determine additional AWB differences. AWB differences can be combined via weighted summation, filtering, or any other computational method to adjust camera settings that enable the capture of an image with improved color saturation. Alternatively, multiple images can be captured using individual adjustments to camera settings and provided to an image merging module. The image merging module then combines portions of the individual images to produce a single image with improved color saturation.

[0009] The functions and concepts of the AWB techniques and devices for camera systems described above can be implemented in any number of different environments, but the embodiments will be explained in the context of the following examples.

[0010] Exemplary system Figure 1 shows an exemplary operating environment 100 for AWB for a camera system 104 of the electronic device 102. The electronic device 102 includes a camera system 104. In the exemplary operating environment 100, the camera system 104 consists of a camera application 106 that utilizes an actuator 108 and a display 110. The camera system 104 further includes an automatic white balance (AWB) module 112 configured to control imaging sensors such as RGB sensors by using a color tone determination module 114 to determine the color tone of a portion of an image and utilizing a white balance controller 116.

[0011] The camera system 104 also incorporates a face frequency (FF) module 118 that works in conjunction with the AWB module 112. The face frequency module 118 includes a face detector 120 configured to detect faces in an image, and a chromatic transient filter 122 configured to determine chromatic data from individual chromatic judgments for the same face in one or more images. The chromatic transient filter 122 can determine the transient data by summing, weighted summing, filtering, or any other computational method. The camera system 104 further includes an image merging module 124 that can generate a single image from multiple images of the same scene.

[0012] FIG. 2 shows an exemplary embodiment 200 of the AWB module 112 and the FF module 118 of FIG. 1 operating in cooperation with each other. In the exemplary embodiment 200, the camera application 106 provides an image to the FF module 118. The provided image may be a preview image of the scene or an image captured previously. The face detector 120 detects faces in the image. The FF module can determine values such as a face frequency value associated with the detected face. The face frequency value is particularly useful in instances where at least one additional detected face is detected in the preview image. The face frequency value is determined based on the number of preceding images determined to include faces of the same person as the detected face in the preview image. When processed, the detected face and image are provided to the camera application 106.

[0013] The camera application 106 provides these images to the AWB module 112, and the AWB module 112 processes the detected face together with the data from the image and determines a value including a color tone associated with the detected face in the image. The color tone is determined using the color tone determination module 114, but the AWB module 112 can further determine a confidence value associated with the detected face in the image using ambient data (e.g., light data and face size data). The determined value including the color tone is output to the camera application 106 and maintained within the camera application 106, provided to the FF module 118, or both.

[0014] When provided to the FF module 118, the FF module 118 can process the value into color data using the color temporary filter 122. The color data includes a more accurate color calculated using the color values associated with the detected faces in one or more images. Further, more accurate color data can be calculated by combining the color values associated with the detected faces in one or more images and the confidence values associated with the detected faces in the same images. These values can be combined using a weighted sum, filtering, or any other computational method. Further, the color data can include a face frequency value determined to be the number of images containing the same face as the detected face.

[0015] The color data can be provided to the AWB module 112 and can be compared to the color of the detected faces in the reference image. The WB controller 116 can determine the WB difference between the color of the detected faces in the reference image and the color data based on this comparison. Further, the WB controller 116 can adjust the camera settings in the camera application 106 based on the WB difference.

[0016] Figure 3 shows a detailed example of the operating environment 100 in Figure 1. Specifically, the electronic devices 102 are shown by various examples including a smartphone 102-1, a tablet 102-2, a laptop 102-3, a desktop computer 102-4, a smartwatch 102-5, digital glasses 102-6, an electronic controller 102-7, a home automation and control system 102-8, and a microwave oven 102-9. These are only a few of the many electronic devices that can use AWB for camera systems, and other devices such as televisions, entertainment systems, sound systems, automobiles, drones, trackpads, drawing pads, netbooks, e-readers, home security systems, and other household appliances may also be included. Note that the electronic devices 102 may be mobile, wearable, non-wearable but mobile, or relatively immobile (e.g., desktops and electrical appliances).

[0017] The electronic device 102 further includes one or more processors 302, a computer-readable medium 304, and one or more sensors 306. The computer-readable medium 304 includes a memory and storage medium 308, an application 310, an automatic white balance (AWB) module medium 312, a face frequency (FF) module medium 314, and an image merging (IM) module medium 316. The application 310, implemented as computer-readable instructions on the computer-readable medium 304, can be executed by the computer processor 302 to provide some or all of the functions described herein. For example, the application 310 may include, or operate together with, a camera application (e.g., camera application 106), an AWB module (e.g., AWB module 112), or an FF module (e.g., FF module 118) to implement AWB for a camera system. Furthermore, one or more sensors 306 may include one or more cameras 320 configured to capture images, video, and sound.

[0018] Figure 4 shows an exemplary embodiment for capturing a scene 402, and a precursor image 404 within the context of AWB for a camera system. The shown electronic device 102 comprises a camera application 106 active on a display 110. The camera application includes an actuator 108 that enables control on the camera application. Using the electronic device 102, the precursor image 404 of scene 402 is captured using one or more sensors. For example, one or more sensors include a camera, and the precursor image 404 is a digital image captured by that camera. However, one or more sensors may be low-resolution RGB sensors, or any other sensors that can be used to capture details from the scene that enable face detection and chromatic determination for detected faces. Furthermore, due to improper WB, the color and sharpness of one or more parts of the precursor image may appear different from the scene.

[0019] Figure 5 shows an exemplary embodiment for detecting a face in a precursor image (e.g., precursor image 404 in Figure 4). The shown electronic device 102 comprises a camera application 106 which is active on a display 110 and presenting the precursor image 404. Using the face detector 120, a first detected face 502 is detected in the precursor image 404. Optionally, the face detector 120 can also detect additional detected faces, such as the additionally detected face 504.

[0020] It should be noted that the face detector 120 does not need to recognize or track the detected faces or the identities associated with them. Instead, the face detector 120 can classify the features associated with the detected faces and later temporarily match those features against features from other images. The classified features can be used to match the detected faces against faces from other images, but it is not necessary to use the classified features associated with the detected faces to determine the identities associated with them. Rather, the classified features can simply be used to match the detected faces against faces from other images, without the actual identity of the person being used. This method eliminates the need for server storage, face identification, and face recognition, and thus protects the identities associated with faces detected by the face detector 120.

[0021] Upon detection, the first detected face 502 is provided to the hue determination module 114, which determines the hue of the first detected face 502 in the preceding image 404. The hue determination module can then determine additional hues for any additionally detected faces 504 in the preceding image 404. Due to inappropriate white balance in the preceding image, the first detected face 502 may appear with an incorrect or altered hue relative to the hue of the same face in the scene (e.g., scene 402).

[0022] Referring to Figure 6, Figure 6 shows an exemplary embodiment of color determination using a color determination module (e.g., color determination module 114). The color determination module 114 is provided with detected faces 602 (e.g., detected face 602-1, detected face 602-2, detected face 602-3). The color determination module 114 determines the color for the detected faces 602 by correlating the detected faces 602 with a color 606 from a set of colors 604. Once the detected faces 602 are correlated with a color 606 from the set of colors 604, that color 606 can be provided to one or more modules (e.g., AWB module 112 and FF module 118) or one or more applications (e.g., camera application 106).

[0023] The set of hues 604 can be generated by predetermined values, machine learning techniques, or any other computation method. Similarly, the hue determination for the detected faces 602 can be determined by a predetermined set of decisions (e.g., a decision tree), machine learning, etc. The use of machine learning can include supervised or unsupervised learning through the use of neural networks, including perceptrons, feedforward neural networks, convolutional neural networks, radial basis function neural networks, or recurrent neural networks. In the example, a machine learning model can be trained by supervised machine learning. In the supervised machine learning example, a machine learning model can be built by being given a labeled set of preceding image captures that identify detected faces in an image. With regard to training the machine learning model and building the set of hues 604, these faces can be classified by their hue values. Through this supervised machine learning, detected faces input to the hue determination module 114 can receive more accurate hue determinations. Future hue determinations can be fed back to the model and marked as correct or incorrect to further train the model. Furthermore, the color determination module 114 can determine if the color 606 of the detected face 602 is not within the color set 604 and add it to the color set 604.

[0024] When a first detected face (e.g., first detected face 502) is detected and its hue is determined, a group of preceding images 702 is retrieved. Figure 7 shows an exemplary embodiment in which the same face as the first detected face 502 is detected within the group of preceding images 702. The group of preceding images 702, captured under preceding ambient conditions, is retrieved. The preceding images 702 can be retrieved from any image storage medium, including on-device image storage and cloud-based image storage. Once the group of preceding images 702 is retrieved, a face detector (e.g., face detector 120) is used to determine the group of preceding images 702 that contain the same face 706 (e.g., same face 706-1, same face 706-2, and same face 706-3) as the first detected face 502, to classify them, if present, into a group of matching images 704. Prior images 708, such as prior images 708-1, 708-2, and 708-3, which are determined to contain the same face 706 as the first detected face 502, are classified into group 704 of matching images. Group 704 of matching images can be provided to one or more modules (e.g., AWB module 112 and FF module 118) or one or more applications (e.g., camera application 106).

[0025] Figure 8 shows an exemplary embodiment in which the group of matching images 802 is determined to contain the same face as the additionally detected face 504 in the preceding image (e.g., preceding image 404). Similar to Figure 7, the preceding group of images 702 is retrieved from the image storage medium. In this example, the preceding group of images 702 is the same as the group of images in Figure 7, but this group may be entirely or partially different from the preceding group of images 702 used for the first detected face (e.g., first detected face 502). In this case as well, using a face detector (e.g., face detector 120), it is determined that the preceding image 804 contains the same face 806 as the additionally detected face 504. Therefore, the preceding image 804 is classified into the group of matching images 802.

[0026] In Figure 9, the tint determination module 114 is used to determine a tint 606 for each identical face 706 (e.g., identical face 706-1, identical face 706-2, identical face 706-3) within a group of matching images (e.g., group of matching images 704). Similar to the exemplary embodiment in Figure 6, the tint determination module 114 determines a tint 606 from a set of tints 604 for each identical face 706. For example, using the tint determination module 114, it is determined that identical face 706-1 has tint 606-1. Similarly, it is determined that identical face 706-2 and identical face 706-3 have the same tint 606-2. These tints 606 associated with identical faces 706 can be provided to one or more modules (e.g., AWB module 112 and FF module 118) or one or more applications (e.g., camera application 106).

[0027] Figure 10 shows an example in which tint data 1002 is determined from a group of matching images 704 that are determined to contain the same face 706 as the detected face (e.g., the first detected face 502). The tint data 1002 is a set of data determined from the group of matching images 704 that are associated with the detected face (e.g., the first detected face 502). The group of matching images 704 is a group of preceding images 708 that are determined to contain the same face 706 as the detected face (e.g., the first detected face 502). In the simplified example, the tint data 1002 is constructed simply from more accurate tints 1004. More accurate tints 1004 are determined by combining tints 606 for each of the same faces 706. The tints 606 for each of the same faces 706 can be combined by any computation method, the simplest of which is averaging the tints to obtain a more accurate tint value.

[0028] More complex examples of color tone data, such as the color tone data 1002 from Figure 10, include face frequency values ​​1006. Face frequency values ​​1006 are computed by determining the number of preceding images 708 in a group of matching images 704. Broadly speaking, face frequency values ​​906 are based on the number of preceding images 708 that are determined to contain the same person's face 706 as the detected person's face (e.g., the first detected face 502). Face frequency values ​​1006 are particularly useful in instances where one or more additional detected faces (e.g., additional detected faces 504) are detected in the preceding images. In such instances, face frequency values ​​1006 can be used to determine the most important detected faces (e.g., the first detected face 502 and the additional detected faces 504).

[0029] For example, a face detector detects a first detected face and additional detected faces in a precursor image, instructs the disclosed camera system to determine the hue of each detected face in the precursor image, and determines a group of matching images for each detected face. The group of matching images is used to create hue data containing more accurate hue values ​​and face frequency values ​​using the disclosed technique. The hue of each detected face in the precursor image is compared to the more accurate hue value in the hue data associated with each detected face to determine camera setting adjustments that minimize the difference between the detected face hue and the associated more accurate hue. In this example, two sets of camera adjustments are provided, with a camera adjustment provided for each detected face in the precursor image. The face frequency values ​​are used to generate priority values ​​for determining one or more of the most important faces in the precursor image. These priority values ​​can be used to combine the two sets of camera adjustments using a weighted sum. In this method, the detected face with the highest priority value and therefore determined to be the most important will have the greatest impact on the camera setting adjustments.

[0030] Priority values ​​can be calculated using different methods, such as assigning priority values ​​based solely on frequency values. Alternatively, or in conjunction with other methods, priority values ​​can also be determined using the face size of detected faces in the precursor image. Specifically, detected faces with larger face sizes are given higher priority values.

[0031] Figure 11 shows a more complex example of a set of matched images determined to contain the same face as the first detected face 502. As in Figure 7, a group of preceding images 1104 is taken out and faces 1106 (e.g., same face 1106-1, same face 1106-2, same face 1106-3) identical to the first detected face 502 are searched for. Again, preceding images 1108 (e.g., preceding image 1008-1, preceding image 1008-2, and preceding image 1008-3) determined to contain the same face 1106 as the first detected face 502 are collected into the group of matched images 1102. Unlike in Figure 7, the preceding images 1108 in the group of matched images 1102 contain the same face 1106 exposed to various ambient conditions 1110 such as lighting and face size, which alter the image elements.

[0032] Figure 12 shows an example of hue data 1002 from group 1102 of matched images in Figure 11. Similar to Figure 10, hue data 1002 is a set of data including a more accurate hue 1004 determined from group 1102 of matched images associated with a detected face (e.g., the first detected face 502). Hue data 1002 may also include face frequency values ​​1006, and, similar to Figure 10, each of the same faces 1106 in the preceding image 1108 from group 1102 of matched images is associated with a hue 606. In contrast to Figure 10, the more accurate hue 1004 is determined here by combining a confidence value 1202 with the hue 606. The confidence value 1202 represents the certainty in determining the hue 606 from each of the same faces 1106.

[0033] It should be noted that the group of matching images 1102 includes the same face 1106 exposed to various ambient conditions 1110. Using data such as face size 1206 and illumination 1204 collected from the ambient conditions 1110, a confidence value 1202 for the hue 606 of each individual identical face 1106 can be determined. For example, a preceding image 1108 in which ambient conditions 1110 include low light conditions may be given a smaller confidence value due to the additional variability of hue determination in the illumination 1204. Alternatively, or in conjunction with this, a preceding image 1108 in which ambient conditions 1110 include a smaller face size 1206 may be given a smaller confidence value due to the additional variability of hue determination in smaller portions of the image.

[0034] Once the hue 606 and associated confidence value 1202 for each identical face 1106 are determined, a more accurate hue 1004 for a detected face (e.g., a first detected face 502) can be determined. The more accurate hue 1004 can be determined by any combination of hue 606 and confidence value 1202. An exemplary method of combination involves a weighted sum in which the hue 606 of each identical face 1106 is weighted by the associated confidence value 1202 and then summed. Another exemplary method involves filtering in which the more accurate hue 1004 is determined using only a subset of the hue 606 of identical faces 1106 that have the highest associated confidence value 1202. Furthermore, filtering can be used in combination with the weighted sum so that the hue 606 is filtered and then weighted by the confidence value 1202.

[0035] Figure 13 shows an exemplary embodiment of AWB for a camera system in an electronic device. Once a more accurate hue determination is made for a detected face (e.g., a first detected face 502), the hue of the detected face is compared to the more accurate hue value. The comparison determines the WB difference between the hue of the detected face in the precursor image and the more accurate hue determined for the detected face. Based on the WB difference, the camera settings of the electronic device 102 are adjusted to enable the capture of an image with improved hue 1302. Camera setting adjustments may necessarily involve some adjustments that enable the capture of an image with improved hue 1302, such as changing the RGB sensor gains, including the red sensor gain, green sensor gain, and blue sensor gain. The camera can then capture an image with improved hue 1302 and provide the captured image to the display 110.

[0036] In a simple example of AWB for a camera system, only one detected face is detected in the lead image. Therefore, one tint data is extracted, one WB difference is determined, and the camera settings are adjusted based on that single WB difference. However, a more complex example includes one or more additional detected faces in the lead image. In this example, multiple tint data is extracted, multiple WB differences are determined, and the camera settings are adjusted based on these multiple WB differences. Priority values ​​can be determined for the first detected face in the lead image and for each additional detected face. Priority values ​​can be determined based on tint data, the lead image, or a combination of the two. For example, if the tint data associated with a detected face contains a higher face frequency value, indicating that the detected face is frequently captured by the camera system, that detected face can be given a higher priority value. Additionally, or separately, if a detected face has a larger face size in the lead image, that detected face can also be given a higher priority value. Once the priority value for each detected face is determined, the camera settings are adjusted based on the WB differences and priority values. WB differences and priority values ​​can be combined using any computation method, including weighted summation and filtering.

[0037] In a specific example, the first detected face is detected in the lead image along with 1 to 19 additional detected faces. Tint data is extracted for each detected face, and AWB differences are determined based on this tint data. A priority value is similarly determined for each detected face in the lead image. Up to 20 detected faces are filtered based on the priority values, generating a subset of detected faces with associated tint data and AWB differences. Once filtered, camera settings can be adjusted based on a weighted sum of the five highest priority value subsets of detected faces, so that WB differences are weighted by the priority values. Note that detected faces and associated tint data can be filtered to any desired subset, and the provided example is merely to illustrate a specific method of combining WB differences across multiple detected faces.

[0038] Figure 14 shows another exemplary embodiment of AWB for a camera system in an electronic device 102, where multiple images (e.g., a first image 1402 and an additional image 1404) are captured with different camera setting adjustments. In one embodiment, a first detected face 502 and an additional detected face 504 are detected in a precursor image. Saturation data is extracted for each detected face and used to determine the WB difference associated with the first detected face 502 and the WB difference associated with the additional detected face 504. The camera settings are adjusted based on the WB difference associated with the first detected face 502, thereby capturing the first image 1402 with improved saturation of the first detected face 502. Next, the camera settings are adjusted based on the WB difference associated with the additional detected face 504, and the additional image 1404 is captured with improved saturation of the additional detected face 504. It should be noted that the first image 1402 and the additional image 1404 can be captured with the same camera or a different camera, one or more images can be stored in memory, and the camera system can capture one or more additional images using the adjusted camera settings.

[0039] Figure 15 shows an exemplary embodiment of an image merging module 124 of a camera system performing AWB. Two images are shown: a first image 1402 has improved saturation in the first detected face 502, and an additional image 1404 has improved saturation in the additional detected face 504. The first image 1402 and the additional image 1404 are provided to the image merging module 124. The image merging module 124 uses image suturing to combine the first image 1402 and the additional image 1404 into a single image 1504 having improved saturation in the first detected face 502 and the additional detected face 504. The image merging module 124 incorporates the first image 1402 into the first detected face 502 and the additional image 1404 into the additional detected face 504. Once the single image 1504 is generated, it is provided to a display 110 on an electronic device 102 for digital display. Furthermore, a single image 1504 can be stored within an electronic device, or it can be stored via any other image storage medium.

[0040] Furthermore, additional images can be provided to the image merging module 124, and by incorporating the additional images for the additionally detected faces in the preceding image, a single image 1502 can be generated via the image merging module 124. Thus, the single image 1502 will have improved color saturation for each of its detected faces.

[0041] Estimative method Figure 16 shows an exemplary method of AWB for a camera system. In Figure 1602, the detected face is detected in a preceding image of the scene. Face detection can be performed by any face detection method, including knowledge-based face detection, feature-based face detection, template matching face detection, or appearance-based face detection.

[0042] In step 1604, the hue of the detected face is determined within the preceding image. The hue of the detected face can be determined from a predetermined set of hue values ​​or from a continuously constructed set of hue values. Hue determination can be performed by machine learning methods, including supervised or unsupervised learning. Furthermore, hue determination can be pre-trained or trained over time.

[0043] In step 1606, tint data associated with the detected face is retrieved. A group of matching images is determined using a group of preceding images captured under preceding ambient conditions. The group of matching images is determined to contain the same face as the detected person's face. The same face as the detected face in the preceding image can be detected in the preceding image using the same or a similar method. The tint data contains tints that can be used to determine more accurate tint values, but the tint data may also contain additional data about the same face in the preceding image and the preceding image itself. This data may include ambient conditions such as the size of the same face in the preceding image and the lighting in the preceding image. For example, ambient conditions can be used to determine confidence values ​​for the tint associated with the same face in each of the preceding images. The confidence values ​​can be used in combination with the tint of the same face to determine more accurate tints. Furthermore, the tint data may include data that can be used to assist in determining priority values ​​such as face frequency values ​​associated with the detected face.

[0044] In step 1608, the difference in white balance (WB) between the detected face hue and the color data is determined. This difference can be determined solely based on the difference between the detected face hue and a more accurate color from the color data, or partially based on that difference. The WB difference is the determined difference in camera settings to match the detected face hue in the precursor image to a more accurate color from the color data. The WB difference can be expressed as a change in the RGB sensor gain values, including the red, green, and blue sensors.

[0045] In 1610, camera settings are adjusted based on the difference in WB. Camera setting adjustments may include adjusting camera sensor gain values, including RGB sensor gain. By adjusting the camera settings, the camera system can capture images of the scene with improved color tones compared to, for example, images of the scene that would be captured by the camera if the camera settings were not adjusted. Furthermore, by adjusting the camera settings, the camera can capture images of the scene with improved color tones. The captured images can be provided to a display for digital display, stored in an electronic device, or both.

[0046] Figure 17 shows an exemplary method of AWB for a camera system in which one or more additional detected faces are detected in a preceding image. Similar to the method shown in Figure 16, in 1602, the detected faces are detected in the preceding image. In 1604, the hue for the detected faces in the preceding image is determined, and the hue data associated with the detected faces is retrieved. In 1608, the difference in WB between the hue of the detected faces and the hue data associated with the detected faces is determined. Unlike Figure 16, a priority value associated with the detected faces is determined. The importance of the detected faces in the preceding image is determined using the priority value. The priority value can be determined based on the hue data, the preceding image, or a combination of the two. For example, if the hue data associated with a detected face contains a higher face frequency value, indicating that the detected face is frequently captured by the camera system, that detected face can be given a higher priority value. Additionally, or separately, if a detected face has a larger face size in the preceding image, that detected face can also be given a higher priority value.

[0047] Optionally, additional faces can be detected in the lead image at 1702. At 1604, the hue for these additional faces is determined, and the hue data associated with these additional faces is retrieved at 1606. At 1608, additional AWB differences are determined based on the difference between the hue of the additionally detected faces in the lead image and the hue data associated with these additionally detected faces. At 1704, the priority value of the additionally detected faces is determined. The method can optionally involve detecting additional faces in the lead image at 1702 and repeating the step for these additionally detected faces, or adjusting the camera settings at 1610. If the camera settings are adjusted, the adjustment can be based on a combination of WB differences and the priority values ​​of the detected and additionally detected faces. The WB differences and priority values ​​can be combined by any computation method, including weighted summation and filtering.

[0048] Figure 18 shows another exemplary method of AWB for a camera system in which one or more additional detected faces are detected in a precursor image. Similar to Figure 17, at 1602 a first face is detected in the precursor image, and at 1604 the hue for the detected face in the precursor image is determined. At 1606 the hue data associated with the detected face is retrieved, and the WB difference between the hue of the detected face in the precursor image and the hue data associated with the detected face is determined. However, the priority value is not determined. Instead, at 1802, a first image with improved hue in the detected face is captured using camera settings adjusted based on the WB difference. At 1702 an additional detected face is detected in the precursor image, and the above steps are repeated for the additional detected face. Thus, at 1608, an additional WB difference is determined based on the hue of the additional detected face and the hue data associated with the additional detected face. At 1802, an additional image is captured using camera settings adjusted based on the additional WB difference. The additional image has improved color rendering in the newly detected faces. The newly detected faces can be detected within the preceding image, and the step can be repeated.

[0049] In step 1804, the first image and the additional image are provided to the image merging module. The image merging module merges the first image and the additional image to produce a single image with improved color tones. For example, the image merging module can incorporate the first image into detected faces and the additional image into additional detected faces. Thus, the resulting single image can have improved color tones in its detected faces and additional detected faces. The single image can be provided to a display for digital display, stored in an electronic device, or both.

[0050] In general, all components, modules, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuits), manual processing, or any combination thereof. Some operations of exemplary methods can be described in the general context of executable instructions stored on computer-readable memory that are local and / or remote to a computer processing system, and embodiments may also include software applications, programs, functions, etc. Alternatively or additionally, all functions described herein can be implemented by one or more hardware logic components, including, but not limited to, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-chip systems (SoCs), composite programmable logic devices (CPLDs), etc.

[0051] The following are some examples. Example 1 A method for automatic white balance (AWB) in a camera system of an electronic device, the method comprising: detecting a first human face in a precedent image; determining a first hue of the first detected human face in the precedent image; and retrieving first hue data associated with the detected first human face, the first hue data being determined from a group of precedent images captured under precedent ambient conditions, wherein the precedent images are determined to contain the same human face as the first detected human face; the method further comprises: determining a first difference in white balance (WB) between the first hue of the first detected human face in the precedent image and the first hue data based on the first hue data; and adjusting camera settings based on the first difference in WB, thereby enabling the capture of an image with improved hue.

[0052] Example 2 The method described in Example 1, further comprising detecting additional human faces in a precursor image, determining the hue of the additionally detected human faces in the precursor image, and retrieving hue data associated with the additionally detected human faces, wherein the hue data is associated with the additionally detected human faces determined from a group of precursor images captured under preceding ambient conditions, and the precursor images are determined to contain the same human faces as the additionally detected human faces, and the method further comprises determining an additional AWB difference between the hue of the additionally detected human faces in the precursor image and the hue data associated with the additionally detected human faces, based on the hue data associated with the additionally detected human faces, and the camera settings are adjusted based on the first WB difference and the additional WB difference.

[0053] Example 3: The method described in Example 2, further comprising determining a first priority value for a first detected human face in the precursor image and determining an additional priority value for an additional detected human face in the precursor image, wherein the camera setting adjustment is based on a weighted sum of a first difference in WB weighted by the first priority value and an additional difference in WB weighted by the additional priority value.

[0054] Example 4: The method described in Example 3, wherein the first priority value is determined based on the first size of the first detected human face in the precursor image, and the additional priority value is determined based on the size of the additionally detected human face in the precursor image.

[0055] Example 5 The method described in Example 3, wherein the first priority value is determined based on the first determined face frequency value, the first determined face frequency value is based on the number of preceding images determined to contain the same person's face as the first detected person's face, and the additional priority value is determined based on the determined face frequency value of the additionally detected person's face, the determined face frequency value of the additionally detected person's face is based on the number of preceding images determined to contain the same person's face as the additionally detected person's face.

[0056] Example 6: The method described in Example 2, wherein detecting additional human faces includes detecting up to 19 human faces, and detecting additional priority values ​​includes determining up to 19 priority values.

[0057] Example 7 A method described in Example 1, wherein by adjusting camera settings, the electronic device captures a first image having a first detected human face having improved color tone, the method further comprising detecting an additional human face in a precursor image, determining the color tone of the additional detected human face in the precursor image, and retrieving color tone data associated with the additional detected human face, the color tone data associated with the additional detected human face being determined from a group of precursor images captured under preceding ambient conditions, the precursor images being determined to contain the same human face as the additional detected human face, the method further comprising determining an additional AWB difference between the color tone of the additional detected human face in the precursor image and the color tone data associated with the additional detected human face based on the color tone data associated with the additional detected human face, causing the electronic device to capture an additional image having an additional detected human face having improved color tone by adjusting camera settings based on the additional WB difference, and providing the first image and the additional image to an image merging module to generate a single image incorporating the first image for the first detected human face and the additional image for the additional detected human face.

[0058] Example 8 The method described in Example 1, wherein the first color data includes a confidence value, the confidence value is based on the face size of the same person's face as the first detected person's face in a preceding image determined to contain the same person's face as the first detected person's face, or the ambient light conditions in the preceding image determined to contain the same person's face as the first detected person's face.

[0059] Example 9 is the method described in Example 8, wherein the determination of the first difference in WB is based on filtered first hue data, and the filtered first hue data is a subset of the first hue data filtered by confidence values.

[0060] Example 10 The method described in Example 8, in which the determination of the first difference in WB is based on a weighted sum of the first hue data weighted by confidence values.

[0061] Example 11 The method described in Example 1, wherein the detection of a first human face in the precursor image is initiated by an actuator.

[0062] Example 12: The method described in Example 1, wherein the electronic device captures an image with improved color tones by adjusting the camera settings.

[0063] Example 13: The method described in Example 12, further comprising providing an image having improved color tones to a display.

[0064] Example 14 The method described in Example 1, wherein adjusting the camera settings includes adjusting the red sensor gain, green sensor gain, or blue sensor gain.

[0065] Example 15 The method described in Example 1, wherein detecting the face of a first person includes using a low-resolution sensor.

[0066] Example 16: A camera system in an electronic device capable of performing automatic white balance (AWB) comprises a processor, a computer-readable medium, a sensor capable of capturing a preceding image, a face frequency module capable of detecting human faces in an image, a storage device capable of storing preceding images, a color determination module capable of determining the color tone of detected human faces, and a white balance controller capable of adjusting camera settings.

[0067] conclusion While the embodiments of the AWB for the camera system have been described above in a language specific to functions and / or methods, the subject matter of the appended claims is not necessarily limited to the specific functions or methods described. Rather, the specific functions and methods are disclosed as exemplary embodiments of the claimed AWB for the camera system, and other equivalent functions and methods are also intended to be within the scope of the appended claims. Furthermore, it should be noted that various embodiments have been described, and each described embodiment can be implemented independently or in conjunction with one or more other described embodiments.

Claims

1. A method for automatic white balance (AWB) in a camera system of an electronic device, Detecting the face of a first person in the preceding image and additional faces of people in the preceding image (1602), Determining the first hue of the first detected person's face in the preceding image and the additional hue of the additional detected person's face in the preceding image (1604), The method includes (1606) retrieving first color tone data associated with the detected first human face and additional color tone data associated with the additional detected human face, wherein the first color tone data is determined from one or more matching images in a group of preceding images captured under preceding ambient conditions, wherein the one or more matching images are determined to contain the same human face as the first detected human face, and the additional color tone data is determined from one or more matching additional images in the group of preceding images or another group of preceding images captured under preceding ambient conditions, wherein the one or more matching additional images are determined to contain the same human face as the additional detected human face, and the method further includes Based on the first color data, a first difference in white balance (WB) between the first color of the first detected human face in the preceding image and the first color data is determined (1608), Based on the additional color tone data associated with the additional detected human faces, determine the additional AWB difference between the additional color tone of the additional detected human faces in the preceding image and the additional color tone data associated with the additional detected human faces. Determining the first priority value of the first detected human face in the preceding image and the additional priority value of the additional detected human face in the preceding image (1702), A method comprising adjusting camera settings based on the first difference in WB, the additional difference in WB, the first priority value, and the additional priority value (1610), wherein by adjusting, an image having improved color tone can be captured.

2. The method according to claim 1, wherein the adjustment of the camera settings is based on a combination of the first difference of WB and the additional difference of WB, the combination being determined using the first priority value and the additional priority value.

3. The method according to claim 1, wherein the adjustment of camera settings is based on a combination of the first difference of WB weighted by the first priority value and the additional difference of WB weighted by the additional priority value.

4. The method according to claim 1, wherein the adjustment of the camera settings is based on a weighted sum of the first difference of the WB weighted by the first priority value and the additional difference of the WB weighted by the additional priority value.

5. Determining the first priority value is based on the first size of the first detected human face in the preceding image. Determining the additional priority value is based on the size of the additional detected human face in the preceding image. The method according to any one of claims 1 to 4.

6. The determination of the first priority value is based on the first determined face frequency value, and the first determined face frequency value is based on the number of preceding images in the group of preceding images that have been determined to contain the same person's face as the first detected person's face. Determining the additional priority value is based on the additional determined face frequency value of the additional detected person's face, and the additional determined face frequency value of the additional detected person's face is based on the number of preceding images in the other group of preceding images that have been determined to contain the same person's face as the additional detected person's face. The method according to any one of claims 1 to 5.

7. Detecting additional human faces involves detecting up to 19 human faces. Determining additional priority values ​​involves determining up to 19 priority values. The method according to any one of claims 1 to 6.

8. The first color data includes a confidence value, and the confidence value is The face size of the face of the same person as the first detected person in the preceding image, which is determined to include the same person's face as the first detected person's face, or Ambient light conditions in the preceding image that were determined to contain the same person's face as the first detected person's face. The method according to any one of claims 1 to 7, based on the present invention.

9. The method according to claim 8, wherein the determination of the first difference in WB is based on filtered first hue data, the filtered first hue data being a subset of the first hue data filtered by the confidence value.

10. The method of claim 8, wherein determining the first difference in WB is based on a weighted sum of the first color data weighted by the confidence value.

11. The method according to any one of claims 1 to 10, wherein the detection of the first human face in the preceding image is initiated by an actuator.

12. The method according to any one of claims 1 to 11, wherein the electronic device captures an image having the improved color tones by adjusting the camera settings.

13. The method according to claim 12, further comprising providing the image having the improved color tones to a display.

14. Adjusting camera settings is Red sensor gain, Green sensor gain, or Blue sensor gain The method according to any one of claims 1 to 13, including adjusting the

15. The method according to any one of claims 1 to 14, wherein detecting a first person's face includes using a sensor having sufficient resolution to enable face detection and determination of the hue of the detected face.

16. A camera system in an electronic device capable of performing automatic white balance (AWB), Processor (302), Computer-readable media (304) and A sensor (306) capable of capturing a precursor image and providing the precursor image to the processor, A face frequency module (118) that can detect human faces in an image, A memory device (308) capable of storing a preceding image, A color tone determination module (114) that can determine the color tone of a detected person's face, The camera (320) is equipped with a white balance controller (116) that can adjust the settings. The computer-readable medium stores instructions that, when executed by the processor, cause the camera system to perform the following method, the method being: The first human face in the preceding image received from the sensor and additional human faces in the preceding image are detected using the face frequency module. The first hue of the detected first person's face and the additional hue of the additional detected person's face are determined using the hue determination module, The method further includes retrieving from the storage device first color tone data associated with the first person's face and additional color tone data associated with the additional detected person's face, wherein the first color tone data is determined from one or more matching images in a group of preceding images captured under preceding ambient conditions, wherein the one or more matching images are determined to contain the same person's face as the first detected person's face, and the additional color tone data is determined from one or more matching additional images in the group of preceding images or another group of preceding images captured under preceding ambient conditions, wherein the one or more matching additional images are determined to contain the same person's face as the additional detected person's face, and the method further Determining a first difference in white balance (WB) between the first color tone and the first color tone data, To determine the additional AWB difference between the additional color and the additional color data, Determining the first priority value of the first detected human face in the preceding image and the additional priority value of the additional detected human face in the preceding image, A camera system comprising adjusting the camera settings using the white balance controller based on the first difference of WB, the additional difference of WB, the first priority value, and the additional priority value, in order to enable subsequent capture of improved images.

17. A camera (320) system in an electronic device configured to carry out the method described in any one of claims 1 to 15.

18. A computer program that, when executed by one or more processors (302), includes instructions causing the one or more processors (302) to perform the method described in any one of claims 1 to 15.

Images