Spatially varying white balance and color adjustment using machine learning

Abstract

A method includes obtaining an image depicting a subject and determining, based on the image, a skin segmentation mask representing skin regions of the subject as depicted in the image. The skin regions may include (i) a face skin region on a face of the subject and (ii) a body skin region on at least one body part of the subject. The method also includes determining, based on a plurality of skin pixels of the image corresponding to the skin regions as represented by the skin segmentation mask, one or more color gain parameters for the image. The method further includes adjusting a pixel value of a skin pixel of the plurality of skin pixels of the image based on the one or more color gain parameters.

Description

Spatially Varying White Balance and Color Adjustment Using Machine LearningBACKGROUND

[0001] The objective visual quality of an image may be improved by adjusting a white balance and / or a color of the image. Adjusting the white balance of an image may involve determining a white balance gain that, when applied to pixel(s) of the image that represent a white feature and / or a gray feature of the physical world, adjusts the pixel(s) to also appear white and / or gray, respectively, in the image. Similarly, correcting the color of the image may involve determining color correction gains that, when applied to pixel(s) of the image that represent a feature of the physical world having a particular color, adjust the pixel(s) to also have the particular color in the image. One set of white balance and / or color correction gains may be insufficient to improve the visual quality of an entire image.SUMMARY

[0002] An image processing system may be configured to perform a spatially-variable adjustment of a white balance and / or color of an image based on the semantic content and / or illumination conditions present in different parts of the image. Specifically, the image processing system may be configured to determine, based on an image that depicts a subject, a skin segmentation mask that indicates a face skin region of the subject and a body skin region of the subject. Based on the skin segmentation mask, the image processing system may be configured to determine color gain parameters for the image, and apply the color gain parameters to skin pixels in the skin regions of the image. By representing both the face skin region and the body skin region, the skin segmentation mask may increase a number of skin tone samples available to the image processing system, thus allowing for more accurate determination of the color gain parameters. In some cases, the color gain parameters may be applied to a subject region of the image that depicts the subject and includes both the skin and non-skin regions, thereby uniformly adjusting the entire subject region.

[0003] In a first example embodiment, a method includes obtaining an image depicting a subject. The method also includes determining, based on the image, a skin segmentation mask representing skin regions of the subject as depicted in the image. The skin regions may include (i) a face skin region on a face of the subject and (ii) a body skin region on at least one body part of the subject. The method additionally includes determining, based on a plurality of skin pixels of the image corresponding to the skin regions as represented by the skin segmentation mask, one or more color gain parameters for the image. The method further includes adjustinga pixel value of a skin pixel of the plurality of skin pixels of the image based on the one or more color gain parameters.

[0004] In a second example embodiment, a system may include a processor and a non- transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations in accordance with the first example embodiment.

[0005] In a third example embodiment, a non-transitory computer-readable medium may have stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations in accordance with the first example embodiment.

[0006] In a fourth example embodiment, a system may include various means for carrying out each of the operations of the first example embodiment.

[0007] These, as well as other embodiments, aspects, advantages, and alternatives, will become apparent to those of ordinary skill in the art by reading the following detailed description, with reference where appropriate to the accompanying drawings. Further, this summary and other descriptions and figures provided herein are intended to illustrate embodiments by way of example only and, as such, that numerous variations are possible. For instance, structural elements and process steps can be rearranged, combined, distributed, eliminated, or otherwise changed, while remaining within the scope of the embodiments as claimed.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 illustrates a computing device, in accordance with examples described herein.

[0009] Figure 2 illustrates a computing system, in accordance with examples described herein.

[0010] Figure 3 illustrates an image processing system, in accordance with examples described herein.

[0011] Figure 4A illustrates an input image, in accordance with examples described herein.

[0012] Figure 4B illustrates a subject segmentation mask, in accordance with examples described herein.

[0013] Figure 4C illustrates a skin segmentation mask, in accordance with examples described herein.

[0014] Figure 4D illustrates a subject-and-skin segmentation mask, in accordance with examples described herein.

[0015] Figure 4E illustrates a background segmentations mask, in accordance with examples described herein.

[0016] Figure 4F illustrates an illumination segmentation mask, in accordance with examples described herein.

[0017] Figure 4G illustrates a color gain parameter map, in accordance with examples described herein.

[0018] Figure 5 illustrates a training system, in accordance with examples described herein.

[0019] Figure 6 illustrates a flow chart, in accordance with examples described herein.DETAILED DESCRIPTION

[0020] Example methods, devices, and systems are described herein. It should be understood that the words “example” and “exemplary” are used herein to mean “serving as an example, instance, or illustration.” Any embodiment or feature described herein as being an “example,” “exemplary,” and / or “illustrative” is not necessarily to be construed as preferred or advantageous over other embodiments or features unless stated as such. Thus, other embodiments can be utilized and other changes can be made without departing from the scope of the subject matter presented herein.

[0021] Accordingly, the example embodiments described herein are not meant to be limiting. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

[0022] Further, unless context suggests otherwise, the features illustrated in each of the figures may be used in combination with one another. Thus, the figures should be generally viewed as component aspects of one or more overall embodiments, with the understanding that not all illustrated features are necessary for each embodiment.

[0023] Additionally, any enumeration of elements, blocks, or steps in this specification or the claims is for purposes of clarity. Thus, such enumeration should not be interpreted to require or imply that these elements, blocks, or steps adhere to a particular arrangement or are carried out in a particular order. Unless otherwise noted, figures are not drawn to scale.I. Overview

[0024] An image processing system may be configured to determine and apply one or more color gain parameters to an image to correct and / or improve a visual appearance of portions of the image. For example, the color gain parameters may be selected to make a scene, as represented in the image, appear more similar to how the scene itself is perceived by a human. For example, the one or more color gain parameters may include a white balance gain parameter configured to improve a white balance of the image, and / or a color correction gain parameter configured to improve a color balance / quality of the image. The color gain parameters may be selected to correct for color artifacts resulting from optical properties of the camera and / or variations in lighting conditions across images and / or across different portions of the same image.

[0025] Inaccurate and / or unnatural colors of an image may be especially apparent in regions that depict the skin of human subjects. Thus, some image processing systems may determine color gain parameters that are primarily and / or exclusively based on a face region of interest (ROI) of the image that depicts a subject’s face. However, color gain parameters generated in this manner may nevertheless be inaccurate due to presence of non-skin pixels (e.g., clothing, masks, glasses, hair, etc.) in the face ROI and / or the face ROI being of insufficient size to be representative of the color properties of the subject’s face. Additionally, color gain parameters selected based on the face ROI may be inaccurate for other portions of the image, especially where the other portions of the image include different lighting conditions and / or objects of different types.

[0026] Accordingly, the image processing system may be improved by using a skin segmentation mask to determine color gain parameters based on skin regions of a subject in the image. The skin regions represented by the skin segmentation mask may include a face skin region on the face of the subject and one or more body skin regions on one or more body parts of the subject. The skin segmentation mask may represent arbitrarily-shaped skin regions, and may thus more accurately include skin regions, and exclude non-skin regions, of the subject’s face. Additionally, by also representing body skin regions, the skin segmentation mask may identify additional skin pixels that provide the image processing system with additional visual data representative of the subject’s skin tone, thus allowing for determination of more accurate color gain parameters.

[0027] In some implementations, the color gain parameters determined based on the skin regions may be applied to both the skin regions and non-skin regions of the subject. For example, the image processing system may also determine a subject segmentation mask, ofwhich the skin segmentation mask may be a subset, and may apply the color gain parameters to all pixels of the image that correspond to the subject segmentation mask, on the assumption that both the subject’s skin and non-skin regions experience the same or similar illumination. The color gain parameters may be determined independently of pixel values in the non-skin regions, thus favoring color correction of the skin regions, but may nevertheless also improve the color quality of the non-skin regions. The image processing system may be configured to determine different color gain parameters for (background) regions outside of the subject segmentation mask, thus providing for spatially-variable color correction of the image according to the semantic content thereof.

[0028] Additionally or alternatively, the image processing system may be configured to determine an illumination segmentation mask based on the image, and use the illumination segmentation mask to further improve determination of the color gain parameters. Specifically, the illumination segmentation mask may partition the image into two or more illumination regions each of which is illuminated by light of a corresponding color temperature, with the color temperature varying across the two or more illumination regions. Each illumination region may be associated with an estimated color temperature, thus providing an indication of how different colors should appear within different parts of the image. The image processing system may be configured to use the illumination segmentation mask to determine a corresponding set of color gain parameters for each illumination region, with the color gain parameters possibly differing among illumination regions.

[0029] The illumination segmentation mask may be used in combination with the subject segmentation mask and / or the skin segmentation mask to further improve the color quality of the image. For example, the illumination segmentation mask may partition the subject segmentation mask and / or the skin segmentation mask into two or more skin illumination regions, thus indicating that different parts of the subject are illuminated using light sources of differing temperatures. Accordingly, the image processing system may be configured to determine and apply a corresponding (different) set of color gain parameters for each skin illumination region, thus adjusting a color quality of the image based both on the subject’s skin tone and the scene’s lighting conditions.II. Example Computing Devices and Systems

[0031] Figure 1 illustrates an example computing device 100. Computing device 100 is shown in the form factor of a mobile phone. However, computing device 100 may be alternatively implemented as a laptop computer, a tablet computer, and / or a wearable computing device, among other possibilities. Computing device 100 may include variouselements, such as body 102, display 106, and buttons 108 and 110. Computing device 100 may further include one or more cameras, such as front-facing camera 104 and rear-facing camera 112.

[0032] Front-facing camera 104 may be positioned on a side of body 102 typically facing a user while in operation (e.g., on the same side as display 106). Rear-facing camera 112 may be positioned on a side of body 102 opposite front-facing camera 104. Referring to the cameras as front and rear facing is arbitrary, and computing device 100 may include multiple cameras positioned on various sides of body 102.

[0033] Display 106 could represent a cathode ray tube (CRT) display, a light emitting diode (LED) display, a liquid crystal (LCD) display, a plasma display, an organic light emitting diode (OLED) display, or any other type of display known in the art. In some examples, display 106 may display a digital representation of the current image being captured by front-facing camera 104 and / or rear-facing camera 112, an image that could be captured by one or more of these cameras, an image that was recently captured by one or more of these cameras, and / or a modified version of one or more of these images. Thus, display 106 may serve as a viewfinder for the cameras. Display 106 may also support touchscreen functions that may be able to adjust the settings and / or configuration of one or more aspects of computing device 100.

[0034] Front-facing camera 104 may include an image sensor and associated optical elements such as lenses. Front-facing camera 104 may offer zoom capabilities or could have a fixed focal length. In other examples, interchangeable lenses could be used with front-facing camera 104. Front-facing camera 104 may have a variable mechanical aperture and a mechanical and / or electronic shutter. Front-facing camera 104 also could be configured to capture still images, video images, or both. Further, front-facing camera 104 could represent, for example, a monoscopic, stereoscopic, or multiscopic camera. Rear-facing camera 112 may be similarly or differently arranged. Additionally, one or more of front-facing camera 104 and / or rear-facing camera 112 may be an array of one or more cameras.

[0035] Computing device 100 could be configured to use display 106 and front-facing camera 104 and / or rear-facing camera 112 to capture images of a target object. The captured images could be a plurality of still images or a video stream. The image capture could be triggered by activating button 108, pressing a softkey on display 106, or by some other mechanism. Depending upon the implementation, the images could be captured automatically at a specific time interval, for example, upon pressing button 108, upon appropriate lighting conditions of the target object, upon moving computing device 100 a predetermined distance, or according to a predetermined capture schedule.

[0036] Figure 2 is a simplified block diagram showing some of the components of an example computing system 200. By way of example and without limitation, computing system 200 may be a cellular mobile telephone (e.g., a smartphone), a computer (such as a desktop, notebook, tablet, server, or handheld computer), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, a gaming console, a robotic device, a vehicle, or some other type of device. Computing system 200 may represent, for example, aspects of computing device 100.

[0037] As shown in Figure 2, computing system 200 may include communication interface 202, user interface 204, processor 206, data storage 208, and camera components 224, all of which may be communicatively linked together by a system bus, network, or other connection mechanism 210. Computing system 200 may be equipped with at least some image capture and / or image processing capabilities. It should be understood that computing system 200 may represent a physical image processing system, a particular physical hardware platform on which an image sensing and / or processing application operates in software, or other combinations of hardware and software that are configured to carry out image capture and / or processing functions.

[0038] Communication interface 202 may allow computing system 200 to communicate, using analog or digital modulation, with other devices, access networks, and / or transport networks. Thus, communication interface 202 may facilitate circuit-switched and / or packet-switched communication, such as plain old telephone service (POTS) communication and / or Internet protocol (IP) or other packetized communication. For instance, communication interface 202 may include a chipset and antenna arranged for wireless communication with a radio access network or an access point. Also, communication interface 202 may take the form of or include a wireline interface, such as an Ethernet, Universal Serial Bus (USB), or High- Definition Multimedia Interface (HDMI) port, among other possibilities. Communication interface 202 may also take the form of or include a wireless interface, such as a Wi-Fi, BLUETOOTH®, global positioning system (GPS), or wide-area wireless interface (e.g., WiMAX or 3GPP Long-Term Evolution (LTE)), among other possibilities. However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used over communication interface 202. Furthermore, communication interface 202 may comprise multiple physical communication interfaces (e.g., a Wi-Fi interface, a BLUETOOTH® interface, and a wide-area wireless interface).

[0039] User interface 204 may function to allow computing system 200 to interact with a human or non-human user, such as to receive input from a user and to provide output to theuser. Thus, user interface 204 may include input components such as a keypad, keyboard, touch-sensitive panel, computer mouse, trackball, joystick, microphone, and so on. User interface 204 may also include one or more output components such as a display screen, which, for example, may be combined with a touch-sensitive panel. The display screen may be based on CRT, LCD, LED, and / or OLED technologies, or other technologies now known or later developed. User interface 204 may also be configured to generate audible output(s), via a speaker, speaker jack, audio output port, audio output device, earphones, and / or other similar devices. User interface 204 may also be configured to receive and / or capture audible utterance(s), noise(s), and / or signal(s) by way of a microphone and / or other similar devices.

[0040] In some examples, user interface 204 may include a display that serves as a viewfinder for still camera and / or video camera functions supported by computing system 200. Additionally, user interface 204 may include one or more buttons, switches, knobs, and / or dials that facilitate the configuration and focusing of a camera function and the capturing of images. It may be possible that some or all of these buttons, switches, knobs, and / or dials are implemented by way of a touch-sensitive panel.

[0041] Processor 206 may comprise one or more general purpose processors - e.g., microprocessors - and / or one or more special purpose processors - e.g., digital signal processors (DSPs), graphics processing units (GPUs), floating point units (FPUs), network processors, application-specific integrated circuits (ASICs), and / or tensor processing units (TPUs). In some instances, special purpose processors may be capable of image processing, image alignment, and merging images, among other possibilities. Data storage 208 may include one or more volatile and / or non-volatile storage components, such as magnetic, optical, flash, or organic storage, and may be integrated in whole or in part with processor 206. Data storage 208 may include removable and / or non-removable components.

[0042] Processor 206 may be capable of executing program instructions 218 (e.g., compiled or non-compiled program logic and / or machine code) stored in data storage 208 to carry out the various functions described herein. Therefore, data storage 208 may include a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by computing system 200, cause computing system 200 to carry out any of the methods, processes, or operations disclosed in this specification and / or the accompanying drawings. The execution of program instructions 218 by processor 206 may result in processor 206 using data 212.

[0043] By way of example, program instructions 218 may include an operating system222 (e.g., an operating system kernel, device driver(s), and / or other modules) and one or moreapplication programs 220 (e.g., camera functions, address book, email, web browsing, social networking, audio-to-text functions, text translation functions, and / or gaming applications) installed on computing system 200. Similarly, data 212 may include operating system data 216 and application data 214. Operating system data 216 may be accessible primarily to operating system 222, and application data 214 may be accessible primarily to one or more of application programs 220. Application data 214 may be arranged in a file system that is visible to or hidden from a user of computing system 200.

[0044] Application programs 220 may communicate with operating system 222 through one or more application programming interfaces (APIs). These APIs may facilitate, for instance, application programs 220 reading and / or writing application data 214, transmitting or receiving information via communication interface 202, receiving and / or displaying information on user interface 204, and so on.

[0045] In some cases, application programs 220 may be referred to as “apps” for short. Additionally, application programs 220 may be downloadable to computing system 200 through one or more online application stores or application markets. However, application programs can also be installed on computing system 200 in other ways, such as via a web browser or through a physical interface (e.g., a USB port) on computing system 200.

[0046] Camera components 224 may include, but are not limited to, an aperture, shutter, recording surface (e.g., photographic film and / or an image sensor), lens, shutter button, infrared projectors, and / or visible-light projectors. Camera components 224 may include components configured for capturing of images in the visible-light spectrum (e.g., electromagnetic radiation having a wavelength of 380 - 700 nanometers) and / or components configured for capturing of images in the infrared light spectrum (e.g., electromagnetic radiation having a wavelength of 701 nanometers - 1 millimeter), among other possibilities. Camera components 224 may be controlled at least in part by software executed by processor 206.

[0047] In further examples, one or more remote cameras 230 may be controlled by computing system 200. For instance, computing system 200 may transmit control signals to the one or more remote cameras 230 through a wireless or wired connection. Such signals may be transmitted as part of an ambient computing environment. In such examples, inputs received at the computing system 200 (for instance, physical movements of a wearable device) may be mapped to movements and / or other functions of the one or more remote cameras 230. Images captured by the one or more remote cameras 230 may be transmitted to the computing system200 for further processing. Such images may be treated as images captured by cameras physically located on the computing system 200.III. Example Image Processing System

[0048] Figure 3 illustrates aspects of an example image processing system 300. Image processing system 300 may include foreground segmentation model 304, background segmentation model 314, illumination segmentation model 318, color gain parameter selector 322, and color adjuster 326. Aspects of image processing system 300 may be implemented as part of an image signal processor (ISP) of a camera device (e.g., camera 112 and / or 104) and / or as an application executed by a general purpose processor (e.g., as part of application programs 220). Aspects of image processing system 300 may be implemented using hardware, software, and / or a combination thereof.

[0049] Image processing system 300 may be configured to generate output image 330 based on input image 302. Specifically, output image 330 may represent input image 302 with one or more color adjustments to modify, correct, and / or improve a white balance and / or a color of input image 302. Accordingly, output image 330 may more accurately represent the objective color properties of the underlying scene than input image 302. An example of input image 302 is illustrated in Figure 4A.

[0050] Foreground segmentation model 304 may include subject segmentation model 306 and skin segmentation model 310. Subject segmentation model 306 may be configured to determine subject segmentation mask 308 based on input image 302. Subject segmentation mask 308 may indicate portions of input image 302 that represent a subject (e.g., a human) depicted in input image 302. Specifically, each respective pixel of subject segmentation mask 308 may indicate whether corresponding one or more pixels of input image 302 represent the subject or some visual feature other than the subject. Subject segmentation mask 308 may indicate both skin regions and non-skin regions of the subject. An example of subject segmentation mask 308 is illustrated in Figure 4B.

[0051] Skin segmentation model 310 may be configured to determine skin segmentation mask 312 based on input image 302 and / or subject segmentation mask 308. Skin segmentation mask 312 may indicate portions of input image 302 that represent skin of the subject depicted in input image 302. Skin segmentation mask 312 may indicate the skin regions of the subject, but not the non-skin regions, and may thus be a subset of subject segmentation mask 308. Specifically, each respective pixel of skin segmentation mask 312 may indicate whether corresponding one or more pixels of input image 302 represent the subject’s skin or some visual feature other than the subject’s skin.

[0052] The skin regions may include a face skin region representing a face of the subject and a body skin region representing one or more body parts (other than the face) of the subject. In some implementations, skin segmentation mask 312 may also indicate, for each respective pixel in the skin regions, a body part (e.g., face, arm, leg, torso, etc.) represented by the respective skin pixel. An example of skin segmentation mask 312 is illustrated in Figure 4C.

[0053] In some implementations, subject segmentation mask 308 and skin segmentation mask 312 may be represented using a combined subject-and-skin segmentation mask that represents skin regions of the subject using first one or more values and / or channels, non-skin regions of the subject using second one or more values and / or channels, and nonsubject regions using third one or more values and / or channels. An example of the combined subject-and-skin segmentation mask is illustrated in Figure 4D. Additionally or alternatively, the operations performed by subject segmentation model 306 and skin segmentation model 310 may be performed using a combined subject-and-skin segmentation model configured to generate subject segmentation mask 308, skin segmentation mask 312, and / or the combined subject-and-skin segmentation mask.

[0054] Background segmentation model 314 may be configured to determine background segmentation mask 316 based on input image 302. Background segmentation mask 316 may represent one or more classes of objects depicted in at least a background region of image 302. The term “background” is used herein to denote portions of input image 302 that do not represent the subject, and may be used synonymously with the term “non-subject” portions / regions. Background segmentation mask 316 may partition pixels of input image 302 according to the object type / class represented by each pixel. Specifically, each respective pixel of background segmentation mask 316 may indicate a type / class of object represented by corresponding one or more background pixels of input image 302.

[0055] In some implementations, background segmentation mask 316 might not represent the subject (e.g., the subject region represented by subject segmentation mask 308 might not be processed by background segmentation model 314). In other implementations, background segmentation mask 316 may provide an additional representation of the subject region occupied by the subject, and this additional representation of the subject region may be used or ignored by color gain parameter selector 322.

[0056] In some implementations, subject segmentation mask 308, skin segmentation mask 312, the combined subject-and-skin segmentation mask, and / or background segmentation mask 316 may be represented as part of a combined semantic segmentation mask.Specifically, each respective pixel of this semantic segmentation mask may indicate, using a corresponding set of one or more values and / or channels, a visual feature (e.g., subject, skin, first object type, second object type, etc.) represented by corresponding one or more pixels of input image 302. Further, the operations of foreground segmentation model 304 and background segmentation model 314 may be performed using a combined semantic segmentation model configured to generate the semantic segmentation mask.

[0057] Illumination segmentation model 318 may be configured to determine illumination segmentation mask 320 based on input image 302. Illumination segmentation mask 320 may partition input image 302 according to the color temperature of light by which different portions of input image 302 are illuminated. Specifically, each respective pixel of illumination segmentation mask 320 may indicate a color temperature of light (e.g., a correlated color temperature (CCT) and / or a distance in color space from the CCT curve) by which a visual feature represented by corresponding one or more pixels of input image 302 is illuminated. Thus, illumination segmentation mask 320 may indicate one or more regions of input image 302, each of which is illuminated by light of a corresponding color temperature. For example, illumination segmentation mask 320 may partition input image 302 into a plurality of illumination regions, thus representing a plurality of portions of input image 302 that are illuminated by a plurality of different illumination sources each having a different CCT.

[0058] Foreground segmentation model 304, subject segmentation model 306, skin segmentation model 310, background segmentation model 314, and / or illumination segmentation model 318 may be implemented using one or more artificial intelligence (Al) algorithms and / or models, machine learning (ML) algorithms and / or models (e.g., artificial neural networks arranged using various possible architectures), computer vision algorithms and / or models, heuristics, and / or other algorithms and / or models. Accordingly, in some cases, each respective model of these models may be trained using corresponding training data that illustrates desired output of the respective model, and / or one or more loss functions that quantify the respective model’s ability to generate the desired output.

[0059] In some cases, subject segmentation mask 308, skin segmentation mask 312, background segmentation mask 316, and / or illumination segmentation mask 320 may be generated at a first resolution that is smaller than a second resolution of input image 302, and may thus be upsampled from the first resolution to the second resolution before being used. For example, the upsampling may be performed by the corresponding model and / or by color gain parameter selector 322.

[0060] Color gain parameter selector 322 may be configured to determine color gain parameter(s) 324 based on input image 302, subject segmentation mask 308, skin segmentation mask 312, background segmentation mask 316, and / or illumination segmentation mask 320. Color gain parameter(s) 324 may include a plurality of color gain parameters that vary spatially in accordance with a type of content and / or an illumination thereof depicted by input image 302. For example, for each respective semantically-distinct portion of input image 302, color gain parameter(s) 324 may include a corresponding set of one or more color gain parameters configured to adjust a white balance and / or color of the respective semantically-distinct portion of input image 302.

[0061] Color gain parameter(s) 324 may be represented using one or more white balance matrices, one or more color correction matrices, and / or one or more CCT values. The one or more white balance matrices may be represented using one or more white balance lookup tables, and the one or more color correction matrices may be represented using one or more color correction look-up tables, among other possibilities.

[0062] Color gain parameter selector 322 may be configured to determine corresponding color gain parameter(s) for at least the skin regions indicated by skin segmentation mask 312 based on visual statistics associated with pixels of input image 302 in both the face skin region and the body skin region. Skin segmentation mask 312 may represent skin regions having arbitrary shapes (e.g., irregular polygons), and may thus provide a more accurate delineation between skin regions and non-skin regions than, for example, a rectangular face region of interest (RO I) and / or another ROI having a predetermined shape. Thus, by using skin segmentation mask 312, color gain parameter selector 322 may be able to more accurately distinguish between the visual statistics of skin regions (to be included in the computation of the corresponding color gain parameter(s) for at least the skin regions) and the visual statistics of non-skin regions (to be excluded from the computation of the corresponding color gain parameter(s) for at least the skin regions).

[0063] Additionally, considering the visual statistics of both the face skin region and the body skin regions may improve the quality and / or accuracy of the corresponding color gain parameters for at least the skin regions. Specifically, in some input images, the face skin region by itself might not provide color gain parameter selector 322 with sufficient visual statistics to accurately correct the white balance and / or color of the skin regions. For example, the face skin region may be too small (e.g., represented by less than a threshold number of skin pixels) due to the subject occupying a small portion of the input image, and / or the subject’s face being partially covered up (e.g., by clothing, mask, glasses, and / or other obstructions). Accordingly,the body skin regions may provide color gain parameter selector 322 with additional pixels that represent the visual properties of the subject’s skin.

[0064] Thus, in some implementations, color gain parameter selector 322 may be configured to determine a number of pixels in the face region of image 302, as indicated by skin segmentation mask 312. When the number of face pixels in the face region is greater than a threshold number of skin pixels, color gain parameter selector 322 may determine corresponding color gain parameter(s) for at least the skin regions based on the visual statistics of face pixels in the face region, and might not consider the visual statistics of body pixels in the body regions. When the number of face pixels in the face region is not greater than the threshold number of skin pixels, color gain parameter selector 322 may determine corresponding color gain parameter(s) for at least the skin regions based on both (i) the visual statistics of face pixels in the face region and (ii) the visual statistics of body pixels in the body regions. In some cases, a sum of the face pixels and the body pixels may exceed the threshold number of skin pixels, and may thus provide color gain parameter selector 322 with a sufficient number of samples to provide a more accurate white balance correction and / or color correction.

[0065] Color gain parameter selector 322 may be configured to determine the corresponding color gain parameter(s) for the skin regions by determining a tone of the skin regions. For example, color gain parameter selector 322 may define a plurality of skin tones and may be configured to determine a skin tone for the skin regions based on the extent to which the visual statistics associated with the skin regions match the plurality of skin tones. For example, based on the visual statistics associated with the skin regions, color gain parameter selector 322 may be configured to determine, for each respective skin tone of the plurality of skin tones, a corresponding probability value indicating a likelihood that the skin regions exhibit the respective skin tone. Each respective skin tone of the plurality of skin tones may be associated with one or more corresponding color gain values. Color gain parameter selector 322 may be configured to (i) determine the skin tone for the skin regions by interpolating the plurality of skin tones according to the corresponding probability values thereof and (ii) select the corresponding color gain parameter(s) for the selected skin tone by interpolating the corresponding color gain values of the plurality of skin tones according to the corresponding probability values thereof.

[0066] The visual statistics of a pixel group having one or more pixels may include one or more Bayer grid statistics associated with the pixel group, color properties of the pixel group (e.g., color(s), tone(s), hue(s), tint(s), and / or other color metrics, expressed in one or more color spaces), brightness value(s) associated with the pixel group, exposure value(s) associated withthe pixel group, a scene type represented by the pixel group, and / or properties of the image sensor, among other possibilities. Determining the visual statistics may include determining a mean, median, average, minimum, maximum, frequency, and / or other metrics of the quantities described above.

[0067] In some implementations, color gain parameter selector 322 may be configured to apply different weights to face skin pixels and the body skin pixels when determining the visual statistics. For example, face skin pixels may be weighted more heavily than the body skin pixels to allow for prioritization of the face skin tone while also using the body skin tone to provide additional visual information. In some implementations, the weight of a respective body skin pixel may be based on a confidence value associated with the respective pixel (e.g., the weight may be equal to the confidence value), where the confidence value indicates a likelihood that the respective pixel represents the subject’s skin (rather than, e.g., background objects or the subject’s clothing, etc.).

[0068] In some implementations, the visual statistics of the face skin region may be used to select a subset of the body skin pixels to be used in determining color gain parameter(s) 324. For example, values of the face skin pixels may be used to identify an average color of the face region. Body skin pixels having color values that fall within a predetermined distance (in color space) of the average color of the face region may be used in determining color gain parameter(s) 324, while body skin pixels having color values that do not fall within the predetermined distance of the average color may be excluded from determining color gain param eter(s) 324.

[0069] In some implementations, the corresponding color gain parameter(s) determined for the skin regions may be applied to the entirety of the subject region indicated by subject segmentation mask 308, including both the skin regions and the non-skin regions of the subject. Specifically, color gain parameter selector 322 may be configured to determine the corresponding color gain parameter(s) for the entirety of the subject region based on visual statistics associated with skin pixels in the skin regions, but independently of the visual statistics of non-skin pixels in non-skin portions of the subject region. Thus, the corresponding color gain parameter(s) for the subject region may be selected and / or optimized to preferentially improve an appearance of the skin regions, since inaccurate and / or abnormal colors may be more apparent to viewers in skin regions than in non-skin regions. Since both the skin regions and the non-skin regions of the subject are likely to be illuminated by the same or similar light source, applying the corresponding color gain parameter(s) to both the skin andnon-skin regions of the subject may improve the visual quality and accuracy of the entirety of the subject region.

[0070] In cases where input image 302 includes multiple different subjects, image processing system 300 may be configured to independently adjust the white balance and / or the color of each subject. Accordingly, subject segmentation mask 308 and / or skin segmentation mask 312 may be configured to differentiate between multiple subjects by indicating, for each respective pixel of a plurality of pixels thereof, which of the multiple subjects the respective pixel corresponds to. Color gain parameter selector 322 may be configured to independently determine a corresponding set of color gain param eter(s) for each of the multiple subjects.

[0071] Color gain parameter selector 322 may also be configured to determine corresponding color gain parameter(s) for each semantically-distinct region indicated by background segmentation mask 316. Specifically, for each respective type of object represented in background portions of input image 302, color gain parameter selector 322 may be configured to determine a corresponding set of one or more color gain parameter(s) based on visual statistics associated with pixels of input image 302 that represent the respective type of object. The corresponding set of color gain param eter(s) selected for a first object type may be different from the corresponding set of color gain parameter(s) selected for a second object type. Further, the color gain parameter(s) determined for background regions of input image 302 may differ from the color gain parameter(s) determined for the subject and / or skin regions of input image 302.

[0072] Color gain parameter selector 322 may be configured to determine color gain parameter(s) 324 for any part of input image 302 based additionally on illumination segmentation mask 320. In cases where the entirety of input image 302 is illuminated using a single illumination source, color gain parameter(s) 324 may be based on the CCT of the single illumination source. In cases where different parts of input image 302 are illuminated using multiple different illumination sources, each having a different color temperature, the color gain param eter(s) of a respective part of input image 302 may be based on the CCT of the light by which the respective part is illuminated. For example, the color gain parameter(s) of a first background object illuminated by a first light source may be based on a first CCT of the first light source, while the color gain param eter(s) of a second background object illuminated by a second light source may be based on a second CCT of the second light source. The CCT of a light source may represent the closest point along the correlated color temperature curve that is nearest to a color of the light source and / or a distance in color space of the color of the light source from the closest point.

[0073] In some cases, a semantically-distinct portion of input image 302 may be partitioned by illumination segmentation mask 320 into two or more regions, each illuminated with light of a different CCT. For example, a first portion of the subject of input image 302 may be illuminated with light of a first CCT, while a second portion of the subject may be illuminated with light of a second CCT. Accordingly, corresponding color gain parameter(s) for the first portion of the subject may be based on the first CCT, while corresponding color gain parameter(s) for the second portion of the subject may be based on the second CCT. Thus, color gain parameter(s) 324 may vary spatially across the area of input image 302 based on both (i) the semantic content of input image 302 and (ii) illumination conditions represented in input image 302.

[0074] In some implementations, color gain parameter selector 322 may be configured to determine color gain parameter(s) 324 based additionally on one or more user preferences. For example, color gain parameter selector 322 may be configured to determine initial values of color gain parameter(s) 324 that accurately correct a white balance and / or color of different portions of input image 302. However, some users may prefer that output image 330 include certain color hues, tints, tones, and / or shades, rather than having output image 330 accurately depicting the color of the corresponding scene. Accordingly, color gain parameter selector 322 may be configured to generate user-preferred values of color gain parameter(s) 324 by adjust the initial values of color gain parameter(s) 324. The user preferences may be predefined based on feedback from a plurality of sample users, and / or may be adjustable by individual users.

[0075] Color gain parameter selector 322 may be configured to determine color gain parameter(s) 324 using one or more of predetermined mapping(s), heuristic(s), machine learning model(s), and / or other function(s). Thus, color gain parameter selector 322 may represent a function that, for any given one or more pixels of input image 302, maps segmentation mask values and / or visual statistics values associated with the given one or more pixels to corresponding color gain parameter(s) that, when applied to the given one or more pixels, improve a white balance and / or a color of the given one or more pixels.

[0076] Color adjuster 326 may be configured to apply color gain parameter(s) 324 to input image 302 and thereby generate output image 330. For example, color adjuster 326 may be configured to multiply a corresponding value of each respective pixel of input image 302 by a color gain parameter corresponding to the respective pixel, thus performing a spatially- varying adjustment of a white balance and / or color of input image 302. In some cases, color adjuster 326 may be configured to first modify a white balance of input image 302 by multiplying the pixels thereof with corresponding white balance matrices, and then modify acolor of the white balanced version of input image 302 by multiplying the white balanced pixels thereof with corresponding color correction matrices.IV. Example Images and Masks

[0077] Figure 4A illustrates an example input image 400 that may be processed by image processing system 300 to improve a white balance and / or a color thereof. Input image 400 includes subject 402 (e.g., a human), object 410, window 412, sky region 408, landscape region 406, and illumination source 404 (e.g., a light fixture). Input image 400 is one example of input image 302.

[0078] Input image 400 represents a semantically -variable scene in that each of subject 402, object 410, sky region 408, and landscape region 406 belongs to a different object class, and thus might benefit from a class-specific white balance correction and / or color correction. Further, input image 400 represents a mixed-illumination scene in that a first part of input image 400 is illuminated by natural light coming through window 412 while a second part of input image 400 is illuminated by the artificial light generated by illumination source 404. As discussed above, image processing system 300 may be configured to adjust the white balance and / or the color of input image 400 based on the semantic variability and / or the illumination variability thereof.

[0079] Figure 4B illustrates an example subject segmentation mask 420 corresponding to input image 400. Subject segmentation mask 420 represents subject region 422 occupied by subject 402 in input image 400 using white pixels, and represents the background region 424 (i.e., non-subject region) using black pixels. Subject segmentation mask 420 provides one example of subject segmentation mask 308.

[0080] Figure 4C illustrates an example skin segmentation mask 430 corresponding to input image 400. Skin segmentation mask 430 represents skin regions of subject 402 using white pixels, and non-skin regions using black pixels. Skin segmentation mask 430 provides one example of skin segmentation mask 312. Skin segmentation mask 430 represents face skin region 432, arm skin regions 434A and 434B, and leg skin regions 436A and 436B. In the illustrated example, the combination of skin regions 434A, 434B, 436A, and 436B includes at least the same number of pixels as face skin region 432. Thus, considering both (i) face skin region 432 and (ii) body skin regions 434A, 434B, 436A, and 436B at least double the number of skin pixels based on which visual statistics can be determined, thereby improving the accuracy of resulting white balance and / or color adjustments.

[0081] Figure 4D illustrates a combined subject-and-skin segmentation mask 440 corresponding to input image 400. Combined subject-and-skin segmentation mask 440represents skin regions of subject 402 using pixels with a dotted fill pattern, non-skin subject regions using white pixels, and non-subject regions using black pixels. Thus, combined subject- and-skin segmentation mask 440 represents the skin regions of skin segmentation mask 430 imposed over subject segmentation mask 420.

[0082] In some implementations, color gain parameter(s) determined based on the skin regions indicated by skin segmentation mask 430 may be applied to the skin regions, but not to the non-skin regions of subject segmentation mask 420. A separate set of color gain param eter(s) may be determined based on the non-skin regions of subject region 422, and may be applied to the non-skin regions of subject region 422. In other implementations, the color gain parameter(s) determined based on the skin regions of skin segmentation mask 430 may be applied to both the skin regions and the non-skin regions of subject segmentation mask 420 (e.g., to the entirety of subject region 422). By determining the color gain parameter(s) based on the skin regions, the visual appearance of skin regions may be prioritized, and by applying these color gain parameter(s) to the entirety of subject region 422, the visual appearance of the entirety of subject 402 may be uniformly adjusted and / or improved while reducing the amount of computing resources and / or energy expended in determining the color gain parameter(s).

[0083] Figure 4E illustrates a background segmentation mask 450 corresponding to input image 400. Background segmentation mask 450 represents region 454 occupied by object 410 using white pixels, region 458 occupied by sky region 408 using pixels with a hatched pattern, region 456 occupied by landscape region 406 using pixels with a dotted pattern, subject region 422 occupied by subject 402 (and / or representing the space behind subject 402) using black pixels, and region 452 occupied by the remainder of the scene (e.g., other portions of the environment in which subject 402 is located) using pixels with a diagonally-lined pattern. In some cases, subject region 422 may indicate that subject 402 is not part of the scene’s background, and / or may be omitted from background segmentation mask 450. Background segmentation mask 450 provides one example of background segmentation mask 316.

[0084] Figure 4F illustrates an illumination segmentation mask 460 corresponding to input image 400. Illumination segmentation mask 460 represents regions 462 (corresponding to window 412) and 464 (corresponding to a left half of subject 402) illuminated by natural light coming through window 412 using white pixels, and region 466 illuminated by illumination source 404 using black pixels. Region 466 includes portions of the background, object 410, and a right half of subject 402. Thus, illumination segmentation mask 460 partitions subject 402 into two regions, each illuminated by light of a different color temperature.Illumination segmentation mask 460 provides one example of illumination segmentation mask 320.

[0085] In some cases, the illumination segmentation mask may represent an illumination region that is associated with light (e.g., flash) emitted by the camera capturing an input image. For example, the subject and / or some objects in the scene may be illuminated by the light emitted by the camera and having a first color temperature, while portions of the background and / or other objects in the scene may be illuminated by another light source present in the scene (e.g., the sun, a light fixture, etc.) and having a second color temperature. Thus, the illumination segmentation mask may allow image processing system 300 to more accurately adjust the input image for color artifacts associated with using camera flash. In cases where the subject is primarily illuminated by the camera flash, while other portions of the scene are illuminated by another light source, the subject segmentation mask may indirectly operate to segment the scene based on illumination, and may thus be used by image processing system 300 to generate color gain param eter(s) that account for differences in illumination color temperature.

[0086] The pixels of segmentation masks 420, 430, 440, 450, and / or 460 may include binary values and / or grayscale values. The grayscale value of a given pixel may indicate a confidence value generated by a corresponding model and representing a likelihood of a corresponding portion of input image 400 representing a particular type of visual feature.

[0087] Figure 4G illustrates color gain parameter map 470 that represents a spatial distribution of the different color gain parameters determined for different portions of input image 400 based on the semantic and / or illumination information associated therewith. Specifically, color gain parameter map 470 includes region 472 (corresponding to sky region 408), region 474 (corresponding to landscape region 406), region 476 (corresponding to object 410), region 478 (corresponding to the left portion of subject 402 illuminated by the natural light coming through window 412), region 480 (corresponding to the right portion of subject 402 illuminated by illumination source 404), and region 482 (corresponding to the remainder of the background of input image 400).

[0088] Each of regions 472, 474, 476, 478, 480, and 482 may have associated therewith corresponding one or more white balance gain parameters (e.g., expressed as a white balance correction matrix and / or look-up table), corresponding one or more color correction parameters (e.g., expressed as a color correction matrix and / or look-up table), and / or corresponding one or more correlated color temperature parameters. The color gain parameters may be different for each of regions 472, 474, 476, 478, 480, and 482, thus allowing different regions of input image400 to be adjusted to different extents. Within a given region of regions 472, 474, 476, 478, 480, and 482, the color gain parameters may be the same so that semantically-consistent regions exposed to the same or similar illumination are adjusted in a consistent and / or uniform manner.V. Example Training System

[0089] Figure 5 illustrates training system 500 configured to train segmentation model 516 based on training input image 502 and ground-truth segmentation mask 504. Segmentation model 516 may represent foreground segmentation model 304, subject segmentation model 306, skin segmentation model 310, background segmentation model 314, and / or illumination segmentation model 318 of image processing system 300. Training system 500 may be configured to train segmentation model 516 to identify the pixels of training input image 502 that represent a particular visual feature (e.g., subject, skin, different background objects, illumination conditions, etc.), and / or associate with each respective pixel of training input image 502 a confidence value indicative of a likelihood that the respective pixel represents the particular visual feature.

[0090] Training system 500 may include loss function(s) 508 and model parameter adjuster 512. Segmentation model 516 may be configured to generate training segmentation mask 506 based on training input image 502. Training input image 502 may be analogous to input image 302, but may be processed at training time rather than at inference time. Training segmentation mask 506 may be analogous to subject segmentation mask 308, skin segmentation mask 312, background segmentation mask 316, and / or illumination segmentation mask 320 (depending on which visual features segmentation model 516 is configured to identify), but may be determined at training time rather than at inference time.

[0091] Loss function(s) 508 may be configured to generate loss value 510 based at least on training segmentation mask 506 and ground-truth segmentation mask 504. For example, loss function(s) 508 may include a mean squared error loss, a mean absolute errors loss, a cross entropy loss, and / or a Dice loss, among other possibilities. Thus, loss function(s) 508 may be configured to incentivize segmentation model 516 to generate training segmentation mask 506 that matches ground-truth segmentation mask 504.

[0092] Model parameter adjuster 512 may be configured to determine updated model parameters 514 based on loss value 510. Specifically, updated model parameters 514 may be selected such that, during a subsequent iteration of processing of training input image 502, training segmentation mask 506 more closely matches ground-truth segmentation mask 504.Updated model parameters 514 may include one or more updated parameters of any trainable component of segmentation model 516.

[0093] Model parameter adjuster 512 may be configured to determine updated model parameters 514 by, for example, determining a gradient of loss function(s) 508. Based on this gradient and loss value 510, model parameter adjuster 512 may be configured to select updated model parameters 514 that are expected to reduce loss value 510, and thus improve a performance of segmentation model 516. After applying updated model parameters 514 to segmentation model 516, the operations discussed above may be repeated to compute another instance of loss value 510 and, based thereon, another instance of updated model parameters 514 may be determined and applied to segmentation model 516 to further improve the performance thereof. Such training of segmentation model 516 may be repeated until, for example, loss value 510 is reduced to below a target loss value.VI. Additional Example Operations

[0094] Figure 6 illustrates a flow chart of operations related to using segmentation masks to perform a white balance adjustment and / or color adjustment of an image. The operations may be carried out by computing device 100, computing system 200, and / or image processing system 300, among other possibilities. The embodiments of Figure 6 may be simplified by the removal of any one or more of the features shown therein. Further, these embodiments may be combined with features, aspects, and / or implementations of any of the previous figures or otherwise described herein.

[0095] Block 600 may involve obtaining an image depicting a subject.

[0096] Block 602 may involve determining, based on the image, a skin segmentation mask representing skin regions of the subject as depicted in the image. The skin regions may include (i) a face skin region on a face of the subject and (ii) a body skin region on at least one body part of the subject.

[0097] Block 604 may involve determining, based on a plurality of skin pixels of the image corresponding to the skin regions as represented by the skin segmentation mask, one or more color gain parameters for the image.

[0098] Block 606 may involve adjusting a pixel value of a skin pixel of the plurality of skin pixels of the image based on the one or more color gain parameters.

[0099] In some examples, a subject segmentation mask representing a subject region of the image occupied by the subject may be determined based on the image. The subject region may include the skin regions and a non-skin region of the subject. Adjusting the pixel value ofthe skin pixel may include adjusting a corresponding pixel value of each respective subject pixel of a plurality of subject pixels in the subject region of the image.

[0100] In some examples, the skin segmentation mask may be determined based on the subject segmentation mask. The skin regions may form a subset of the subject region.

[0101] In some examples, the one or more color gain parameters may be (i) determined independently of non-skin pixels of the non-skin region of the subject and (ii) used to adjust a plurality of non-skin pixels in the non-skin region.

[0102] In some examples, determining the subject segmentation mask may include determining a first subject segmentation mask having a first resolution by processing the image using a machine learning model configured to detect the subject region, and determining a second subject segmentation mask having a second resolution larger than the first resolution by upsampling the first subject segmentation mask.

[0103] In some examples, determining the one or more color gain parameters may include determining that the face skin region includes less than a threshold number of pixels and, based on determining that the face skin region comprises less than the threshold number of pixels, determining the one or more color gain parameters based on (i) one or more face skin pixels of the image corresponding to the face skin region and (i) one or more body skin pixels of the image corresponding to the body skin region.

[0104] In some examples, a sum of the one or more face skin pixels and the one or more body skin pixels may exceed the threshold number of pixels.

[0105] In some examples, a background segmentation mask may be determined based on the image. The background segmentation mask may represent, for each respective object class of one or more object classes depicted in a first background region of the image, a corresponding object region of the image occupied by the respective object class. For each respective object class of the one or more object classes, corresponding one or more classbased color gain parameters may be determined based on a plurality of object pixels of the corresponding object region of the respective object class. For each respective object class of the one or more object classes, a respective pixel value of a respective object pixel of the corresponding object region may be adjusted based on the corresponding one or more classbased color gain parameters.

[0106] In some examples, an illumination segmentation mask may be determined based on the image. The illumination segmentation mask may represent a plurality of illumination regions in the image. Each respective illumination region may be illuminated by light of a corresponding color temperature. The one or more color gain parameters may be determinedbased on a corresponding color temperature of at least one illumination region of the illumination segmentation mask.

[0107] In some examples, the illumination segmentation mask may partition the skin regions into one or more (e.g., two or more) skin illumination regions each illuminated by light of a different color temperature. Determining the one or more color gain parameters may include determining, for each respective skin illumination region of the one or more skin illumination regions, corresponding one or more color gain parameters based on the corresponding color temperature of the respective skin illumination region. Adjusting the pixel value of the skin pixel may include adjusting, for each respective skin illumination region of the one or more skin illumination regions, a corresponding pixel value of a respective skin pixel of the respective skin illumination region based on the corresponding one or more color gain parameters.

[0108] In some examples, the illumination segmentation mask may partition a second background region around the subject into one or more (e.g., two or more) background illumination regions each illuminated by light of a different color temperature. For each respective background illumination region of the one or more background illumination regions, corresponding one or more background color gain parameters may be determined based on the corresponding color temperature of the respective background illumination region. For each respective background illumination region of the one or more background illumination regions, a respective pixel value of a respective background pixel of the respective background illumination region may be adjusted based on the corresponding one or more background color gain parameters.

[0109] In some examples, the plurality of illumination regions represented by the illumination segmentation mask may include (i) a camera flash region illuminated by light emitted from a camera flash source that and (ii) a scene light source region illuminated by a light source present in a scene represented by the input image.

[0110] In some examples, determining the one or more color gain parameters may include determining, based on the plurality of skin pixels, a skin tone of the skin regions, and determining the one or more color gain parameters based on the skin tone.[OHl] In some examples, determining the skin segmentation mask may include determining a first skin segmentation mask having a first resolution by processing the image using a machine learning model configured to detect the skin regions, and determining a second skin segmentation mask having a second resolution larger than the first resolution by upsampling the first skin segmentation mask.

[0112] In some examples, determining the one or more color gain parameters may include determining, based on the plurality of skin pixels, one or more initial color gain parameters for the image, and determining the one or more color gain parameters by adjusting the one or more initial color gain parameters based on one or more user preferences corresponding to the skin regions.

[0113] In some examples, the one or more color gain parameters may include a white balance gain parameter configured to adjust a white balance of the plurality of skin pixels.

[0114] In some examples, the one or more color gain parameters may include a color correction gain parameter configured to adjust a color of the plurality of skin pixels.

[0115] In some examples, the one or more color gain parameters may be determined further based on one or more of: (i) Bayer grid statistics associated with the plurality of skin pixels, (ii) one or more brightness values associated with the image, (iii) output of a spectral sensor, or (iv) a classification of a scene represented by the image.

[0116] In some examples, the image may depict two or more subjects. The skin segmentation mask may (i) represent, for each respective subject of the two or more subjects, corresponding skin regions of the respective subject as depicted in the image and (ii) distinguish the corresponding skin regions of different subjects. The corresponding skin regions of each respective subject may include (i) a corresponding face skin region on a corresponding face of the respective subject and (ii) a corresponding body skin region on at least one corresponding body part of the respective subject. Determining the one or more color gain parameters may include determining, for each respective subject of the two or more subjects, corresponding one or more color gain parameters for the respective subject based on a corresponding plurality of skin pixels of the corresponding skin regions of the respective subject. Adjusting the pixel value of the skin pixel may include adjusting, for each respective subject of the two or more subjects, a corresponding pixel value of a corresponding skin pixel of the corresponding plurality of skins pixels of the respective subject based on the corresponding one or more color gain parameters for the respective subject.VII. Conclusion

[0117] The present disclosure is not to be limited in terms of the particular embodiments described in this application, which are intended as illustrations of various aspects. Many modifications and variations can be made without departing from its scope, as will be apparent to those skilled in the art. Functionally equivalent methods and apparatuses within the scope of the disclosure, in addition to those described herein, will be apparent tothose skilled in the art from the foregoing descriptions. Such modifications and variations are intended to fall within the scope of the appended claims.

[0118] The above detailed description describes various features and operations of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The example embodiments described herein and in the figures are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

[0119] With respect to any or all of the message flow diagrams, scenarios, and flow charts in the figures and as discussed herein, each step, block, and / or communication can represent a processing of information and / or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages can be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved. Further, more or fewer blocks and / or operations can be used with any of the message flow diagrams, scenarios, and flow charts discussed herein, and these message flow diagrams, scenarios, and flow charts can be combined with one another, in part or in whole.

[0120] A step or block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical operations or actions in the method or technique. The program code and / or related data may be stored on any type of computer readable medium such as a storage device including random access memory (RAM), a disk drive, a solid state drive, or another storage medium.

[0121] The computer readable medium may also include non-transitory computer readable media such as computer readable media that store data for short periods of time like register memory, processor cache, and RAM. The computer readable media may also includenon-transitory computer readable media that store program code and / or data for longer periods of time. Thus, the computer readable media may include secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, solid state drives, compactdisc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.

[0122] Moreover, a step or block that represents one or more information transmissions may correspond to information transmissions between software and / or hardware modules in the same physical device. However, other information transmissions may be between software modules and / or hardware modules in different physical devices.

[0123] The particular arrangements shown in the figures should not be viewed as limiting. It should be understood that other embodiments can include more or less of each element shown in a given figure. Further, some of the illustrated elements can be combined or omitted. Yet further, an example embodiment can include elements that are not illustrated in the figures.

[0124] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purpose of illustration and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method comprising: obtaining an image depicting a subject; determining, based on the image, a skin segmentation mask representing skin regions of the subject as depicted in the image, wherein the skin regions comprise (i) a face skin region on a face of the subject and (ii) a body skin region on at least one body part of the subject; determining, based on a plurality of skin pixels of the image corresponding to the skin regions as represented by the skin segmentation mask, one or more color gain parameters for the image; and adjusting a pixel value of a skin pixel of the plurality of skin pixels of the image based on the one or more color gain parameters.

2. The computer-implemented method of claim 1, further comprising: determining, based on the image, a subject segmentation mask representing a subject region of the image occupied by the subject, wherein the subject region includes the skin regions and a non-skin region of the subject, and wherein adjusting the pixel value of the skin pixel comprises: adjusting a corresponding pixel value of each respective subject pixel of a plurality of subject pixels in the subject region of the image.

3. The computer-implemented method of claim 2, wherein the skin segmentation mask is determined based on the subject segmentation mask, and wherein the skin regions form a subset of the subject region.

4. The computer-implemented method of any one of claims 2-3, wherein the one or more color gain parameters are (i) determined independently of non-skin pixels of the non-skin region of the subject and (ii) used to adjust a plurality of non-skin pixels in the non-skin region.

5. The computer-implemented method of any of claims 1-4, wherein determining the one or more color gain parameters comprises: determining that the face skin region comprises less than a threshold number of pixels; andbased on determining that the face skin region comprises less than the threshold number of pixels, determining the one or more color gain parameters based on (i) one or more face skin pixels of the image corresponding to the face skin region and (i) one or more body skin pixels of the image corresponding to the body skin region.

6. The computer-implemented method of claim 5, wherein a sum of the one or more face skin pixels and the one or more body skin pixels exceeds the threshold number of pixels.

7. The computer-implemented method of any of claims 1-6, further comprising: determining, based on the image, a background segmentation mask representing, for each respective object class of one or more object classes depicted in a first background region of the image, a corresponding object region of the image occupied by the respective object class; determining, for each respective object class of the one or more object classes, corresponding one or more class-based color gain parameters based on a plurality of object pixels of the corresponding object region of the respective object class; and adjusting, for each respective object class of the one or more object classes, a respective pixel value of a respective object pixel of the corresponding object region based on the corresponding one or more class-based color gain parameters.

8. The computer-implemented method of any of claims 1-7, further comprising: determining, based on the image, an illumination segmentation mask representing a plurality of illumination regions in the image, wherein each respective illumination region is illuminated by light of a corresponding color temperature, and wherein the one or more color gain parameters are determined based on a corresponding color temperature of at least one illumination region of the illumination segmentation mask.

9. The computer-implemented method of claim 8, wherein: the illumination segmentation mask partitions the skin regions into one or more skin illumination regions each illuminated by light of a different color temperature, determining the one or more color gain parameters comprises determining, for each respective skin illumination region of the one or more skin illumination regions, corresponding one or more color gain parameters based on the corresponding color temperature of the respective skin illumination region, andadjusting the pixel value of the skin pixel comprises adjusting, for each respective skin illumination region of the one or more skin illumination regions, a corresponding pixel value of a respective skin pixel of the respective skin illumination region based on the corresponding one or more color gain parameters.

10. The computer-implemented method of any of claims 8-9, wherein the illumination segmentation mask partitions a second background region around the subject into one or more background illumination regions each illuminated by light of a different color temperature, and wherein the method further comprises: determining, for each respective background illumination region of the one or more background illumination regions, corresponding one or more background color gain parameters based on the corresponding color temperature of the respective background illumination region; and adjusting, for each respective background illumination region of the one or more background illumination regions, a respective pixel value of a respective background pixel of the respective background illumination region based on the corresponding one or more background color gain parameters.

11. The computer-implemented method of any of claims 8-10, wherein the plurality of illumination regions represented by the illumination segmentation mask comprise (i) a camera flash region illuminated by light emitted from a camera flash source that and (ii) a scene light source region illuminated by a light source present in a scene represented by the input image.

12. The computer-implemented method of any of claims 1-11, wherein determining the one or more color gain parameters comprises: determining, based on the plurality of skin pixels, a skin tone of the skin regions; and determining the one or more color gain parameters based on the skin tone.

13. The computer-implemented method of any of claims 1-12, wherein determining the skin segmentation mask comprises: determining a first skin segmentation mask having a first resolution by processing the image using a machine learning model configured to detect the skin regions; and determining a second skin segmentation mask having a second resolution larger than the first resolution by upsampling the first skin segmentation mask.

14. The computer-implemented method of any of claims 1-13, wherein determining the one or more color gain parameters comprises: determining, based on the plurality of skin pixels, one or more initial color gain parameters for the image; and determining the one or more color gain parameters by adjusting the one or more initial color gain parameters based on one or more user preferences corresponding to the skin regions.

15. The computer-implemented method of any of claims 1-14, wherein the one or more color gain parameters comprise a white balance gain parameter configured to adjust a white balance of the plurality of skin pixels.

16. The computer-implemented method of any of claims 1-15, wherein the one or more color gain parameters comprise a color correction gain parameter configured to adjust a color of the plurality of skin pixels.

17. The computer-implemented method of any of claims 1-16, wherein the one or more color gain parameters are determined further based on one or more of:Bayer grid statistics associated with the plurality of skin pixels; one or more brightness values associated with the image; output of a spectral sensor; or a classification of a scene represented by the image.

18. The computer-implemented method of any of claims 1-17, wherein: the image depicts two or more subjects; the skin segmentation mask (i) represents, for each respective subject of the two or more subjects, corresponding skin regions of the respective subject as depicted in the image and (ii) distinguishes the corresponding skin regions of different subjects; the corresponding skin regions of each respective subject comprise (i) a corresponding face skin region on a corresponding face of the respective subject and (ii) a corresponding body skin region on at least one corresponding body part of the respective subject; determining the one or more color gain parameters comprises determining, for each respective subject of the two or more subjects, corresponding one or more color gainparameters for the respective subject based on a corresponding plurality of skin pixels of the corresponding skin regions of the respective subject; and adjusting the pixel value of the skin pixel comprises adjusting, for each respective subject of the two or more subjects, a corresponding pixel value of a corresponding skin pixel of the corresponding plurality of skins pixels of the respective subject based on the corresponding one or more color gain parameters for the respective subject.

19. A system comprising: a processor; and a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform the method of any of claims 1-18.

20. A non-transitory computer-readable medium having stored thereon instructions that, when executed by a computing device, cause the computing device to perform the method of any of claims 1-18.

Images