Human segmentation based exposure improvements

Abstract

A method includes obtaining a first image depicting a subject. The method further includes determining, based on the first image, a subject segmentation mask representing regions of the subject as depicted in the first image. The method additionally includes determining, based on the subject segmentation mask, one or more exposure gain parameters for the first, image. The method also includes obtaining a second image depicting the subject. The method further includes, based on determining that the second image omits a particular region of the subject, applying the one or more exposure gain parameters for the first image to the second image, wherein the particular region is included in the first image.

Description

Human Segmentation Based Exposure Improvements BACKGROUND

[0001] Many modern computing devices, including mobile phones, personal computers, and tablets, include image capturing devices. Some image capturing devices are configured with telephoto capabilities. SUMMARY

[0002] In an embodiment, a method includes obtaining a first image depicting a subject. The method further includes determining, based on the first image, a subject segmentation mask representing regions of the subject as depicted in the first image. The method additionally includes determining, based on the subject segmentation mask, one or more exposure gain parameters for the first image. The method also includes obtaining a second image depicting the subject. The method further includes, based on determining that the second image omits a particular region of the subject, applying the one or more exposure gain parameters for the first image to the second image, wherein the particular region is included in the first image.

[0003] In another embodiment, a system includes a processor and a non-transitory computer- readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations. The operations include obtaining a first image depicting a subject. The operations also include determining, based on the first image, a subject segmentation mask representing regions of the subject as depicted in the first image. The operations additionally include determining, based on the subject segmentation mask, one or more exposure gain parameters for the first image. The operations further include obtaining a second image depicting the subject. The operations also include, based on determining that the second image omits a particular region of the subject, applying the one or more exposure gain parameters for the first image to the second image, wherein the particular region is included in the first image.

[0004] In a further embodiment, a non-transitory computer readable medium stores program instructions that, when executed by a computing device, cause the computing device to perform operations. The operations include obtaining a first image depicting a subject. The operations also include determining, based on the first image, a subject segmentation mask representing regions of the subject as depicted in the first image. The operations also include determining, based on the subject segmentation mask, one or more exposure gain parameters for the first image. The operations additionally include obtaining a second image depicting the subject. The operations also include, based on determining that the second image omits a particular regionof the subject, applying the one or more exposure gain parameters for the first image to the second image, wherein the particular region is included in the first image.

[0005] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the figures and the following detailed description and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 illustrates an example computing device, in accordance with example embodiments.

[0007] Figure 2 is a simplified block diagram showing some of the components of an example computing system.

[0008] Figure 3 is a diagram illustrating a training phase and an inference phase of one or more trained machine learning models in accordance with example embodiments.

[0009] Figure 4 is a flow chart of a method, in accordance with example embodiments.

[0010] Figure 5 depicts a first image, in accordance with example embodiments.

[0011] Figure 6 depicts a first segmentation mask, in accordance with example embodiments.

[0012] Figure 7 depicts a second image, in accordance with example embodiments.

[0013] Figure 8 depicts a second segmentation mask, in accordance with example embodiments. DETAILED DESCRIPTION

[0014] Example methods, devices, and systems are described herein. It should be understood that the words “example” and “exemplary” are used herein to mean “serving as an example, instance, or illustration.” Any embodiment or feature described herein as being an “example” or “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or features unless indicated as such. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein.

[0015] Thus, the example embodiments described herein are not meant to be limiting. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

[0016] Throughout this description, the articles “a” or “an” are used to introduce elements of the example embodiments. Any reference to “a” or “an” refers to “at least one,” and any reference to “the” refers to “the at least one,” unless otherwise specified, or unless the contextclearly dictates otherwise. The intent of using the conjunction “or” within a described list of at least two terms is to indicate any of the listed terms or any combination of the listed terms.

[0017] The use of ordinal numbers such as “first,” “second,” “third” and so on is to distinguish respective elements rather than to denote a particular order of those elements. For the purpose of this description, the terms “multiple” and “a plurality of” refer to “two or more” or “more than one.”

[0018] Further, unless context suggests otherwise, the features illustrated in each of the figures may be used in combination with one another. Thus, the figures should be generally viewed as component aspects of one or more overall embodiments, with the understanding that not all illustrated features are necessary for each embodiment. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. Further, unless otherwise noted, figures are not drawn to scale and are used for illustrative purposes only. Moreover, the figures are representational only and not all components are shown. For example, additional structural or restraining components might not be shown.

[0019] Additionally, any enumeration of elements, blocks, or steps in this specification or the claims is for purposes of clarity. Thus, such enumeration should not be interpreted to require or imply that these elements, blocks, or steps adhere to a particular arrangement or are carried out in a particular order. I. Overview

[0020] A computing system may be configured to adjust exposure gains to cause an image to appear more visually pleasing, to make details of the image more apparent, improve the contrast between different parts of the image, and / or to cause the image to appear more similar to how the scene represented by the image would be perceived by a human. The exposure gains may include digital gain values that are applied to digital pixel values of an image. A computing system may adjust an image to increase and / or decrease the exposure of the image by the amount indicated by the exposure gain. For example, the computing system may multiply the digital pixel values by corresponding exposure gains.

[0021] Adjusting exposure gains of an image may improve lighting in an image. For instance, the image may appear washed out if the exposure is too high. The computing system may apply an exposure gain (e.g., exposure gain < 1) configured to reduce the amount of light represented by the image, which may help extract details of the image that were previously washed out. Alternatively, an image may appear too dark if the exposure is too low. The computing system may apply another exposure gain (e.g., exposure gain > 1) configured to increase the brightnessof the image, which may help extract details of the image that were previously too dark to be apparent.

[0022] However, in some examples, adjusting exposure of an image to correct the brightness of some image portions may cause other areas of the image to be too dark and / or too bright. For example, for an image of a person in front of a window, the window may initially be well exposed and the person may be underexposed. After adjusting the exposure, the person may be well exposed, while the window may become overexposed. Further, in some examples, the computing system may be unable to determine the main subject of the image to which the exposure should be calibrated. For instance, if only the body of a person is in the image (and the face is not), the computing system may be unable to determine the area for which the exposure should be calculated and / or the computing system may be unable to determine the exposure gain parameters to apply to the image.

[0023] Provided herein are methods of improving the exposure of an image by applying varying exposure gain parameters across the image and / or by selectively applying previously determined exposure gain parameters. In particular, the computing system may determine a segmentation mask based on the image, which may be used to determine exposure gain parameters. The segmentation mask may represent a spatial variability of the semantic information in the image, and may thus provide information for determining spatially varying exposure gain parameters. Exposure gain parameters may be considered spatially varying where different pixels of a given image are adjusted using different exposure gain parameters. The computing system may then apply the determined exposure gain parameters to the image. Additionally or alternatively, the computing system may store the exposure gain parameters determined based on a first image and use the stored exposure gain parameters for a second image based on detecting that a particular region and / or visual feature is not included in the second image.

[0024] For example, the computing system may receive an image that depicts a person in front of a window. The computing system may apply a machine learning model and / or other algorithm to determine a segmentation mask based on the image. The segmentation mask may identify a plurality of regions and corresponding classifications thereof. For example, the segmentation mask may identify one or more subject regions, including one or more skin regions in the image that depict a person’s skin and / or one or more non-skin regions in the image that depict the non-skin parts of the person (e.g., a shirt, a hat, pants, etc.). In some examples, the skin region may represent visible skin on at least one body part of a subject depicted in the image, and the subject region may represent both skin and non-skin parts of thesubject (e.g., a person’s hair covering parts of their face or a shirt that is worn by a person in the image).

[0025] Based on the segmentation mask, the computing system may determine exposure gain parameters for the image by determining an average weight of regions with a particular classification. For example, for a segmentation mask that includes one or more skin regions and one or more non-skin regions, the computing system may determine the average weight of skin regions by averaging the probability that each skin region depicts skin, and the computing system may determine the average weight of the non-skin regions by averaging the probability that each non-skin region depicts a non-skin region. After having determined the average weights for the regions, the computing system may determine the luma values for each classification of regions based on the determined weights of each classification of regions and one or more visual statistics associated with the regions.

[0026] Based on the luma values, the computing system may adjust the exposure gain parameters. In particular, the computing system may increase and / or decrease the exposure of the regions associated with the subject of the image. For example, if the subject of the image is a person, the computing system may adjust the exposure of regions of the image that are classified as skin based on a skin luma value and the regions of the image that are classified as non-skin based on a non-skin luma value. The computing system may also store the exposure gain parameters for use in a subsequent image.

[0027] In some examples, the computing system may determine whether to use the stored exposure gain parameters or determine new exposure gain parameters based on whether an image represents a particular region. For example, the computing system may determine whether a first image includes a particular region (e.g., head, face, etc.) of a subject represented by the first image. If the computing system determines that the first image includes the particular region, the computing system may apply the process above. For a second, perhaps subsequent, image, the computing system may determine that the second image does not include the particular region of the subject represented in the first image. Accordingly, the computing system may apply the exposure gain parameters used in the first image in lieu of determining new exposure gain parameters for the second image.

[0028] The computing system may also continue to apply the exposure gain parameters used in the first image in lieu of determining new exposure gain parameters in additional images after the first and second image. The exposure may be locked until the computing system determines that the scene depicted in the final image is different from the scene depicted in thefirst image, perhaps by comparing one or more factors determined for the first image with one or more factors determined for the final image.

[0029] Applying previously determined exposure gain parameters when a particular region is not detected in the second image may help maintain improved and / or optimal exposure. In particular, for an image that includes a person as the subject, the particular region may be a face of the person. Exposure gain parameters determined when a face of the person is depicted in the image may be more accurate than exposure gain parameters determined when a face of the person is not depicted in the image. Accordingly, the computing system may apply the previously determined exposure gain parameters to an image that does not depict the face of the person, thereby improving the visual quality of the image on the basis of information available in other temporally-proximate images. II.Example Systems and Methods

[0030] Figure 1 illustrates an example computing device 100. In examples described herein, computing device 100 may be an image capturing device and / or a video capturing device. Computing device 100 is shown in the form factor of a mobile phone. However, computing device 100 may be alternatively implemented as a laptop computer, a tablet computer, and / or a wearable computing device, among other possibilities. Computing device 100 may include various elements, such as body 102, display 106, and buttons 108 and 110. Computing device 100 may further include one or more cameras, such as front-facing camera 104 and at least one rear-facing camera 112. In examples with multiple rear-facing cameras such as illustrated in Figure 1, each of the rear-facing cameras may have a different field of view. For example, the rear facing cameras may include a wide angle camera, a main camera, and a telephoto camera. The wide angle camera may capture a larger portion of the environment compared to the main camera and the telephoto camera, and the telephoto camera may capture more detailed images of a smaller portion of the environment compared to the main camera and the wide angle camera.

[0031] Front-facing camera 104 may be positioned on a side of body 102 typically facing a user while in operation (e.g., on the same side as display 106). Rear-facing camera 112 may be positioned on a side of body 102 opposite front-facing camera 104. Referring to the cameras as front and rear facing is arbitrary, and computing device 100 may include multiple cameras positioned on various sides of body 102.

[0032] Display 106 could represent a cathode ray tube (CRT) display, a light emitting diode (LED) display, a liquid crystal (LCD) display, a plasma display, an organic light emitting diode (OLED) display, or any other type of display known in the art. In some examples, display 106may display a digital representation of the current image being captured by front-facing camera 104 and / or rear-facing camera 112, an image that could be captured by one or more of these cameras, an image that was recently captured by one or more of these cameras, and / or a modified version of one or more of these images. Thus, display 106 may serve as a viewfinder for the cameras. Display 106 may also support touchscreen functions that may be able to adjust the settings and / or configuration of one or more aspects of computing device 100.

[0033] Front-facing camera 104 may include an image sensor and associated optical elements such as lenses. Front-facing camera 104 may offer zoom capabilities or could have a fixed focal length. In other examples, interchangeable lenses could be used with front-facing camera 104. Front-facing camera 104 may have a variable mechanical aperture and a mechanical and / or electronic shutter. Front-facing camera 104 also could be configured to capture still images, video images, or both. Further, front-facing camera 104 could represent, for example, a monoscopic, stereoscopic, or multiscopic camera. Rear-facing camera 112 may be similarly or differently arranged. Additionally, one or more of front-facing camera 104 and / or rear-facing camera 112 may be an array of one or more cameras.

[0034] One or more of front-facing camera 104 and / or rear-facing camera 112 may include or be associated with an illumination component that provides a light field to illuminate a target object. For instance, an illumination component could provide flash or constant illumination of the target object. An illumination component could also be configured to provide a light field that includes one or more of structured light, polarized light, and light with specific spectral content. Other types of light fields known and used to recover three-dimensional (3D) models from an object are possible within the context of the examples herein.

[0035] Computing device 100 may also include an ambient light sensor that may continuously or from time to time determine the ambient brightness of a scene that cameras 104 and / or 112 can capture. In some implementations, the ambient light sensor can be used to adjust the display brightness of display 106. Additionally, the ambient light sensor may be used to determine an exposure length of one or more of cameras 104 or 112, or to help in this determination.

[0036] Computing device 100 could be configured to use display 106 and front-facing camera 104 and / or rear-facing camera 112 to capture images of a target object. The captured images could be a plurality of still images or a video stream. The image capture could be triggered by activating button 108, pressing a softkey on display 106, or by some other mechanism. Depending upon the implementation, the images could be captured automatically at a specific time interval, for example, upon pressing button 108, upon appropriate lighting conditions ofthe target object, upon moving computing device 100 a predetermined distance, or according to a predetermined capture schedule.

[0037] Figure 2 is a simplified block diagram showing some of the components of an example computing system 200, such as an image capturing device and / or a video capturing device. By way of example and without limitation, computing system 200 may be a cellular mobile telephone (e.g., a smartphone), a computer (such as a desktop, notebook, tablet, server, or handheld computer), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, a gaming console, a robotic device, a vehicle, or some other type of device. Computing system 200 may represent, for example, aspects of computing device 100.

[0038] As shown in Figure 2, computing system 200 may include communication interface 202, user interface 204, processor 206, data storage 208, and camera components 224, all of which may be communicatively linked together by a system bus, network, or other connection mechanism 210. Computing system 200 may be equipped with at least some image capture and / or image processing capabilities. It should be understood that computing system 200 may represent a physical image processing system, a particular physical hardware platform on which an image sensing and / or processing application operates in software, or other combinations of hardware and software that are configured to carry out image capture and / or processing functions.

[0039] Communication interface 202 may allow computing system 200 to communicate, using analog or digital modulation, with other devices, access networks, and / or transport networks. Thus, communication interface 202 may facilitate circuit-switched and / or packet-switched communication, such as plain old telephone service (POTS) communication and / or Internet protocol (IP) or other packetized communication. For instance, communication interface 202 may include a chipset and antenna arranged for wireless communication with a radio access network or an access point. Also, communication interface 202 may take the form of or include a wireline interface, such as an Ethernet, Universal Serial Bus (USB), or High-Definition Multimedia Interface (HDMI) port, among other possibilities. Communication interface 202 may also take the form of or include a wireless interface, such as a Wi-Fi, BLUETOOTH®, global positioning system (GPS), or wide-area wireless interface (e.g., WiMAX or 3GPP Long- Term Evolution (LTE)), among other possibilities. However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used over communication interface 202. Furthermore, communication interface 202 may comprisemultiple physical communication interfaces (e.g., a Wi-Fi interface, a BLUETOOTH® interface, and a wide-area wireless interface).

[0040] User interface 204 may function to allow computing system 200 to interact with a human or non-human user, such as to receive input from a user and to provide output to the user. Thus, user interface 204 may include input components such as a keypad, keyboard, touch-sensitive panel, computer mouse, trackball, joystick, microphone, and so on. User interface 204 may also include one or more output components such as a display screen, which, for example, may be combined with a touch-sensitive panel. The display screen may be based on CRT, LCD, LED, and / or OLED technologies, or other technologies now known or later developed. User interface 204 may also be configured to generate audible output(s), via a speaker, speaker jack, audio output port, audio output device, earphones, and / or other similar devices. User interface 204 may also be configured to receive and / or capture audible utterance(s), noise(s), and / or signal(s) by way of a microphone and / or other similar devices.

[0041] In some examples, user interface 204 may include a display that serves as a viewfinder for still camera and / or video camera functions supported by computing system 200. Additionally, user interface 204 may include one or more buttons, switches, knobs, and / or dials that facilitate the configuration and focusing of a camera function and the capturing of images. It may be possible that some or all of these buttons, switches, knobs, and / or dials are implemented by way of a touch-sensitive panel.

[0042] Processor 206 may comprise one or more general purpose processors – e.g., microprocessors – and / or one or more special purpose processors – e.g., digital signal processors (DSPs), graphics processing units (GPUs), floating point units (FPUs), network processors, or application-specific integrated circuits (ASICs). In some instances, special purpose processors may be capable of image processing, image alignment, and merging images, among other possibilities. Data storage 208 may include one or more volatile and / or non-volatile storage components, such as magnetic, optical, flash, or organic storage, and may be integrated in whole or in part with processor 206. Data storage 208 may include removable and / or non-removable components.

[0043] Processor 206 may be capable of executing program instructions 218 (e.g., compiled or non-compiled program logic and / or machine code) stored in data storage 208 to carry out the various functions described herein. Therefore, data storage 208 may include a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by computing system 200, cause computing system 200 to carry out any of the methods, processes, or operations disclosed in this specification and / or the accompanying drawings. Theexecution of program instructions 218 by processor 206 may result in processor 206 using data 212.

[0044] By way of example, program instructions 218 may include an operating system 222 (e.g., an operating system kernel, device driver(s), and / or other modules) and one or more application programs 220 (e.g., camera functions, address book, email, web browsing, social networking, audio-to-text functions, text translation functions, and / or gaming applications) installed on computing system 200. Similarly, data 212 may include operating system data 216 and application data 214. Operating system data 216 may be accessible primarily to operating system 222, and application data 214 may be accessible primarily to one or more of application programs 220. Application data 214 may be arranged in a file system that is visible to or hidden from a user of computing system 200.

[0045] Application programs 220 may communicate with operating system 222 through one or more application programming interfaces (APIs). These APIs may facilitate, for instance, application programs 220 reading and / or writing application data 214, transmitting or receiving information via communication interface 202, receiving and / or displaying information on user interface 204, and so on.

[0046] In some cases, application programs 220 may be referred to as “apps” for short. Additionally, application programs 220 may be downloadable to computing system 200 through one or more online application stores or application markets. However, application programs can also be installed on computing system 200 in other ways, such as via a web browser or through a physical interface (e.g., a USB port) on computing system 200.

[0047] Camera components 224 may include, but are not limited to, an aperture, shutter, recording surface (e.g., photographic film and / or an image sensor), lens, shutter button, infrared projectors, and / or visible-light projectors. Camera components 224 may include components configured for capturing of images in the visible-light spectrum (e.g., electromagnetic radiation having a wavelength of 380 - 700 nanometers) and / or components configured for capturing of images in the infrared light spectrum (e.g., electromagnetic radiation having a wavelength of 701 nanometers - 1 millimeter), among other possibilities. Camera components 224 may be controlled at least in part by software executed by processor 206.

[0048] Histogram processing algorithm(s) 226 may include one or more stored algorithms programmed to process histogram information to facilitate autofocus as described herein. In some examples, histogram processing algorithm(s) 226 may include one or more trained machine learning models. In other examples, histogram processing algorithm(s) 226 may bebased on heuristics without the use of machine learning. In further examples, a combination of different types of histogram processing algorithm(s) 226 may be used as well.

[0049] In further examples, one or more remote cameras 230 may be controlled by computing system 200. For instance, computing system 200 may transmit control signals to the one or more remote cameras 230 through a wireless or wired connection. Such signals may be transmitted as part of an ambient computing environment. In such examples, inputs received at the computing system 200 (for instance, physical movements of a wearable device) may be mapped to movements or other functions of the one or more remote cameras 230. Images captured by the one or more remote cameras 230 may be transmitted to the computing system 200 for further processing. Such images may be treated as images captured by cameras physically located on the computing system 200.

[0050] Figure 3 shows diagram 300 illustrating a training phase 302 and an inference phase 304 of trained machine learning model(s) 332, in accordance with example embodiments. Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions about (patterns in the) training data. The resulting trained machine learning algorithm can be termed as a trained machine learning model. For example, Figure 3 shows training phase 302 where one or more machine learning algorithms 320 are being trained on training data 310 to become trained machine learning model 332. Producing trained machine learning model(s) 332 during training phase 302 may involve determining one or more hyperparameters, such as one or more stride values for one or more layers of a machine learning model as described herein. Then, during inference phase 304, trained machine learning model 332 can receive input data 330 and one or more inference / prediction requests 340 (perhaps as part of input data 330) and responsively provide as an output one or more inferences and / or predictions 350. The one or more inferences and / or predictions 350 may be based in part on one or more learned hyperparameters, such as one or more learned stride values for one or more layers of a machine learning model as described herein

[0051] As such, trained machine learning model(s) 332 can include one or more models of one or more machine learning algorithms 320. Machine learning algorithm(s) 320 may include, but are not limited to: an artificial neural network (e.g., a herein-described convolutional neural networks, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system). Machine learningalgorithm(s) 120 may be supervised or unsupervised, and may implement any suitable combination of online and offline learning.

[0052] In some examples, machine learning algorithm(s) 320 and / or trained machine learning model(s) 332 can be accelerated using on-device coprocessors, such as graphic processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and / or application specific integrated circuits (ASICs). Such on-device coprocessors can be used to speed up machine learning algorithm(s) 320 and / or trained machine learning model(s) 332. In some examples, trained machine learning model(s) 332 can be trained, reside and execute to provide inferences on a particular computing device, and / or otherwise can make inferences for the particular computing device.

[0053] During training phase 302, machine learning algorithm(s) 320 can be trained by providing at least training data 310 as training input using unsupervised, supervised, semi- supervised, and / or reinforcement learning techniques. Unsupervised learning involves providing a portion (or all) of training data 310 to machine learning algorithm(s) 320 and machine learning algorithm(s) 320 determining one or more output inferences based on the provided portion (or all) of training data 310. Supervised learning involves providing a portion of training data 310 to machine learning algorithm(s) 320, with machine learning algorithm(s) 320 determining one or more output inferences based on the provided portion of training data 310, and the output inference(s) are either accepted or corrected based on correct results associated with training data 310. In some examples, supervised learning of machine learning algorithm(s) 320 can be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or set of labels may be used to correct inferences of machine learning algorithm(s) 320.

[0054] Semi-supervised learning involves having correct results for part, but not all, of training data 310. During semi-supervised learning, supervised learning is used for a portion of training data 310 having correct results, and unsupervised learning is used for a portion of training data 310 not having correct results.

[0055] Reinforcement learning involves machine learning algorithm(s) 320 receiving a reward signal regarding a prior inference, where the reward signal can be a numerical value. During reinforcement learning, machine learning algorithm(s) 320 can output an inference and receive a reward signal in response, where machine learning algorithm(s) 320 are configured to try to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing an expected total of the numerical values provided by the reward signal over time. In some examples, machinelearning algorithm(s) 320 and / or trained machine learning model(s) 332 can be trained using other machine learning techniques, including but not limited to, incremental learning and curriculum learning.

[0056] In some examples, machine learning algorithm(s) 320 and / or trained machine learning model(s) 332 can use transfer learning techniques. For example, transfer learning techniques can involve trained machine learning model(s) 332 being pre-trained on one set of data and additionally trained using training data 310. More particularly, machine learning algorithm(s) 320 can be pre-trained on data from one or more computing devices and a resulting trained machine learning model provided to computing device CD1, where CD1 is intended to execute the trained machine learning model during inference phase 304. Then, during training phase 302, the pre-trained machine learning model can be additionally trained using training data 310. This further training of the machine learning algorithm(s) 320 and / or the pre-trained machine learning model using training data 310 of CD1’s data can be performed using either supervised or unsupervised learning. Once machine learning algorithm(s) 320 and / or the pre- trained machine learning model has been trained on at least training data 310, training phase 302 can be completed. The trained resulting machine learning model can be utilized as at least one of trained machine learning model(s) 332.

[0057] In particular, once training phase 302 has been completed, trained machine learning model(s) 332 can be provided to a computing device, if not already on the computing device. Inference phase 304 can begin after trained machine learning model(s) 332 are provided to computing device CD1.

[0058] During inference phase 304, trained machine learning model(s) 332 can receive input data 330 and generate and output one or more corresponding inferences and / or predictions 350 about input data 330. As such, input data 330 can be used as an input to trained machine learning model(s) 332 for providing corresponding inference(s) and / or prediction(s) 350. For example, trained machine learning model(s) 332 can generate inference(s) and / or prediction(s) 350 in response to one or more inference / prediction requests 340. In some examples, trained machine learning model(s) 332 can be executed by a portion of other software. For example, trained machine learning model(s) 332 can be executed by an inference or prediction daemon to be readily available to provide inferences and / or predictions upon request. Input data 330 can include data from computing device CD1 executing trained machine learning model(s) 332 and / or input data from one or more computing devices other than CD1.

[0059] Figure 4 is a flow chart of method 400. Method 400 may be executed by one or more computing systems (e.g., computing system 200 of Figure 2, computing system 100 of Figure1) and / or one or more processors (e.g., processor 206 of Figure 2). Aspects of method 400 may be implemented using software, hardware, and / or a combination thereof.

[0060] At block 402, method 400 includes obtaining a first image depicting a subject. In some examples, the first image depicting the subject may be a high dynamic range (HDR) image, such that one area of the image may be very bright, whereas another area of the image may be very dark. The subject depicted by the first image may be the focal point of the first image. For example, the subject may include one or more people, plants, animals, books, and / or objects, among others.

[0061] For example, Figure 5 depicts first image 500. Figure 5 depicts person 502 standing in front of window 504. The area of first image 500 depicting window 504 may be very bright, whereas the area of first image 500 depicting person 502 may be less bright. A computing system may attempt to increase the brightness of first image 500 so that the subject, person 502, is more apparent and / or more accurately represented in first image 500. However, increasing the brightness of first image 500 as a whole may cause some parts of first image 500 to be washed out. Accordingly, the computing system may apply the process described herein to adjust the exposure of first image 500 by determining exposure gains that, in some cases, may spatially vary (across first image 500) depending on the visual features represented by different portions of first image 500.

[0062] In some examples, first image 500 may depict person 502, including a face of person 502. Determining exposure gain parameters when the face of the person is depicted in first image 500 may be more accurate than exposure gain parameters determined when the face of the person is not depicted in first image 500. In general, in an image, the face of the person may be the main region of interest, which may be the focal point of the image and the region for which the exposure gain parameters should be calibrated. Accordingly, exposure gain parameters determined for first image 500 may also apply to subsequent images that do not include the face of the person.

[0063] Referring back to Figure 4, at block 404, method 400 includes determining, based on the first image, a subject segmentation mask representing regions of the subject as depicted in the first image. A computing system may determine a subject segmentation mask representing regions of the subject as depicted in the first image based on applying a machine learning model. The machine learning model may segment the first image into one or more regions and determine corresponding classification(s) for each of the one or more regions. The one or more regions and the classification(s) thereof may be represented by the subject segmentation mask.

[0064] Figure 6 depicts segmentation mask 600. The computing system may be configured to determine segmentation mask 600 by applying a machine learning model to first image 500 and / or by applying another segmentation method to first image 500. Segmentation mask 600 may include region 602 (face), region 604 (torso), region 606 (arm(s)), region 608 (leg(s)), and region 610 (shoe(s) / feet). Each region may be associated with a classification, including, for example, a skin region on at least one body part of the subject in the image and a non-skin region on the subject. For example, in segmentation mask 600, region 602 may be classified as a skin region of person 502 depicted in first image 500, region 604 may be classified as a non- skin region of person 502 being depicted in first image 500, regions 606 and 608 may be classified as a skin regions, and region 610 may be classified as a non-skin region. In addition, segmentation mask 600 may also include one or more classifications for region 612, perhaps as a background region or a region not associated with a person or a subject in the image. In some cases, region 612 may include a plurality of sub-regions, each of which is classified according to the visual and / or semantic content represented thereby.

[0065] In some examples, segmentation mask 600 may include one or more probability values, where each probability value is associated with a region and indicates a likelihood that the region belongs to a particular classification. For example, region 602 may have an associated probability value that the area (e.g., group of pixels in first image 500) represented by region 602 is a skin region. Region 604 may likewise have an associated probability value that the area represented by region 604 is a non-skin region. Region 606 may have an associated probability value that the area represented by region 606 is a skin region, and region 608 may have an associated probability that the area represented by region 608 is a skin region. Additionally, region 610 may have an associated probability value that the area represented by region 610 is a non-skin region.

[0066] Referring back to Figure 4, at block 406, method 400 includes determining, based on the subject segmentation mask, one or more exposure gain parameters for the first image. In some examples, the computing system may determine exposure gain parameters based on one or more probability values associated with the segmentation mask. Additionally and / or alternatively, the computing system may determine exposure gain parameters based on a count of regions with a particular classification.

[0067] To determine exposure gain parameters based on the probability values associated with each region identified by segmentation mask 600 of Figure 6, the computing system may determine an average weight of regions with a particular classification based on an average of the probability values of the regions with the particular classification. For example, thecomputing system may determine the average of the probability values of regions 602, 606, and 608, and may use the average of these probability values to determine a weight associated with skin regions. The computing system may also determine the average probability value of regions 604 and 610, and may use the average of these probability values to determine a weight associated with the non-skin regions of segmentation mask 600.

[0068] The computing system may determine luma values for each classification of regions based on the determined weights of each classification and one or more visual statistics associated with the regions. In particular, the computing system may, for each region associated with the particular classification, multiply the weight for the particular classification by one or more statistics determined for the respective region. The computing system may sum the products, which the computing system may divide by the sum of the weights associated with regions with the particular classification.

[0069] The computing system may also apply one or more thresholds. For example, the computing system may determine a count of regions associated with a skin classification. If the number of skin regions (e.g., a number of skin pixels, and / or a number of disjoint groups of skin pixels) is less than a minimum skin region count, the computing system may set the exposure gain parameter of regions associated with a skin classification to one, which may cause the exposure of the regions associated with a skin classification to not be adjusted. If the number of skin regions is greater than a minimum skin region count, the computing system may set the exposure gain parameter of regions associated with a skin classification to a value other than one (e.g., < 1 to cause exposure attenuation, or > 1 to cause exposure amplification), which may cause the exposure of the regions associated with a skin classification to be adjusted.

[0070] Additionally, the computing system may determine a count of regions (e.g., a number of non-skin pixels on the subject, and / or a number of disjoint groups of non-skin pixels on the subject) associated with a non-skin classification. If the number of non-skin regions is less than a minimum non-skin region count, the computing system may set the exposure gain parameter of regions associated with a non-skin classification to one, which may cause the exposure of the regions associated with a non-skin classification to not be adjusted. If the number of non- skin regions is greater than a non-skin region count, the computing system may set the exposure gain parameter of the regions associated with the non-skin classification to a value other than one, which may cause the exposure of the regions associated with a non-skin classification to be adjusted. Additionally and / or alternatively, the computing system may assign a predetermined value to the exposure gain parameters of regions having less than a particular number of regions (e.g., a predetermined value other than one).

[0071] In some examples, the exposure gain parameters for one or more regions may be set to one. The computing system might not increase or decrease the exposure of the regions associated with the exposure gain parameters of one. However, the computing system may nonetheless adjust the exposure of those regions (perhaps together with the exposure of other regions) based on exposure gain parameters determined using other processes. For example, the computing system might not adjust the digital exposure gain of some regions, but the exposure time and / or analog gain of these regions may nevertheless be adjusted and / or controlled by the computing system.

[0072] In some examples, the computing system may temporally filter the exposure gain parameters from frame to frame, so that abrupt changes in the exposure gain parameters from frame to frame are reduced and / or avoided. For example, the computing system may combine (i) the exposure gain parameters determined for a previous image that was received prior to first image 500 with (ii) initial exposure gain parameters determined for first image 500 (e.g., considered in isolation, independently of other images) to thereby determine temporally- filtered exposure gain parameters to apply to first image 500. For example, the computing system may combine the exposure gain parameters of the previous image and the initial exposure gain parameters of first image 500 in a predetermined ratio to determine temporally- filtered exposure gain parameters for first image 500. Temporally filtering the exposure gain parameters from frame to frame may help facilitate maintaining stable exposure levels across frames.

[0073] The computing system may apply the determined exposure gain parameters to the image. In particular, taking first image 500 and segmentation mask 600 as examples, the computing system may apply the determined exposure gain parameters associated with skin regions to skin regions of first image 500, the determined exposure gain parameters associated with non-skin regions to non-skin regions of first image 500, and so on. The computing system may also adjust regions of the image not associated with the region of interest (e.g., the region including the skin regions and the non-skin regions) based on exposure gain parameters determined through other exposure adjustment processes.

[0074] In some examples, the computing system may store the determined exposure gain parameters (whether temporally filtered or not) in memory. The computing system may later retrieve the stored exposure gain parameters for use in a future image if needed. The stored exposure gain parameters may include, for each classification of visual content represented by first image 500, a corresponding set of exposure gain parameters. When the exposure gain parameters are later retrieved, the stored exposure gain parameters associated with a givenclassification may be applied to one or more regions with the given classification, both in first image 500 and / or other images.

[0075] Referring back to Figure 4, at block 408, method 400 includes obtaining a second image depicting the subject. In some examples, the second image may be received subsequent to or some time after the first image (e.g., first image 500). The second image may include a subject that was also depicted in the first image.

[0076] Figure 7 depicts second image 700, in accordance with example embodiments. Second image 700 includes person 702 and window 704. The computing system may receive second image 700 after having received first image 500 of Figure 5. The subject depicted in second image 700, i.e., person 702, might not include a face. As mentioned above, exposure gain parameters determined for an image while a face was represented in the image may be more accurate than exposure gain parameters determined when the face is not represented in the image. The face of the person may be the main region of interest, which may be the focal point of the image and the region for which the exposure gain parameters should be calibrated.

[0077] Referring back to Figure 4, at block 410, method 400 includes, based on determining that the second image omits a particular region of the subject, applying the one or more exposure gain parameters for the first image to the second image, wherein the particular region is included in the first image. For example, the computing system may apply a machine learning model to image 700 to determine that the face of person 702 is not included in image 700. The machine learning model used to determine whether a face of person 702 is included in image 700 may be a different or the same machine learning model as the machine learning model used to segment first image 500 to segmentation mask 600.

[0078] Figure 8 depicts segmented second image 800. As shown in Figure 8, segmented second image 800 includes region 806 associated with a classification of skin, region 804 associated with a classification of non-skin, region 808 associated with a classification of skin, and region 810 associated with a classification of non-skin. Segmented second image 800 may also include region 812 associated with a classification of background 812. Further, background 812 may be segmented into one or more regions with associated classifications based on what other objects are present in the scene.

[0079] Based on segmented second image 800 not including a face, the computing system may apply the previously stored exposure gain parameters of first image 500. Determining that second image 700 and / or segmentation mask 800 omits a face may involve determining that the face has migrated and / or is positioned out of the frame of second image 700.

[0080] In some examples, the computing system may verify one or more factors, including whether the exposure level of image 700 is within a threshold level from the exposure level of first image 500. In addition, the computing system may determine one or more tuning parameters for the first image and for the second image. The tuning parameters may quantify how different the subject or region of interest is from the rest of the scene. For example, the computing system may determine whether the one or more tuning parameters of the second image are within a threshold value of the respective tuning parameter determined for the first image. Further, the computing system may receive and verify sensor metadata captured during the receiving period for the images. The computing system may compare the sensor metadata associated with the first image with the sensor data associated with the second image, and the computing system may verify that the differences between sensor metadata is below a threshold level. These verifications may help ensure that the image has not significantly changed between frames. If the computing system determines that the image does not pass verification, the computing system may apply the process described in the context of first image 500 to determine exposure gain parameters.

[0081] By implementing a segmentation process for a subject in the image, the computing system may help facilitate an exposure correction process that does not rely on face detection. In some instances, face detection may fail. By applying the method described herein, the computing system may nevertheless determine an image having a corrected exposure that appears closer to what a human observing the scene would observe and / or more accurately represents the content of the scene.

[0082] In general, the computing system may be used to improve the exposure of two or more images that differ in the portions of a subject represented by the two or more images. Specifically, exposure gain parameters may be determined based on a representation of a given portion of the subject in a first image, and these exposure gain parameters may be reused across one or more other images that do not represent the given portion of the subject. Thus, when the given portion of the subject allows for determination of more accurate gain parameters, the exposure of multiple images may be more accurately corrected regardless of whether the given portion of the subject remains visible across the multiple images.

[0083] In some examples, the computing system may receive a stream of images including the first image and the second image. The computing system may receive one or more images subsequent to the first image, including the second image. The one or more images received subsequent to the first image may be received consecutively. If the computing system determines that the one or more images do not include a particular region of the subject, thecomputing system may lock the exposure gain parameters applied to the first image to also be applied to the one or more subsequent images as well. For each of these subsequent images, the computing system may determine one or more factors mentioned above and compare those factors with one or more factors determined for the first image. If the computing system determines that the factors determined for an image of the one or more images are not within threshold with the factors determined for the first image, the computing system may unlock the exposure gain parameter and determine exposure gain parameters for subsequent images.

[0084] Referring back to Figure 4, in some examples, the regions comprise (i) a skin region on at least one body part of the subject and (ii) a non-skin region of the subject.

[0085] In some examples, the subject segmentation mask includes, for each respective region of the one or more regions, a probability value representing a likelihood that the respective region belongs to a particular classification.

[0086] In some examples, the subject comprises a person and the particular region comprises a face of the person.

[0087] In some examples, determining the subject segmentation mask comprises applying a machine learning model to the first image.

[0088] In some examples, method 400 further comprises determining that the second image omits the particular region of the subject by determining, using an additional machine learning model, whether the second image depicts the particular region of the subject.

[0089] In some examples, the regions comprise one or more regions corresponding to a first classification and one or more regions corresponding to a second classification, wherein the one or more exposure gain parameters are determined based on determining a number of the one or more regions corresponding to the first classification and a number of the one or more regions corresponding to the second classification.

[0090] In some examples, the regions comprise one or more regions corresponding to a first classification and one or more regions corresponding to a second classification, wherein determining the one or more exposure gain parameters comprises determining a first average probability of the one or more regions corresponding to the first classification and a second average probability of the one or more regions corresponding to the second classification.

[0091] In some examples, method 400 further comprises applying the one or more exposure gain parameters for the first image to the first image.

[0092] In some examples, method 400 further comprises receiving the second image subsequent to receiving the first image.

[0093] In some examples, applying the one or more exposure gain parameters for the first image to the second image further comprises verifying that an exposure level of the second image is within a threshold level from the exposure level of the first image.

[0094] In some examples, determining the one or more exposure gain parameters comprises determining one or more first tuning parameters representing the one or more regions in the first image, where method 400 further comprises determining one or more second tuning parameters representing one or more regions of the second image, where applying the one or more exposure gain parameters for the first image to the second image comprises verifying that each tuning parameter of the one or more first tuning parameters are within a threshold level from a respective tuning parameter of the one or more second tuning parameters.

[0095] In some examples, method 400 further comprises receiving sensor metadata associated with the first image, receiving sensor data associated with the second image, and applying the one or more exposure gain parameters for the first image to the second image based on comparing the sensor metadata associated with the first image and the sensor metadata associated with the second image.

[0096] In some examples, method 400 further comprises storing the one or more exposure gain parameters for the first image, wherein applying the one or more exposure gain parameters for the first image to the second image comprises receiving the exposure gain parameters for the first image from storage.

[0097] In some examples, applying the one or more exposure gain parameters for the first image to the second image comprises applying the one or more exposure gain parameters to areas of the second image determined to depict the subject.

[0098] In some examples, the one or more exposure gain parameters are temporally filtered, and wherein the one or more exposure gain parameters for the first image are determined further based on an image received prior to the first image.

[0099] In some examples, method 400 further comprises receiving a third image, based on determining that the third image includes the particular region of the subject, determining one or more additional exposure gain parameters for the third image, and applying the one or more additional exposure gain parameters for the third image to the third image.

[0100] In some examples, determining one or more exposure gain parameters for the first image comprises determining a number of one or more regions with a particular classification, determining whether the number is above a threshold value, and based on determining that the number is not above the threshold value, assigning a predetermined value to the one or more exposure gain parameters.

[0101] In some examples, a system may comprise a processor and a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform the operations of method 400.

[0102] In some examples, a non-transitory computer readable medium may store program instructions that, when executed by a computing device, cause the computing device to perform the operations of method 400. III. Conclusion

[0103] The present disclosure is not to be limited in terms of the particular embodiments described in this application, which are intended as illustrations of various aspects. Many modifications and variations can be made without departing from its scope, as will be apparent to those skilled in the art. Functionally equivalent methods and apparatuses within the scope of the disclosure, in addition to those described herein, will be apparent to those skilled in the art from the foregoing descriptions. Such modifications and variations are intended to fall within the scope of the appended claims.

[0104] The above detailed description describes various features and operations of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The example embodiments described herein and in the figures are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

[0105] With respect to any or all of the message flow diagrams, scenarios, and flow charts in the figures and as discussed herein, each step, block, and / or communication can represent a processing of information and / or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages can be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved. Further, more or fewer blocks and / or operations can be used with any of the message flow diagrams, scenarios, and flow charts discussed herein, and these message flow diagrams, scenarios, and flow charts can be combined with one another, in part or in whole.

[0106] A step or block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical operations or actions in the method or technique. The program code and / or related data may be stored on any type of computer readable medium such as a storage device including random access memory (RAM), a disk drive, a solid state drive, or another storage medium.

[0107] The computer readable medium may also include non-transitory computer readable media such as computer readable media that store data for short periods of time like register memory, processor cache, and RAM. The computer readable media may also include non- transitory computer readable media that store program code and / or data for longer periods of time. Thus, the computer readable media may include secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, solid state drives, compact- disc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.

[0108] Moreover, a step or block that represents one or more information transmissions may correspond to information transmissions between software and / or hardware modules in the same physical device. However, other information transmissions may be between software modules and / or hardware modules in different physical devices.

[0109] The particular arrangements shown in the figures should not be viewed as limiting. It should be understood that other embodiments can include more or less of each element shown in a given figure. Further, some of the illustrated elements can be combined or omitted. Yet further, an example embodiment can include elements that are not illustrated in the figures.

[0110] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for the purpose of illustration and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

CLAIMS 1. A computer-implemented method comprising: obtaining a first image depicting a subject; determining, based on the first image, a subject segmentation mask representing regions of the subject as depicted in the first image; determining, based on the subject segmentation mask, one or more exposure gain parameters for the first image; obtaining a second image depicting the subject; and based on determining that the second image omits a particular region of the subject, applying the one or more exposure gain parameters for the first image to the second image, wherein the particular region is included in the first image.

2. The computer-implemented method of claim 1, wherein the regions comprise (i) a skin region on at least one body part of the subject and (ii) a non-skin region of the subject.

3. The method of claim 1, wherein the subject segmentation mask includes, for each respective region of the one or more regions, a probability value representing a likelihood that the respective region belongs to a particular classification.

4. The computer-implemented method of claim 1, wherein the subject comprises a person and the particular region comprises a face of the person.

5. The computer-implemented method of claim 1, wherein determining the subject segmentation mask comprises applying a machine learning model to the first image.

6. The method of claim 5, wherein the computer-implemented method further comprises: determining that the second image omits the particular region of the subject by determining, using an additional machine learning model, whether the second image depicts the particular region of the subject.

7. The method of claim 1, wherein the regions comprise one or more regions corresponding to a first classification and one or more regions corresponding to a second classification, wherein the one or more exposure gain parameters are determined based ondetermining a number of the one or more regions corresponding to the first classification and a number of the one or more regions corresponding to the second classification.

8. The method of claim 1, wherein the regions comprise one or more regions corresponding to a first classification and one or more regions corresponding to a second classification, wherein determining the one or more exposure gain parameters comprises determining a first average probability of the one or more regions corresponding to the first classification and a second average probability of the one or more regions corresponding to the second classification.

9. The method of claim 1, further comprising: applying the one or more exposure gain parameters for the first image to the first image.

10. The method of claim 1, further comprising: receiving the second image subsequent to receiving the first image.

11. The method of claim 1, wherein applying the one or more exposure gain parameters for the first image to the second image further comprises verifying that an exposure level of the second image is within a threshold level from the exposure level of the first image.

12. The method of claim 1, wherein determining the one or more exposure gain parameters comprises determining one or more first tuning parameters representing the one or more regions in the first image, wherein the method further comprises: determining one or more second tuning parameters representing one or more regions of the second image, wherein applying the one or more exposure gain parameters for the first image to the second image comprises verifying that each tuning parameter of the one or more first tuning parameters are within a threshold level from a respective tuning parameter of the one or more second tuning parameters.

13. The method of claim 1, further comprising: receiving sensor metadata associated with the first image; receiving sensor data associated with the second image; andapplying the one or more exposure gain parameters for the first image to the second image based on comparing the sensor metadata associated with the first image and the sensor metadata associated with the second image.

14. The method of claim 1, further comprising: storing the one or more exposure gain parameters for the first image, wherein applying the one or more exposure gain parameters for the first image to the second image comprises receiving the exposure gain parameters for the first image from storage.

15. The method of claim 1, wherein applying the one or more exposure gain parameters for the first image to the second image comprises applying the one or more exposure gain parameters to areas of the second image determined to depict the subject.

16. The method of claim 1, wherein the one or more exposure gain parameters are temporally filtered, and wherein the one or more exposure gain parameters for the first image are determined further based on an image received prior to the first image.

17. The method of claim 1, further comprising: receiving a third image; based on determining that the third image includes the particular region of the subject, determining one or more additional exposure gain parameters for the third image; and applying the one or more additional exposure gain parameters for the third image to the third image.

18. The method of claim 1, wherein determining one or more exposure gain parameters for the first image comprises: determining a number of one or more regions with a particular classification; determining whether the number is above a threshold value; and based on determining that the number is not above the threshold value, assigning a predetermined value to the one or more exposure gain parameters.

19. A system comprising: a processor; anda non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform the method of any of claims 1-18.

20. A non-transitory computer readable medium storing program instructions that, when executed by a computing device, cause the computing device to perform the method of any of claims 1-18.

Images