Techniques for determining settings for a content capture device

Abstract

To provide techniques for determining settings for a content capture device.SOLUTION: Methods, systems, and computer program products for determining one or more settings of a content capture device are provided. In some examples, luminance values of pixels of the image from the content capture device may be identified to determine the one or more settings. Objects in the image and information associated with the objects may also be identified. Information associated with an object may be divided into categories of information. Then, using the objects and information, a separate weighting array for each category of information may be computed. Separate weighting arrays may be combined to create a total weighting array to augment the luminance values. The augmented luminance values may be aggregated to calculate a weighted luminance average of the image. Based on the weighted luminance average and target difference, one or more settings may be adjusted.SELECTED DRAWING: FIG. 1A

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 62 / 438,926, filed December 23, 2016, entitled "Method and System For Determining Exposure Levels," the disclosure of which is incorporated herein by reference in its entirety for all purposes.

[0002] This disclosure generally relates to determining settings (such as exposure settings) for content capture devices. Exposure settings may relate to the amount of light received by a sensor of the content capture device when content (e.g., an image or video) is captured. Examples of exposure settings include shutter speed, aperture setting, or International Organization for Standardization (ISO) speed. [Background technology]

[0003] Traditional solutions for setting exposure rely on the user, for example, the user adjusting the exposure settings to their own preferences, but this has proven unreliable and often produces suboptimal results.

[0004] Currently, automatic exposure control (AEC) is a standard feature on cameras. AEC automatically determines exposure settings for an image without user input. Using AEC, a camera may determine exposure settings for the camera. However, AEC typically measures the amount of light in the field of view without reference to what is in the field of view. Thus, there is a need in the art for an improved AEC. Summary of the Invention [Means for solving the problem]

[0005] Techniques are provided for determining one or more settings (e.g., exposure settings and / or gain settings) for a content capture device. In some embodiments, brightness values ​​of pixels of an image from the content capture device may be identified to determine one or more settings. Objects in the image and information associated with the objects may also be identified. The information associated with the objects may be divided into categories. A separate weighted array for each category of information may then be calculated using the objects and information. The separate weighted arrays may be combined to create a total weighted array and augment the brightness values. The augmented brightness values ​​may be aggregated to calculate a weighted brightness average for the image. Based on the difference between the weighted brightness average and the target, one or more settings may be adjusted.

[0006] In other embodiments, rather than calculating a separate weight array for each category of information, information associated with each object may be used in a separate learning-based model. The output of each learning-based model may be combined to create a total weight array to augment the brightness values ​​of the image. The augmented brightness values ​​may be aggregated to calculate a weighted brightness average for the image. Based on the difference between the weighted brightness average and the target, one or more settings may be adjusted.

[0007] In other examples, a weighting model as described herein may be used for objects in an image of a scene, and the same weighting model may then be used for objects in other images of the scene so that an image stitcher may combine the image and the other images together to create an optimized image.

[0008] Numerous benefits are achieved through the present disclosure compared to conventional techniques. For example, embodiments of the present disclosure provide better exposure of images by focusing on higher priority objects. The present disclosure also provides properly exposed images as part of a mixed reality or augmented reality scene. In some examples, the present disclosure may even learn and adapt priorities assigned to objects within an image.

[0009] Some embodiments allow exposure to dynamically adapt based on the user's line of sight as the line of sight vector changes. Embodiments may also dynamically reorder object priorities based on the user's movement or resizing of the focus reticle within the image.

[0010] The present disclosure also enables an object-based high dynamic range (HDR) method in which multiple high priority objects are properly exposed. These and other embodiments of the present disclosure, along with many of its advantages and features, are described in more detail in conjunction with the following text and accompanying figures.

[0011] Techniques are provided for updating settings of a content capture device. For example, the method may include receiving an image captured by the content capture device. In some embodiments, the image may include a plurality of pixels. In some embodiments, the image may not be presented to a user. The method may further include identifying a target brightness value for the image. In some embodiments, the target brightness value may be determined based on at least one of an exposure setting or a gain setting of the content capture device.

[0012] The method may further include identifying an object in the image, dividing a plurality of pixels of the image into a plurality of pixel groups, and calculating a pixel group intensity value for each of the plurality of pixel groups. The method may further include defining a first set of pixels not associated with the object, setting a weight for each pixel group in the first set of pixel groups, defining a second set of pixels associated with the object, and setting a weight for each pixel group in the second set of pixel groups. In some embodiments, the weight for each pixel group in the second set of pixel groups may be based on an association between the second set of pixels and the object. In some embodiments, the number of pixels in the first pixel group may be equal to the number of pixels in the second pixel group. In some embodiments, the first pixel group may be different from the second pixel group.

[0013] The method may further include calculating an image luminance value for each of the plurality of pixel groups using the weights and the pixel group luminance values. The method may further include updating settings (e.g., gain and / or exposure) of the content capture device based on the calculated difference.

[0014] In some embodiments, the method may further include identifying a second object in the image, defining a third set of pixels associated with the second object, and setting a weight for each pixel in the third set of pixels. The weight set for the second object may be used when calculating the image brightness value.

[0015] In some embodiments, the method may further include identifying additional information associated with the object. In such embodiments, the additional information may be a category associated with the object, a size of the object, a distance of the object from the content capture device, or a distance from a focusing reticle of the content capture device at which the object is located. In some embodiments, the weighting for each pixel group in the first set of pixels may be based on the additional information. In some embodiments, the weighting for each pixel group in the first set of pixels may be further based on second additional information. In such embodiments, the additional information may be different from the second additional information.

[0016] As an example, the method may further include identifying additional information associated with the object. The weighting for each pixel group in the first set of pixels may be based on the additional information. In another example, the additional information may include a category associated with the object, a size of the object, a distance of the object from the content capture device, or a distance from a focusing reticle of the content capture device at which the object is located.

[0017] In some embodiments, the method may further include identifying a direction in which the user is looking, determining a location on the image that corresponds to the direction in which the user is looking, and determining a distance from the location at which the object is located. In some embodiments, the weighting for each pixel group in the first set of pixels may be based on additional information. The weighting for each pixel group in the first set of pixels may be based on second additional information that is different from the second additional information.

[0018] For another embodiment, a method may include receiving an image captured by a content capture device. In some embodiments, the image may include a plurality of pixels. The method may further include identifying a target brightness value for the image. In some embodiments, the target brightness value may be based on a field of view. The method may further include identifying an object in the image, identifying one or more attributes of the object, and calculating a weight for the object using a neural network. In some embodiments, the neural network may use the one or more attributes as inputs. In such embodiments, the attributes of the one or more attributes of the object may include object priority, object distance, or object size. In some embodiments, the neural network may be a multilayer perceptron. The method may further include dividing a plurality of pixels of the image into a plurality of pixel groups. In some embodiments, each pixel group of the plurality of pixel groups may be the same size.

[0019] The method may further include defining a first set of pixels not associated with the object and defining a second set of pixels associated with the object. The method may further include calculating a pixel group luminance value for each pixel in the second set of pixels. The method may further include calculating a pixel group luminance value for each pixel in the second set of pixels. The method may further include multiplying the pixel group luminance value by a weighting for each pixel in the second set of pixels to provide a weighted pixel group luminance value.

[0020] The method may further include calculating a total luminance value for the image. In some embodiments, the total luminance value may include a sum of weighted pixel group luminance values. The method may further include calculating a difference between the total luminance value and a target luminance value, and updating a setting of the content capture device based on the calculated difference. In some embodiments, the setting of the content capture device may be associated with exposure or gain.

[0021] The method may further include identifying a second object in the image, identifying one or more second attributes of the second object, defining a third set of pixels associated with the second object, and calculating second weights for the second object using a second neural network. In some embodiments, the second neural network may use the one or more second attributes as input. The method may further include calculating a second pixel group luminance value for each pixel in the third set of pixels. The method may further include multiplying the second pixel group luminance value by a second weight for each pixel in the third set of pixels to provide a weighted second pixel group luminance value. In some embodiments, the total luminance value may further include a sum of the weighted second pixel group luminance values.

[0022] For another example, a method may include receiving a first image captured by a content capture device, identifying a first object in the first image, and determining a first update to a first setting of the content capture device. In some examples, the first update may be determined with respect to the first object. In some examples, the first update may be determined using a neural network. The method may further include receiving a second image captured by the content capture device. In some examples, the second image may be captured after the first image. The method may further include identifying a second object in the second image and determining a second update to a second setting of the content capture device. In some examples, the second update may be determined with respect to the second object. In some examples, the first setting and the second setting may be associated with exposure or gain. In some examples, the first setting may be the second setting. In some examples, the first update may differ from the second update. In some examples, the first image and the second image may be in the same field of view. The method may further include performing a first update to a first setting of the content capture device and receiving a third image captured by the content capture device. In some embodiments, the third image may be captured after the first update is performed. The method may further include performing a second update to a second setting of the content capture device and receiving a fourth image captured by the content capture device. In some embodiments, the fourth image may be captured after the second update is performed. The method may further include combining the third image and the fourth image into a single image. In some embodiments, the third image and the fourth image may be combined using an image stitcher.

[0023] According to one embodiment of the present invention, a method is provided. The method includes receiving a first image captured by a content capture device and identifying a predetermined number of priority objects in the first image. The predetermined number is two or more. The method also includes determining, for each of the predetermined number of priority objects, one or more updates for one or more settings of the content capture device. The method further includes iteratively updating the content capture device with each of the one or more updates and receiving a predetermined number of images captured by the content capture device with each of the one or more updates. Additionally, the method includes stitching the predetermined number of images together to form a composite image.

[0024] In another embodiment, a method may include receiving an image captured by a content capture device. The image may include a plurality of pixels. The method may further include identifying a target luminance value for the image, dividing the plurality of pixels of the image into a plurality of pixel groups, calculating a pixel group luminance value for each of the plurality of pixel groups, and identifying a location within the image. The location may correspond to a point at which a user is looking within an environment corresponding to the image. In some embodiments, the location may be identified based on an image of one or more eyes of the user. In other embodiments, the location may be identified based on a direction of the user's gaze. In other embodiments, the location may be identified based on a location of an object identified within the image. The method may further include setting a weight for each of the plurality of pixel groups based on the identified location, calculating an image luminance value for each of the plurality of pixel groups using the weight and the pixel group luminance value, calculating a difference between the image luminance value and the target luminance value, and updating a setting of the content capture device based on the calculated difference. The setting may be related to gain or exposure. In some embodiments, the method may further include dividing the plurality of pixels of the image into a plurality of patches, each of which may include one or more pixel groups. In such embodiments, setting the weight may be further based on distance from a patch containing the location. In some embodiments, setting the weight may be further based on distance from the location.

[0025] In another embodiment, a method may include receiving an image captured by a content capture device, the image including a plurality of pixels. The method may further include identifying a target luminance value for the image, dividing the plurality of pixels of the image into a plurality of pixel groups, calculating a pixel group luminance value for each of the plurality of pixel groups, receiving a depth map corresponding to the image, setting weights for each of the plurality of pixel groups based on the depth map, calculating an image luminance value for each of the plurality of pixel groups using the weights and the pixel group luminance values, calculating a difference between the image luminance value and the target luminance value, and updating a setting of the content capture device based on the calculated difference. The setting may be related to gain or exposure. In some embodiments, the depth map may indicate distances from a point in space for one or more points, each of the one or more points corresponding to one or more pixels of the image. In some embodiments, the method may further include capturing the depth map simultaneously with image capture. In other embodiments, the method may further include capturing the depth map before the images are captured, the depth map being used to set weights for the plurality of images. In some embodiments, setting the weights may be further based on data indicative of the location of the object from the image, which may be determined by analyzing the pixels of the image and identifying one or more pixels of the image that match one or more pixels of the stored image of the object.

[0026] For another example, a method may include receiving an image captured by a content capture device, the image including a plurality of pixels. The method may further include identifying a target luminance value for the image, dividing the plurality of pixels of the image into a plurality of pixel groups, and identifying a plurality of patches within the image. The plurality of patches may include a first patch and a second patch, where the first patch includes one or more pixel groups and the second patch includes one or more pixel groups that are different from the one or more pixel groups of the first patch. In some examples, the plurality of patches may be identified based on a plurality of pixels. In some examples, the plurality of patches may be identified based on one or more objects identified in the image, where the first patch includes pixels associated with the first object. The method may further include calculating one or more weights for the first patch using a first model and calculating one or more weights for the second patch using a second model. In some examples, the first model may be based on one or more attributes determined for pixels included in the first patch. In such embodiments, the second model may be based on one or more attributes determined for pixels included in the second patch, and the one or more attributes associated with the first model differ from the one or more attributes associated with the second model. In some embodiments, the first model is a neural network based on one or more attributes determined for pixels included in the first patch. In some embodiments, the one or more weights for the first patch may be calculated simultaneously with the one or more weights for the second patch. In some embodiments, the first patch may be a different size than the second patch. The method may further include, for each pixel group, calculating a pixel group luminance value and multiplying the pixel group luminance value by the weight to provide a weighted pixel group luminance value. The method may further include calculating a total luminance value for the image, wherein the total luminance value comprises a sum of the weighted pixel group luminance values.The method may further include calculating a difference between the total luminance value and the target luminance value, and updating a setting of the content capture device based on the calculated difference. The setting may be related to gain or exposure.

[0027] Although methods are described above, it should be appreciated that a computer product may include a computer-readable medium storing a plurality of instructions for controlling a computer system and performing the operations of any of the methods described above. In addition, a system may include a computer product and one or more processors for executing instructions stored on the computer-readable medium. In addition, a system may include means for performing any of the methods described above. In addition, a system may be configured to perform any of the methods described above. In addition, a system may include modules that each perform a step of any of the methods described above.

[0028] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter, which subject matter should be understood by reference to appropriate portions of the entire specification, drawings, and any or all of the claims of this patent.

[0029] The foregoing, together with other features and embodiments, will be explained in more detail in the following specification, claims, and accompanying drawings. The present invention provides, for example, the following. (Item 1) 1. A method comprising: receiving an image captured by a content capture device, the image including a plurality of pixels; identifying a target luminance value for the image; identifying an object within the image; dividing a plurality of pixels of the image into a plurality of pixel groups; calculating a pixel group luminance value for each of the plurality of pixel groups; defining a first set of pixels not associated with the object; setting a weight for each pixel in the first set of pixels; defining a second set of pixels associated with the object; setting a weight for each of the pixels in the second set of pixels; calculating, for each of the plurality of pixel groups, an image luminance value using the weights and the pixel group luminance values; calculating a difference between the image luminance value and the target luminance value; updating a configuration of the content capture device based on the calculated difference; and A method comprising: (Item 2) Item 10. The method of item 1, wherein the weighting of the pixels in the second set of pixels is based on the association between the second set of pixels and the object. (Item 3) Item 10. The method of item 1, wherein the setting relates to gain or exposure. (Item 4) identifying a second object within the image; and defining a third set of pixels associated with the second object; setting a weight for each of the pixels in the third set of pixels; Item 1, the method of claim 1 further comprising: (Item 5) Item 10. The method of item 1, wherein the target brightness value is determined based on at least one of an exposure setting or a gain setting of the content capture device. (Item 6) Item 1. The method of item 1, wherein the number of pixels in the first group of pixels is equal to the number of pixels in the second group of pixels. (Item 7) Item 1. The method of item 1, wherein the first group of pixels is different from the second group of pixels. (Item 8) Item 10. The method of item 1, further comprising identifying additional information associated with the object. (Item 9) 9. The method of claim 8, wherein the additional information is a category associated with the object, a size of the object, a distance of the object from the content capture device, or a distance from a focusing reticle of the content capture device at which the object is located, and wherein a weighting for each pixel group in the first set of pixels is based on the additional information. (Item 10) Item 9. The method of item 8, wherein the weighting for each of the pixel groups in the first set of pixel groups is based at least in part on the additional information. (Item 11) Identifying the direction in which a user is looking; determining a location on the image corresponding to a direction the user is looking; determining a distance from the location where the object is located, wherein a weight for each of the pixels in the first set of pixels is based on the distance; Item 1, the method of claim 1 further comprising: (Item 12) Item 10. The method of item 1, wherein the image is not presented to the user. (Item 13) 1. A method comprising: receiving an image captured by a content capture device, the image including a plurality of pixels; identifying a target luminance value for the image; identifying an object within the image; identifying one or more attributes of the object; calculating a weight for the object using a neural network, the neural network using the one or more attributes as input; dividing a plurality of pixels of the image into a plurality of pixel groups; defining a first set of pixels not associated with the object; defining a second set of pixels associated with the object; For each pixel in the second set of pixels, calculating pixel group luminance values; multiplying the pixel group luminance values ​​by the weights to provide weighted pixel group luminance values; calculating a total luminance value of the image, the total luminance value comprising a sum of the weighted pixel group luminance values; calculating a difference between the total luminance value and the target luminance value; updating a configuration of the content capture device based on the calculated difference; and A method comprising: (Item 14) identifying a second object within the image; and identifying one or more attributes of the second object; defining a third set of pixels associated with the second object; calculating weights for the second object using a second neural network, the second neural network using one or more attributes of the second object as input; For each pixel in the third set of pixels, Calculating a second pixel group luminance value; multiplying the second pixel group luminance values ​​by weights for the second object to provide weighted second pixel group luminance values, the total luminance value further comprising a sum of the weighted second pixel group luminance values; Item 14. The method of item 13, further comprising: (Item 15) Item 14. The method according to item 13, wherein the neural network is a multilayer perceptron. (Item 16) Item 14. The method of item 13, wherein the attribute of the one or more attributes of the object includes object priority, object distance from a viewer, object distance from a reticle, object distance from a line of sight, or object size. (Item 17) Item 14. The method of item 13, wherein the target luminance value is based on a field of view. (Item 18) Item 14. The method of item 13, wherein the settings of the content capture device are associated with exposure or gain. (Item 19) Item 14. The method of item 13, wherein each pixel group of the plurality of pixel groups is the same size. (Item 20) 1. A method comprising: receiving a first image captured by a content capture device; identifying a predetermined number of priority objects in the first image, the predetermined number being greater than or equal to two; determining, for each of the predetermined number of priority objects, one or more updates for one or more settings of the content capture device; Repeatedly, updating the content capture device with each of the one or more updates; receiving the predetermined number of images captured by the content capture device using each of the one or more updates; stitching said predetermined number of images together to form a composite image; A method comprising: (Item 21) 21. The method of claim 20, further comprising displaying the composite image. (Item 22) 21. The method of claim 20, wherein the first image includes a predetermined number of portions, each of the predetermined number of portions being associated with one of the predetermined number of priority objects. (Item 23) Item 23. The method of item 22, wherein stitching the predetermined number of images together includes combining the predetermined number of portions. (Item 24) 21. The method of claim 20, wherein a setting of the one or more settings is associated with exposure or gain. (Item 25) 21. The method of claim 20, wherein one or more of the one or more updates are determined using a neural network. (Item 26) 21. The method of claim 20, wherein one of the one or more updates is different from another of the one or more updates. (Item 27) 21. The method of claim 20, wherein the first image and the predetermined number of images are within the same field of view. (Item 28) 1. A method comprising: receiving an image captured by a content capture device, the image including a plurality of pixels; identifying a target luminance value for the image; dividing a plurality of pixels of the image into a plurality of pixel groups; calculating a pixel group luminance value for each of the plurality of pixel groups; identifying a location within the image, the location corresponding to a point at which a user is looking within an environment corresponding to the image; setting a weight for each of the plurality of pixel groups based on the identified location; calculating, for each of the plurality of pixel groups, an image luminance value using the weights and the pixel group luminance values; calculating a difference between the image luminance value and the target luminance value; updating a configuration of the content capture device based on the calculated difference; and A method comprising: (Item 29) 29. The method of claim 28, wherein the location is identified based on an image of one or more eyes of the user. (Item 30) Item 29. The method of item 28, wherein the location is identified based on the direction of the user's gaze. (Item 31) Item 29. The method of item 28, wherein the location is identified based on the location of an object identified in the image. (Item 32) 29. The method of claim 28, further comprising dividing the plurality of pixels of the image into a plurality of patches, each patch including one or more groups of pixels, and setting weights further based on distance from a patch including the location. (Item 33) 29. The method of claim 28, wherein setting the weight is further based on distance from the location. (Item 34) 29. The method of claim 28, wherein the setting relates to gain or exposure. (Item 35) 1. A method comprising: receiving an image captured by a content capture device, the image including a plurality of pixels; identifying a target luminance value for the image; dividing a plurality of pixels of the image into a plurality of pixel groups; calculating a pixel group luminance value for each of the plurality of pixel groups; receiving a depth map corresponding to the image; setting a weight for each of the plurality of pixel groups based on the depth map; calculating, for each of the plurality of pixel groups, an image luminance value using the weights and the pixel group luminance values; calculating a difference between the image luminance value and the target luminance value; updating a configuration of the content capture device based on the calculated difference; and A method comprising: (Item 36) Item 36. The method of item 35, wherein the depth map indicates distance from a point in space for one or more points, each of the one or more points corresponding to one or more pixels of the image. (Item 37) Item 36. The method of item 35, further comprising capturing the depth map simultaneously with image capture. (Item 38) 36. The method of claim 35, further comprising capturing the depth map before the images are captured, the depth map being used to set weights for multiple images. (Item 39) Item 36. The method of item 35, wherein setting the weights is further based on data indicating the location of the object from the image. (Item 40) The data indicating the location of the object may include: analyzing pixels of the image to identify one or more pixels of the image that match one or more pixels of the stored image of the object; Item 39. The method of item 39, wherein the method is determined by (Item 41) Item 36. The method of item 35, wherein the setting relates to gain or exposure. (Item 42) 1. A method comprising: receiving an image captured by a content capture device, the image including a plurality of pixels; identifying a target luminance value for the image; dividing a plurality of pixels of the image into a plurality of pixel groups; identifying a plurality of patches within the image, the plurality of patches including a first patch and a second patch, the first patch including one or more groups of pixels, and the second patch including one or more groups of pixels that differ from one or more groups of pixels of the first patch; calculating one or more weights for the first patch using a first model; and calculating one or more weights for the second patch using a second model; and For each pixel group, calculating pixel group luminance values; multiplying the pixel group luminance values ​​by the weights to provide weighted pixel group luminance values; calculating a total luminance value of the image, the total luminance value comprising a sum of the weighted pixel group luminance values; calculating a difference between the total luminance value and the target luminance value; updating a configuration of the content capture device based on the calculated difference; and A method comprising: (Item 43) Item 43. The method of item 42, wherein the first model is based on one or more attributes determined for pixels included in the first patch, and the second model is based on one or more attributes determined for pixels included in the second patch, and one or more attributes associated with the first model differ from one or more attributes associated with the second model. (Item 44) Item 43. The method of item 42, wherein the first model is a neural network based on one or more attributes determined for pixels contained in the first patch. (Item 45) Item 43. The method of item 42, wherein the one or more weights for the first patch are calculated simultaneously with the one or more weights for the second patch. (Item 46) Item 43. The method of item 42, wherein the first patch is a different size than the second patch. (Item 47) Item 43. The method of item 42, wherein the plurality of patches are identified based on the plurality of pixels. (Item 48) Item 43. The method of item 42, wherein the plurality of patches are identified based on one or more objects identified in the image, and the first patch includes pixels associated with a first object. (Item 49) Item 43. The method of item 42, wherein the setting relates to gain or exposure. [Brief explanation of the drawings]

[0030] Illustrative embodiments are described in detail below with reference to the following figures:

[0031] [Figure 1A] FIG. 1A illustrates an example process for updating one or more settings of a content capture device using automatic exposure control.

[0032] [Figure 1B] FIG. 1B illustrates an example process for determining how to update one or more settings of a content capture device.

[0033] [Figure 2] FIG. 2 illustrates examples of various photometric techniques for weighting luminance values.

[0034] [Figure 3] FIG. 3 illustrates an example of a priority weighting array for objects.

[0035] [Figure 4] FIG. 4 illustrates an example of a priority weighting array for multiple objects.

[0036] [Figure 5] FIG. 5 illustrates an example of a focal reticle weighting array.

[0037] [Figure 6] FIG. 6 illustrates an example of a line-of-sight weighted array.

[0038] [Figure 7] FIG. 7 illustrates an example of a normalized total weight array.

[0039] [Figure 8] FIG. 8 is a flow chart illustrating an embodiment of a process for automatic exposure control using a first weighting model.

[0040] [Figure 9] FIG. 9 illustrates an example of a first portion of a second weighting model that may be used for automatic exposure control.

[0041] [Figure 10] FIG. 10 illustrates an example of a second portion of a second weighting model that may be used for automatic exposure control.

[0042] [Figure 11] FIG. 11 is a flow chart illustrating an embodiment of a process for automatic exposure control using a second weighting model.

[0043] [Figure 12A] FIG. 12A illustrates an example of an image stitching process that may use multiple instances of automatic exposure control.

[0044] [Figure 12B] FIG. 12B illustrates another example of an image stitching process that may use multiple instances of automatic exposure control.

[0045] [Figure 12C] FIG. 12C is a flowchart illustrating an embodiment of a process for using multiple instances of automatic exposure control.

[0046] [Figure 13] FIG. 13 illustrates an example of an image stream that can be used in conjunction with the image stitching process.

[0047] [Figure 14] FIG. 14 is a flow chart illustrating an embodiment of a process for automatic exposure control using an image stitching process.

[0048] [Figure 15] FIG. 15 illustrates an embodiment of a block diagram of a computer system.

[0049] [Figure 16] FIG. 16 is a flow chart illustrating an embodiment of a process for automatic exposure control using locations identified based on a user's gaze.

[0050] [Figure 17] FIG. 17 is a flow chart illustrating an embodiment of a process for automatic exposure control using a depth map.

[0051] [Figure 18] FIG. 18 is a flow chart illustrating an embodiment of a process for automatic exposure control using multiple models. DETAILED DESCRIPTION OF THE INVENTION

[0052] In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of embodiments of the present disclosure. However, it will be apparent that various embodiments may be practiced without these specific details. The illustrations and description are not intended to be limiting.

[0053] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. For example, while the description may describe pixel information, images, and / or displays, it should be recognized that audio may be generated by an augmented reality device and presented to a user instead of, or in addition to, visual content. It should also be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present disclosure as set forth in the appended claims.

[0054] The present disclosure generally relates to determining exposure settings for a content capture device. The exposure settings may relate to the amount of light received by a sensor of the content capture device when content (e.g., an image or video) is captured. Examples of exposure settings include shutter speed, aperture setting, or International Organization for Standardization (ISO) speed.

[0055] Traditional solutions for setting exposure rely on the user, for example, the user adjusting the exposure settings to their own preferences, but this has proven unreliable and often produces suboptimal results.

[0056] Automatic exposure control (AEC) is now a standard feature on cameras. AEC automatically determines exposure settings for an image without user input. Using AEC, the camera may determine the exposure settings for the camera. In some embodiments, AEC may be activated in conjunction with automatic focus control (AF) and / or automatic white balance control (AWB) for the field of view. In such embodiments, AEC may first be used to calculate an estimate of the amount of exposure for the field of view. After the estimate is calculated, AF may be performed to determine the amount of focus for the field of view. In some embodiments, after the amount of focus is determined, AEC may continue to run to fine-tune the exposure settings for the field of view. In some embodiments, AWB may run at least partially in parallel with AEC. In such embodiments, AWB may finish before or after AF. In some embodiments, AWB may begin running after AF has finished. While the field of view is described above, it should be appreciated that the scene captured in the image may constitute the field of view.

[0057] Further described below are techniques for determining one or more settings (e.g., exposure settings and / or gain settings) for a content capture device. In some embodiments, brightness values ​​of pixels of an image from the content capture device may be identified to determine one or more settings. Objects in the image and information associated with the objects may also be identified. The information associated with the objects may be divided into categories. A separate weighted array for each category of information may then be calculated using the objects and information. The separate weighted arrays may be combined to create a total weighted array and augment the brightness values. The augmented brightness values ​​may be aggregated to calculate a weighted brightness average for the image. Based on the difference between the weighted brightness average and the target, one or more settings may be adjusted.

[0058] In other embodiments, rather than calculating a separate weight array for each category of information, information associated with each object may be used in a separate learning-based model. The output of each learning-based model may be combined to create a total weight array to augment the brightness values ​​of the image. The augmented brightness values ​​may be aggregated to calculate a weighted brightness average for the image. Based on the difference between the weighted brightness average and the target, one or more settings may be adjusted.

[0059] In other examples, a weighting model as described herein may be used for an object in an image of a scene, and the same weighting model may then be used for other objects in other images of the scene so that an image stitcher may combine the image and other images together to create an optimized image.

[0060] 1A illustrates an example of a process 100 for updating one or more settings of a content capture device using automatic exposure control (AEC). In some examples, the one or more settings may include an exposure setting, a gain setting, or any combination thereof. In such examples, the exposure setting may be a shutter speed, an ISO speed, or any combination thereof. The gain setting may be a digital gain, an analog gain, or any combination thereof.

[0061] Process 100 may include receiving (110) an image. In some embodiments, the image may be received from a sensor of a content capture device. In other embodiments, the image may be included in a feed provided to process 100. It should be appreciated that the image may be received from a number of devices and systems.

[0062] In some embodiments, the image may be received in an image buffer. In such embodiments, the image may be a bitmap image that includes pixel information for each pixel of the image. In some embodiments, the image buffer may be a size equal to the pixel height and pixel width of the image. By way of example, the image buffer size may be 640x480, where 640 may correspond to the width in number of pixels of the image and 480 may correspond to the height in number of pixels of the image.

[0063] Process 100 may further include dividing the image into pixel groups (120). The size and shape of each pixel group may be predefined. In some embodiments, the size and shape of each pixel group may be the same or may vary. For illustrative purposes, the pixel groups will be described as rectangular. However, it should be appreciated that the pixel groups may be any shape that divides the image into multiple portions. For example, the pixel groups may be radial from the center of the image. In such embodiments, each pixel group may include a different range of diameter (e.g., a first radius may be 0-1 unit from the center, a second radius may be 1-2 units from the center, and a third radius may be 2-3 units from the center). For another embodiment, a pixel group may be associated with each object (e.g., a first object may be a first pixel group, a second object may be a second pixel group, and the remainder of the image may be a third pixel group). It should also be appreciated that the pixel groups may be any other shape that divides the image into multiple portions. In some embodiments, the pixel groups may be arranged such that two or more pixel groups overlap.

[0064] In one illustrative example, an image may be divided into 96 pixel groups (12 pixel groups by 8 pixel groups, where 12 corresponds to the number of pixel groups along the width of the image and 8 corresponds to the number of pixel groups along the height of the image). In such an example, having 12 by 8 pixel groups in a 640 by 480 image would mean that each pixel group would have a height of approximately 50 pixels and a width of approximately 60 pixels. While this example shows that the width and height of each pixel group would be different, it should be understood that the width of a pixel group can be the same as the height of the pixel group.

[0065] Process 100 may further include calculating (130) an average luminance pixel group value for each pixel group of the image. In some embodiments, the average luminance pixel group value may be calculated by accumulating luminance values ​​for each pixel in the pixel group. In such embodiments, the luminance value may represent the brightness of the image (e.g., an achromatic portion of the image). In some embodiments, the luminance value may be a representation of the image without color components. For example, in a YUV color space, the luminance value may be Y. In some embodiments, the luminance value is a weighted sum of the gamma-compressed RGB components of the image. In such embodiments, the luminance value may be referred to as gamma-corrected luminance. In some embodiments, the accumulation may be performed by software or hardware by summing the luminance values ​​for each pixel in the pixel group. Once the luminance values ​​for the pixels are accumulated, the total may be divided by the number of pixels in the pixel group to calculate an average luminance pixel group value for the pixel group. This process may be repeated for each pixel group in the image.

[0066] Process 100 may further include performing AEC (140). The AEC may receive as input an average luminance pixel group value for each pixel group of the image. In some embodiments, the AEC method may apply weights to the average luminance pixel group values ​​described above using a weighting array. In such embodiments, the AEC may also receive as input a weighting array that may identify a weight to apply to each pixel group of the image.

[0067] In some embodiments, the weight array may include pixel groups corresponding to pixel groups created by dividing the image. For example, if the image is divided into 25 pixel groups (5 pixel groups by 5 pixel groups), the weight array may include weights for the 25 pixel groups (5 pixel groups by 5 pixel groups). In such embodiments, the top-left most pixel group in the image may correspond to the top-left most pixel group in the weight array, etc. In some embodiments, the values ​​for each pixel group in the weight array may be based on several techniques discussed herein, including photometry, object priority, focus reticle, line of sight, normalized total weight array, learning-based methods, the like, or any combination thereof.

[0068] In some embodiments, a weight array (e.g., the weight arrays illustrated in FIGS. 1-7 and 10) may be combined with the average luminance pixel group values ​​to calculate weighted luminance pixels. For example, each average luminance pixel group value may be multiplied by a corresponding weight. In other embodiments, a weight array may be combined with luminance values ​​of pixels based on the pixel group being created. In such embodiments, the weight to apply to a pixel may be determined based on the pixel group that includes the pixel. For example, if a pixel is in the top-left pixel group in an image, the weight associated with the top-left pixel group in the weight array may be applied to the pixel. In some embodiments, the weight associated with a pixel group may be multiplied by each pixel in the corresponding pixel group of the image to calculate a weighted luminance value for each pixel in the corresponding pixel group.

[0069] In some embodiments, the weighted luminance values ​​may be averaged together to create a weighted luminance average for the image. In one embodiment, the weighted luminance average is [ka] where WLA is the weighted luminance average, M is the height of the pixel group, N is the width of the pixel group, w[r,c] is the location r and location c in the weight array, and luma avg [r,c] is the average luminance value of the pixel at location r and location c.

[0070] In other embodiments, the weighting array may be used for local tone mapping. For example, a local tone mapping system may use the weighting array to identify portions of the field of view that should be brightened. This technique may address portions of the field of view rather than the entire field of view as in some embodiments of the averaging technique described above. In some embodiments, settings (e.g., exposure settings and / or gain settings) may be applied to one or more portions based on the weighting array. In such embodiments, the weighting array may be used as a guide for local tone mapping.

[0071] In some embodiments, local tone mapping may be performed on pixels that are above a predetermined threshold, the threshold corresponding to a weight in the weighting array. For example, a pixel that is given a weight above the threshold may cause the local tone mapping system to determine an adjustment of a setting (e.g., an exposure setting and / or a gain setting) for the pixel. In some embodiments, a pixel and one or more neighboring pixels may be used when comparing to the threshold. For example, a pixel and one or more neighboring pixels would need to be above the threshold for local tone mapping to be applied to the pixel and / or one or more neighboring pixels. In some embodiments, local tone mapping may be supported in software and / or hardware.

[0072] In some embodiments, process 100 may further include updating one or more settings. As described above, the one or more settings may include an exposure setting, a gain setting, or any combination thereof. In such embodiments, the exposure setting may be a shutter speed, an ISO speed, or any combination thereof. In some embodiments, the shutter speed may be a global shutter or a rolling shutter. A global shutter may indicate a duration for exposing all pixels in the field of view. A rolling shutter may indicate a duration for exposing a row (either horizontally or vertically) of the field of view. With a rolling shutter, lines of an image may be scanned in a rolling manner, rather than a snapshot of the field of view. The gain setting may be a digital gain, an analog gain, or any combination thereof. For example, a gain setting may be 8x by having 2x analog gain and 4x digital gain. In some embodiments, the exposure setting may be adjusted before the gain setting when increasing exposure. In such embodiments, the gain setting may be adjusted before the gain setting when decreasing exposure.

[0073] 1B illustrates an example process 160 for determining how to update one or more settings of a content capture device. The process may include comparing a target luminance value to a luminance value (e.g., a weighted luminance average) (step 170).

[0074] If the weighted luminance average is less than the target luminance average by a predetermined first threshold, one or more settings may be adjusted to brighten the image (step 172). For example, if the exposure setting of one or more settings is not at the maximum allowable exposure setting, the exposure setting may be increased (e.g., incrementally increased) to the maximum allowable exposure setting (steps 174 and 176). In one illustrative example, the maximum allowable exposure setting may be 16.6 milliseconds for a frame rate of 30 frames per second. However, it should be recognized that the maximum allowable exposure setting may be different even for a frame rate of 30 frames per second. The maximum allowable exposure setting may be based on the ISO speed and the content capture device. In some examples, the software and / or hardware of the content capture device may determine the maximum allowable exposure setting (e.g., the maximum allowable exposure setting may be less than the frame period (1 / frame rate) minus the time to transfer the image between the sensor and the host processor). In some examples, the maximum allowable exposure setting may be less than what the software and / or hardware allows.

[0075] If the exposure setting is at the maximum allowable exposure setting, the gain setting of one or more settings may be increased (e.g., incrementally increased) up to the maximum allowable gain setting (steps 174 and 178). In one illustrative example, the maximum allowable gain setting may be 8x. However, it should be recognized that the maximum allowable gain setting may vary. The maximum allowable gain setting may be based on the desired image quality (e.g., noise in the image may increase as the gain setting increases) and the software and / or hardware of the content capture device (e.g., a sensor may support up to a certain gain setting).

[0076] If the weighted luminance average exceeds the target luminance average by a predetermined second threshold (which may be the same as or different from the first threshold), one or more settings may be adjusted to make the image darker (step 180). For example, if the gain setting is not at the minimum allowable gain setting (e.g., 1x), the gain setting may be decreased (e.g., incrementally decreased) to the minimum allowable gain setting (steps 182 and 184). In some embodiments, the minimum allowable gain setting may be determined based on the software and / or hardware of the content capture device (e.g., a sensor in the content capture device).

[0077] If the gain setting is at the minimum allowable gain setting, the exposure setting may be decreased (e.g., incrementally decreased) to the minimum allowable exposure setting (e.g., 20 microseconds) (steps 182 and 186). The minimum allowable exposure may correspond to the amount of time the field of view should be exposed based on the software and / or hardware of the content capture device (e.g., the sensor of the content capture device). The amount of adjustment for either situation may be based on the amount of difference of the weighted luminance average from the target luminance average.

[0078] In some embodiments, a target brightness average may be provided. In such embodiments, a target brightness average may be provided and used until a new target brightness average is provided. In other embodiments, multiple target brightness averages may be provided for different situations. For example, the target brightness average may be slightly higher outdoors than indoors because it may be brighter outdoors. Thus, in some embodiments, two different target brightness averages may be provided or determined, one for indoors and one for outdoors. In other embodiments, a target brightness average may be determined. In such embodiments, the current physical environment may be determined based on one or more sensors. For example, the one or more sensors may detect the amount of light in the current physical environment. In some embodiments, any combination of one or more settings may be used to determine the target brightness value.

[0079] FIG. 2 illustrates examples of various photometric techniques for weighting luminance values. For various photometric techniques, a weighting scale 210 is provided to serve as a legend for the particular weights. Each level of the weighting scale 210 includes a color that, when included in the weighting array, corresponds to a particular weight. For example, the top box in the weighting scale 210 is the darkest color and represents a weighting of 1, and the bottom box in the weighting scale 210 is the lightest color and represents a weighting of 0. While the weighting scale 210 appears to vary linearly from the top box to the bottom box, it should be understood that the weighting scale 210 may vary as long as the weighting scale 210 is consistent throughout a single image. In some embodiments, two or more of the photometric techniques described below may be combined in a single weighting array.

[0080] The first metering technique may be spot metering. In some examples, spot metering may refer to giving the same weight to each pixel in a selected region of an image. In such examples, spot metering may be implemented to give decreasing weights the further away from the selected region you move. In some examples, the selected region may be identified based on user movement (e.g., by performing a screen-touching motion within the field of view). In such examples, a finger moving in the air from a first position to a second position may correspond to selecting the region. There would be no actual screen the user is touching, but the motion itself may be detected. Another way a region may be selected may be by interlocking fingers together in a manner that roughly creates a circle, with everything inside the circle being the selection. In other examples, a virtual frame may appear that can be moved by the user. In such examples, a circle that can be moved and / or resized may be shown on the user's display.

[0081] Spot metering may involve assigning a first weight to one or more first pixel groups and a second weight to one or more second pixel groups. Typically, the one or more first pixel groups are identified based on a user selection. For example, a user may place a finger on the screen to indicate a spot. The location of the finger may define the one or more first pixel groups used for spot metering. The result of the spot metering may be spot metering weight array 220. As can be seen, spot metering weight array 220 includes spot 222. Spot 222 may be a location (e.g., one or more first pixel groups) identified based on a user selection. Spot 222 includes a first weight, which may be 1. The remainder of spot metering weight array 220 (e.g., one or more second pixel groups) includes a second weight, which may be 0. It should be understood that different weights may be used. It should also be understood that more than one spot may be identified.

[0082] A second metering technique may be central metering, which in some embodiments may refer to giving greater weight to pixels in the center of an image with decreasing weight moving towards the edge of the image.

[0083] Central metering may assign multiple weights to pixel groups based on their distance from a point (e.g., the center) of the weight array. The result of central metering may be central metering weight array 230. Central metering weight array 230 may include a first weight for a particular distance away from the center. The first weight may be the maximum weight. In central metering weight array 230, the first weight is included in two pixel groups vertically from the center and three pixel groups horizontally from the center. This illustrates that the distance from the center may vary horizontally and vertically. It should also be understood that central metering may also vary in other directions, including diagonally.

[0084] The central photometry weight array 230 may also include second, third, fourth, and fifth weights, each one pixel away from the previous weight. In some embodiments, each successive level of weighting may have decreasing weights. For example, the fifth weight may be less than the fourth weight, which may be less than the third weight, which may be the second weight, which may be less than the first weight. Again, it should be understood that the central photometry weight array is merely an example, and other configurations of the weights and specific weights for each level may also be used.

[0085] A third photometry technique may be image photometry. Image photometry may cause a single weight to be assigned to all pixels in a weight array. The result of the image photometry may be image metering weight array 240. Image metering weight array 240 includes a single weight for all pixels. Image photometry may produce an average exposure for the entire scene.

[0086] In some embodiments, in addition to (or instead of) photometry, one or more objects in an image may be assigned a priority that influences the weighting array for the image. By assigning weights for the weighting array based on one or more objects, the one or more objects may remain properly exposed throughout multiple images even if the one or more objects are moving relative to the images and the weighting array. The one or more objects may be properly exposed throughout multiple images because the weights are not applied to a single position each time, but rather vary based on one or more objects.

[0087] FIG. 3 illustrates an example of a priority weighting array 320 for objects in an image. As above, a weighting scale 310 is provided as a legend for particular weights. In some examples, objects in an image may be identified by an object recognition system. In such examples, the object recognition system may identify a category of the object (e.g., a person, an animal, a chair, or the like). In other examples, the identification may identify one or more additional details of the object (in addition to the object's category). For example, the identity of a person, the identity of an animal (e.g., this is a golden retriever, or this is a Spot), or the like may be identified. The object recognition system may also determine a group of pixels that include the object.

[0088] In one illustrative embodiment, a table such as the one illustrated below may be used to assign weights to identified objects: Using such a table, when an object is identified, one or more groups of pixels identified for the object are assigned a priority weight within the table. [Table 1-1] [Table 1-2]

[0089] In some examples, an active object (as shown in the table above) may represent an object that is actively in use by an application. For example, a surface may be in use by a mixed reality application. In such an example, a character may be dancing on the surface. To ensure that the surface takes priority, the weighting associated with the surface may exceed the weighting for other areas of the image.

[0090] In some embodiments, the user-selected object (as shown in the table above) may represent the object selected by the user, similar to the spot metering described above, but unlike spot metering, the user-selected object may include all pixels within which the object is located, rather than just the pixels covered by the finger.

[0091] In some examples, an identified individual (as shown in the table above) may represent an individual in an image who has been identified by a facial recognition system (or other identification system). For example, the facial recognition system may recognize an individual as Clark Kent. In such an example, Clark Kent may be included in a list of identified individuals who should be assigned a higher weighting. Additionally, in some examples, certain individuals may be assigned different weightings. In some examples, individuals not included in the list and / or not identified by name may be given a different weighting than identified individuals (as indicated by "Person" in the table above).

[0092] Similarly, an identified pet (as shown in the table above) may represent a pet in the image that has been identified by a facial recognition system (or other identification system). For example, the facial recognition system may recognize the pet as Dixie. In such an embodiment, Dixie may be included in a list of identified pets that should be assigned a higher weighting. Additionally, in some embodiments, specific pets may be assigned different weightings. In some embodiments, pets not included in the list and / or not identified by name may be given a different weighting than identified pets (as shown by "Pet" in the table above).

[0093] In some embodiments, one or more other categories of identified objects (e.g., pets, cars, flowers, buildings, insects, trees / shrubs, artwork, furniture, or the like) (as shown in the table above) may be assigned different weights. In such embodiments, each category may be assigned a weighting that may be applied to all pixel groups that contain objects. In some embodiments, any pixel groups that do not contain objects may be given a different weighting (as indicated by "Other" in the table above). In such embodiments, the different weighting may be zero or some other non-zero number. In most embodiments, the different weighting may be less than one or more of the other weightings. In some embodiments, the different weighting may vary depending on the distance away from the identified object.

[0094] In some examples, the weights, priorities, and object types of the table may be predefined. In some examples, the weights, priorities, and object types may adapt over time based on learning from user actions. For example, when a user primarily captures images containing a particular object, that object may be assigned a higher priority. For another example, the weights may change based on one or more user actions in relation to the number of images of similar scenes. For example, based on an image being deleted, the priority weights may be updated to match the user's preferences. Similar to deletion, an image shared with others may indicate that one or more settings for the image were optimal.

[0095] 3 , an area of ​​the image that includes an object 322 may be identified. Based on the identification of the object 322, one or more pixels in the priority weighting array 320 that include the object 322 may be assigned a first weight. In some embodiments, one or more pixels in the priority weighting array 320 that are not included in the object 322 may be assigned a second weight. Although the priority weighting array 320 illustrates the object 322 with a weight of 1 and one or more pixels not included in the object 322 with a weight of 0, it should be appreciated that any weight may be assigned to the object 322 and one or more pixels not included in the object 322.

[0096] FIG. 4 illustrates an example of a priority weighting array 420 for multiple objects. As above, a weighting scale 410 is provided as a legend for particular weights. In some examples, one or more objects may be identified in the image. In such examples, the AEC may determine to run using a subset of one or more objects identified in the image. For example, if more than one object is identified in the image, the AEC may determine to run using only one of the objects. In some examples, the AEC may determine to run based on the object with the highest priority. In other examples, the AEC may determine to run based on one or more of the identified objects, even if not all of the objects are identified. In other examples, the AEC may determine to run based on all of the identified objects.

[0097] In one illustrative example, the one or more objects may include a first object 422, a second object 424, and a third object 426. In some examples, each object may be assigned a weighting. In some examples, two or more weightings may be similar and / or two or more weightings may be different. For example, the first object 422 may be an active object (as described in the table above), the second object 424 may be a user-selected object (as described in the table above), and the third object 426 may be a car (as described in the table above).

[0098] FIG. 5 illustrates an example of a focus reticle weighting array 530. As above, a weighting scale 510 is provided as a legend for particular weights. In some examples, focus reticle weighting array 530 may identify the location of focus reticle 522 within image 520. In some examples, focus reticle 522 may be adjusted based on a hand gesture, automatic focus control (AF) (as described above), and / or a resize of focus reticle 522. In some examples, a hand gesture may form or draw focus reticle 522. In such examples, if an object is inside focus reticle 522, the object's pixels may be given a higher weight. If the object is on the boundary of focus reticle 522, the object's pixels may be given a lower weight. If the object is outside focus reticle 522, the object's pixels may be given the lowest weight. In some embodiments, the focus reticle 522 may be resized using commands either on the content capture device or remotely from the content capture device (e.g., hand gestures, a remote device, or the like). In other embodiments, the AF may indicate an area of ​​the image 520 that is the focus of the image 520. In such embodiments, objects within the area may be given a higher weighting.

[0099] In some embodiments, similar to center metering weight array 230, the weights in focus reticle weight array 530 may decrease as distance increases from the center of focus reticle 522. For example, the center of focus reticle 522 may have the highest weight. Pixels around the center may also have the highest weight. However, as distance increases from the center, pixels may receive decreasing amounts of weight.

[0100] In other embodiments, focus reticle weight array 530 may be used in conjunction with an object identification system to increase the weighting of pixels that contain an object that is at least partially within focus reticle 522. For example, if the object overlaps or is completely contained within focus reticle 522, the weighting of the pixels that contain the object may be increased. If the object is completely outside focus reticle 522, the weighting of the pixels that contain the object may be decreased.

[0101] FIG. 6 illustrates an example of a gaze weighting array 690. As above, a weighting scale 610 is provided as a legend for particular weights. To implement the gaze weighting array 690, a system may include one or more eye capture devices. The eye capture devices may be used to capture one or more images and / or one or more videos of one or more eyes (e.g., eyes 660) of a user. The images and / or videos may be further processed to determine a line of sight 670 of the eye 660. In some examples, the line of sight 670 may indicate the direction the eye 660 is looking or where the eye 660 is looking.

[0102] In some embodiments, the line of sight 670 may also indicate the depth at which the eye 660 is looking, based on the user's eye 660. In some embodiments, by looking at the left and right eyes, the system may determine that the line of sight 670 is looking at a particular object at a particular depth. In some embodiments, similar to the central metering weight array 230, the weights in the line of sight weight array 690 may decrease as the distance increases from where the line of sight 670 is looking. For example, the center of where the line of sight 670 is looking may have the highest weight. The pixels around the center may also have the highest weight. However, as the distance increases from the center, the pixels may receive decreasing amounts of weight.

[0103] Illustrating a line of sight weighting array 690, line of sight 670 may point to a second set of pixels 630, which may include a second object 632. The pixels that include the second object 632 may be assigned the highest weight. Weights may then be assigned based on distance away from the second object 632 and / or based on other objects that are identified.

[0104] For example, a first object 622 in the first set of pixels 620 and a third object 642 in the third set of pixels 640 may be identified. In such an embodiment, the first set of pixels 620 may be assigned a weight corresponding to the distance from the second object 632 and the identification of the first object 622. If the first object 622 is an object with a high priority, the weight of the first set of pixels may be increased. On the other hand, if the first object 622 is an object with a low priority, the weight of the first set of pixels may be decreased. Similar operations may be performed with respect to the third set of pixels 640 and the third object 642. In some embodiments, no object may be identified in the set of pixels (e.g., the fourth set of pixels 650). When no object is identified, the set of pixels may be assigned a default value or a value based on the distance from one or more of the objects (e.g., the second object 632) or the location of the line of sight 670.

[0105] 16 is a flowchart illustrating an embodiment of a process 1600 for automatic exposure control using a location identified based on a user's gaze. In some aspects, the process 1600 may be performed by a computing device (e.g., a content capture device such as a camera).

[0106] Process 1600 is illustrated as a logical flow diagram, whose operations represent a sequence of actions that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the actions represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described actions. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular data types. The order in which the actions are described is not intended to be construed as a limitation, and any number of the described actions may be combined in any order and / or in parallel to implement a process.

[0107] Additionally, process 1600 may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As noted above, the code may be stored on a machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The machine-readable storage medium may be non-transitory.

[0108] Process 1600 may include receiving (1610) an image captured by a content capture device. In some embodiments, the image may include multiple pixels. In some embodiments, the image may be received from a camera or other content capture device. In other embodiments, the image may be received from a feed (or stream) of images. In such embodiments, the feed may be current or past images. FIG. 6 illustrates the image as including a first set of patches 620, a second set of patches 630, a third set of patches 640, and a fourth set of patches 650. It should be appreciated that the image may include more or fewer sets of patches than illustrated in FIG. 6.

[0109] Process 1600 may further include identifying (1620) a target luminance value for the image. In some embodiments, the target luminance value may indicate an optimal amount of luminance for the image. In such embodiments, each pixel in the image may be associated with a luminance value. In some embodiments, the target luminance value may correspond to an average of the luminance values ​​of the image. In other embodiments, the target luminance value may correspond to a weighted average of the luminance values ​​of the image. In other embodiments, the target luminance value may correspond to a number that would result from multiplying the weight array by the luminance value.

[0110] In some embodiments, the target brightness value may be predefined. In other embodiments, the target brightness value may be determined based on one or more sensors. For one illustrative embodiment, a sensor may detect the amount of light in an environment. Based on the amount of light, the target brightness value may be set. In such embodiments, there may be a threshold for the target brightness value. For example, if the amount of light is above a certain amount, the target brightness value may be the certain amount.

[0111] Process 1600 may further include dividing (1630) the plurality of pixels of the image into a plurality of pixel groups. In some embodiments, the pixel groups may represent adjacent (or adjacent / neighboring) groups of pixels. In such embodiments, the shapes of the pixel groups may vary. For example, each pixel group may be rectangular or square, such that the image is divided into rows and columns. In other embodiments, each pixel group may include a certain number of pixels from the center of the image. In some embodiments, different pixel groups may be different shapes (e.g., if the pixel groups are created to include objects). In some embodiments, the pixel groups may be different sizes. In some embodiments, the pixel groups may be arranged such that two or more pixel groups overlap.

[0112] Process 1640 may further include calculating (1640) a pixel group luminance value for each of the plurality of pixel groups. In some embodiments, the pixel group luminance value for a pixel group may comprise an average of the luminance values ​​for each pixel of the pixel group. In other embodiments, the pixel group luminance value may be a sum of the luminance values ​​for each pixel of the pixel group. In other embodiments, the pixel group luminance value may be the difference between the luminance value of each pixel and an average luminance value for the image. For example, an average luminance value may be calculated for the image considering all pixels. The average luminance value may then be used to calculate a pixel-by-pixel difference such that a difference from the average luminance value is generated for the pixel group. It should be appreciated that other summary values ​​of the luminance values ​​of the pixel groups may also be used.

[0113] Process 1600 may further include identifying (1650) a location within the image. The location may correspond to a point the user is looking at within the environment (i.e., the physical area in which the user is located) corresponding to the image. For example, the image may correspond to a general direction the user is looking. The location may be a position within the image where the user is determined to be actually looking.

[0114] In some examples, the location may be identified based on an eye image of one or more eyes of a user. The eye image may be separate from the image described above. In such examples, the eye image may be analyzed to identify where the user is looking. For example, each set of one or more pixels of the eye image may be determined to correspond to a different set of one or more pixels of the image. In such examples, if one or more eyes in the eye image are determined to be looking at a particular set of one or more pixels of the eye image, a location in the image may be identified as the set of one or more pixels of the image that corresponds to the particular set of one or more pixels of the eye image. It should be appreciated that more than one eye image may be used. For example, an eye image of a first eye and an eye image of a second eye may be used. In some examples, the location may correspond to the location of an object identified in the image. It should also be appreciated that other methods for identifying where a user is looking based on an eye image of one or more eyes of a user may also be used.

[0115] In other examples, the location may be identified based on the direction of the user's gaze. In such examples, instead of using an eye image of one or more eyes of the user, the orientation of the device worn by the user may be used. In one illustrative example, the orientation may include one or more from the set of pitch, yaw, and roll. In such examples, a sensor (e.g., a gyroscope) that measures the orientation of the device may be included with the device. It should be appreciated that other methods for identifying the location where a user is looking based on the orientation of the device may also be used.

[0116] Process 1600 may further include setting (1660) a weight for each of the plurality of pixel groups based on the identified location. For example, pixel groups corresponding to locations closer to the identified location may be set with a higher weight than pixel groups corresponding to locations farther from the identified location. It should be appreciated that the weights may be set according to other methods. Figure 6 illustrates the weights set for pixel groups in eye weight array 690.

[0117] In some embodiments, process 1600 may further include dividing the pixels of the image into patches, where a patch includes one or more groups of pixels. In such embodiments, setting the weights may further be based on distance from the patch that includes the location. FIG. 6 illustrates dividing the pixels of the image into patches. For example, a first patch may be a first set of patches 620, a second patch may be a second set of patches 630, a third patch may be a third set of patches 640, and a fourth patch may be a fourth set of patches 650. It should be appreciated that the image may be divided into more or fewer sets of patches than illustrated in FIG. 6.

[0118] Process 1600 may further include calculating (1670) an image luminance value for each of the plurality of pixel groups using the weights and the pixel group luminance values. In some embodiments, calculating the image luminance value may include summing each of the weighted pixel group luminance values, and the weighted pixel group luminance value may be calculated by multiplying the weight associated with the pixel group by the luminance value associated with the pixel group.

[0119] Process 1600 may further include calculating (1680) a difference between the image brightness value and the target brightness value and updating (1690) a setting of the content capture device based on the calculated difference. In some embodiments, the setting may be a gain setting or an exposure setting. In such embodiments, the exposure setting may be a shutter speed, an ISO speed, or any combination thereof. In some embodiments, the shutter speed may be a global shutter or a rolling shutter. A global shutter may indicate a duration for exposing all pixels in the field of view. A rolling shutter may indicate a duration for exposing a row (either horizontally or vertically) of the field of view. With a rolling shutter, a line of the image may be scanned in a rolling manner rather than a snapshot of the field of view. The gain setting may be a digital gain, an analog gain, or any combination thereof. For example, the gain setting may be 8x by having 2x analog gain and 4x digital gain. In some embodiments, the exposure setting may be adjusted before the gain setting when increasing exposure. In such embodiments, the gain setting may be adjusted before the gain setting when decreasing exposure. In some embodiments, the difference in the adjustment may be proportional to the difference. For example, the adjustment may be greater if the difference between the image luminance value and the target luminance value is greater.

[0120] 7 illustrates an example of a normalized total weight array 730 with respect to an image 720. As above, a weight scale 710 is provided as a legend for the particular weights. In some embodiments, the normalized total weight array 730 may be a combination of two or more weight arrays. In such embodiments, the two or more weight arrays may be generated based on any method, including those described above (e.g., photometry, line of sight, focusing reticle, and the like).

[0121] In some examples, normalized total weighted array 730 may be based on image 720. In such examples, one or more objects (e.g., woman 722, child 724, cat 726, toy house 728, and toy truck 729) may be identified in image 720. Based on identifying the one or more objects, one or more weighted arrays may be generated.

[0122] For example, an object priority weighting array 740 (similar to priority weighting array 420) may be generated. The object priority weighting array 740 may assign weights to pixels that include one or more objects. In some embodiments, the weightings may be based on a table such as that described above in FIG. 3. For example, people may be associated with higher weights than animals. Thus, pixels that include a woman 722 and a child 724 may be given a higher weight than pixels that include a cat 726. Additionally, toys may be associated with lower weights than animals, and thus, pixels that include a toy house 728 and a toy truck 729 may have a lower weight than pixels that include a cat 726. Other pixels that do not include identified objects may have a lower weight than toys. For example, the illustration in FIG. 7 shows other pixels having a weight of 0. However, weights may be non-zero.

[0123] For another embodiment, object distance weighting array 750 may be generated using one or more identified objects. In such an embodiment, object distance weighting array 750 may further be based on a depth map. A depth map may be created from an image (or set of images) and describe the depth of each pixel. Using the depth map, identified objects may be associated with depths. Objects closer to the image source (e.g., the content capture device capturing image 720) may then be assigned higher weights.

[0124] For example, the pixels of toy house 728 may be given the highest weight because the toy house is the closest object. The pixels of woman 722, child 724, and cat 726 may be assigned a similarity weight that is less than the weight for toy house 728 because these objects are the same distance from the image source but farther away than toy house 728. Similarly, the pixels of toy truck 729 may be assigned the smallest weight of the identified objects because toy truck 729 is the farthest object. In some embodiments, other pixels that do not contain objects may be assigned either a weight of zero or some other non-zero weight that is less than the weight assigned to the pixels of the object.

[0125] In other embodiments, each group of pixels may be assigned a weight regardless of whether an object is identified. In such embodiments, the object does not need to be identified, as each weight may correspond to a value in the depth map. It should be appreciated that other methods for weighting objects based on distance may also be used.

[0126] For another embodiment, an object size weighting array 760 may be generated using one or more identified objects. While the size of an object may be calculated in several different ways, one method for determining the size of an object is to count the number of pixels identified in the object size weighting array 760 for the object.

[0127] In some embodiments, after calculating the size of each identified object, pixels of each identified object may be assigned a weight proportional to its size. For example, toy house 728 may be given the highest weight because it is the largest object. In one illustrative embodiment, woman 722, child 724, and toy truck 729 may all be given the same weight because they are all similarly sized. And cat 726 may be given the lowest weight because it is the smallest. Other pixels that do not contain an identified object may be given either a zero weight or a non-zero weight that is less than the minimum weight for the identified object.

[0128] In other embodiments, after calculating the size of each identified object, the pixels of each identified object may be given a weighting percentage for each pixel group identified for the identified object. For example, if each pixel group is worth a weighting of 0.1, an object with four pixels would be weighted 0.4. In other embodiments, the amount of weighting each pixel group deserves may be a percentage of the largest object. For example, the largest object may be given a weighting of 1. And if the object is 10 pixels in size, an object with five pixels may be given a weighting of 0.5. It should be appreciated that other methods for weighting objects based on size may also be used.

[0129] For another embodiment, one or more photometric weighting arrays may be generated as illustrated in Figure 2. In one illustrative embodiment, a central photometric weighting array 770 (similar to central photometric weighting array 230) may be generated.

[0130] For another embodiment, a gaze weighting array 790 (similar to gaze weighting array 690) may be generated.

[0131] In some embodiments, image 720 may also include information associated with a focusing reticle such as that described in Figure 4. The focusing reticle may be used alone or in combination with the identified objects to generate a focusing reticle weighting array 780 (similar to focusing reticle weighting array 530).

[0132] As described above, after the two or more weight arrays are generated, the two or more weight arrays may be combined to create a normalized total weight array 730. For example, the weights for the pixels across the multiple weight arrays may be combined (e.g., multiplied together) to generate updated weights for the pixels. The normalized total weight array 730 may be combined with an average luminance pixel group value to calculate a weighted luminance pixel group, as described above in FIG. 1. In some embodiments, the normalized total weight array 730 may be: [ka] where N w is the number of weight arrays. In some embodiments, normalization of the total weight array is performed by normalizing the maximum weight with a value of 1 and w T A scaling may be implemented such that the total weight array is scaled according to the smallest weight having the smallest value of [r, c] (e.g., 0). In some embodiments, normalization may be omitted, as the weighted luminance average automatically scales the result by dividing by the sum of the total weight array.

[0133] 8 is a flowchart illustrating an embodiment of a process for automatic exposure control using a first weighting model. In some aspects, process 800 may be implemented by a computing device.

[0134] Process 800 is illustrated as a logical flow diagram, whose operations represent a sequence of actions that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the actions represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described actions. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular data types. The order in which the actions are described is not intended to be construed as a limitation, and any number of the described actions may be combined in any order and / or in parallel to implement a process.

[0135] Additionally, process 800 may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As noted above, the code may be stored on a machine-readable storage medium, e.g., in the form of a computer program comprising a plurality of instructions executable by one or more processors. The machine-readable storage medium may be non-transitory.

[0136] Process 800 may include receiving (805) an image captured by a content capture device. In some embodiments, the image may include multiple pixels. In some embodiments, the image may be received from a camera or other content capture device. In other embodiments, the image may be received from a feed (or stream) of images. In such embodiments, the feed may be current or past images.

[0137] Process 800 may further include identifying (810) a target luminance value for the image. In some embodiments, the target luminance value may indicate an optimal amount of luminance for the image. In such embodiments, each pixel in the image may be associated with a luminance value. In some embodiments, the target luminance value may correspond to an average of the luminance values ​​of the image. In other embodiments, the target luminance value may correspond to a weighted average of the luminance values ​​of the image. In other embodiments, the target luminance value may correspond to a number that would result from multiplying the weight array by the luminance value.

[0138] In some embodiments, the target brightness value may be predefined. In other embodiments, the target brightness value may be determined based on one or more sensors. For one illustrative embodiment, a sensor may detect the amount of light in an environment. Based on the amount of light, the target brightness value may be set. In such embodiments, there may be a threshold for the target brightness value. For example, if the amount of light is above a certain amount, the target brightness value may be the certain amount.

[0139] Process 800 may further include identifying an object in the image (815). In some embodiments, the object may be identified using an object identification system. In such embodiments, the object identification system may identify the type of object (e.g., a person, an animal, a building, or the like). In other embodiments, the object identification system may identify one or more attributes of the object that indicate more than the type of object (e.g., that the object is Bill Nye). In some embodiments, the object identification system may identify whether the object is being processed by another system (such as a mixed reality system that is determining where to place a virtual object).

[0140] Process 800 may further include dividing (820) the plurality of pixels of the image into a plurality of pixel groups. In some embodiments, the pixel groups may represent adjacent (or adjacent / neighboring) groups of pixels. In such embodiments, the shapes of the pixel groups may vary. For example, each pixel group may be rectangular or square, such that the image is divided into rows and columns. In other embodiments, each pixel group may include a certain number of pixels from the center of the image. In some embodiments, different pixel groups may be different shapes (if the pixel groups are created to include objects). In some embodiments, the pixel groups may be different sizes. In some embodiments, the pixel groups may be arranged such that two or more pixel groups overlap.

[0141] Process 800 may further include calculating (825) a pixel group luminance value for each of the plurality of pixel groups. In some embodiments, the pixel group luminance value for a pixel group may comprise an average of the luminance values ​​for each pixel of the pixel group. In other embodiments, the pixel group luminance value may be a sum of the luminance values ​​for each pixel of the pixel group. In other embodiments, the pixel group luminance value may be the difference between the luminance value of each pixel and an average luminance value for the image. For example, an average luminance value may be calculated for the image considering all pixels. The average luminance value may then be used to calculate a pixel-by-pixel difference such that a difference from the average luminance value is generated for the pixel group. It should be appreciated that other summary values ​​of the luminance values ​​of the pixel groups may also be used.

[0142] Process 800 may further include defining (830) a first set of pixels that are not associated with the object. In some embodiments, the first set of pixels may include one or more other objects and / or may not include any objects. In some embodiments, not being associated with the object may indicate that no pixels of one or more pixels identified as the object are included in the first set of pixels.

[0143] Process 800 may further include setting a weight for each pixel group in the first set of pixels (835). In some embodiments, the weight for each pixel group in the first set of pixels may be based on a table such as that described above in FIG. 3. In other embodiments, the weight for each pixel group in the first set of pixels may be automatically set to zero because no objects are included in the pixels. It should be appreciated that other methods for determining weights for pixels without objects may also be used.

[0144] Process 800 may further include defining 840 a second set of pixels associated with the object. In some embodiments, being associated with the object may indicate that a pixel of the one or more pixels identified as the object is included in the second set of pixels.

[0145] Process 800 may further include setting a weight for each pixel group in the second set of pixels (845). Similar to that described above, the weight for each pixel group in the second set of pixels may be based on a table such as that described above with respect to FIG. 3. In other embodiments, the weight for each pixel group in the second set of pixels may be automatically set to 1 because the object is included in the pixel group. It should be appreciated that other methods for determining the weight for pixels with objects may also be used.

[0146] Process 800 may further include calculating (850) an image luminance value for each of the plurality of pixel groups using the weights and the pixel group luminance values. In some embodiments, calculating the image luminance value may include summing each of the weighted pixel group luminance values, and the weighted pixel group luminance value may be calculated by multiplying the weight associated with the pixel group by the luminance value associated with the pixel group.

[0147] Process 800 may further include calculating (855) a difference between the image brightness value and the target brightness value and updating (860) a setting of the content capture device based on the calculated difference. In some embodiments, the setting may be a gain setting or an exposure setting. In such embodiments, the exposure setting may be a shutter speed, an ISO speed, or any combination thereof. In some embodiments, the shutter speed may be a global shutter or a rolling shutter. A global shutter may indicate a duration for exposing all pixels in the field of view. A rolling shutter may indicate a duration for exposing a row (either horizontally or vertically) of the field of view. With a rolling shutter, a line of the image may be scanned in a rolling manner rather than a snapshot of the field of view. The gain setting may be a digital gain, an analog gain, or any combination thereof. For example, the gain setting may be 8x by having 2x analog gain and 4x digital gain. In some embodiments, the exposure setting may be adjusted before the gain setting when increasing exposure. In such embodiments, the gain setting may be adjusted before the gain setting when decreasing exposure. In some embodiments, the difference in the adjustment may be proportional to the difference. For example, the adjustment may be greater if the difference between the image luminance value and the target luminance value is greater.

[0148] Using the weighted luminance average equation described above with respect to FIG. 1, a specific weight may be disabled for the first weighting model by setting the array associated with the weight to unity.

[0149] 17 is a flowchart illustrating an embodiment of a process 1700 for automatic exposure control using a depth map. The depth map may include one or more distances, measured from a single point for each of the one or more distances. Each distance may correspond to one or more pixels in the image, such that each distance indicates the distance from the single point that content contained in the one or more pixels is located.

[0150] In some aspects, process 1700 may be performed by a computing device (e.g., a content capture device such as a camera). Process 1700 is illustrated as a logical flow diagram, whose operations represent a sequence of actions that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the actions represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described actions. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular data types. The order in which the actions are described is not intended to be construed as a limitation, and any number of the described actions may be combined in any order and / or in parallel to implement the process.

[0151] Additionally, process 1700 may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As noted above, the code may be stored on a machine-readable storage medium, e.g., in the form of a computer program comprising a plurality of instructions executable by one or more processors. The machine-readable storage medium may be non-transitory.

[0152] Process 1700 may include receiving 1710 an image captured by a content capture device. In some embodiments, the image may include multiple pixels. In some embodiments, the image may be received from a camera or other content capture device. In other embodiments, the image may be received from a feed (or stream) of images. In such embodiments, the feed may be current or past images.

[0153] Process 1700 may further include identifying (1720) a target luminance value for the image. In some embodiments, the target luminance value may indicate an optimal amount of luminance for the image. In such embodiments, each pixel in the image may be associated with a luminance value. In some embodiments, the target luminance value may correspond to an average of the luminance values ​​of the image. In other embodiments, the target luminance value may correspond to a weighted average of the luminance values ​​of the image. In other embodiments, the target luminance value may correspond to a number that would result from multiplying the weight array by the luminance value.

[0154] In some embodiments, the target brightness value may be predefined. In other embodiments, the target brightness value may be determined based on one or more sensors. For one illustrative embodiment, a sensor may detect the amount of light in an environment. Based on the amount of light, the target brightness value may be set. In such embodiments, there may be a threshold for the target brightness value. For example, if the amount of light is above a certain amount, the target brightness value may be the certain amount.

[0155] Process 1700 may further include dividing (1730) the plurality of pixels of the image into a plurality of pixel groups. In some embodiments, the pixel groups may represent adjacent (or adjacent / neighboring) groups of pixels. In such embodiments, the shapes of the pixel groups may vary. For example, each pixel group may be rectangular or square, such that the image is divided into rows and columns. In other embodiments, each pixel group may include a certain number of pixels from the center of the image. In some embodiments, different pixel groups may be different shapes (if the pixel groups are created to include objects). In some embodiments, the pixel groups may be different sizes. In some embodiments, the pixel groups may be arranged such that two or more pixel groups overlap.

[0156] Process 1740 may further include calculating (1740) a pixel group luminance value for each of the plurality of pixel groups. In some embodiments, the pixel group luminance value for a pixel group may comprise an average of the luminance values ​​for each pixel of the pixel group. In other embodiments, the pixel group luminance value may be a sum of the luminance values ​​for each pixel of the pixel group. In other embodiments, the pixel group luminance value may be the difference between the luminance value of each pixel and an average luminance value for the image. For example, an average luminance value may be calculated for the image considering all pixels. The average luminance value may then be used to calculate a pixel-by-pixel difference such that a difference from the average luminance value is generated for the pixel group. It should be appreciated that other summary values ​​of the luminance values ​​of the pixel groups may also be used.

[0157] Process 1700 may further include receiving (1750) a depth map corresponding to the image. As described above, the depth map may include one or more distances, the distances being measured from a single point for each of the one or more distances. Each distance may correspond to one or more pixels in the image, such that each distance indicates the distance from the single point that content contained in the one or more pixels is located.

[0158] In some embodiments, a depth map may be captured simultaneously with the image capture described above. For example, a depth map may also be captured each time an image is captured for use in determining settings for the content capture device. In other embodiments, a depth map may be captured at some point before an image is captured so that the depth map can be used to set weights for multiple images. In such embodiments, the depth map may be used based on the orientation at which the image was captured. For example, one or more depth maps may each correspond to a different orientation. When an image is captured at a particular orientation, the depth map (or a portion of the depth map) corresponding to the particular orientation may be used when setting the weights used to determine settings for the content capture device.

[0159] 7, a depth map may indicate the distance that a woman 722, a child 724, a cat 726, a toy house 728, and a toy truck 729 are each located from the content capture device. It should be appreciated that the distances are not object-based, but may be pixel-based, such that the distances relate to pixels rather than objects. It should also be appreciated that the depth map may indicate depth in different ways than described above.

[0160] Process 1700 may further include setting (1760) weights for each of the plurality of pixel groups based on the depth map. The weights may be set higher for a closer set of one or more pixels than for a more distant set of one or more pixels. In some embodiments, setting the weights may also be based on data indicative of the location of the object from the image. For example, as illustrated in FIG. 7, the weights in object distance weight array 750 are set such that only pixels identified as being associated with the object are given a weight. It should be appreciated that other pixels may also be set with different weights greater than zero. In some embodiments, the data indicative of the location of the object is determined by analyzing the pixels of the image and identifying one or more pixels of the image that match one or more pixels of a stored image of the object. It should also be appreciated that the distance weight array is a result of setting the weights, and that the distance weight array is not based on the objects identified in the image. It should also be appreciated that other methods for setting weights using a depth map may be used besides those described above.

[0161] Process 1700 may further include calculating (1770) an image luminance value for each of the plurality of pixel groups using the weights and the pixel group luminance values. In some embodiments, calculating the image luminance value may include summing each of the weighted pixel group luminance values, and the weighted pixel group luminance value may be calculated by multiplying the weight associated with the pixel group by the luminance value associated with the pixel group.

[0162] Process 1700 may further include calculating (1780) a difference between the image brightness value and the target brightness value and updating (1790) a setting of the content capture device based on the calculated difference. In some embodiments, the setting may be a gain setting or an exposure setting. In such embodiments, the exposure setting may be a shutter speed, an ISO speed, or any combination thereof. In some embodiments, the shutter speed may be a global shutter or a rolling shutter. A global shutter may indicate a duration for exposing all pixels in the field of view. A rolling shutter may indicate a duration for exposing a row (either horizontally or vertically) of the field of view. With a rolling shutter, a line of the image may be scanned in a rolling manner rather than a snapshot of the field of view. The gain setting may be a digital gain, an analog gain, or any combination thereof. For example, the gain setting may be 8x by having 2x analog gain and 4x digital gain. In some embodiments, the exposure setting may be adjusted before the gain setting when increasing exposure. In such embodiments, the gain setting may be adjusted before the gain setting when decreasing exposure. In some embodiments, the difference in the adjustment may be proportional to the difference. For example, the adjustment may be greater if the difference between the image luminance value and the target luminance value is greater.

[0163] FIG. 9 illustrates an example of a first portion of a second weighting model used for automatic exposure control (AEC). In some examples, the second weighting model may treat each object in an image independently of all others in the image. In such examples, a learning-based model (e.g., neural network, clustering, or the like) may be used for each object (or a subset of objects) in the image. The learning-based model may output values ​​that are used as weights for each pixel group for the object. After weights are generated for each object in the image, a final weighting array may be created by summing each one of the weighting arrays for each object. This differs from the first weighting model, which multiplied the weighting arrays together. In other examples, a learning-based model may be used for each pixel group of an object. In such examples, the remainder of the process would be performed similarly to having a learning-based model for each object, except that rather than having a single value used for all pixels of the object, a value would be calculated for each pixel group. In other examples, a learning-based model may be used for each weight for an object (e.g., object distance, object priority, photometry, or the like). In such an embodiment, the weights for a single object would need to be combined as discussed above with respect to the first weighting model. After the weights per object are combined, the remainder of the process may continue similarly with the second weighting model.

[0164] In one illustrative example, the second weighting model may utilize one or more neural networks (e.g., multi-layer perceptrons) to determine the weighting array. In some examples, the second weighting model may create a neural network for each of one or more identified objects in the image. In such examples, the inputs to the neural network may include weights similar to those described above with respect to the weighting array. In one illustrative example, the weights may include object priority—{.1...1}; object size—small, medium, or large {.5,.75,1}; object distance—near, medium, far {.5,.75,1}; photometry—edge to center {.9...1}; focus reticle—medium, near, or outside {.5,.75, or 1}; and line of sight—“at vector,” “near vector,” or “outside vector” {.5,.75,1}.

[0165] In the above embodiments, the object priority weighting may indicate the priority of the object. The priority of the object may be between 0 and 1. In some embodiments, the priority may be defined by a table such as that described above with respect to FIG.

[0166] The object size weighting may indicate the size of the object. The size of the object may be one of an enumerated set (e.g., small, medium, or large). In some embodiments, the size corresponding to each one of the enumerated sets may be predefined (e.g., if the object is contained within three or fewer pixel groups, the object is small and receives a weight of 0.5; if the object is contained within four to six pixel groups, the object is medium and receives a weight of 0.75; if the object is contained within six or more pixel groups, the object is large and receives a weight of 1). In other embodiments, the size of an object may simply be the number of pixel groups the object is contained within.

[0167] The object distance weighting may indicate a distance from an image source (e.g., a content capture device). The object distance may be one of an enumerated set (e.g., near, medium, or far). In some examples, the distance corresponding to each one of the enumerated sets may be predefined (e.g., if the object is within 10 feet of the image source, the object is near and receives a weighting of 0.5; if the object is within 10-50 feet of the image source, the object is medium and receives a weighting of 0.75; if the object is more than 50 feet from the image source, the object is far and receives a weighting of 1). In some examples, the distance may be calculated using a depth map generated using a depth capture device.

[0168] The photometric weighting may indicate a value associated with the object from one or more photometric techniques as described above. For example, if the object is near the edge of the image's field of view, it may be assigned a weighting value of 0.9. If the object is in the center of the field of view, it may be assigned a weighting value of 1.0.

[0169] The focal reticle weight may indicate the location of the object relative to the focal reticle. The focal reticle value may be one of an enumerated set (e.g., inside, near, or outside). In some embodiments, the values ​​of the enumerated set may be predefined (e.g., if the object is inside the focal reticle, the object is inside and receives a weight of 0.5; if the object overlaps the focal reticle, the object is near and receives a weight of 0.75; if the object is outside the focal reticle, the object is outside and receives a weight of 1).

[0170] The gaze weight may indicate where the user is looking relative to the object. The gaze value may be one of an enumerated set (e.g., "at the vector," "near the vector," or "outside the vector"). In some examples, the values ​​of the enumerated set may be predefined (e.g., if the gaze is a second distance (greater than the first distance) from the object, the gaze is "outside the vector" and receives a weighting of 0.5; if the gaze is a first distance away from the object, the gaze is "near the vector" and receives a weighting of 0.75; if the gaze is towards the object, the gaze is "at the vector" and receives a weighting of 1).

[0171] 9, a first neural network for a first object (910) may be used. For the first neural network, each weighting described above for the first object may be one of the inputs 920. For example, the object priority weighting may be exposure weighting parameter 1 (922), the object size weighting may be exposure weighting parameter 2 (924), and the gaze weighting may be exposure weighting parameter n (926). It should be appreciated that any number of weightings may be used as inputs 920.

[0172] The input 920 may then be passed to the hidden layer 930. The hidden layer 930 is illustrated as one level, but there may be more levels, depending on the implementation of the neural network. For each level in the hidden layer 930, each value from the previous level (which would be the input 920 for the first level) is multiplied by a value determined by the neural network for each node in the level (e.g., node 932, node 934, node 936, and node 938). At each node, a function is applied to the node's input. After the hidden layer 930 is complete, the final value is multiplied by each output, and the results are combined to create the Object 1 Exposure Weights (922).

[0173] As can be seen in FIG. 9, a similar process as described above may be performed for each object in the image (e.g., Object 2 neural network (950) and Object M neural network (960)).

[0174] 10 illustrates an example of a second portion of a second weighting model that may be used for automatic exposure control (AEC). The second portion may illustrate what occurs after one or more neural networks are implemented for each object in an image. The neural networks (e.g., object 1 neural network 1030, object 2 neural network 1040, object 3 neural network 1050, object 4 neural network 1060, and object 5 neural network 1070) may each output a single value (as described above). The single value may be a weight for the object associated with the neural network.

[0175] A single value may be applied to all of the pixels of an object (as shown in total weight array 1080). Illustrated in equation form, the total weight array w T may be the sum of the outputs from the neural network of each object, e.g. [ka] where N o is the number of objects.

[0176] After the total weighted array 1080 is generated, the total weighted array 1080 may be compared to the target as described above with reference to Figure 1. Additionally, using the weighted intensity average equation described above with reference to Figure 1, a specific weight may be disabled for the second weighting model by setting the array associated with the weight to zero.

[0177] 11 is a flowchart illustrating an embodiment of a process 1100 for automatic exposure control (AEC) using the second weighting model. In some aspects, the process 1100 may be implemented by a computing device.

[0178] Process 1100 is illustrated as a logical flow diagram, whose operations represent a sequence of actions that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the actions represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described actions. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular data types. The order in which the actions are described is not intended to be construed as a limitation, and any number of the described actions may be combined in any order and / or in parallel to implement a process.

[0179] Additionally, process 1100 may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As noted above, the code may be stored on a machine-readable storage medium, e.g., in the form of a computer program comprising a plurality of instructions executable by one or more processors. The machine-readable storage medium may be non-transitory.

[0180] Process 1100 may include receiving (1105) an image captured by a content capture device. In some embodiments, the image may include multiple pixels. In some embodiments, the image may be received from a camera or other content capture device. In other embodiments, the image may be received from a feed (or stream) of images. In such embodiments, the feed may be current or past images.

[0181] Process 1100 may further include identifying (1110) a target luminance value for the image. In some embodiments, the target luminance value may indicate an optimal amount of luminance for the image. In such embodiments, each pixel in the image may be associated with a luminance value. In some embodiments, the target luminance value may correspond to an average of the luminance values ​​of the image. In other embodiments, the target luminance value may correspond to a weighted average of the luminance values ​​of the image. In some embodiments, the target luminance value may be predefined. In other embodiments, the target luminance value may be determined based on one or more sensors. For one illustrative embodiment, the sensor may detect the amount of light in the environment.

[0182] Process 1100 may further include identifying (1115) an object in the image. In some examples, the object may be identified using an object identification system. In such examples, the object identification system may identify the type of object (e.g., an individual, an animal, a building, or the like). In other examples, the object identification system may identify one or more attributes or characteristics of the object beyond just the type of object (e.g., that the object is Shania Twain). In some examples, the object identification system may identify whether the object is being processed by another system (such as a mixed reality system that is determining where to place a virtual object).

[0183] Process 1100 may further include identifying (1120) one or more attributes of the object. In some embodiments, the attribute may be associated with a weighting array, such as a weighting array for object priority, object size, object distance, photometry, focus reticle, line of sight, or the like (as described with reference to FIG. 9). In such embodiments, the attribute may be a single value representing the average of the respective weights of a group of pixels of the object.

[0184] Process 1100 may further include calculating (1125) weights for the object using a neural network. In some embodiments, the neural network may use one or more attributes as input (as described with reference to FIG. 9). In such embodiments, one or more attributes may be associated with a single object. Additional neural networks may be used for one or more other objects (or each of the one or more other objects). In other embodiments, a neural network may be associated with multiple objects.

[0185] Process 1100 may further include dividing (1130) the plurality of pixels of the image into a plurality of pixel groups. In some embodiments, the pixel groups may represent adjacent (or adjacent / neighboring) groups of pixels. In such embodiments, the shape of the pixel groups may vary. For example, each pixel group may be rectangular or square, such that the image is divided into rows and columns. In other embodiments, each pixel group may include a certain number of pixels from the center of the image. In some embodiments, different pixel groups may be different shapes (if the pixel groups are created to include objects). In some embodiments, the pixel groups may be different sizes.

[0186] Process 1100 may further include defining (1135) a first set of pixels that are not associated with the object. In some embodiments, the first set of pixels may include one or more other objects and / or may not include any objects. In some embodiments, not being associated with the object may indicate that no pixels of one or more pixels identified as the object are included in the first set of pixels.

[0187] Process 1100 may further include defining 1140 a second set of pixels associated with the object. In some embodiments, being associated with the object may indicate that a pixel of the one or more pixels identified as the object is included in the second set of pixels.

[0188] Process 1100 may further include calculating (1145A) a pixel group luminance value for each pixel group in the second set of pixels. In some embodiments, the pixel group luminance value for the pixel group may comprise an average of the luminance values ​​for each pixel of the pixel group. In other embodiments, the pixel group luminance value may be a sum of the luminance values ​​for each pixel of the pixel group. In other embodiments, the pixel group luminance value may be the difference between the luminance value of each pixel and an average luminance value for the image. For example, an average luminance value may be calculated for a frame considering all pixels. The average luminance value may then be used to calculate a pixel-by-pixel difference such that a difference from the average luminance value is generated for the pixel group. It should be appreciated that other summary values ​​of the luminance values ​​of the pixel group may also be used.

[0189] Additionally, process 1100 may further include, for each pixel group in the second set of pixels, multiplying the pixel group intensity value by a weighting to provide a weighted pixel group intensity value (1145B). In some embodiments, the weighting may be the same for each pixel group associated with the object.

[0190] Process 1100 may further include calculating (1150) a total luminance value for the image. In some embodiments, the total luminance value may comprise a sum of the weighted pixel group luminance values. In some embodiments, calculating the image luminance value may comprise summing each of the weighted pixel group luminance values, and the weighted pixel group luminance value may be calculated by multiplying the weight associated with the pixel group by the luminance value associated with the pixel group.

[0191] Process 1100 may further include calculating (1155) a difference between the total luminance value and the target luminance value and updating (1160) a setting of the content capture device based on the calculated difference. In some embodiments, the setting may be a gain setting or an exposure setting. In such embodiments, the exposure setting may be a shutter speed, an ISO speed, or any combination thereof. In some embodiments, the shutter speed may be a global shutter or a rolling shutter. A global shutter may indicate a duration for exposing all pixels in the field of view. A rolling shutter may indicate a duration for exposing a row (either horizontally or vertically) of the field of view. With a rolling shutter, a line of the image may be scanned in a rolling manner rather than a snapshot of the field of view. The gain setting may be a digital gain, an analog gain, or any combination thereof. For example, the gain setting may be 8x by having 2x analog gain and 4x digital gain. In some embodiments, the exposure setting may be adjusted before the gain setting when increasing exposure. In such embodiments, the gain setting may be adjusted before the gain setting when decreasing exposure. In some embodiments, the amount of adjustment may be proportional to the difference. For example, the adjustment may be greater if the difference between the image luminance value and the target luminance value is greater.

[0192] 18 is a flowchart illustrating an embodiment of a process 1800 for automatic exposure control using multiple models. When using multiple models, two or more of the multiple models may be the same or different types of models. For example, multiple neural networks (as described above in FIGS. 9-11) may be used. When multiple models are used (according to this figure), the multiple models may each provide a weighting for a particular portion of the image such that the weightings associated with each of the multiple models do not overlap.

[0193] In some aspects, process 1800 may be implemented by a computing device. Process 1800 is illustrated as a logical flow diagram, whose operations represent a sequence of actions that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the actions represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described actions. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular data types. The order in which the actions are described is not intended to be construed as a limitation, and any number of the described actions may be combined in any order and / or in parallel to implement the process.

[0194] Additionally, process 1800 may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As noted above, the code may be stored on a machine-readable storage medium, e.g., in the form of a computer program comprising a plurality of instructions executable by one or more processors. The machine-readable storage medium may be non-transitory.

[0195] Process 1800 may include receiving 1802 an image captured by a content capture device. In some embodiments, the image may include multiple pixels. In some embodiments, the image may be received from a camera or other content capture device. In other embodiments, the image may be received from a feed (or stream) of images. In such embodiments, the feed may be images that have either been recently taken or stored.

[0196] Process 1800 may further include identifying (1804) a target luminance value for the image. In some embodiments, the target luminance value may indicate an optimal amount of luminance for the image. In such embodiments, each pixel in the image may be associated with a luminance value. In some embodiments, the target luminance value may correspond to an average of the luminance values ​​of the image. In other embodiments, the target luminance value may correspond to a weighted average of the luminance values ​​of the image. In some embodiments, the target luminance value may be predefined. In other embodiments, the target luminance value may be determined based on one or more sensors. For one illustrative embodiment, the sensor may detect the amount of light in the environment.

[0197] Process 1800 may further include dividing (1806) the plurality of pixels of the image into a plurality of pixel groups. In some embodiments, the pixel groups may represent adjacent (or adjacent / neighboring) groups of pixels. In such embodiments, the shapes of the pixel groups may vary. For example, each pixel group may be rectangular or square, such that the image is divided into rows and columns. In other embodiments, each pixel group may include a certain number of pixels from the center of the image. In some embodiments, different pixel groups may be different shapes (if the pixel groups are created to include objects). In some embodiments, the pixel groups may be different sizes.

[0198] Process 1800 may further include identifying (1808) multiple patches within the image. The multiple patches may include a first patch and a second patch. Each patch may include one or more groups of pixels. In some embodiments, some patches may include more or fewer groups of pixels than other patches. In some embodiments, the patches may be identified based on the content of the image. For example, one or more objects may be identified within the image, with each patch including pixels of a different object. In other embodiments, each patch may be a set of one or more pixels, with each set being used for whatever image is received. It should be appreciated that other methods for identifying patches may also be used.

[0199] Process 1800 may further include calculating 1810 one or more weights for the first patch. The one or more weights for the first patch may be calculated using a first model. The first model may be any model described herein, such as an object priority weighting array, an object size weighting array, an object distance weighting array, a photometry weighting array, a focus reticle weighting array, a line of sight weighting array, a neural network based on content contained in the first patch, or the like.

[0200] Process 1800 may further include calculating one or more weights for the second patch (1812). The one or more weights for the second patch may be calculated using a second model, which may be the same as the first model or a different model from the first model. It should be appreciated that more than two models may be used. It should also be appreciated that some patches may use the same model, while other patches in the same image may use different models. In such cases, patches determined to have one or more particular characteristics in common may ultimately use the same model.

[0201] Process 1800 may further include calculating (1814) a pixel group luminance value for each pixel group in the second set of pixels. In some embodiments, the pixel group luminance value for the pixel group may comprise an average of the luminance values ​​for each pixel of the pixel group. In other embodiments, the pixel group luminance value may be a sum of the luminance values ​​for each pixel of the pixel group. In other embodiments, the pixel group luminance value may be the difference between the luminance value of each pixel and an average luminance value for the image. For example, an average luminance value may be calculated for a frame considering all pixels. The average luminance value may then be used to calculate a pixel-by-pixel difference such that a difference from the average luminance value is generated for the pixel group. It should be appreciated that other summary values ​​of the luminance values ​​of the pixel group may also be used.

[0202] Additionally, process 1800 may further include, for each pixel group in the second set of pixels, multiplying the pixel group luminance value of the pixel group by a weight for the pixel group to provide a weighted pixel group luminance value (1816). In some embodiments, the weight may be the same for each pixel group associated with the object.

[0203] Process 1800 may further include calculating 1818 a total luminance value for the image. In some embodiments, the total luminance value may include a sum of the weighted pixel group luminance values. In some embodiments, calculating the image luminance value may include summing each of the weighted pixel group luminance values, and the weighted pixel group luminance value may be calculated by multiplying a weight associated with a pixel group by the luminance value associated with the pixel group.

[0204] Process 1800 may further include calculating (1820) a difference between the total luminance value and the target luminance value and updating (1822) a setting of the content capture device based on the calculated difference. In some embodiments, the setting may be a gain setting or an exposure setting. In such embodiments, the exposure setting may be a shutter speed, an ISO speed, or any combination thereof. In some embodiments, the shutter speed may be a global shutter or a rolling shutter. A global shutter may indicate a duration for exposing all pixels in the field of view. A rolling shutter may indicate a duration for exposing a row (either horizontally or vertically) of the field of view. With a rolling shutter, a line of the image may be scanned in a rolling manner rather than a snapshot of the field of view. The gain setting may be a digital gain, an analog gain, or any combination thereof. For example, the gain setting may be 8x by having 2x analog gain and 4x digital gain. In some embodiments, the exposure setting may be adjusted before the gain setting when increasing exposure. In such embodiments, the gain setting may be adjusted before the gain setting when decreasing exposure. In some embodiments, the amount of adjustment may be proportional to the difference. For example, the adjustment may be greater if the difference between the image luminance value and the target luminance value is greater.

[0205] FIG. 12A illustrates an image stitching process 1200. In some embodiments, the image stitching process 1200 may include multiple instances of an automatic exposure control (AEC), each adjusting one or more settings, whether the same or different. Any weighting model, including the first and second weighting models discussed above, may be used for the AEC instances. In some embodiments, the image stitching process 1200 may create an output image 1204 that combines two or more images. In such embodiments, the image stitching process 1200 may combine images captured based on adjustments from each of the multiple instances of the AEC. For example, an image may be captured after one or more settings are set due to an instance of the AEC, and then combined with other images similarly captured after one or more settings are set due to other instances of the AEC.

[0206] In some embodiments, the content capture device 1210 may be configured in a high dynamic range (HDR) mode. HDR mode may facilitate capturing multiple images of similar scenes and stitching the multiple images together. In such embodiments, HDR mode may include the ability to change one or more settings for a particular image. In a traditional HDR mode, software and / or hardware associated with the content capture device may be programmed to automatically adjust the exposure value (e.g., a number representing the combination of the shutter speed and f-stop of the content capture device, sometimes abbreviated as EV) for a sequence of three or five frames by + / -1 EV or + / -1 EV and + / -3E. In some embodiments, the EV may be an exposure step of a predetermined size. For example, a three-frame HDR mode may adjust the EV to -1 for the first frame, 0 for the second frame, and +1 for the third frame. In such embodiments, the traditional HDR mode may blindly adjust the exposure by a fixed amount in an attempt to increase the contrast of the image. The object-based exposure method described herein can provide appropriate settings for the number of objects in an image. For example, a three-frame HDR mode may adjust the EV for a first object in a first frame, a second object in a second frame, and the remaining objects in a third frame.

[0207] In some examples, an AEC instance may be associated with one or more objects in an image. In such examples, the AEC instance may determine to update one or more settings of the content capture device 1210 for the one or more objects. In some examples, each instance of the AEC may be associated with a different image and one or more different objects, such that updates from the instance are customized for one or more objects in the instance. In such examples, images captured using different updates (each update for one or more different objects) may be combined to create images that are individually and separately customized for the different objects.

[0208] In some embodiments, the image stitching process 1200 may include receiving three images (e.g., image n (1212), image n+1 (1214), and image n+2 (1216)) from the content capture device 1210. The three images may be received at once or sequentially. In addition, it should be appreciated that more or less than three images may be received.

[0209] In some embodiments, the three images may be received by the image controller 1220. The image controller 1220 may determine the AEC instance to which to send the images. In some embodiments, the image controller 1220 may send the images to the AEC instances in a loop as the images are received. For example, the loop may cause a first image (e.g., image n (1212)) to be sent to the first AEC instance 1230, a second image (e.g., image n+1 (1214)) to be sent to the second AEC instance 1240, and a third image (e.g., image n+2 (1216)) to be sent to the third AEC instance 1250. Using the loop order, the fourth image may be sent to the first AEC instance 1230 (similar to the first image), the fifth image may be sent to the second AEC instance 1240 (similar to the second image), and the sixth image may be sent to the third AEC instance 1250 (similar to the third image).

[0210] It should be appreciated that objects in an image can be divided into larger or smaller groups so that more or fewer images are included in one complete loop. For example, a loop of size 2 would cause the third image to be sent to the first AEC instance rather than the third AEC instance. In some embodiments, the images of the image stitching process 1200 may be associated with approximately the same scene so that the images can be easily combined together.

[0211] In some examples, the first AEC instance 1230 may determine one or more first settings 1232 (e.g., exposure settings, gain settings, or any combination thereof) for one or more first objects from the first image. The one or more first settings 1232 may be sent to the content capture device 1210 to be used for future images. In some examples, the future image may be the next image. In other examples, the future images may be based on the number of AEC instances. For example, if there are three AEC instances, the future images may be three images in the future.

[0212] In some examples, the second AEC instance 1240 may determine one or more second settings 1242 (e.g., exposure settings, gain settings, or any combination thereof) for one or more second objects from the second image. In such examples, the second image may be captured before or after the first image. The one or more second settings 1242 may be sent to the content capture device 1210 to be used for future images, similar to what is described above with respect to the first AEC instance.

[0213] In some examples, the third AEC instance 1250 may determine one or more third settings 1252 (e.g., exposure settings, gain settings, or any combination thereof) for one or more third objects from the third image. In such examples, the third image may be captured before or after the second image. The one or more third settings 1252 may be sent to the content capture device 1210 to be used for future images, similar to what is described above with respect to the first AEC instance.

[0214] In some embodiments, after an adjustment to content capture device 1210 from an AEC instance, content capture device 1210 may continue to capture images that are sent to image controller 1220. In such embodiments, as the adjustment is made, the new image may be optimized for the particular object, thereby incrementally improving the exposure for the image.

[0215] In some embodiments, image controller 1220 may also send the images to image stitcher 1202. Image stitcher 1202 may combine the images based on the identified objects. For example, image stitcher 1202 may receive a first image optimized for one or more first objects, a second image optimized for one or more second objects, and a third image optimized for other portions not included in the first and second objects and / or the third image as a whole. In such embodiments, image stitcher 1202 may create output image 1204 using portions of the first image associated with one or more first objects, portions of the second image associated with one or more second objects, and portions of the third image.

[0216] In some embodiments, the image stitcher 1202 may wait before stitching images together for a number of images corresponding to the number of AEC instances. For example, if there are three AEC instances, the image stitcher 1202 may wait until it receives three images before combining (or stitching) the images.

[0217] In some examples, the image stitcher 1202 may recognize the identified objects and their relative priorities. In such examples, the image stitcher 1202 may output the output image 1204 based on the identified objects and their relative priorities. For example, the image stitcher 1202 may first generate a low-priority portion and then generate a high-priority portion over the low-priority portion. In other words, an image adjusted for a high-priority object is overlaid on an image adjusted for a low-priority object. In some examples, the image stitcher 1202 may generate an image. In such examples, the image stitcher 1202 may be programmed to operate on three images with assigned priorities. As the AEC accepts each of the three images, the image stitcher 1202 may generate a progressively better stitched image. The stitched image may be used in a variety of ways, including for display to a user (e.g., in a viewfinder of the content capture device or on a remote display) as input to an object identification system so that objects are identified in the stitched image rather than in an unoptimized image, or for storage in memory (local or remote from the content capture device) for later use.

[0218] According to one embodiment of the present invention, a method is provided. The method includes receiving a first image captured by a content capture device, identifying a first object in the first image, and determining a first update to a first setting of the content capture device, the first update being determined with respect to the first object. The method also includes receiving a second image captured by the content capture device, the second image being captured after the first image, identifying a second object in the second image, and determining a second update to a second setting of the content capture device, the second update being determined with respect to the second object. The method further includes implementing the first update to the first setting of the content capture device, receiving a third image captured by the content capture device, the third image being captured after the first update was implemented, and implementing the second update to the second setting of the content capture device. The method additionally includes a step of receiving a fourth image captured by the content capture device, the fourth image being captured after the second update is performed, and a step of combining the third image and the fourth image into a single image.

[0219] As an example, the first setting and the second setting can be associated with exposure or gain. The first setting can be the second setting. The third image and the fourth image can be combined using an image stitcher. The first update can be determined using a neural network. In an embodiment, the first update is different from the second update. For example, the first image and the second image can have the same field of view.

[0220]

[00130] Figure 12B illustrates another example of an image stitching process that may use multiple instances of automatic exposure control. The process illustrated in Figure 12B shares some similarities with that illustrated in Figure 12A, and the description provided with respect to Figure 12A is applicable to Figure 12B, where appropriate. With reference to Figure 12B, a content capture device 1210 (referred to as a camera in Figure 12B for clarity purposes) is used to capture a first image (image N 1261).

[0221] Image N 1261 is provided to image controller 1220. Image controller 1220 identifies up to a predetermined number of priority objects in image N, the predetermined number being two or more. As an example, two to five priority objects can be identified in an embodiment. The example illustrated in FIG. 12B uses three priority objects 1267, 1268, and 1269 in composite image 1266. Image 1266 may have three priority objects because 1) that is the maximum allowed (i.e., the predetermined number of priority objects was three, and even if more than three potential priority objects were identified, only the top three options were selected), or 2) only three objects that could be considered priority objects were identified in the image even if the maximum was not reached (e.g., the predetermined number of priority objects was five), or 3) the system will always return the predetermined number of priority objects regardless of what is contained in the scene (i.e., the system will force the selection of three). While three priority objects are illustrated in Figure 12B, it will be apparent to one skilled in the art that fewer (e.g., two) or more (e.g., four or more) may be utilized in embodiments of the present invention. In some examples, a set of priority objects is selected for use from a larger set of priority objects identified in the first image. To reduce system complexity and computational burden, the number of priority objects may be limited to a given number, e.g., a maximum of five priority objects.

[0222] The image controller 1220 sends the image to multiple AEC instances 1230, 1240, and 1250, where the number of AEC instances is equal to the number of priority objects identified. In some embodiments, the image controller 1220 may send image N to N AEC instances simultaneously in a serial manner.

[0223] An AEC instance is used to determine updates for one or more settings of the content capture device for each of a predetermined number of priority objects. As shown in FIG. 12B , image N 1261 can be analyzed using three different AEC instances 1230, 1240, and 1250 to determine updates for one or more settings, each customized for one of the priority objects. Thus, AEC instances are used to determine settings for the content capture device, with each AEC instance performing a function customized for one of the specific priority objects. The settings may be exposure settings, gain settings, or any combination thereof.

[0224] An iterative process is then performed, the number of iterations equaling a predetermined number. As illustrated in FIG. 12B , sets of images N+1, N+2, and N+3 are captured as the content capture device is updated using each of a predetermined number of updates (i.e., three updates). The first image is customized to a first priority object and, in embodiments, provides optimized exposure settings for the first priority object. In an iterative manner, the content capture device is updated using the next update for one or more settings of the content capture device, and the next image is acquired. This process continues with each update until the predetermined number of images have been captured. Thus, in FIG. 12B , sets of images N+1, N+2, and N+3 are each captured using one or more settings customized (e.g., optimized) to individual priority objects 1267, 1268, and 1269. Of course, more than three settings and images can be captured.

[0225] As would be apparent to one skilled in the art, the set of images 1265 would each include a portion associated with each of a predetermined number of priority objects. As an example, image N+1 may be associated with priority object 1267, image N+2 may be associated with priority object 1268, and image N+3 may be associated with priority object 1269. The sets of images, each captured using settings customized (e.g., optimized) for a particular priority object, are provided to image stitcher 1260. Image stitcher 1260 is used to stitch the predetermined number of images (i.e., set of images 1265) together into a composite image 1266. As illustrated in FIG. 12B , the set of images 1265 can be used to extract portions of each image associated with one of the priority objects, and these portions can then be stitched together to form composite image 1266 combining the portions into a single image. Since each portion of the set of images is optimized for one of the priority objects, a single image will include an optimized image for each priority object. The composite image can then be displayed to the user. Also, in some implementations, the functionality of the image controller 1220 and the image stitcher 1260 can be combined in a single master image processing unit.

[0226] FIG. 12C is a flowchart illustrating an embodiment of a process for using multiple instances of automatic exposure control. Method 1270 includes receiving an image from a content capture device (1272) and identifying a predetermined number (n) of priority objects within the image (1274). In an embodiment, the identified priority objects can be the top n priority objects within the image. While three priority objects are illustrated in FIG. 12B, embodiments of the invention are not limited to three, and n can be greater than or less than three. The method also includes determining content capture device settings for priority object 1 (1276), determining content capture device settings for priority object 2 (1278), and determining content capture device settings for priority object n (1280). By way of example, the settings, also referred to as one or more settings, can be optimized settings in the sense that they provide exposure values ​​that result in a desired (e.g., best) exposure of the priority object of interest. As illustrated in FIG. 12B, AEC instances 1230, 1240, and 1250 can be utilized to provide these settings, which may include exposure settings, gain settings, or any combination thereof.

[0227] One or more first settings may be sent to the content capture device to adjust the settings. Thus, the settings for the content capture device are adjusted for priority object 1 (1282), priority object 2 (1284), and priority object 3 (1286). Using the updated set of settings, a set of images is captured using the content capture device. An image adapted for priority object 1 is captured (1288), an image adapted for priority object 2 is captured (1290), and an image adapted for priority object 3 is captured (1292). Because these new images are optimized for the particular priority object, the exposure for each of these images is improved for each priority object.

[0228] The method also includes stitching portions of the set of images together to form a composite image adapted for priority objects 1 through n 1294. Thus, the image stitcher 1202 illustrated in Figure 12B may use a portion of a first image associated with a first priority object, a portion of a second image associated with a second priority object, and a portion of a third image associated with a third priority object to create a single image 1266. The discussion related to image stitching in Figure 12A is applicable, where appropriate, to the method described with respect to Figure 12C.

[0229] Thus, using a single image received in block 1272 that includes multiple priority objects, one or more AEC instances can be used to determine updated (e.g., optimized) settings that are specific to each of the priority objects identified in the single image. Then, using the updated settings, a set of images can be captured, with each image in the set customized (e.g., optimized) for each priority object. Stitching together portions of each image in the set, where the portions are associated with one of the priority objects, allows for the generation of a stitched image that can be displayed to a user 1296, where exposure settings vary throughout the composite image and are customized to the priority object.

[0230] 13 illustrates an example of an image stream that may be used in conjunction with an image stitching process (e.g., image stitching process 1200). The image stream illustrates multiple images that may be received by an image controller (e.g., image controller 1220). In the example of FIG. 13, an object in the image with a dotted line indicates that an AEC instance is not centered on the object, while an object in the image with a solid line indicates that an AEC instance is implemented based on the object.

[0231] The images in Figure 13 are separated into images sent to AEC instances. For example, image 1 (1332) is sent to a first AEC instance 1330. Image 1 (1332) is illustrated as having a first object 1362 with a solid line, indicating that the first AEC instance 1330 is implemented based on the first object 1362. Image 1 (1332) also has a second object 1364 and a third object 1366. The second object 1364 and the third object 1366 are illustrated as having dotted lines, indicating that the first AEC instance 1330 treats both objects as unidentified.

[0232] The first AEC instance 1330 may determine adjustments for one or more settings (e.g., exposure settings, gain settings, or any combination thereof) of a content capture device (e.g., content capture device 1210). The adjustments may be sent to the content capture device 1210 such that the content capture device 1210 adjusts the one or more settings of the content capture device 1210. In some examples, the one or more settings may be sent to the content capture device 1210 just before the content capture device 1210 captures an image. In such examples, in response to receiving the one or more settings, the content capture device 1210 may adjust its settings. In other examples, the content capture device 1210 may be configured to receive the one or more settings and wait until it is time to capture an image with the one or more settings (such as based on a loop as described above).

[0233] Image 2 (1368) may be sent to the second AEC instance 1340. The second AEC instance 1340 may be implemented based on an object corresponding to the second object 1364. Image 3 (1370) may be sent to the third AEC instance 1350. The third AEC instance 1350 may be implemented based on an object corresponding to the third object 1366. The second and third AEC instances (1340, 1350) may also determine and send one or more settings to the content capture device 1210 (similarly as described above with respect to the first AEC instance 1330).

[0234] Image 1 (1360), image 2 (1368), and image 3 (1370) may represent a loop. After one or more settings are set due to an AEC instance, additional images may be captured with the one or more settings. After the additional images are captured, the image controller may determine the AEC instance to which the one or more settings were set. The image controller may then send the additional images to the determined AEC instance. For example, the first AEC instance 1330 may set one or more first settings based on image 1 (1360). When the image controller receives image 4 (1372), which may be captured using the one or more first settings, the image controller may send image 4 (1372) to the first AEC instance 1330. Similarly, image 5 (1374) may be sent to the second AEC instance 1340, and image 6 (1376) may be sent to the third AEC instance 1350.

[0235] Images 4, 5, and 6 (1372, 1374, 1376) may represent a second loop. This process may continue until images n, n+1, and n+2 (1378, 1380, 1382) are received. A similar process may occur, including determining the AEC instance associated with the new image and sending the new image to the AEC instance that generated the one or more settings used to capture the new image. For example, image n (1378) may be sent to the first AEC instance 1330, image n+1 (1380) may be sent to the second AEC instance 1340, and image n+2 (1382) may be sent to the third AEC instance 1350.

[0236] 14 is a flowchart illustrating an embodiment of a process 1400 for automatic exposure control using the third weighting model. In some aspects, the process 1400 may be implemented by a computing device.

[0237] Process 1400 is illustrated as a logical flow diagram, whose operations represent a sequence of actions that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the actions represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described actions. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular data types. The order in which the actions are described is not intended to be construed as a limitation, and any number of the described actions may be combined in any order and / or in parallel to implement a process.

[0238] Additionally, process 1400 may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As noted above, the code may be stored on a machine-readable storage medium, e.g., in the form of a computer program comprising a plurality of instructions executable by one or more processors. The machine-readable storage medium may be non-transitory.

[0239] Process 1400 may include receiving 1405 a first image captured by a content capture device. In some embodiments, the content capture device may be in a high dynamic range (HDR) mode. HDR mode may cause multiple photographs to be taken close in time with varying settings (e.g., exposure settings, gain settings, or any combination thereof).

[0240] Process 1400 may further include identifying (1410) a first object in the first image. In some examples, the first object may be identified using an object identification system. In such examples, the object identification system may identify the type of the first object (e.g., an individual, an animal, a building, or the like). In other examples, the object identification system may identify one or more attributes or characteristics of the first object beyond just the type of the first object (e.g., that the first object is Bob Marley). In some examples, the object identification system may identify whether the first object is being processed by another system (such as a mixed reality system that is determining where to place a virtual object).

[0241] Process 1400 may further include determining (1415) a first update to a first setting of the content capture device. In some embodiments, the first update may be determined with respect to the first object. In such embodiments, the first update may be determined using any of the weighting models described herein, including the first weighting model and / or the second weighting model. In some embodiments, the first setting may be an exposure setting, a gain setting, or any combination thereof. In such embodiments, the exposure setting may be a shutter speed, an ISO speed, or any combination thereof. The gain setting may be a digital gain, an analog gain, or any combination thereof.

[0242] Process 1400 may further include receiving (1420) a second image captured by the content capture device. In some embodiments, the second image may be captured before or after the first image. In some embodiments, the second image may be captured according to an HDR mode.

[0243] Process 1400 may further include identifying a second object in the second image (1425). In some embodiments, the second object may be identified using the object identification system described above.

[0244] Process 1400 may further include determining (1430) a second update to a second configuration of the content capture device. In some embodiments, the second update may be determined with respect to a second object. In some embodiments, the second update may be determined using any of the weighting models described herein, including the first weighting model and / or the second weighting model. In such embodiments, the weighting model used for the second update may be the same as or different from the weighting model used for the first update.

[0245] Process 1400 may further include performing (1435) a first update to the first settings of the content capture device. Performing the first update may include sending one or more instructions to the content capture device that cause the content capture device to change the first settings. In some embodiments, the one or more instructions may be sent that cause the first settings to be updated at a future time (e.g., when an image that would have used the first settings is about to be taken). In other embodiments, the one or more instructions may be sent after an image is captured (or received) from the content capture device, the capture indicating that a new image that may use the first settings is about to be captured.

[0246] Process 1400 may further include receiving (1440) a third image captured by the content capture device. In some embodiments, the third image may be captured after the first update is implemented. In such embodiments, the first update may be applied to the content capture device such that the third image is captured with an exposure based on the first update.

[0247] Process 1400 may further include performing (1345) a second update to the second settings of the content capture device. Performing the second update may include sending one or more instructions to the content capture device that cause the content capture device to change the second settings. In some embodiments, the one or more instructions may be sent that cause the second settings to be updated at a future time (e.g., when an image that would use the second settings is about to be captured). In other embodiments, the one or more instructions may be sent after an image is captured (or received) from the content capture device, the capture indicating that a new image that may use the second settings is about to be captured.

[0248] Process 1400 may further include receiving (1450) a fourth image captured by the content capture device. In some embodiments, the fourth image may be captured after the second update is implemented. In such embodiments, the second update may be applied to the content capture device such that the fourth image is captured with an exposure based on the second update.

[0249] Process 1400 may further include combining (1455) the third image and the fourth image into a single image for viewing. In some examples, the combining (e.g., stitching) may include capturing a portion of the third image and a portion of the fourth image. For example, the third image may correspond to a first image optimized for a first object. In such examples, the portion of the captured third image may be associated with the first object, such that the combining includes capturing a portion of the third image associated with the first object. Similarly, the fourth image may correspond to a second image optimized for a second object. Thus, the portion of the captured fourth image may be associated with a second object, such that the combining may include capturing a portion of the fourth image associated with the second object. After capturing the portions of the third and fourth images associated with the first and second objects, respectively, other portions of the third and fourth images may be averaged between the third and fourth images. The other portions may not be associated with either the first object or the second object, such that the other portions were not optimized by the first update and / or the second update. However, it should be recognized that images may be stitched (or combined) in a variety of ways.

[0250] In some embodiments, process 1400 may include a first cycle and a second cycle. The first cycle may include a first image and a second image. The second cycle may include a third image and a fourth image. In some embodiments, each new cycle may include two more images so that operations continue to be performed with each new cycle. In some embodiments, each new cycle may be used to update settings of the content capture device (as with the first and second images) and may be combined (as with the third and fourth images). For example, determining updates (as described with respect to the first and second images) may be performed with respect to the third and fourth images.

[0251] 15 illustrates an example of a block diagram of a computer system. In this example, computer system 1500 includes a monitor 1510, a computer 1520, a keyboard 1530, a user input device 1540, and one or more computer interfaces 1550 or the like. In this example, user input device 1540 is typically embodied as a computer mouse, trackball, trackpad, joystick, wireless remote, drawing tablet, voice command system, eye tracking system, and the like. User input device 1540 typically allows a user to select objects, icons, text, and the like that appear on monitor 1510 via commands such as clicking a button or the like.

[0252] Examples of computer interface 1550 typically include an Ethernet card, a modem (telephone, satellite, cable, ISDN), an (asynchronous) digital subscriber line (DSL) unit, a FireWire interface, a USB interface, and the like. For example, computer interface 1550 may be coupled to computer network 1555, a FireWire bus, or the like. In other embodiments, computer interface 1550 may be a software program, such as soft DSL, that may be physically integrated on the motherboard of computer 1520, or the like.

[0253] In various embodiments, computer 1520 typically includes well-known computer components such as a processor 1560, a memory storage device such as random access memory (RAM) 1570, a disk drive(s) 1580, and a system bus 1590 interconnecting the above components.

[0254] RAM 1570 and disk drive 1580 are examples of tangible media configured to store data, such as embodiments of the present disclosure, including executable computer code, human-readable code, or the like. Other types of tangible media include floppy disks, removable hard disks, optical storage media such as CD-ROMs, DVDs, and barcodes, semiconductor memory such as flash memory, read-only memories (ROMS), battery-backed volatile memory, networked storage devices, and the like.

[0255] In various embodiments, computer system 1500 may also include software that enables communication over a network, such as HTTP, TCP / IP, RTP / RTSP protocols, and the like. In alternative embodiments of the present disclosure, other communication software and transport protocols, e.g., IPX, UDP, or the like, may also be used.

[0256] The features described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations thereof. The apparatus may also be implemented in a computer program product tangibly embodied in an information carrier (e.g., in a machine-readable storage device) for execution by a programmable processor, and the method steps may be performed by the programmable processor executing a program of instructions to perform the functions of the described implementation by operating on input data and generating output. The described features may advantageously be implemented in one or more computer programs executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used directly or indirectly by a computer to perform an activity or bring about a result. The computer program may be written in any form of programming language, including a compiled or interpreted language, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use within a computing environment.

[0257] Suitable processors for the execution of a program of instructions include, by way of example, both general-purpose and special-purpose microprocessors, and the sole processor or one of multiple processors of any kind of computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data files; such devices will include magnetic disks, such as internal hard disks and removable disks, magneto-optical disks, and optical disks. Suitable storage devices for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, ASICs (application-specific integrated circuits).

[0258] To provide for interaction with a user, features may be implemented on a computer having a display device such as a CRT (cathode ray tube), LCD (liquid crystal display), LED (light emitting diode) monitor, etc. for displaying information to the user, and a keyboard and pointing device such as a mouse or trackball, by which the user may provide input to the computer.

[0259] Features may be implemented in a computer system that includes back-end components such as data servers, or includes middleware components such as application servers or Internet servers, or includes front-end components such as client computers having a graphical user interface or Internet browser, or any combination thereof. The components of the system may be connected by any form or medium of digital data communication, such as a communications network. Examples of communications networks include LANs, WANs, and the computers and networks forming the Internet.

[0260] The computer system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a network, such as that described. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. Although several implementations have been described in detail above, other modifications are possible.

[0261] Additionally, the logic flows depicted in the figures do not require the particular order or sequential order shown to achieve desirable results. Additionally, other steps may be provided or steps may be eliminated from the described flows, and other components may be added to or removed from the described systems. Thus, other implementations are within the scope of the following claims.

[0262] Where a component is described as being configured to perform a certain operation, such configuration may be accomplished, for example, by designing electronic circuitry or other hardware to perform the operation, by programming programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or by any combination thereof.

[0263] Although several embodiments of the present disclosure have been described, it will nevertheless be understood that various modifications may be made without departing from the scope of the present disclosure.

Claims

1. 1. A method for calculating a total weight array, the method being performed on a computer system, the computer system including one or more processors, the method comprising: receiving, by the one or more processors, image frames captured by a content capture device; the one or more processors identifying a plurality of objects in the image frame, each object of the plurality of objects being represented by a group of pixels corresponding to a shape of each object of the plurality of objects; the one or more processors providing a plurality of neural networks; the one or more processors calculating, for each object of the plurality of objects, an object weight using a corresponding one of the plurality of neural networks; the one or more processors calculating the total weight array by summing the object weights for each of the plurality of objects; A method comprising:

2. The method of claim 1 , wherein each of the plurality of neural networks comprises a different neural network.

3. The method of claim 1 , wherein each of the plurality of objects is associated with a row r and a column c of the image frame.

4. The total weight array is [Number 100] and 4. The method of claim 3, wherein N0 is a number of objects and wi[r,c] is the object weight for each of the plurality of objects.

5. The method of claim 1 , wherein each object weight comprises a single value.

6. The method of claim 5 , wherein the single value is applied to all pixels in each of the groups of pixels of the plurality of groups of pixels.

7. The method comprises: the one or more processors identifying a target luminance value for the image frame; the one or more processors calculating an image brightness value using the total weight array; the one or more processors calculating a difference between the image luminance value and the target luminance value; the one or more processors updating a configuration of the content capture device based on the calculated difference; and The method of claim 1 further comprising:

8. 1. A method executed on a computer system, the computer system including one or more processors, the method comprising: receiving, by the one or more processors, an image captured by a content capture device; the one or more processors identifying a target luminance value for the image; the one or more processors providing a plurality of neural networks; the one or more processors identifying a plurality of objects in the image, each of the plurality of objects being represented by a group of pixels corresponding to a shape of each object of the plurality of objects; the one or more processors calculating, for each object of the plurality of objects, an object weight using a corresponding one of the plurality of neural networks; the one or more processors defining a first set of pixels associated with the plurality of objects; the one or more processors defining a second set of pixels associated with the plurality of objects; the one or more processors calculating, for each pixel group in the first set of pixels, a pixel group luminance value; the one or more processors multiplying the pixel group luminance values ​​by the object weights to provide weighted pixel group luminance values ​​for each pixel group in the first set of pixels; the one or more processors calculating a total luminance value for the image; A method comprising:

9. The method comprises: the one or more processors calculating a difference between the total luminance value and the target luminance value; the one or more processors updating a configuration of the content capture device based on the calculated difference; and The method of claim 8 further comprising:

10. The method of claim 8 , wherein the image comprises one image of a stream of images.

11. The method of claim 8 , wherein the image comprises a plurality of pixels, each of the plurality of pixels having a pixel luminance value, and the target luminance value corresponds to an average of the pixel luminance values.

12. The method of claim 8 , wherein the image comprises a plurality of pixels, each of the plurality of pixels having a pixel luminance value and a weighting, and the target luminance value corresponds to a weighted average of the pixel luminance values.

13. The method of claim 8 , wherein the method further comprises identifying one or more attributes for each of the plurality of objects in the image.

14. 14. The method of claim 13, wherein the one or more attributes include at least one of a priority weighting array for object priority, a size weighting array for object size, a distance weighting array for object distance, or a gaze weighting array for eye gaze.

15. The method of claim 13 , wherein each corresponding neural network uses the one or more attributes as input.

16. The method of claim 8 , wherein the pixel group luminance value comprises an average of the luminance values ​​for each pixel of the pixel group.

17. 9. The method of claim 8, wherein the total luminance value is equal to the sum of the weighted pixel group luminance values ​​for each pixel in the first set of pixels multiplied by a pixel group luminance value for each pixel in the first set of pixels.

18. The method of claim 8 , wherein each of the plurality of neural networks comprises a different neural network.

19. The method comprises: each neural network of the plurality of neural networks receiving a plurality of inputs corresponding to the object associated with the corresponding neural network; each neural network of the plurality of neural networks outputs a single weight for the object associated with the corresponding neural network; The method of claim 1 further comprising:

20. The method of claim 1 , wherein the method further comprises each neural network of the plurality of neural networks receiving a plurality of attributes as inputs to each neural network of the plurality of neural networks.

Images