Machine-learned model to determine high dynamic range gain map

Abstract

A computer-implemented method includes receiving, by a computing system, an input image (2100) having a first dynamic range; applying, by the computing system, a machine-learned model (2300) to the input image, to obtain a gain map (2600) for content having a second dynamic range, the second dynamic range being greater than the first dynamic range; and storing, by the computing system, the gain map in metadata associated with the input image.

Description

MACHINE-LEARNED MODEL TO DETERMINEHIGH DYNAMIC RANGE GAIN MAPFIELD

[0001] The disclosure relates to machine-learned models configured to predict high dynamic range (HDR) image rendition given only a lower resolution (e.g., low dynamic range (LDR) or standard dynamic range (SDR)) inputs. In particular, computing systems and methods described herein which implement the machine-learned models can generate gain map metadata for legacy (e.g., SDR) content so that legacy content can be rendered with improved contrast and higher peak brightness for HDR display.BACKGROUND

[0002] Modem high dynamic range (HDR) compatible displays enable the presentation of image and video content with higher contrast (ratio of peak white level to black level) along with higher peak brightness than traditional standard dynamic range (SDR) displays.

[0003] Recently, a new image format for HDR imagery' has been developed which is referred to as Ultra HDR image format which encodes a logarithmic range gain map image (a HDR gain map) in an image file. Legacy readers that don't support the new image format read and display the conventional low dynamic range image from the image file. Readers that support the new image format combine the primary image with the HDR gain map and render a high dynamic range image on compatible displays. For example, the HDR gain map and its associated metadata can be stored in the extensible metadata platform (XMP) metadata of an image (e.g., a JPEG image).

[0004] Existing works directed to reconstructing HDR images using machine-learned models describe recovering missing light energy' (pixel values) in clipped / saturated regions, and assume the input images are already in radiometrically linear space. For example, many methods assume that the tone curve of the input LDR image is a global operator (the camera response function or response curve). For example, some methods leam to invert an unknown global tone-mapping operator, however they do not handle inputs where local tonemapping (LTM) operators have been applied.SUMMARY

[0005] Aspects and advantages of embodiments of the disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0006] One example aspect of the disclosure is directed to a computer-implemented method for determining a gain map for rendering an output image having a dynamic range which is greater than an input image from which the gain map is derived. The method includes receiving, by a computing system, an input image having a first dynamic range; applying, by the computing system, a machine-learned model to the input image, to obtain a gain map for content having a second dynamic range, the second dynamic range being greater than the first dynamic range; and storing, by the computing system, the gain map in metadata associated with the input image.

[0007] In some implementations, the first dynamic range is a standard dynamic range or a low dynamic range, and the second dynamic range is a high dynamic range.

[0008] In some implementations, the method includes reducing a size of the input image a predetermined amount before applying the machine-learned model to the input image.

[0009] In some implementations, the method includes rendering an output image having the second dynamic range by applying the gain map stored in the metadata associated with the input image to the input image.

[0010] In some implementations, the method includes editing the input image by modifying a portion of the input image to produce a modified input image with the modified portion; applying the machine-learned model to the modified portion of the modified input image, to obtain a partial gain map for content having the second dynamic range, the partial gain map corresponding to the modified portion; and modifying the gain map by replacing gain map values corresponding to the portion of the input image with gain map values from the partial gain map corresponding to the modified portion.

[0011] In some implementations, the method includes editing the input image by modifying a portion of the input image to produce a modified input image with the modified portion; when a size of the portion of the input image is less than a threshold amount: applying the machine-learned model to the modified portion of the modified input image, to obtain a partial gain map for content having the second dynamic range, the partial gain map corresponding to the modified portion; and when a size of the portion of the input image is greater than the threshold amount: applying the machine-learned model to an entirety of themodified input image, to obtain an updated gain map for content having the second dynamic range, the updated gain map corresponding to the entirety of the modified input image.

[0012] In some implementations, the method includes receiving a plurality of input images, wherein each of the plurality of input images have the first dynamic range; applying the machine-learned model to each of the plurality of input images, to obtain a plurality of gain maps corresponding to the plurality of input images, the plurality of gain maps being for content having the second dynamic range; storing the plurality of gain maps in metadata associated with the plurality of input images; and rendering video content having the second dynamic range by applying the plurality of gain maps stored in the metadata associated with the plurality of input images to the plurality of input images.

[0013] In some implementations, the method includes receiving a plurality of input images, wherein each of the plurality of input images have the first dynamic range; applying the machine-learned model to every n-th input image of the plurality of input images, to obtain a plurality of first gain maps, wherein the plurality of first gain maps are for content having the second dynamic range and n is a value greater than two; obtaining a plurality of second gam maps for input images among the plurality of input images other than the every n-th input image among the plurality7of inputs images, based on the plurality of first gain maps; storing the plurality7of first gain maps and the plurality of second gain maps in metadata associated with corresponding input images among the plurality of input images; and rendering video content having the second dynamic range by applying the plurality of first gain maps and the plurality of second gain maps to the corresponding input images among the plurality of input images.

[0014] In some implementations, storing the gain map in metadata associated with the input image comprises storing the gain map in extensible metadata platform metadata of the input image.

[0015] In some implementations, the method includes applying, by the computing system, the machine-learned model to the input image, to predict a maximum content boost value; and applying, by the computing system, the maximum content boost value to the gain map.

[0016] In some implementations, the method includes storing the maximum content boost value in extensible metadata platform metadata of the input image, and storing the gain map in metadata associated with the input image comprises storing the gain map in the extensible metadata platform metadata of the input image.

[0017] In some implementations, the method includes receiving, by the computing system, an input via a user interface modifying the maximum content boost value predicted via the machine-learned model to change a degree of contrast enhancement applied to the input image.

[0018] Another example aspect of the disclosure is directed to a computing system for determining a gain map for rendering an output image having a dynamic range which is greater than an input image from which the gain map is derived. The computing system includes one or more processors; and one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising: receiving an input image having a first dynamic range; applying a machine-learned model to the input image, to obtain a gain map for content having a second dynamic range, the second dynamic range being greater than the first dynamic range; and storing the gain map in metadata associated with the input image.

[0019] In some implementations the first dynamic range is a standard dynamic range or a low dynamic range, and the second dynamic range is a high dynamic range.

[0020] In some implementations the operations further comprise: rendering an output image having the second dynamic range by applying the gain map stored in the metadata associated with the input image to the input image.

[0021] In some implementations the operations further comprise: editing the input image by modifying a portion of the input image to produce a modified input image with the modified portion; when a size of the portion of the input image is less than a threshold amount: applying the machine-learned model to the modified portion of the modified input image, to obtain a partial gain map for content having the second dynamic range, the partial gain map corresponding to the modified portion; and when a size of the portion of the input image is greater than the threshold amount: applying the machine-learned model to an entirety of the modified input image, to obtain an updated gain map for content having the second dynamic range, the updated gain map corresponding to the entirety of the modified input image.

[0022] In some implementations the operations further comprise: receiving, by the computing system, a plurality of input images, wherein each of the plurality of input images have the first dynamic range; applying the machine-learned model to at least two of the plurality of input images to obtain a plurality of gain maps, the plurality of gain maps being for content having the second dynamic range; and rendering video content having the seconddynamic range by applying the plurality of gain maps to the at least two of the plurality of input images.

[0023] In some implementations the operations further comprise: applying, by the computing system, the machine-learned model to the input image, to predict a maximum content boost value; and applying, by the computing system, the maximum content boost value to the gain map.

[0024] In some implementations the operations further comprise: storing the maximum content boost value in extensible metadata platform metadata of the input image, and wherein storing the gain map in metadata associated with the input image comprises storing the gain map in the extensible metadata platform metadata of the input image.

[0025] Other aspects of the disclosure are directed to various systems, apparatuses, non- transitory computer-readable media, user interfaces, and electronic devices. In one or more example embodiments, a computer-readable medium (e.g., a non-transitory computer- readable medium) which stores instructions that are executable by one or more processors of a computing system or computing device is provided. In some implementations the computer-readable medium stores instructions which may include instructions to cause the one or more processors to perform one or more operations of any of the methods described herein (e.g., operations of the computing system or of the computing device). The computer- readable medium may store additional instructions to execute other aspects of the computing system or of the computing device and corresponding methods of operation, as described herein.

[0026] For example, an aspect of the disclosure is directed to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising: receiving an input image having a first dynamic range; applying a machine- learned model to the input image to obtain a gain map for content having a second dynamic range, the second dynamic range being greater than the first dynamic range; and storing the gain map in metadata associated with the input image.

[0027] These and other features, aspects, and advantages of various embodiments of the disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the disclosure and, together with the description, serve to explain the related principles.BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Detailed discussion of embodiments directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the appended drawings, in which:

[0029] FIG. 1 A depicts a block diagram of an example computing system that performs various tasks using a machine-learned model, according to example embodiments of the disclosure.

[0030] FIG. IB depicts a block diagram of an example computing device that performs various tasks using a machine-learned model, according to example embodiments of the disclosure.

[0031] FIG. 1C depicts a block diagram of an example computing device that performs various tasks using a machine-learned model, according to example embodiments of the disclosure.

[0032] FIG. 2 depicts an example system for predicting a gain map and associated metadata for dynamic image rendering, according to example embodiments of the disclosure.

[0033] FIG. 3 depicts an example system for predicting a gain map and associated metadata for dynamic image rendering, according to example embodiments of the disclosure.

[0034] FIG. 4 depicts an example system for predicting a gain map and associated metadata for dynamic image rendering, according to example embodiments of the disclosure.

[0035] FIG. 5 depicts an example system for predicting a gain map and associated metadata for dynamic image rendering, according to example embodiments of the disclosure.

[0036] FIG. 6 depicts a flow chart diagram of an example method to perform according to example embodiments of the disclosure.

[0037] FIG. 7 depicts example images illustrating input SDR images, corresponding ground truth HDR gain maps, and predicted HDR gain maps generated according to example embodiments of the disclosure.

[0038] FIG. 8 depicts example input SDR images, corresponding ground truth HDR images, and predicted HDR images rendered according to example embodiments of the disclosure.

[0039] FIG. 9 depicts example images illustrating input SDR images, predicted HDR images rendered according to example embodiments of the disclosure, and predicted HDR gain maps generated according to examples of the disclosure.

[0040] Reference numerals that are repeated across plural drawings are intended to identify the same features in various implementations.DETAILED DESCRIPTIONOverview

[0041] Reference now will be made to embodiments of the disclosure, one or more examples of which are illustrated in the drawings, wherein like reference characters across drawings are intended to denote like features in various implementations. Each example is provided by way of explanation of the disclosure and is not intended to limit the disclosure.

[0042] Terms used herein are used to describe the example embodiments and are not intended to limit and / or restrict the disclosure. The singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. In this disclosure, terms such as "including", "having", “comprising”, and the like are used to specify features, numbers, steps, operations, elements, components, or combinations thereof, but do not preclude the presence or addition of one or more of the features, elements, steps, operations, elements, components, or combinations thereof.

[0043] It will be understood that, although the terms first, second, third, etc., may be used herein to describe various elements, the elements are not limited by these terms. Instead, these terms are used to distinguish one element from another element. For example, without departing from the scope of the disclosure, a first element may be termed as a second element, and a second element may be termed as a first element.

[0044] It will be understood that when an element is referred to as being “connected” to another element, the expression encompasses an example of a direct connection or direct coupling, as well as a connection or coupling with one or more other elements interposed therebetween.

[0045] The term "and I or" includes a combination of a plurality of related listed items or any item of the plurality of related listed items. For example, the scope of the expression or phrase "A and / or B" includes the item "A", the item "B", and the combination of items "A and B”.

[0046] In addition, the scope of the expression or phrase "at least one of A or B" is intended to include all of the following: (1) at least one of A, (2) at least one of B, and (3) at least one of A and at least one of B. Likewise, the scope of the expression or phrase "at least one of A, B, or C" is intended to include all of the following: (1) at least one of A, (2) at least one of B, (3) at least one of C, (4) at least one of A and at least one of B, (5) at least one of A and at least one of C, (6) at least one of B and at least one of C, and (7) at least one of A, at least one of B, and at least one of C.

[0047] Some existing photography processing pipelines produce images using HDRphotography techniques (e.g., merging multiple frames across time using several short exposures, or merging images at different exposures, etc.) to increase the dynamic range of the image signal captured for a particular scene. Thus, HDR information is available within such pipelines. In these implementations, the increased dynamic range is compressed before display using tone-mapping techniques, as standard displays typically render 8-bit imagery. Dynamic range compression reduces contrast to match the display able range of pixel values while preserving image details. Many existing images (e.g., stored in various databases, published on the internet, stored in older computing devices, etc ), do not contain HDR gain map metadata required for HDR rendering. Furthermore, most legacy video content is SDR.

[0048] According to examples of the disclosure, a computing system may include a machine-learned model configured to predict HDR image rendition given only SDR inputs, thereby leveraging the improved HDR display capabilities of modem devices. From this inferred HDR image rendition, the computing system including the machine-learned model can generate gain map metadata for legacy or SDR content. Thus, both SDR images and SDR videos can be rendered with improved contrast and higher peak brightness, with plausible re-tone-mapping for HDR display.

[0049] Aspects of the disclosure solve the problem of understanding the HDR rendition of an image given only SDR or LDR (low dynamic range) information. In some implementations, aspects of the disclosure are applied to legacy content where HDR information has already been compressed into the standard dynamic range available in the typical 8-bit SDR image, or to images where HDR information is not present. For example, legacy photographs captured with older computing devices (e.g., older smartphones) may include images with the SDR information or cell phone videos taken before the era of HDR video recording, where light often saturates an image sensor for a particular single exposure used over the course of the capture, may not include HDR information. For legacy content such as this, the computing system can be configured to “hallucinate"’ or “infer” or “predict” the HDR gain map and its associated metadata using one or more machine-learned models as described herein.

[0050] Different from previous methods, aspects of the disclosure disclose a computing system which is configured to combine an HDR inference (given only SDR / LDR imagery) with the HDR gain map format, for compatibility with viewing images on the web (e.g., in a web browser) or on mobile devices (e.g., a photograph applications on a smartphone). That is, the disclosed computing system is configured to predict an HDR gain map and associatedmetadata for dynamic HDR rendering, rather than merely trying to predict HDR imagery from SDR imagery.

[0051] Because the computing system can infer the HDR gain map, the HDR rendition recovered using the machine-learned model inference can be ensured to be adaptable to both SDR displays and HDR displays with different HDR capacities, peak luminance levels, etc., as this is a property of the gain map format.

[0052] In some implementations, the computing system can be configured to enable a maximum brightness value to be modified (e.g., with a user-controlled slider or other input method) based on the HDR gain map format. Therefore, the "‘degree'’ of HDR contrast enhancement applied to the SDR image can be manually controlled.

[0053] Some previous methods describe recovering missing light energy (pixel values) in clipped / saturated regions, and assume the input images are already in radiometrically linear space. For example, many methods assume that the tone curve of the input LDR image is a global operator (the camera response function or response curve). For example, some methods learn to invert an unknown global tone-mapping operator, however they do not handle inputs where local tone-mapping (LTM) operators have been applied, unlike the technique implemented according to the disclosed computing system and methods described herein. That is, the disclosed computing system and methods make no assumption about the global tone curve of the input images, even permitting local-tone-mapping, and make adjustments even to non-clipped regions to faithfully enhance contrast across all pixel values.

[0054] According to examples of the disclosure a machine-learned model is trained by taking as an input a single in-the-wild. arbitrary image and recovers (outputs) an HDR rendition of the image based on a learned prior. From this HDR rendition, the computing system is configured to compute an HDR gain map and associated metadata that allow the computing system to promote or enhance the image to HDR in the gain map based adaptive rendering scheme. For example, the input image may be an image formed from a modem HDR imaging pipeline, with both global and local tone-mapping operations applied to compress the dynamic range of the original HDR scene. For example, the majority' of mobile photography images have LTM operators applied, so the disclosed computing systems and methods can be applied to the majority’ of imagery’ captured by users using mobile devices.

[0055] The machine-learned model disclosed herein may be trained in a supervised manner using a large dataset. In some implementations, an example dataset may includethousands of images (e.g., 25,000 images, 50,000 images, 100,000 images, etc.) for an initial training evaluation, randomly sampled over many years, where the HDR image rendition for a given photograph is available as ground truth. These training images can contain diverse subject matters including landscapes, individuals, groups of people, pets, food, etc. HDR images can be captured as a single image with high dynamic range or as bursts of multiple images from a physical camera sensor, which are then combined together to form a HDR image such that their dynamic range exceeds the dynamic range of each individual image.

[0056] In some implementations, the machine-learned model may be trained for all image input types. In some implementations, the machine-learned model may be trained for image input types of a particular category, genre, etc. (e.g., models trained specifically for portraits, HDR landscapes, etc.). For example, each HDR image can be processed by the computing system to generate a respective SDR image rendition (e.g., by using an image processing pipeline that removes noises and enhances tone, color, contrast, etc.) and including the local tone-mapping (LTM) operation. The computing system can also be configured to generate the ground truth HDR gain map and associated metadata for each image. The gain map encodes a low-resolution (typically, approximately quarter-resolution) per-pixel map that, when employed with its metadata, allows the computing system to determine how many stops brighter each individual pixel should be when rendered on any HDR display given its known peak luminance.

[0057] In some implementations, the disclosed machine-learned models can include a fully-convolutional machine-learned model. In some implementations, the disclosed machine-learned models can implement image-to-image translation using a U-Net architecture (e.g., a U-Net encoder / decoder with skip connections) to effectively translate the input image into an output image while preserving spatial information and capturing both low-level and high-level features.

[0058] In some implementations, the disclosed machine-learned models can take as an input a single image. For example, the image may be a gamma-encoded, SDR input image at quarter-resolution. In some implementations, a first example machine-learned model can be configured to output a predicted image (e.g., approximately quarter-resolution, per-pixel, and single channel log boost image, which is a map that indicates how many stops brighter each pixel should be in the new HDR rendition). The inferred log boost may be constrained to be 0 or greater with a softplus output activation function (e.g., Y = log(l + ex)). As 2° = 1, this constraint ensures that, relative to the input SDR rendition, the HDR rendition can only beequal to or brighter than the input at each pixel. Thus, the network activation function implemented by the first example machine-learned model ensures that no pixels are darkened. For example, in some implementations the computing system can implement the single channel prediction to minimize any color changes, only altering per-pixel brightness.

[0059] From the predicted log boost image output by the first example machine-learned model, the computing system can be configured to compute the HDR gain map and log2_max_content_boost, such that the input image (e.g., a full-resolution SDR JPEG image) can be saved with the metadata required for HDR gain map based adaptive HDR rendering on HDR-capable display devices. For example, the amount of apparent brightening can be tuned by adjusting the log2_max_content_boost parameter during rendering. In some implementations, the log2_max_content_boost parameter can be combined with a user interface input method (e.g., a Ul-based slider) to enable the manual adjustment of the log2_max_content_boost parameter. Thus, a larger contrast enhancing operation than is initially predicted by the model can be obtained, for an enhanced HDR effect. However, as the final rendition is controlled by the HDR gain map rendering logic, the amount of contrast enhancement for a particular image may still be adaptive, based not only on the log2 max content boost, but also on the HDR capacity of the display.

[0060] In some implementations, a second example machine-learned model can be configured to output the predicted HDR gam map and the log2_max_content_boost directly. In this second model architecture, the disclosed machine-learned model can implement image-to-image translation using a U-Net architecture (e.g., a U-Net encoder / decoder to directly predict the HDR gain map). The encoded latent features at the bottleneck of the encoder / decoder network are then passed to a subsequent neural network including several downsampling and convolution layers, followed by a series of fully-connected layers to output the log2_max_content_boost value of the HDR gain map metadata. The HDR gain map and the log2_max_content_boost can then be stored by the computing system in the XMP metadata of the original image for HDR rendering.

[0061] In some implementations, a user can manually override the predicted log2 max content boost value to apply more or less of an HDR / contrast enhancement effect (e.g., via a user interface input method).

[0062] In some implementations, legacy video content (e.g., in SDR) can be promoted or enhanced to HDR format using the disclosed computing systems and methods. For example.in some implementations, a machine-learned model can be configured to process each frame of the video individually. For videos, the per-frame HDR rendition can be directly encoded and saved in any HDR video format (e.g., HDR10). In some implementations, a machine- learned model can be configured to process every N frames (e.g., every 2 frames, every 3 frames, every 4 frames, etc.), thus reducing the number of inference times, while propagating information from previous frames.

[0063] In some implementations, the first example machine-learned model may be trained using a VGG perceptual feature loss comparing the ground truth HDR rendition with the model-predicted HDR rendition. For example, the predicted and ground truth HDR images may first be tone-mapped using a differentiable global tone-mapping technique called Adaptive Logarithmic Mapping (ALM), which remaps HDR imagery with potentially very' large pixel values into the [0, 1] range expected by the VGG network. Other differentiable tone-mapping operators besides ALM may be implemented, which serve the role of compressing the HDR range into the [0, 1] range expected by the VGG network. The first example machine-learned model may also be trained with an LI image reconstruction loss comparing the ground truth and predicted HDR images after they have been encoded in a scale that transforms the values in the SDR (<=1.0) range according to a power function, and the values in the HDR range (>1.0) according to a logarithmic function, similar to the hybrid log gamma (HLG) HDR encoding function. The first example machine-learned model may also be trained with a direct LI or L2 supervision loss comparing the predicted and ground truth HDR gain map directly, and the same on the predicted vs. ground truth log2_max_content_boost.

[0064] In the second example machine-learned model network architecture, the computing system can be configured to train the UNet using LI loss on the HDR gain map compared with ground truth, the VGG perceptual loss comparing the HDR image reconstructed using the predicted HDR gain map and the real / ground truth log2_max_content_boost, and a direct LI loss comparing the predicted and ground truth log2_max_content_boost.

[0065] The disclosed computing systems and methods can also be applied for corrective HDR inpainting. For example, the machine-learned model can be configured to predict or hallucinate HDR content for images that are subject to a destructive editing operation. For example, if an SDR image with an HDR Gain Map and associated metadata is edited using an editing application, new pixels can be inpainted in the SDR color image. The editingapplication can include any editing application that removes unwanted distractions from an image (e.g., automatically or with user guidance), and replaces these distractions with harmonious image content so as to form a plausible complete image. However, such editing applications do not inpaint the pixels of the HDR gain map. In some implementations, the computing system can be configured to implement a blending strategy that keeps the HDR gain map pixels of the original image, while adding ML-inferred HDR gain map pixels for new content that replaces the content which is removed via the editing operation. In some implementations, the computing system can be configured to infer a totally new HDR gain map for the new image after the edit operation. For example, if a threshold amount of the image (e.g., 50% or greater) is replaced via the editing operation, the computing system can be configured to infer an entirely new HDR gain map for the new image after the edit operation.

[0066] The disclosure provides numerous technical effects and benefits. For example, in some implementations, the machine-learned models are configured to improve the quality and clarity of an image or video. The content can include photographs, digital renditions of paintings, Al-generated imagery, etc. For example, in some implementations, the machine- learned models are configured to generate a HDR gain map and associated metadata for an image that does not include such information. Experimental results show that machine- learned models trained according to the examples described herein can predict HDR images having an image quality which is comparable to ground truth HDR images and vastly improved over the corresponding SDR image. Experimental results show that machine- learned models trained according to the examples described herein can predict HDR gain maps having an image quality which is comparable to ground truth HDR gain maps. The disclosed machine-learned models leverage existing datasets and can promote SDR content to HDR content, and can increase the dynamic range and contrast of legacy content to optimize them for HDR displays. The disclosed models can generate predictions for the HDR gain maps and HDR images quickly (e.g., within 1 second, e.g., between 30 ms to 100 ms, between 30 ms to 80 ms).

[0067] With reference now to the drawings, example embodiments of the disclosure will be discussed in further detail.Example Devices and Systems

[0068] FIG. 1A depicts a block diagram of an example computing system 100 that performs various tasks using a machine-learned model (e.g., a gain map prediction model) according to example embodiments of the disclosure. The system 100 includes a user computing system 102, a server computing system 130, and a training computing system 150 that are communicatively coupled over a network 180.

[0069] The user computing system 102 can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0070] The user computing system 102 includes one or more processors 112 and a memory 114. The one or more processors 1 12 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 114 can include one or more non- transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 which are executed by the processor 112 to cause the user computing system 102 to perform operations.

[0071] In some implementations, the user computing system 102 can store or include one or more machine-learned models 120 (e.g., one or more prediction models). For example, the machine-learned models 120 can be or can otherwise include various machine- learned models such as neural networks (e.g., deep neural networks) or other ty pes of machine-learned models, including non-linear models and / or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multiheaded self-attention models (e.g.. transformer models). Example machine-learned models 120 are discussed with reference to the drawings herein.

[0072] In some implementations, the one or more machine-learned models 120 can be received from the server computing system 130 over network 180, stored in the memory7114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing system 102 can implement multiple parallel instances ofa single machine-learned model 120 (e.g., to perform parallel tasks across multiple instances of the machine-learned model 120).

[0073] More particularly, the machine-learned models disclosed herein (e.g., including one or more gain map prediction models), may be implemented to perform various tasks related to an input image or a plurality of input images which relates to imagery. According to examples of the disclosure, a computing system may include a machine-learned model configured to predict HDR image rendition given only SDR inputs, thereby leveraging the improved HDR display capabilities of modem devices. From this inferred HDR image rendition, the computing system including the machine-learned model can generate gain map metadata for legacy or SDR content. Thus, both SDR images and SDR videos can be rendered with improved contrast and higher peak brightness, with plausible re-tone-mapping for HDR display.

[0074] The machine-learned models trained according to the methods described herein may be utilized to predict (estimate, infer, hallucinate, etc.) gain map information (e.g., HDR gain map information) and its associated metadata (e.g., a maximum content boost value) for an input image having a lower dynamic range (e.g., an SDR image or LDR image). An image (e.g., an HDR image) can then be rendered based on the input image and the gain map information and its associated metadata.

[0075] Additionally, or alternatively, one or more machine-learned models 140 can be included in or otherwise stored and implemented by the server computing system 130 that communicates with the user computing system 102 according to a client-server relationship. For example, the machine-learned models 140 can be implemented by the server computing system 130 as a portion of a web service (e.g., a recommendation service, a search service, an image analysis service, and the like). Thus, one or more machine-learned models 120 can be stored and implemented at the user computing system 102 and / or one or more machine- learned models 140 can be stored and implemented at the server computing system 130.

[0076] The user computing system 102 can also include one or more user input components 122 that receives user input. For example, the user input component 122 can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can sen e to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, a mouse, or other devices by which a user can provide user input (e.g., a camera which captures an image).

[0077] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 can be any suitable processing device (e.g.. a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plural ity of processors that are operatively connected. The memory 134 can include one or more non-transi tory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138 which are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0078] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 130 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0079] As described above, the server computing system 130 can store or otherwise include one or more machine-learned models 140. For example, the machine-learned models 140 can be or can otherwise include various machine-learned models. Example machine- learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models). Example machine-learned models 140 are discussed herein with reference to the drawings.

[0080] The user computing system 102 and / or the server computing system 130 can train the machine-learned models 120 and / or 140 via interaction with the training computing system 150 that is communicatively coupled over the network 180. The training computing system 150 can be separate from the server computing system 130 or can be a portion of the server computing system 130.

[0081] The training computing system 150 includes one or more processors 152 and a memory' 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, suchas RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 which are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0082] The training computing system 150 can include a model trainer 160 that trains the machine-learned models 120 and / or 140 stored at the user computing system 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss function can be back propagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations.

[0083] In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer 160 can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

[0084] In particular, the model trainer 160 can train the machine-learned models 120 and / or 140 based on a set of training data 162. The training data 162 can include, for example, various datasets which may be stored remotely or at the training computing system 150. For example, in some implementations an example dataset utilized for training includes a plurality of images (e.g., a series of rapid photos captured in quick succession, capturing multiple exposures of the same scene). For example, the plurality of images may include thousands of images (e.g., 25,000 bursts, 50,000 bursts, etc.). However, other datasets of images may be utilized (e.g., images from external websites). In some implementations, the dataset may be confined to a particular category, genre, landscape, etc. In some implementations, the dataset may contain diverse subject matter including landscapes, individuals, groups of people, pets, food, etc. HDR images can be captured as a single image with high dynamic range or as bursts of multiple images from a physical camera sensor, which are then combined together to form a HDR image such that their dynamic range exceeds the dynamic range of each individual image.

[0085] In some implementations, if the user has provided consent, the training examples can be provided by the user computing system 102. Thus, in such implementations, themachine-learned model 120 provided to the user computing system 102 can be trained by the training computing system 150 on user-specific data received from the user computing system 102. In some instances, this process can be referred to as personalizing the model.

[0086] The model trainer 160 includes computer logic utilized to provide desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general purpose processor. For example, in some implementations, the model trainer 160 includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM, hard disk, or optical or magnetic media.

[0087] The network 180 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over the network 180 can be carried via any type of wired and / or wireless connection, using a wide variety of communication protocols (e.g., TCP / IP, HTTP. SMTP. FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0088] The machine-learned models described in this specification may be used in a variety7of tasks, applications, and / or use cases.

[0089] In some implementations, the input to the machine-learned model(s) of the disclosure can be image data (e.g.. one or more images). The machine-learned model(s) can process the image data to generate an output. As an example, the machine-learned model(s) can process the image data to generate a prediction output. For example, the machine-learned model(s) can process the image data to generate (predict) a gain map for content having a higher dynamic range than the dynamic range of the input image. As an example, the machine-learned model(s) can process the image data to generate (predict) metadata associated wdth the gain map (e.g., a maximum content boost value which is multiplied by the values of the HDR gain map for rendering the input image with extended dynamic range, etc.).

[0090] FIG. 1 A illustrates an example computing system that can be used to implement aspects of the disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing system 102 can include the model trainer 160 and the training data 162. In such implementations, the machine-learned models 120 can be both trained and used locally at the user computing system 102. In some of such implementations,the user computing system 102 can implement the model trainer 160 to personalize the machine-learned models 120 based on user-specific data.

[0091] FIG. IB depicts a block diagram of an example computing device 10 that performs according to example embodiments of the disclosure. The computing device 10 can be a user computing device or a server computing device.

[0092] The computing device 10 includes a number of applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine-learned model(s). For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a social media application, an infotainment application, a browser application, etc.

[0093] As illustrated in FIG. IB, each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0094] FIG. 1C depicts a block diagram of an example computing device 50 that performs according to example embodiments of the disclosure. The computing device 50 can be a user computing device or a server computing device.

[0095] The computing device 50 includes a number of applications (e.g., applications 1 through N). Each application is in communication with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a social media application, an infotainment application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

[0096] The central intelligence layer includes a number of machine-learned models. For example, as illustrated in FIG. 1C, a respective machine-learned model can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device 50.

[0097] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository’ of data for the computing device 50. As illustrated in FIG. 1C, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0098] FIG. 2 depicts an example system for predicting a gain map and associated metadata for dynamic image rendering, according to example embodiments of the disclosure.

[0099] The example system 2000 of FIG. 2 may be implemented by any one of user computing system 102, server computing system 130. or training computing system 150. or a combination thereof. Unless otherwise specified, references to a '‘computing system” herein refers to the user computing system 102 alone, server computing system 130 alone, training computing system 150 alone, or any combination thereof. Thus, each of the components of FIG. 2 can be included in user computing system 102 alone, server computing system 130 alone, training computing system 150 alone, or any combination thereof.

[0100] In the example system 2000 of FIG. 2, an input image 2100 is received by an input image re-sizer 2200. The input image 2100 may have a first dynamic range. The ratio between the maximum and the minimum of a tonal value in an image (e.g., a brightness or luminance level of a specific pixel in the image) can be referred to as the dynamic range, or may simply be determined based on the range of luminosity a display can reproduce, from the black level to its peak brightness. Dynamic range can also be measured in stops, which is a unit of measurement to describe a change in luminance or exposure by a predetermined amount (e.g.. by a factor of 2, either doubling the luminance or exposure or halving it).

[0101] For example, the input image 4100 may have a standard dynamic range (SDR) or a low- dynamic range. Generally, SDR content (images, video, etc.) may have a color depth of 8 bits and can represent a maximum luminance level of around 100 nits. For example, SDR images may have a dynamic range of about 6 stops.

[0102] Conversely, HDR content (images, video, etc.) generally shows more detail in both bright and dark areas and thus has a greater dynamic range of color. Generally, HDR content may have a color depth of 10 bits or higher (e.g., 32 bits) and can represent a maximum luminance level of 400 nits or higher (e.g., 1,000 nits, 10,000 nits, etc.). For example. HDR images may have a dynamic range of about 17 stops.

[0103] The input image re-sizer 2200 may be configured to decrease the resolution (e.g., by downsampling the input image 2100). For example, the resolution of the input image 2100 may be decreased such that the downsized input image output by the input image resizer 2200 and input to the one or more machine-learned models 2300, is one-quarter resolution of the original input image 2100. For example, if the original input image 2100 was 1920x1080 pixels, the quarter-resolution version output by the input image re-sizer 2200 would be 480x270 pixels. Implementation of the input image re-sizer 2200 may help conserve or reduce compute time for future operations with respect to the input image (e g., including computing the gain map of the input image), and conserve storage space (e.g., by taking up less storage space with smaller size images). However, in some implementations, the input image re-sizer 2200 may be omitted and the original input image 2100 may be input to the one or more machine-learned models 2300. In some implementations, the input image re-sizer 2200 may decrease the size (resolution) of the input image 2100 by more than a factor of four (e.g., eight times). In some implementations, the input image re-sizer 2200 may also be configured to subject the input image 2100 to gamma encoding to alter the luminance values of the input image 2100 to cause the input image 2100 to be more perceptually consistent.

[0104] The one or more machine-learned models 2300 may receive the dow nsized input image 2100 from the input image re-sizer 2200. In some implementations, the one or more machine-learned models 2300 can include a fully-convolutional machine-learned model. In some implementations, the one or more machine-learned models 2300 can implement image-to-image translation using a U-Net architecture (e.g., a U-Net encoder / decoder with skip connections) to translate an input image into an output image while preserving spatial information and capturing both low-level and high-level features. For example, skip connections can be configured to enable the one or more machine-learned models 2300 to leam and retain information from earlier layers and address issues like vanishing gradients. The one or more machine-learned models 2300 may be trained based on a set of training data (e.g., training data 162). The training data can include, for example, various datasets which may be stored remotely or at the training computing system 150. For example, in some implementations an example dataset utilized for training includes a plurality of images (e.g., a series of rapid photos captured in quick succession, capturing multiple exposures of the same scene). For example, the plurality of images may include thousands of images (e.g., 25,000 bursts, 50,000 bursts, etc.). However, other datasets of images may be utilized (e.g., images from external websites). In some implementations, thedataset may be confined to a particular category, genre, landscape, etc. In some implementations, the dataset may contain diverse subject matter including landscapes, individuals, groups of people, pets, food, etc. HDR images can be captured as a single image with high dynamic range or as bursts of multiple images from a physical camera sensor, which are then combined together to form a HDR image such that their dynamic range exceeds the dynamic range of each individual image.

[0105] The one or more machine-learned models 2300 may be configured to output a predicted or inferred boost image 2400 (e.g., an approximately quarter-resolution, per-pixel, and single channel log boost image, which is a map that indicates how many stops brighter each pixel should be in the output image having a greater dynamic range, for example, an HDR image). As described herein, a stop may refer to a unit of measurement which describes a change in exposure or luminance by a predetermined amount (e.g., by a factor of 2, either doubling the exposure or luminance or halving it). The one or more machine- learned models 2300 may be configured to constrain the predicted or inferred log boost image have pixel values of 0 or greater with a softplus output activation function (e.g., Y = log(l + ex)). As 2° = 1, this constraint ensures that, relative to the input image 2100, the output image (e.g.. an HDR image rendition) can only be equal to or brighter than the input at each pixel. Thus, the one or more machine-learned models 2300 are configured to implement a network activation function to ensure that no pixels from the input image 2100 are darkened. For example, in some implementations the one or more machine-learned models 2300 are configured to implement the single channel prediction to minimize any color changes, only altering per-pixel brightness.

[0106] Based on the boost image 2400 output by the one or more machine-learned models 2300, the max content boost value generator 2500 may be configured to compute a maximum content boost value of the predicted boost image 2400. For example, the maximum content boost value may be multiplied by the values of the HDR gain map for rendering the input image with extended dynamic range. In some implementations, the maximum content boost value may also be referred to as a log2_max_content_boost parameter. The maximum content boost value may correspond to a maximum allowed ratio of the linear luminance for the target HDR rendition relative to (divided by) that of the SDR image, at a given pixel. For example, if the maximum content boost value is four, then for any given pixel, the linear luminance of the displayed HDR rendition must be, at the most, four times the linear luminance of the SDR rendition. Thus, the brighter parts of a scene canbe show n up to four times brighter.

[0107] Based on the boost image 2400 output by the one or more machine-learned models 2300, the gain map generator 2600 may be configured to generate (e.g., predict or infer) a gain map corresponding to the boost image 2400. The gain map 2610 encodes a low- resolution (typically, approximately quarter-resolution) per-pixel map. The gain map 2610 is configured to indicate how much to brighten each pixel, in the SDR rendition, to produce the target HDR rendition. The gain map 2610 can be single-channel or multi-channel. A multichannel map indicates a separate gain for each color channel, such as red, green, and blue.

[0108] Metadata generator 2700 may be configured to generate metadata for the input image 2100 which includes the gain map 2610 and the maximum content boost value. Modified input image generator 2800 may be configured to modify the input image 2100 to include the metadata generated by the metadata generator 2700. For example, the gain map 2610 and its associated metadata (e.g., the maximum content boost value) can be stored in the extensible metadata platform (XMP) metadata of the input image 2100 (e.g., a SDR JPEG image). Other metadata can include, for example, gamma values, maximum display boost values (which correspond to a maximum available boost supported by a display device), etc.

[0109] The image Tenderer 2900 may be configured to determine how many stops brighter each individual pixel from the input image should be when rendering an output image 2910 having a greater dynamic range than the original input image 2100 for display at a display device (e.g., an HDR display) given the gain map 2610 and maximum content boost value which are stored in the modified input image (e.g., stored in metadata associated with the input image or stored in an image file including the gain map 2610, maximum content boost value, and input image) generated by the modified input image generator 2800. For example, the image Tenderer 2900 may be disposed or provided remotely or locally to components of the computing system which generate the metadata or compute the gain map and / or maximum content boost value. For example, a display device capable of displaying HDR imagery may receive the modified input image and generate the output image (HDR image) having the greater dynamic range based on the information (metadata) included in the modified input image including the gain map and maximum content boost value.

[0110] In some implementations, the computing system may include an input device (e.g., user input component 122, a LT-based slider, etc.) for a user to adjust or tune an amount of apparent brightening by adjusting the maximum content boost value (e.g., duringrendering). For example, the computing system may include a user interface or input device to enable the manual adjustment of the maximum content boost value (e.g..Iog2_max_content_boost parameter). Thus, a larger contrast enhancing operation than is initially predicted by the one or more machine-learned models 2300 can be obtained, for an enhanced HDR effect.

[0111] As an example, an HDR gain map image may be stored in an 8-bit representation, where 0 represents no gain (boost above SDR) and 255 represents the maximum boost above SDR. with grayscale values in between. The computing system may be configured to re-map the 0-255 gain map to the range of 0-1, and then multiply the gain map by the maximum content boost value, and then apply the pre-multiplied gain map to the image. Thus, the computing system is configured to effectively control the brightness of the apparent highlight boost, for example, without changing the HDR gain map, by changing the maximum content boost value.

[0112] FIG. 3 depicts an example system for predicting a gain map and associated metadata for dynamic image rendering, according to example embodiments of the disclosure.

[0113] Aspects of the example system 3000 of FIG. 3 correspond to the example system of FIG. 2, and therefore some details will not be repeated herein for the sake of brevity. The example system 3000 of FIG. 3 may be implemented by any one of user computing system 102, server computing system 130, or training computing system 150, or a combination thereof. Unless otherwise specified, references to a “computing system” herein refers to the user computing system 102 alone, server computing system 130 alone, training computing system 150 alone, or any combination thereof. Thus, each of the components of FIG. 3 can be included in user computing system 102 alone, server computing system 130 alone, training computing system 150 alone, or any combination thereof.

[0114] In the example system 3000 of FIG. 3, an input image 3050 is received by one or more machine-learned models 3100. In some implementations, the input image 3050 may be downsized prior to being received by the one or more machine-learned models 3100 (e.g., as described with respect to input image re-sizer 2200). In some implementations, the one or more machine-learned models 3100 can correspond to one or more machine-learned models 2300 and implement a U-Net architecture (e.g., a U-Net encoder / decoder with skip connections) to translate the input image 3050 into an output image while preserving spatial information and capturing both low-level and high-level features. For example, the U-Net architecture can include skip connections which are configured to enable the one or moremachine-learned models 3100 to learn and retain information from earlier layers and address issues like vanishing gradients.

[0115] Image Tenderer 3300 may be configured to generate an output image having a greater dynamic range than the input image 3050. for example, a high dynamic range image. For example, the output image may be based on the product of the input image 3050 and a boost image (e.g., boost image 2400).

[0116] The one or more machine-learned models 3100 may be trained using a Visual Geometry Group (VGG) perceptual feature loss comparing a ground truth rendition generated by ground truth image Tenderer 3400 with the model-predicted rendition generated by image Tenderer 3300. In some implementations, when the original VGG network is trained on SDR imagery only, the system 3000 may be configured to tone map the predicted and ground truth images via the first tone mapping operator 3700 and second tone mapping operator 3800, respectively (e.g., using a differentiable global tone-mapping technique called Adaptive Logarithmic Mapping (ALM), which remaps imagery (e.g.. HDR imagery) with potentially very large pixel values into the [0, 1] range expected by the VGG network that was trained on SDR imagery). The first VGG feature extractor 3750 and second VGG feature extractor 3850 can extract features from the remapped imagery to determine the LI loss.

[0117] FIG. 4 depicts an example system for predicting a gain map and associated metadata for dynamic image rendering, according to example embodiments of the disclosure.

[0118] The example system 4000 of FIG. 4 may be implemented by any one of user computing system 102, server computing system 130, or training computing system 150, or a combination thereof. Unless otherwise specified, references to a ’‘computing system” herein refers to the user computing system 102 alone, server computing system 130 alone, training computing system 150 alone, or any combination thereof. Thus, each of the components of FIG. 4 can be included in user computing system 102 alone, server computing system 130 alone, training computing system 150 alone, or any combination thereof.

[0119] In the example system 4000 of FIG. 4, an input image 4100 is received by an input image re-sizer 4200. The input image 4100 may have a first dynamic range. The ratio between the maximum and the minimum of a tonal value in an image (e.g.. a brightness or luminance level of a specific pixel in the image) can be referred to as the dynamic range, or may simply be determined based on the range of luminosity a display can reproduce, from the black level to its peak brightness. Dynamic range can also be measured in stops, which is aunit of measurement to describe a change in luminance or exposure by a predetermined amount (e.g.. by a factor of 2, either doubling the luminance or exposure or halving it).

[0120] For example, the input image 2100 may have a standard dynamic range (SDR) or a low dynamic range. Generally, SDR content (images, video, etc.) may have a color depth of 8 bits and can represent a maximum luminance level of around 100 nits. For example, SDR images may have a dynamic range of about 6 stops.

[0121] Conversely, HDR content (images, video, etc.) generally shows more detail in both bright and dark areas and thus has a greater dynamic range of color. Generally, HDR content may have a color depth of 10 bits or higher (e.g., 32 bits) and can represent a maximum luminance level of 400 nits or higher (e.g., 1.000 nits, 10,000 nits, etc.). For example. HDR images may have a dynamic range of about 17 stops.

[0122] The input image re-sizer 4200 may be configured to decrease the resolution (e.g., by downsampling the input image 4100). For example, the resolution of the input image 4100 may be decreased such that the downsized input image output by the input image resizer 4200 and input to the one or more machine-learned models 4300, is one-quarter resolution of the original input image 4100. For example, if the original input image 4100 was 1920x1080 pixels, the quarter-resolution version output by the input image re-sizer 4200 would be 480x270 pixels. Implementation of the input image re-sizer 4200 may help conserve or reduce compute time for future operations with respect to the input image (e.g., including computing the gain map of the input image), and conserve storage space (e.g., by taking up less storage space with smaller size images). However, in some implementations, the input image re-sizer 4200 may be omitted and the original input image 4100 may be input to the one or more machine-learned models 4300. In some implementations, the input image re-sizer 4200 may decrease the size (resolution) of the input image 4100 by more than a factor of four (e.g., eight times). In some implementations, the input image re-sizer 4200 may also be configured to subject the input image 4100 to gamma encoding to alter the luminance values of the input image 4100 to cause the input image 4100 to be more perceptually consistent.

[0123] The one or more machine-learned models 4300 may receive the downsized input image 4100 from the input image re-sizer 4200. In some implementations, the one or more machine-learned models 4300 can include a fully-convolutional machine-learned model. In some implementations, the one or more machine-learned models 4300 can implement image-to-image translation using a U-Net architecture (e.g.. a U-Net encoder / decoder with skip connections) to translate an input image into an output imagewhile preserving spatial information and capturing both low-level and high-level features. For example, skip connections can be configured to enable the one or more machine-learned models 4300 to learn and retain information from earlier layers and address issues like vanishing gradients. The one or more machine-learned models 4300 may be trained based on a set of training data (e.g., training data 162). The training data can include, for example, various datasets which may be stored remotely or at the training computing system 150. For example, in some implementations an example dataset utilized for training includes a plurality of images (e g., a series of rapid photos captured in quick succession, capturing multiple exposures of the same scene). For example, the plurality of images may include thousands of images (e.g., 25,000 bursts, 50,000 bursts, 100,000 bursts, etc.). However, other datasets of images may be utilized (e.g.. images from external websites). In some implementations, the dataset may be confined to a particular category, genre, landscape, etc. In some implementations, the dataset may contain diverse subject matter including landscapes, individuals, groups of people, pets, food, etc.

[0124] Different from the one or more machine-learned models 2300 which can be configured to output a predicted or inferred boost image 2400 from which a gain map and maximum content boost value are derived, the one or more machine-learned models 4300 can be configured to directly predict (infer) a gain map 4400 and maximum content boost value 4600. For example, encoded latent features output by the one or more machine-learned models 4300 may be passed to neural network 4500 which can include a plurality of downsampling and convolution layers followed by a plurality of fully-connected layers to output a maximum content boost value 4600. For example, the maximum content boost value 4600 may correspond to a value which can be multiplied by the values of the gain map 4400 for rendering the input image with extended dynamic range. In some implementations, the maximum content boost value 4600 may also be referred to as a log2_max_content_boost parameter (e.g., Iog2 max content boost corresponds to log base 2(max content boost) where max_content_boost equals 2A(log_base_2_max_content_boost)). In some implementations, the gain map 4400 encodes a low-resolution (typically, approximately quarter-resolution) per-pixel map.

[0125] Metadata generator 4700 may be configured to generate metadata for the input image 4100 which includes the gain map 4400 and the maximum content boost value 4600. Modified input image generator 4800 may be configured to modify the input image 4100 to include the metadata generated by the metadata generator 4700. For example, the gain mapand its associated metadata (e.g., the maximum content boost value) can be stored in the extensible metadata platform (XMP) metadata of the input image 4100 (e.g., a SDR JPEG image).

[0126] The image Tenderer 4900 may be configured to determine how many stops brighter each individual pixel from the input image should be when rendering an output image having a greater dynamic range than the original input image 4100 for display at a display device (e.g., an HDR display) given the gain map and maximum content boost value which are stored in the modified input image (e.g., stored in metadata associated with the input image or stored in an image file including the gain map, maximum content boost value, and input image) generated by the modified input image generator 4800. For example, the image Tenderer 4900 may be disposed remotely or locally to the user. For example, a display device capable of displaying HDR imagery may receive the modified input image and generate the output image (HDR image) having the greater dynamic range based on the information (metadata) included in the modified input image including the gain map and maximum content boost value.

[0127] In some implementations, the computing system may include an input device (e.g., user input component 122, a Ul-based slider, etc.) for a user to adjust or tune an amount of apparent brightening by adjusting the maximum content boost value (e.g., during rendering). For example, the computing system may include a user interface or input device to enable the manual adjustment of the maximum content boost value (e g., Iog2_max_content_boost parameter). Thus, a larger contrast enhancing operation than is initially predicted by the one or more machine-learned models 4300 can be obtained, for an enhanced HDR effect.

[0128] FIG. 5 depicts an example system for predicting a gain map and associated metadata for dynamic image rendering, according to example embodiments of the disclosure.

[0129] Aspects of the example system 5000 of FIG. 5 correspond to the example system of FIG. 4, and therefore some details will not be repeated herein for the sake of brevity. The example system 5000 of FIG. 5 may be implemented by any one of user computing system 102, server computing system 130, or training computing system 150, or a combination thereof. Unless otherwise specified, references to a “computing system” herein refers to the user computing system 102 alone, server computing system 130 alone, training computing system 150 alone, or any combination thereof. Thus, each of the components of FIG. 5 canbe included in user computing system 102 alone, server computing system 130 alone, training computing system 150 alone, or any combination thereof.

[0130] In the example system 5000 of FIG. 5, an input image 5050 is received by one or more machine-learned models 5100. In some implementations, the input image 5050 may be downsized prior to being received by the one or more machine-learned models 5100 (e.g., as described with respect to input image re-sizer 4200). In some implementations, the one or more machine-learned models 5100 can correspond to one or more machine-learned models 4300 and implement a U-Net architecture (e.g., a U-Net encoder / decoder with skip connections) to translate the input image 5050 into an output image while preserving spatial information and capturing both low-level and high-level features. For example, the U-Net architecture can include skip connections which are configured to enable the one or more machine-learned models 5100 to learn and retain information from earlier layers and address issues like vanishing gradients.

[0131] As illustrated in FIG. 5, the one or more machine-learned models 5100 can be trained based on the predicted information and ground truth 5400. For example, the one or more machine-learned models 5100 can be trained by comparing the predicted gain map 5200 predicted directly by the one or more machine-learned models 5100 with a normalized ground truth gain map 5420 for the higher dynamic range (e.g., HDR) image (e.g., [-0.5, 0.5]) to compute a LI loss.

[0132] As illustrated in FIG. 5, the one or more machine-learned models 5100 can be configured to output encoded latent features output by the one or more machine-learned models 5100 may be passed to neural network 5250 which can include a plurality' of downsampling and convolution layers 5252 followed by a plurality of fully-connected layers 5254 to output a maximum content boost value 5300.

[0133] As illustrated in FIG. 5, the neural network 5250 can be trained by comparing the predicted maximum content boost value 5300 with a ground truth maximum content boost value 5430 for the higher dynamic range (e.g., HDR) image to compute a LI loss associated with the predicted maximum content boost value.

[0134] FIG. 5 further represents a stop gradient 5150 which is configured to ensure a UNet encoder of the one or more machine-learned models 5100 does not receive updates based on the maximum content boost value LI loss and thus only the plurality of downsampling and convolution layers 5252 and plurality of fully -connected layers 5254 areimpacted by the LI loss determined by comparing the maximum content boost value 5300 and ground truth maximum content boost value 5430.

[0135] Similar to FIG. 3. in FIG. 5 predicted image Tenderer 5500 may be configured to generate an output image having a greater dynamic range than the input image 5050, for example, a high dynamic range image. For example, the output image may be based on the predicted gain map 5200 and the maximum content boost value 5300. In some implementations, the output image may be based on the predicted gain map 5200 and with knowledge of the ground truth maximum content boost value 5430.

[0136] The one or more machine-learned models 5100 may be trained using a Visual Geometry Group (VGG) perceptual feature loss comparing a ground truth rendition generated by ground truth image Tenderer 5410 with the model-predicted rendition generated by image Tenderer 5500. In some implementations, when the original VGG network is trained on SDR imagery' only, the system 5000 may be configured to tone map the ground truth and predicted images via the first tone mapping operator 5600 and second tone mapping operator 5700, respectively (e.g., using a differentiable global tone-mapping technique called Adaptive Logarithmic Mapping (ALM), which remaps imagery7(e.g., HDR imagery) with potentially very7large pixel values into the [0, 1] range expected by the VGG network that was trained on SDR imagery7). The first VGG feature extractor 5800 and second VGG feature extractor 5900 can extract features from the remapped imagery to determine the LI loss.

[0137] The one or more machine-learned models 3100 may also be trained with an LI image reconstruction loss comparing the ground truth and predicted images after they have been encoded via first non-linear luminance encoder 3500 and second non-linear luminance encoder 3600, respectively, in a scale that transforms the values in the first dynamic range (<=1.0) according to a power function, and the values in the second dynamic range (>1.0) according to a logarithmic function, similar to the hybrid log gamma (HLG) HDR encoding function. For example, the non-linear luminance encoding may be implemented using a power function (e.g., y = xA(l / 2.2) in the first dynamic range (SDR) portion of the image [0, 1]), and a log function (e.g., y = ln(x) / (ln(exp(2.2)) + 1) in the second dynamic range (HDR) [1, float max]).

[0138] The one or more machine-learned models 3100 may also be trained with a direct LI or L2 supervision loss comparing the predicted and ground truth gain map directly,and the same on the predicted maximum content boost value versus ground truth maximum content boost value.Example Methods[01391 FIG- 6 depicts a flow chart diagram of an example method to perform according to example embodiments of the disclosure. Although FIG. 6 depicts operations performed in a particular order for purposes of illustration and discussion, the methods of the disclosure are not limited to the particularly illustrated order or arrangement. The various operations of the method 6000 can be omitted, rearranged, combined, and / or adapted in various ways without deviating from the scope of the disclosure.

[0140] According to some examples of the disclosure, a computing system (e.g., user computing system 102, server computing system 130, training computing system 150, or combinations thereof) may be configured to predict (estimate, infer, hallucinate, etc.) gain map information (e.g., HDR gam map information) and its associated metadata (e.g., a maximum content boost value) for an input image having a lower dynamic range (e.g., an SDR image or LDR image). An image (e.g., an HDR image) can then be rendered based on the input image and the gain map information and its associated metadata.

[0141] For example, at 6100, a computing system (e.g., user computing system 102. server computing system 130, training computing system 150) receives an input image having a first dynamic range (e.g., an SDR image).

[0142] At 6200, the computing system (e g., user computing system 102, server computing system 130, training computing system 150) applies a machine-learned model to the input image, to obtain a gain map for content having a second dynamic range, the second dynamic range being greater than the first dynamic range. For example, the machine-learned model may correspond to one or more machine-learned models 2300, one or more machine- learned models 3100, one or more machine-learned models 4300. or one or more machine- learned models 5100.

[0143] At 6300, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) stores the gain map in metadata associated with the input image. For example, the modified input image generator 2800 or modified input image generator 4800 may be configured to store the gain map (and associated metadata such as the maximum content boost value) in the extensible metadata platform (XMP) metadata of the input image (e.g., a SDR JPEG image).

[0144] In some implementations, the method 6000 can include other operations. For example, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) can reduce a size of the input image a predetermined amount before applying the machine-learned model to the input image. For example, input image re-sizer 2200, 4200 may be configured to decrease the resolution (e.g., by downsampling the input image). For example, the resolution of the input image may be decreased such that the downsized input image output by the input image re-sizer 2200, 4200 and input to the one or more machine-learned models 2300, 4300, is one-quarter resolution of the original input image. In some implementations, the input image re-sizer 2200, 4200 may decrease the size (resolution) of the input image by more than a factor of four (e.g., eight times). In some implementations, the input image re-sizer 2200, 4200 may also be configured to subject the input image to gamma encoding to alter the luminance values of the input image to cause the input image to be more perceptually consistent.

[0145] For example, in some implementations the method 6000 can include rendering an output image having the second dynamic range by applying the gain map (stored or embedded in the input image or stored in metadata associated with the input image) to the input image. For example, the image Tenderer 2900, 4900 may be configured to determine how many stops brighter each individual pixel from the input image should be when rendering an output image having a greater dynamic range than the original input image for display at a display device (e.g., an HDR display) given the gain map and maximum content boost value which are stored in metadata associated with the modified input image generated by the modified input image generator 2800, 4800.

[0146] For example, in some implementations the method 6000 can include editing the input image by modifying a portion of the input image to produce a modified input image with the modified portion; applying the machine-learned model to the modified portion of the modified input image, to obtain a partial gain map for content having the second dynamic range, the partial gain map corresponding to the modified portion; and modify ing the gain map by replacing gain map values corresponding to the portion of the input image with gain map values from the partial gain map corresponding to the modified portion.

[0147] For example, a computing system may be configured to modify a portion of the input image which includes extraneous or unneeded content for a corrective inpainting operation. In some implementations, the computing system may be configured to predict (estimate, infer, hallucinate, etc.) HDR content for an image that is subjected to a destructive editing operation. For example, a portion of an input image (e.g., an SDR image) with a gainmap (e.g., an HDR gain map) and associated metadata (e.g., a maximum content boost value) may be edited using an application and new pixels which replace the edited portion may be inpainted in the SDR color image.

[0148] According to examples of the disclosure, in some implementations the computing system may be configured to keep the same prior gain map pixels from the original image while adding inferred or predicted gain map pixels for the new content predicted utilizing one or more machine-learned models (e.g., one or more machine-learned models 2300, 3100, 4300, 5100). According to examples of the disclosure, in some implementations the computing system may be configured to infer or predict gain map pixels for the entire image having the new content utilizing the one or more machine-learned models. For example, in some implementations, the part of the image to be replaced is greater than a threshold level (e.g., 50% or greater), the computing system may be configured to infer or predict gain map pixels for the entire image having the new content utilizing one or more machine-learned models.

[0149] In some implementations, the method includes editing the input image by modifying a portion of the input image to produce a modified input image with the modified portion. For example, the input image may be edited via an inpainting operation that can be performed automatically by the computing system or based on a user input. When a size of the portion of the input image is less than a threshold amount (e.g., less than 50%), the computing system may be configured to apply one or more machine-learned models (e.g., one or more machine-learned models 2300, 3100, 4300, 5100) to the modified portion of the modified input image, to obtain a partial gain map (e.g., an HDR gain map) for content having the second dynamic range, the partial gain map corresponding to the modified portion. When a size of the portion of the input image is greater than the threshold amount (e.g., 50% or greater), the computing system may be configured to apply the one or more machine- learned models to an entirety of the modified input image, to obtain an updated gain map for content having the second dynamic range (e.g., an HDR gain map), the updated gain map corresponding to the entirety of the modified input image.

[0150] In some implementations with respect to the context of inpainting an HDR image, the maximum content boost value may be available in the HDR image's metadata, and the computing system may be configured to leam the HDR gain map values for the missing pixels and the operation of determining the maximum content boost value (for example, via neural network 4500), can be selectively performed (e.g., the operation can be omitted when the maximum content boost value is available in the HDR image's metadata).

[0151] According to examples of the disclosure, in some implementations the computing system may be configured to predict gain maps for video content. For example, in some implementations, the one or more machine-learned models (e.g., one or more machine- learned models 2300, 3100, 4300, 5100) can be configured to process each frame of the video individually. For videos, the per-frame HDR rendition can be directly encoded and saved in any HDR video format (e.g.. HDR10). In some implementations, the one or more machine- learned models (e.g., one or more machine-learned models 2300, 3100, 4300, 5100) can be configured to process every N frames (e.g., every 2 frames, every 3 frames, every 4 frames, etc.), thus reducing the number of inference times, while propagating information from previous frames.

[0152] In some implementations, the method 6000 includes receiving a plurality of input images (which correspond to video content), wherein each of the plurality of input images have the first dynamic range. The method 6000 can further include applying the one or more machine-learned models (e.g., one or more machine-learned models 2300, 3100, 4300, 5100) to each of the plurality of input images, to obtain a plurality of gain maps (e.g.. HDR gain maps) corresponding to the plurality of input images, the plurality of gain maps being for content having the second dynamic range (e.g., HDR content). The method 6000 can further include storing the plurality of gain maps in metadata associated with the plurality of input images and rendering video content having the second dynamic range by applying the plurality of gain maps stored in the metadata associated with the plurality of input images to the plurality of input images.

[0153] In some implementations, the method 6000 can include receiving a plurality of input images (which correspond to video content), wherein each of the plurality of input images have the first dynamic range. The method 6000 can further include applying the one or more machine-learned models (e.g., one or more machine-learned models 2300, 3100, 4300, 5100) to every n-th input image of the plurality of input images, to obtain a plurality of first gain maps (e.g., HDR gain maps), wherein the plurality' of first gain maps are for content having the second dynamic range and n is a value greater than two. The method 6000 can further include obtaining a plurality of second gain maps for input images among the plurality of input images other than the every n-th input image among the plurality' of inputs images, based on the plurality of first gain maps. For example, the second gain maps can be obtained based on interpolation or another method according to the values of the pixels from the first gain maps. The method 6000 can further include storing the plurality of first gain maps and the plurality of second gain maps in metadata associated with corresponding input imagesamong the plurality of input images; and rendering video content having the second dynamic range by applying the plurality of first gain maps and the plurality of second gain maps to the corresponding input images among the plurality of input images. For example, where n=2, the plurality of first gain maps can be applied to the 2nd, 4th, 6th, and subsequent even- numbered images, and the plurality of first gain maps can be applied to the 1st, 3rd, 5th, 7th, and subsequent odd-numbered images.

[0154] In some implementations, storing the gain map in metadata associated with an input image at operation 6300 includes storing the gain map in the extensible metadata platform metadata of the input image.

[0155] In some implementations, the method 6000 includes applying, by the computing system, the one or more machine-learned models (e.g., one or more machine-learned models 2300, 3100, 4300, 5100) to the input image, to predict a maximum content boost value which can be multiplied by the values of the HDR gain map for rendering the input image with extended dynamic range. For example, as described with respect to FIG. 2, based on the boost image 2400 output by the one or more machine-learned models 2300, the max content boost value generator 2500 may be configured to compute a maximum content boost value. For example, the maximum content boost value can be multiplied by the values of the HDR gain map for rendering the input image with extended dynamic range. For example, as described with respect to FIG. 4. encoded latent features output by the one or more machine- learned models 4300 may be passed to neural network 4500 which can include a plurality of downsampling and convolution layers followed by a plurality of fully-connected layers to output a maximum content boost value 4600. As an example, a predicted maximum pixel value for an image having a second dynamic range (e.g., for HDR content) may have a value of 1000. As the value of 1000 may not be saved into a limited bit range (e.g., an eight bit range), each of the pixels in the gain map may be divided by 1000 such that the pixel values are maintained in a normalized scale (e.g., from zero to one).

[0156] In some implementations, operation 6300 can include storing the maximum content boost value in extensible metadata platform metadata of the input image and storing the gam map in the extensible metadata platform metadata of the input image.

[0157] In some implementations, the method 6000 can include receiving, by the computing system, an input via a user interface modifying the maximum content boost value predicted via the one or more machine-learned models (e.g., one or more machine-learned models 2300, 3100, 4300, 5100) to change a degree of contrast enhancement applied to the input image. For example, the computing system may include an input device (e.g., userinput component 122, a Ul-based slider, etc.) for a user to adjust or tune an amount of apparent brightening by adjusting the maximum content boost value (e.g., during rendering). For example, the computing system may include a user interface or input device to enable the manual adjustment of the maximum content boost value (e.g., Iog2_max_content_boost parameter). Thus, a larger contrast enhancing operation than is initially predicted by the one or more machine-learned models 2300 can be obtained, for an enhanced HDR effect.Experimental Results

[0158] Example machine-learned models as described herein were implemented under various conditions to generate gain maps (e.g., HDR gain maps) and to render content having a dynamic range greater than the dynamic range of the input content.

[0159] FIG. 7 depicts example images 7000 illustrating input SDR images, corresponding ground truth HDR gain maps, and predicted HDR gain maps generated according to the one or more machine-learned models 4300, 5100 described with respect to FIGS. 4 and 5.

[0160] FIG. 8 depicts example images 8000 illustrating an input image (e.g., an SDR image), a corresponding predicted HDR gain map, and a predicted HDR image rendered according to the one or more machine-learned models 4300, 5100 described with respect to FIGS. 4 and 5 based on the predicted HDR gain map and maximum content boost value. In the example of FIG. 8, the HDR images are rendered two stops down to simulate HDR display capability’.

[0161] FIG. 9 depicts example images 9000 illustrating input images, predicted HDR images rendered according to the one or more machine-learned models 4300, 5100 described with respect to FIGS. 4 and 5 based on the predicted HDR gain maps and maximum content boost value, and predicted HDR gain maps generated according to examples of the disclosure. In the example of FIG. 9, the HDR images are rendered two stops down to simulate HDR display capability7.

[0162] According to the example embodiments described herein, the disclosed computing systems, including the one or more machine-learned models 2300, 3100, 4300, 5100, can be applied to any kind of image including photographic images, Al-generated imagery, paintings (including realistic imagery and non-realistic, abstract imagery).Additional Disclosure

[0163] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility’ of computer-based systems allows for a great variety7of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0164] Aspects of the above-described example embodiments may be recorded in non-transitory computer-readable media including program instructions to implement various operations embodied by a computer. The media may also include, alone or in combination with the program instructions, data files, data structures, and the like. Examples of non- transitory computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD ROM disks, Blue-Ray disks, and DVDs; magneto-optical media such as optical discs; and other hardware devices that are specially configured to store and perform program instructions, such as semiconductor memory7, readonly memory (ROM), random access memory (RAM), flash memory, USB memory, and the like. Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter. The program instructions may be executed by one or more processors. The described hardware devices may be configured to act as one or more software modules in order to perform the operations of the above-described embodiments, or vice versa. In addition, a non-transitory computer-readable storage medium may be distributed among computer systems connected through a network and computer-readable codes or program instructions may be stored and executed in a decentralized manner. In addition, the non- transitory computer-readable storage media may also be embodied in at least one application specific integrated circuit (ASIC) or Field Programmable Gate Array (FPGA).

[0165] Each block of the flowchart illustrations may represent a unit, module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of order. For example, two blocks shown in succession may in fact be executed substantially concurrently(simultaneously) or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.

[0166] While the disclosure has been described in detail with respect to various example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations and / or additions to the subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the disclosure covers such alterations, variations, and equivalents.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method, comprising: receiving, by a computing system, an input image having a first dynamic range; applying, by the computing system, a machine-learned model to the input image, to obtain a gain map for content having a second dynamic range, the second dynamic range being greater than the first dynamic range; and storing, by the computing system, the gain map in metadata associated with the input image.

2. The computer-implemented method of claim 1. wherein the first dynamic range is a standard dynamic range or a low dynamic range, and the second dynamic range is a high dynamic range.

3. The computer-implemented method of claim 2, further comprising: reducing a size of the input image a predetermined amount before applying the machine-learned model to the input image.

4. The computer-implemented method of claim 1, further comprising rendering an output image having the second dynamic range by applying the gain map stored in the metadata associated with the input image to the input image.

5. The computer-implemented method of claim 1, further comprising: editing the input image by modifying a portion of the input image to produce a modified input image with the modified portion; applying the machine-learned model to the modified portion of the modified input image, to obtain a partial gain map for content having the second dynamic range, the partial gain map corresponding to the modified portion; and modifying the gain map by replacing gain map values corresponding to the portion of the input image with gain map values from the partial gain map corresponding to the modified portion.

6. The computer-implemented method of claim 1. further comprising:editing the input image by modifying a portion of the input image to produce a modified input image with the modified portion; when a size of the portion of the input image is less than a threshold amount: applying the machine-learned model to the modified portion of the modified input image, to obtain a partial gain map for content having the second dynamic range, the partial gain map corresponding to the modified portion; and when a size of the portion of the input image is greater than the threshold amount: applying the machine-learned model to an entirety of the modified input image, to obtain an updated gain map for content having the second dynamic range, the updated gain map corresponding to the entirety of the modified input image.

7. The computer-implemented method of claim 1, further comprising: receiving a plurality of input images, wherein each of the plurality of input images have the first dynamic range; applying the machine-learned model to each of the plurality of input images, to obtain a plurality of gain maps corresponding to the plurality of input images, the plurality of gam maps being for content having the second dynamic range; storing the plurality of gain maps in metadata associated with the plurality of input images; and rendering video content having the second dynamic range by applying the plurality of gain maps stored in the metadata associated with the plurality of input images to the plurality of input images.

8. The computer-implemented method of claim 1, further comprising: receiving a plurality of input images, wherein each of the plurality of input images have the first dynamic range; applying the machine-learned model to every7n-th input image of the plurality' of input images, to obtain a plurality of first gain maps, wherein the plurality of first gain maps are for content having the second dynamic range and n is a value greater than two; obtaining a plurality of second gain maps for input images among the plurality of input images other than the every7n-th input image among the plurality of inputs images, based on the plurality' of first gain maps;storing the plurality of first gain maps and the plurality' of second gain maps in metadata associated with corresponding input images among the plurality’ of input images; and rendering video content having the second dynamic range by applying the plurality' of first gain maps and the plurality’ of second gain maps to the corresponding input images among the plurality of input images.

9. The computer-implemented method of claim 1, wherein storing the gain map in in the metadata associated with the input image comprises storing the gain map in extensible metadata platform metadata of the input image.

10. The computer-implemented method of claim 1, further comprising: applying, by the computing system, the machine-learned model to the input image, to predict a maximum content boost value; and applying, by the computing system, the maximum content boost value to the gain map.

11. The computer-implemented method of claim 10, further comprising: storing the maximum content boost value in extensible metadata platform metadata of the input image, and wherein storing the gain map in the metadata associated with the input image comprises storing the gain map in the extensible metadata platform metadata of the input image.

12. The computer-implemented method of claim 10, further comprising: receiving, by the computing system, an input via a user interface modifying the maximum content boost value predicted via the machine-learned model to change a degree of contrast enhancement applied to the input image.

13. A computing system, comprising: one or more processors; and one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:receiving an input image having a first dynamic range; applying a machine-learned model to the input image, to obtain a gain map for content having a second dynamic range, the second dynamic range being greater than the first dynamic range; and storing the gain map in metadata associated with the input image.

14. The computing system of claim 13, wherein the first dynamic range is a standard dynamic range or a low dynamic range, and the second dynamic range is a high dynamic range.

15. The computing system of claim 13, wherein the operations further comprise: rendering an output image having the second dynamic range by applying the gain map stored in the metadata associated with the input image to the input image.

16. The computing system of claim 13, wherein the operations further comprise: editing the input image by modifying a portion of the input image to produce a modified input image with the modified portion; when a size of the portion of the input image is less than a threshold amount: applying the machine-learned model to the modified portion of the modified input image, to obtain a partial gain map for content having the second dynamic range, the partial gain map corresponding to the modified portion; and when a size of the portion of the input image is greater than the threshold amount: applying the machine-learned model to an entirety of the modified input image, to obtain an updated gain map for content having the second dynamic range, the updated gain map corresponding to the entirety of the modified input image.

17. The computing system of claim 13, wherein the operations further comprise: receiving, by the computing system, a plurality’ of input images, wherein each of the plurality of input images have the first dynamic range; applying the machine-learned model to at least two of the plurality of input images to obtain a plurality' of gain maps, the plurality' of gain maps being for content having the second dynamic range; and rendering video content having the second dynamic range by applying the plurality of gain maps to the at least two of the plurality of input images.

18. The computing system of claim 13, wherein the operations further comprise: applying, by the computing system, the machine-learned model to the input image, to predict a maximum content boost value; and applying, by the computing system, the maximum content boost value to the gain map.

19. The computing system of claim 18, wherein the operations further comprise: storing the maximum content boost value in extensible metadata platform metadata of the input image, and wherein storing the gain map in the metadata associated with the input image comprises storing the gain map in the extensible metadata platform metadata of the input image.

20. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising: receiving an input image having a first dynamic range; applying a machine-learned model to the input image to obtain a gain map for content having a second dynamic range, the second dynamic range being greater than the first dynamic range; and storing the gain map in metadata associated with the input image.

Images