High dynamic range image format compatible with low dynamic range

The method generates a recovery map from HDR and LDR images to adapt pixel brightness for display on devices with varying dynamic ranges, ensuring high-quality HDR display without specialized hardware, addressing the inefficiencies of current technologies.

JP2024543288AActive Publication Date: 2024-11-21GOOGLE LLC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2023576390
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-01-16
Filing Date
2023-05-31
Publication Date
2024-11-21
Estimated Expiration
2043-05-31

AI Technical Summary

Technical Problem

Existing technologies struggle to efficiently display high dynamic range (HDR) images on devices with varying dynamic range capabilities without degrading visual quality or requiring specialized hardware.

Method used

A method and format that generates a recovery map based on HDR and low dynamic range (LDR) images, allowing devices to display HDR images by applying a brightness gain encoded in the recovery map, which scales pixel brightness to match the display device's capabilities.

Benefits of technology

Enables efficient storage and high-quality display of HDR images on a wide range of devices with varying dynamic ranges, maintaining visual fidelity and reducing storage requirements, while avoiding the need for specialized hardware codecs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024543288000001_ABST
    Figure 2024543288000001_ABST
Patent Text Reader

Abstract

Implementations relate to providing a low dynamic range compatible HDR image format. In some implementations, a computer-implemented method includes acquiring a first image depicting a scene with a first dynamic range, acquiring a second image depicting the scene with a second dynamic range different from the first dynamic range, and generating a restoration map based on the first image and the second image. The restoration map encodes a difference in luminance between corresponding portions of the first image and the second image, the difference being scaled by a range scaling factor including a ratio of a maximum luminance of the second image to a maximum luminance of the first image. The first image and the restoration map are provided to an image container, the image container being readable for display of a derived image, the derived image having a dynamic range different from the first dynamic range based on applying the restoration map to the first image.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 421,155, entitled HIGH DYNAMIC RANGE IMAGE FORMAT WITH LOW DYNAMIC RANGE COMPATIBILITY, filed October 31, 2022, and U.S. Provisional Patent Application No. 63 / 439,271, entitled HIGH DYNAMIC RANGE IMAGE AND VIDEO FORMATS WITH LOW DYNAMIC RANGE COMPATIBILITY, filed January 16, 2023, the entire contents of both applications being incorporated herein by reference. [Background technology]

[0002] background Users of devices such as smartphones or other digital camera devices capture and store many photographs in an image library. Some cameras are capable of capturing high dynamic range (HDR) images that present a greater dynamic range and more realistic image quality than the low dynamic range (LDR) images captured by many other (e.g., older) cameras. HDR images are best viewed on a display device capable of displaying the full dynamic range of HDR images.

[0003] The description of the background of the invention provided herein is intended to generally present the context of the present disclosure. The inventors' work to the extent described in this Background of the Invention section, as well as aspects of the description that do not qualify as prior art at the time of filing, are not admitted, explicitly or implicitly, as prior art to the present disclosure. Summary of the Invention

[0004] overview Implementations described herein relate to methods, apparatus, and computer-readable media for providing HDR image formats compatible with low dynamic ranges. In some implementations, the computer-implemented method includes acquiring a first image depicting a particular scene with a first dynamic range, acquiring a second image depicting the particular scene with a second dynamic range different from the first dynamic range, and generating a restoration map based on the first image and the second image. The restoration map encodes a difference in luminance between a portion of the first image and a corresponding portion of the second image, the difference being scaled by a range scaling factor including a ratio of a maximum luminance of the second image to a maximum luminance of the first image. The first image and the restoration map are provided to a readable image container to display a derived image based on applying the restoration map to the first image, the derived image having a dynamic range different from the first dynamic range.

[0005] Various implementations of the method are described. In some implementations, the second dynamic range is greater than the first dynamic range, and the dynamic range of the derived image is greater than the first dynamic range. In some implementations, the image container is readable to display the first image by a first display device capable of displaying the first dynamic range, and to display the derived image by a second display device capable of displaying a dynamic range greater than the first dynamic range. In some implementations, obtaining the first image includes performing range compression on the second image. In some implementations, generating the restoration map includes encoding the brightness gain such that applying the brightness gain to the brightness of individual pixels of the first image results in corresponding pixels in the second image. In some implementations, generating the restoration map may include encoding the brightness gain in a logarithmic space, where the restoration map value is proportional to the difference in logarithms of the brightness divided by the logarithm of the range scaling factor. In some implementations, the step of generating the recovery map is based on the formula recovery(x,y)=log(pixel_gain(x,y)) / log(range_scaling_factor), where recovery(x,y) is the recovery map for pixel location (x,y) of the second image and pixel_gain(x,y) is the ratio of the luminance of the second image to the luminance of the first image at location (x,y).

[0006] In some implementations of the method, generating the recovery map includes encoding the recovery map into a bidirectional grid, which in some examples includes determining a three-dimensional data structure of grid cells, each grid cell being a vector element that maps to a plurality of pixels of the first image. In some implementations, the method further includes encoding the recovery map into a recovered image that is provided to an image container. In some implementations, the range scaling factor is encoded into the recovered image as metadata. In some implementations, the recovered image has an aspect ratio identical to that of the first image.

[0007] In some implementations, the method further includes obtaining an image container, determining to display a second image by a display device, scaling a plurality of pixel intensities of the first image in the image container based on a particular luminance output and a restoration map of the display device to obtain a derived image, and displaying the derived image by the display device after the scaling step as an output image having a larger dynamic range than the first image. In some implementations, the method includes determining a full luminance display capability of the display device, and the scaling of the pixel intensities includes increasing the luminance of the high-light areas in the first image to a luminance level that is equal to or less than the full luminance display capability. In some implementations, the luminance of the derived image is increased to the full luminance of the second image. In some implementations, the scaling of the plurality of pixel intensities includes increasing the range of the dark areas in the first image to a level that the display device can display.

[0008] In some implementations of the method, the second dynamic range is less than the first dynamic range and the dynamic range of the derived image is less than the first dynamic range. In some implementations, obtaining the second image includes performing range compression on the first image. In some implementations, the image container is readable to display the first image by a first display device capable of displaying the first dynamic range and to display the derived image by a second display device capable of displaying a dynamic range less than the first dynamic range.

[0009] In some implementations, a computer-implemented method includes obtaining an image container including a first image having a first dynamic range and a restoration map. The restoration map encodes a luminance gain of pixels of the first image scaled by a range scaling factor including a ratio of a maximum luminance of the second image to a maximum luminance of the first image. The second image has a second dynamic range different from the first dynamic range and corresponds to the first image of the depicted subject. The method determines whether to display one of the first image or a derived image having a dynamic range greater than the first dynamic range. In response to determining to display the first image, the first image is displayed on the display device. In response to determining to display the derived image, the luminance gain of the restoration map is applied to the luminance of pixels of the first image to determine corresponding pixel values ​​for each of the derived images, and the derived images are displayed on the display device.

[0010] Various implementations of the method are described. In some implementations, the second dynamic range is greater than the first dynamic range, and the dynamic range of the derived image is greater than the first dynamic range. In some implementations, the method includes determining a maximum luminance display capability of the display device. In some implementations, in response to determining that the display device can only display a display dynamic range equal to or less than the first dynamic range, the method determines to display the first image. In some implementations, in response to determining that the display device can display a display dynamic range greater than the first dynamic range, the method determines to display the derived image. In some implementations, the display device can display a display dynamic range greater than the first dynamic range, and applying the gain of the restoration map includes adapting a luminance of pixel values ​​of the derived image to a display dynamic range of the display device.

[0011] In some implementations, the display device is capable of displaying a display dynamic range greater than the first dynamic range, and applying the gain of the restoration map includes matching a luminance of pixel values ​​of the derived image to a display dynamic range of the display device, in some implementations, a maximum value of the dynamic range of the derived image is less than a maximum luminance of the display device and a maximum luminance of the dynamic range of the second image.

[0012] In some implementations, applying the recovery map gain includes scaling the luminance of the first image based on the particular luminance output of the display device and the recovery map. In some implementations, scaling the luminance of the first image is performed based on the following formula: derived_image(x,y)=first_image(x,y)+log(display_factor)*recovery(x,y), where derived_image(x,y) is a log-space version of the derived image, first_image(x,y) is a log-space version of the first image recovered from the image container, display_factor is the smaller of a range scaling factor and a maximum (or desired) display luminance of the display device, and recovery(x,y) is the recovery map for pixel location (x,y) of the first image, where the range scaling factor is a ratio of the maximum luminance of the second image to the maximum luminance of the first image.

[0013] In some implementations, the method further includes decoding the range scaling factor from the image container in response to determining to display the derived image. In some implementations, the method further includes decoding the recovery map from the recovery image included in the image container. In some implementations, the method further includes extracting the recovery map from a bidirectional grid stored in the image container.

[0014] In some implementations, the step of determining whether to display one of the first image or the derived image based on a display dynamic range of a display device included in or coupled to the client device is performed by the server device, and the step of the display device displaying the first image includes a step of sending the first image from the server device to the client device so that the client device displays the first image, and the step of the display device displaying the derived image includes a step of sending the derivative image from the server device to the client device so that the client device displays the derivative image.

[0015] In some implementations, displaying the derivative image by the display device includes reducing a maximum brightness level of the displayed derivative image based on a system setting of the device displaying the derivative image, the system setting being selected by a user. In some implementations, displaying the derivative image by the display device includes displaying the derivative image at a first maximum brightness level that is less than a second maximum brightness level determined for pixel values ​​of the derivative image by applying a brightness gain, and gradually increasing the first maximum brightness level of the derivative image to the second maximum brightness level over a particular period of time.

[0016] In some implementations, a system includes a processor and a memory storing instructions coupled to the processor, the processor executing the instructions to perform operations including obtaining an image container including a first image having a first dynamic range and a restoration map encoding luminance gains of pixels of the first image. The method includes determining to display a derived image on the display device, the derived image corresponding to the first image of the depicted subject and having a dynamic range different from the first dynamic range. In response to determining to display the derived image on the display device, a gain of the restoration map is applied to luminances of pixels of the first image to determine respective corresponding pixel values ​​of the derived image, and the luminances of the first image are scaled based on a particular luminance output of the display device and the restoration map. The derived image is displayed on the display device.

[0017] Various implementations of the system are described. In some implementations, the dynamic range of the derived image is greater than the first dynamic range, and the processor, by instructions, performs operations further including determining a maximum luminance display capability of the display device, and determines to display the derived image in response to determining that the display device is capable of displaying a display dynamic range greater than the first dynamic range. In some implementations, the maximum value of the dynamic range of the derived image is the smaller of the maximum luminance of the display device and the maximum luminance of the dynamic range of the original image used to generate the restoration map. In some implementations, the restoration map encodes the luminance gain of the pixels of the first image scaled by a range scaling factor including a ratio of the maximum luminance of the second image to the maximum luminance of the first image, the second image having a second dynamic range greater than the first dynamic range and corresponding to the first image of the depicted subject. In some implementations, the processor further performs operations including extracting the restoration map from a bidirectional grid stored in the image container. In some implementations, the system includes one or more of the operations and / or features of the method described above.

[0018] Some implementations may include a computing device including a processor and a memory coupled to the processor, the memory storing instructions that the processor executes to perform operations including one or more of the method operations and / or functions described above.

[0019] Some implementations include a non-transitory computer-readable medium storing instructions that cause a processor to execute the instructions to perform operations that may be similar to the operations and / or functionality of the methods, systems, and / or computing devices described above. [Brief description of the drawings]

[0020] [Figure 1]FIG. 1 is a block diagram of an example network environment that may be used for one or more implementations described herein. [Diagram 2] 1 is a flow diagram illustrating an example method for encoding an image into a forward-compatible high dynamic range image format according to some implementations. [Diagram 3] 1 is a flow diagram illustrating an example method for generating a restoration map based on an HDR image and an LDR image, according to some implementations. [Figure 4] 1 is a flow diagram illustrating an example method for encoding a resilience map into a bidirectional grid, according to some implementations. [Diagram 5] 1 is a flow diagram illustrating an example method for decoding an image in a forward-compatible high dynamic range image format and displaying an HDR image according to some implementations. [Figure 6] 1A-1C are diagrams illustrating example images exhibiting high dynamic range, according to some implementations. [Figure 7] 1A-1C are diagrams illustrating example images exhibiting low dynamic range, according to some implementations. [Figure 8] 1A-1C are diagrams illustrating example images exhibiting low dynamic range, according to some implementations. [Figure 9] 1A-1C are diagrams illustrating example images exhibiting low dynamic range, according to some implementations. [Figure 10] 1 is a block diagram of an example computing device that may be used to perform one or more functions described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Detailed Description The present disclosure relates to a forward-compatible HDR image format that provides a container from which both low dynamic range (LDR) and high dynamic range (HDR) images can be obtained and displayed. The image format can be used by an LDR display device to display an LDR version of the image, and by an HDR display device to display an HDR version of the image.

[0022] In some implementations, an image container is generated that implements an image format. An image of a smaller dynamic range (e.g., an LDR image) and a corresponding image of a larger (e.g., higher) dynamic range (e.g., an HDR image) are obtained. In some examples, the HDR image is captured by a camera, and the LDR image is generated from the HDR image, for example, using tone mapping or other processing. The LDR image can be a standard image format, such as JPEG or other formats. The restoration map generated based on the HDR image and the LDR image encodes a difference (e.g., gain) in luminance between a portion (e.g., pixel or pixel value) of the LDR image and a corresponding portion (e.g., pixel or pixel value) of the HDR image, and a range scaling factor that scales the difference includes a ratio of the maximum luminance of the HDR image to the maximum luminance of the LDR image. In some example implementations, the restoration map can be a scalar function that encodes a luminance gain in logarithmic space, and the restoration map value is proportional to the difference in logarithm of the luminance divided by the logarithm of the range scaling factor. In some implementations, the restoration map can be encoded into a data structure or image representation that can reduce the storage space required to store the restoration map. For example, the data structure may be a bidirectional grid where a three-dimensional data structure of grid cells is used.The restored image and the LDR image are provided in an image container that may be stored, for example, as a new HDR image format.

[0023] The image container can be read and processed by a device that displays an output image, which is an LDR image or an output HDR image (e.g., a derived image). A device that reads and displays an image stored in the above image format may use an LDR display device (e.g., a display screen) that can display an LDR image, for example, the display dynamic range of the LDR display device can display up to the low dynamic range of the LDR image, but cannot display the larger dynamic range of the HDR image. Such an LDR display device only accesses the LDR image in the image container and ignores the restored image. Since the LDR image is in a standard format, the device can easily display the LDR image. Thus, in some implementations, the LDR image can still be displayed by a device that does not perform coding to detect or apply the restored image. If the accessing device can use a display device that can display an HDR image (e.g., a display device that can display in its output a luminance area larger than the maximum luminance of the LDR image), it accesses the LDR image and the restoration map in the image container and applies the luminance gain encoded in the restoration map to the pixels of the LDR image to determine the respective corresponding pixel values ​​of the output HDR image to be displayed. In some examples, the device scales pixel luminance of the LDR image based on the luminance output of the HDR display device (e.g., the highest luminance output of the display device) as well as the luminance gain stored in the restoration map. The output HDR image is displayed on an HDR display device having a high dynamic range. In other implementations, the HDR image and the restoration map can be stored in an image container, and the restoration map can be applied to the HDR image to obtain a low dynamic range LDR image that is locally tone mapped by the restoration map and has high visual quality.

[0024] The above features have provided several technical advantages, including enabling efficient storage and high-quality display of high dynamic range images or small (e.g., standard) dynamic range images by a wide range of devices and by using a single image format. For example, image formats are provided that allow any device to display images with a dynamic range appropriate to its display capabilities. Furthermore, images displayed from these formats do not suffer from loss of visual quality (e.g., especially in their detailed local contrast) due to conversion from one dynamic range to another. In various implementations, the output HDR image generated from the image container can be a lossless exact version of the original HDR image used to generate the image container, or this output HDR image can be a (visually) similar version of the original HDR image since the restoration map stores luminance information.

[0025] The function may include using a restoration map derived from the LDR image to produce an output HDR image. The restoration map may include values ​​based on a luminance gain ratio scaled by a range scaling factor, which is the ratio of the maximum luminance of the original HDR image to the maximum luminance of the LDR image. For example, the restoration map indicates the relative amount to scale each pixel, and the range scaling factor indicates the specific amount of scaling to be performed, which provides a context to the relative values ​​in the restoration map (e.g., expand the range of amounts by which pixels can be scaled for a particular image). The range scaling factor is advantageous in that it allows efficient, accurate, and concise specification of the normalized range of restoration map values, which can then be efficiently adjusted for a particular output dynamic range using the range scaling factor. Using the range scaling factor makes efficient use of all the bits to store the restoration map, thus enabling accurate encoding of a greater variety of images.

[0026] Moreover, in some implementations or cases, when displaying the output HDR image, the pixel luminance of the LDR image is also scaled based on a display factor. The display factor may be based on the luminance output of the HDR display device that displays the output image. This allows HDR images with any convenient dynamic range higher than LDR to be generated from the container based on the display capabilities of the display device. The dynamic range of the output HDR image is not bound to any standard high dynamic range or to the dynamic range of the original HDR image. Since HDR display devices may have a significant variation in the brightness at which they can display images, the benefit of this feature is that the image can be displayed with a high quality rendition on a display device of any dynamic range. Moreover, the above feature allows for the change of the dynamic range of the output HDR image (e.g., higher than LDR) in real time based on user input or other conditions, for example to reduce the user's visual stress or fatigue, or for other uses. This display scaling allows the output HDR image to have any dynamic range larger than that of the LDR image.

[0027] In contrast, conventional techniques may encode HDR images for displays with a specific and unchanging dynamic range or maximum brightness. Such techniques cannot take advantage of HDR display devices with a dynamic range or brightness greater than the encoded dynamic range. In addition, if the HDR display device cannot display a dynamic range as high as that encoded in the original HDR image, such conventional techniques may result in lower quality images, and may require techniques such as roll-off curves to reduce the dynamic range of the HDR image for display on the HDR display device, which may, for example, reduce the local contrast in the image in an undesirable way, often resulting in a decrease in the visual quality of the image.

[0028] The features provide a sharable and upwardly compatible mechanism for generating and rendering images containing high dynamic range (HDR) content beyond current formats. The image format enables next-generation consumer HDR content and, in some implementations, may enable support for specialized image workflows without requiring specialized hardware codecs for decoding / encoding. One or more implementations of the HDR format address issues with current HDR transfer functions that assume global tone mapping. In contrast, the features enable local tone mapping, which is better at preserving details in scenes with different levels of brightness. Thus, the HDR format can generate higher fidelity and higher quality HDR-rendered content compared to conventional HDR formats.

[0029] In addition, the above features allow the image format to have small storage requirements. For example, the image format allows a single LDR image to be stored with a small storage requirement recovery map, thus saving significant storage space over formats that store multiple full images. For example, some implementations include compressing the recovery map, and in some of these examples, the recovery map can be coded into a bidirectional grid. Such a grid can store the recovery map with significantly reduced storage space requirements and allows the recovery map to provide an output HDR image with little loss of visual quality compared to the original HDR image.

[0030] A technical effect of one or more of the above implementations is that, compared to conventional systems, a device can display an image having a dynamic range that better corresponds to the dynamic range of a particular output display device being used to display the image. For example, such conventional systems may provide images that do not have a dynamic range that corresponds to the dynamic range of the display device, resulting in poorer image display quality. The functionality described herein can mitigate such inconveniences, for example, by providing a recovery map in the format of the image and / or scaling the image output to set the dynamic range of the image to better suit the particular display device. Moreover, a technical effect of one or more of the above implementations may include a device consuming fewer computational resources to obtain a result. For example, a technical effect of the above techniques may include a reduced consumption of processing and / or storage resources of the system, compared to conventional systems in which one or more of the above techniques or features are not provided. For example, conventional systems may need to store and / or provide full LDR and full HDR images to display images with a dynamic range suitable for a particular display device, which requires additional memory and communication bandwidth resources, compared to the above techniques. In another example, conventional systems rely on storing only HDR images and then tone mapping the HDR images to a smaller dynamic range for LDR displays, but such tone mapping can degrade visual quality when rendering a wide variety of HDR images on an LDR display.

[0031] As mentioned herein, dynamic range is related to the ratio of the brightest and darkest parts of a scene. The dynamic range of an LDR format usually does not exceed a certain low range, such as a standard dynamic range (SDR) file with a low range color space / color profile. Similarly, the displayed dynamic range output of an LDR display device is low. As mentioned herein, an image with a larger (or higher) dynamic range than an LDR image (e.g., JPEG) is considered an HDR image, and a display device capable of displaying the larger dynamic range is considered an HDR display device. An HDR image can store pixel values ​​that span a larger tonal range than an LDR. In some examples, an HDR image can display the dynamic range of a real scene more accurately and / or have a lower dynamic range that is larger than the dynamic range of an LDR image (e.g., an HDR image can generally have more than 8 bits per color channel).

[0032] The use of "log" in this description refers to the logarithm to a particular base; for example, all logs herein can be logarithms to any base, such as logarithms to base 2, logarithms to base 10, etc.

[0033] Further to the description herein, a user is provided with controls that allow the user to choose whether and when to allow the systems, programs, or features described herein to collect user information (e.g., images from the user's library, social networks, social activities, or business, occupation, user preferences, the user's current location, the user's messages, or the characteristics of the user's device) and whether and when to send content or communications from the server. In addition, certain data may be processed in one or more ways such that personal information is removed before being stored or used. For example, the user's identification information may be processed such that personal information about the user cannot be determined, or if location information (such as to the city, zip code, or state level) is obtained, the user's geographic location may be generalized such that the user's specific location cannot be determined. Thus, the user may control what information is collected about the user, how that information is used, and what information is provided to the user.

[0034] 1 shows a block diagram of an example network environment 100 that may be used in some implementations described herein. In some implementations, the network environment 100 includes one or more server systems, such as server system 102 in the example of FIG. 1, and a number of client devices, such as client devices 120-126, each associated with a respective user, such as users U1-U4. The server system 102 and each of the client devices 120-126 may be configured to communicate with a network 130.

[0035] The server system 102 may include a server device 104 and an image database 110. In some implementations, the server device 104 may provide an image application 106a. In FIG. 1 and the remaining figures, a letter following a reference number (e.g., "106a") denotes a reference to the element having that particular reference number. A reference number without a following letter in the text, e.g., "106," denotes a general reference to an embodiment of the element having that reference number.

[0036] Image database 110 may be stored in a storage device that is part of server system 102. In some implementations, image database 110 may be implemented using a relational database, a key-value structure, or other types of database structures. In some implementations, image database 110 may include multiple sections, each corresponding to a respective image library, for each of users 1-4. For example, as seen in FIG. 1, image database 110 may include a first image library (image library 1, 108a) for user 1 and other image libraries (image library 2, ..., image library n) for various other users. Although FIG. 1 illustrates a single image database 110, it will be understood that image database 110 may be implemented as a distributed database, for example across multiple database servers. Additionally, while FIG. 1 illustrates multiple sections, one for each user, in some implementations, each image library may be implemented as a separate database.

[0037] Image library 108a may store a number of images (including video) related to user 1, metadata associated with the images, and one or more other database fields stored in association with the images. Access rights to image library 108a may be restricted so that user 1 can control how images and other data in image library 108a are accessed, for example, by image application 106, other applications, and / or one or more other users. Server system 102 may be configured to enforce access rights such that image data for a particular user cannot be accessed unless authorized by that user.

[0038] An image as referred to herein may include a digital image having pixels with one or more pixel values ​​(e.g., color values, brightness values, etc.). An image may be a still image (e.g., a still photograph, an image having a single frame, etc.), a moving image (e.g., an animation, animated GIF, a cinemagraph in which some of the images include motion while other parts are still images, etc.), or a video (e.g., a series of images or frames of images that may optionally include audio). An image as used herein may be understood to be any of the above. For example, implementations described herein may be used with still images (e.g., photographs or other images), video, or moving images.

[0039] The network environment 100 may include one or more client devices, e.g., client devices 120, 122, 124, and 126, which may communicate with each other and / or with the server system 102 through a network 130. The network 130 may be any type of communication network, including one or more of the Internet, a local area network (LAN), a wireless network, a switched or hub connection, and the like. In some implementations, the network 130 may include peer-to-peer communication between devices using, for example, a peer-to-peer wireless protocol (e.g., Bluetooth, Wi-Fi Direct, and the like). An example of peer-to-peer communication between two client devices 120 and 122 is illustrated by arrow 132.

[0040] In various implementations, users 1, 2, 3, and 4 may communicate with server system 102 and / or with each other using respective client devices 120, 122, 124, and 126. In some examples, users 1, 2, 3, and 4 may interact with applications running on respective client devices and / or with server system 102 through network services, such as, for example, social network services or other types of network services implemented in server system 102. For example, respective client devices 120, 122, 124, and 126 may communicate data with one or more server systems, such as, for example, server system 102.

[0041] In some implementations, the server system 102 may provide appropriate data to the client devices so that each client device can receive the communication or shared content that has been uploaded to the server system 102 and / or the network service. In some examples, Users 1-4 may interact by image sharing, audio or video conferencing, audio, video, or text chat, or other communication modes or applications.

[0042] The network services implemented by the server system 102 may include systems that enable users to conduct various communications, form links and relationships, upload or post shared content such as images, text, audio, and other types of content, and / or perform other functions. For example, a client device may display received data, such as content posted, transmitted, or streamed to the client device from separate client devices, server systems, and / or network services through the server and / or network services (or directly from separate client devices). In some implementations, client devices may communicate directly with each other, for example, using peer-to-peer communication between client devices as described above. In some implementations, a "user" may include one or more programs or virtual entities as well as people interfacing with the system or network.

[0043] In some implementations, any of the client devices 120, 122, 124, and / or 126 may host one or more applications. For example, as shown in FIG. 1, the client device 120 may host an image application 106b. The client devices 122-126 may host similar applications. The image application 106a may be implemented using the hardware and / or software of the client device 120. In various implementations, the image application 106a may be a standalone client application running, for example, on any of the client devices 120-124, or may work in cooperation with the image application 106b hosted on the server system 102.

[0044] The image application 106 may provide a variety of image (including video) related functions performed with user permission. For example, such functions may include one or more of capturing an image using a camera, modifying an image, determining image quality (e.g., based on factors such as surface size, dirt, number of surfaces, image composition, lighting, exposure, etc.), storing images in an image library 108, encoding and decoding images and video into any of a variety of image and video formats (including those formats described herein), providing a user interface for viewing displayed images or image-based generation or collection, etc.

[0045] The client device 120 may include an image library 108b for user 1, which may be a standalone image library. In some implementations, the image library 108b may be usable in combination with the image library 108a of the server system 102. For example, with user permission, the image library 108a and the image library 108b may be synchronized over the network 130. In some implementations, the image library 108 may include a number of images related to the user 1, such as images captured by the user (e.g., using a camera on the client device 120 or another device), images shared with the user 1 (e.g., from the image libraries of the other users 2-4), images downloaded by the user 1 (e.g., from a website, messaging application, etc.), screenshots, and other images. In some implementations, the image library 108b on the client device 120 may include a subset of images in the image library 108a on the server system 102. For example, such an implementation may be advantageous when the amount of available storage space on the client device 120 is limited.

[0046] In various implementations, the client device 120 and / or the server system 102 may include other applications (not shown), which may be applications that provide various types of functionality. User interfaces on the client devices 120, 122, 124, and / or 126 may enable the display of user content and other content, including images, image-based generation, data, and other content, as well as communications, privacy settings, notifications, and other data. Such user interfaces may be displayed using software on the client devices, software on the server device, and / or a combination of client software and server software executing on the server device 104, e.g., application software or client software that communicates with the server system 102. The user interfaces may be displayed by a display device, e.g., a touch screen, other display screen, projector, etc., of the client device or server device. In some implementations, application programs running on the server system may communicate with the client devices to receive user input on the client devices and to output data, e.g., visual data, audio data, etc., to the client devices.

[0047] For ease of explanation, FIG. 1 shows one block of server system 102, server device 104, image database 110, and four blocks of client devices 120, 122, 124, and 126. Server blocks 102, 104, and 110 may represent multiple systems, server devices, and network databases, and the blocks may be arranged in different configurations than shown. For example, server system 102 may represent multiple server systems that may communicate with other server systems over network 130. In some implementations, server system 102 may include, for example, a cloud hosting server. In some examples, image database 110 may be stored in a storage device arranged in a server system block separate from server device 104, which may communicate with server device 104 and other server systems over network 130.

[0048] Also, there may be any number of client devices. Each client device may be any type of electronic device, such as, for example, a desktop computer, a laptop computer, a portable or mobile device, a mobile phone, a smart phone, a tablet computer, a television, a set-top box or entertainment device for a television, a wearable device (e.g., glasses or goggles with a display, a watch, a headset, an armband, jewelry, etc.), a personal digital assistant (PDA), a media player, a gaming device, etc. In some implementations, the network environment 100 may have less than all of the components shown and / or other elements instead of or in addition to those described herein, including other types of elements.

[0049] Other implementations of the functionality described herein may use any type of system and / or service. For example, other networked (e.g., connected to the Internet) services may be used instead of or in addition to social networking services. Any type of electronic device may utilize the functionality described herein. Some implementations may provide one or more of the functionality described herein on one or more client devices or server devices that are disconnected or intermittently connected to a computer network. In some examples, a client device that includes or is connected to a display device may display content posts that have been previously received, for example, over a communication network and stored in a storage device local to the client device.

[0050] FIG. 2 is a flow diagram illustrating an example method 200 of encoding an image into a forward compatible high dynamic range image format, e.g., a low dynamic range compatible HDR image format, according to some implementations. In some implementations, the method 200 may be performed on, for example, the server system 102 shown in FIG. 1. In some implementations, some or all of the method 200 may be performed on one or more client devices, such as client devices 120, 122, 124, or 126 of FIG. 1, one or more server devices, such as server device 104 of FIG. 1, and / or both server devices and client devices. In the above examples, the implementing system includes one or more digital processors or processing circuits ("processors") and one or more storage devices (e.g., databases or other storage mechanisms). In some implementations, separate components of one or more servers and / or clients may perform separate blocks or other portions of the method 200. In some examples, an apparatus is described as performing the blocks of the method 200. Some implementations may have one or more blocks of method 200 performed by one or more other devices (e.g., other client devices or server devices) that may send results or data to the first device.

[0051] In some implementations, method 200, or portions thereof, may be initiated automatically by the system. For example, method (or portions thereof) may be executed periodically or based on one or more specific events or conditions, such as, for example, initiation of image application 106 by a client device, capture of a new image by an image capture device of the client device, receipt of an image over the network by a device, upload of a new image to server system 102, expiration of a predetermined period of time since the last operation of method 200, and / or occurrence of one or more other conditions that may be specified in settings retrieved by method 200.

[0052] User permission may be obtained to use user data in method 200 (blocks 210-220). For example, user data for which permission may be obtained may include images stored on a client device (e.g., any of client devices 120-126) and / or a server device, image metadata, user data associated with use of image applications, other image-based generation, etc. The user is provided with options to selectively allow or not allow access to all or any subset of the user data. If user permission is insufficient with respect to particular user data, method 200 may be performed, for example, without the user data, but using other data (e.g., images unrelated to the user).

[0053] Method 200 may begin at block 210. In block 210, a high dynamic range (HDR) image depicting a particular scene ("original HDR image") is acquired by a device. In some implementations, the HDR image is provided in any of several standard formats for high dynamic range images. In some examples, the HDR image may be captured by an image capture device, such as a camera of the client device or another device. In further examples, the HDR image is received from another device over a network or retrieved from a storage mechanism accessible by the device. In some implementations, the HDR image may be generated based on any of several image generation techniques, such as ray tracing.

[0054] In block 212, a low dynamic range (LDR) image (e.g., an "original LDR image") is acquired by the device, where the LDR image depicts the same scene and / or subject matter as the HDR image, e.g., corresponding to the HDR image of the depicted subject matter. The LDR image has a smaller dynamic range than the HDR image. In some implementations, the LDR image may have the dynamic range of a standard LDR image format or may have a somewhat smaller dynamic range than the HDR image. For example, the LDR image may be provided in a standard image format, e.g., JPEG, AV1 Image File Format (AVIF), TIFF, etc. In some examples, the LDR image has a dynamic range provided by 8 bits per channel per pixel, and the HDR image has a higher dynamic range (e.g., typically a bit depth of 10 bits or more, a wide range of color space, and an HDR-oriented transfer function) than the dynamic range of the LDR image.

[0055] In various implementations, the HDR image of block 210 may be acquired before or after the LDR image acquisition. For example, in some implementations, an HDR image is acquired and an LDR image is derived from the HDR image. In some examples, the LDR image may be generated based on the HDR image using local tone mapping or other processing. Local tone mapping, for example, changes each pixel according to its local characteristics in the image (rather than according to a global tone mapping function that changes each pixel in the same way). Any of a variety of local tone mapping techniques may be used to reduce the gray values ​​of the HDR image to those suitable for an LDR image having a smaller dynamic range. For example, local Laplacian filtering and / or other techniques may be used in the tone mapping process. In some implementations, different global tone curves (or digital gains) may be used in different regions of the image. In some implementations, different tone mapping techniques may be used for different types of image data, for example, different techniques may be used for still images and video images. In other examples, exposure fusion techniques may be used to generate an LDR image from an HDR image and / or multiple captured or generated images, e.g., combining portions of multiple images captured at different exposure levels or synthetically generated to generate a combined LDR image including those portions. In some example implementations, a combination of tone mapping and exposure fusion techniques may be used to generate an LDR image (e.g., from separate images having separate exposure levels using a Laplacian tone mapping pyramid to provide a weighted blend of the pyramid and collapse the blended pyramid).

[0056] In other examples, the LDR image may be obtained from other sources. For example, the HDR image may be captured by a camera device, and the LDR image may be captured by the same camera device, for example, immediately before or after the capture of the HDR image, or at least partially simultaneously. In some examples, the camera device may capture the HDR image and the LDR image based on the camera settings, for example, with a dynamic range dictated by the camera settings or user-selected settings. Block 212 may be followed by block 214.

[0057] In block 214, a restoration map is generated based on the HDR image and the LDR image. The restoration map encodes the luminance difference between a portion (e.g., pixel) of the HDR image and a corresponding portion (e.g., pixel) of the LDR image. The restoration map is used to convert the luminance of the LDR image to the luminance of the HDR image. For example, a luminance gain and a range scaling factor may be encoded in the restoration map such that applying the luminance gain to the luminance value of an individual pixel of the LDR image results in the corresponding pixel of the HDR image. An example of generating a restoration map is described in more detail below with reference to FIG. 3. Block 214 may be followed by block 216.

[0058] In block 216, the recovery map may be encoded into a recovery element. In various implementations, the recovery element may be an image, a data structure, an algorithm, a neural network, etc., that includes or provides the recovery map information. For example, the recovery element may be a recovered image compressed into a standard format, with an aspect ratio identical to that of the LDR image. The recovered image may have any resolution, e.g., the same as or different from the LDR image. For example, the recovered image may have a lower resolution than the LDR image, e.g., a quarter of the resolution of the LDR image (e.g., 480x270 recovered image versus 1920x1080 for the LDR image).

[0059] In some examples, the recovered image may be encoded as single-channel 8-bit unsigned integer values, each value representing a recovered value and stored in one pixel of the recovered image. In some example implementations, the encoding may result in a continuous representation of -2.0 to 2.0, which may be compressed, e.g., JPEG compressed. In some implementations, the encoding may include one bit indicating the positive / negative of the map value, and the remaining bits may be interpreted as the magnitude of the recovered map value (e.g., ranging from -1 to +1). In some implementations, such as this example, the magnitude representation may exceed 1.0 or -1.0, since some pixels may require a larger attenuation or gain than that represented by the range scaling factor to accurately recover or represent the luminance region of the original HDR image.

[0060] In single-channel coding, a channel can specify a luminance (lightness)-based adjustment to a target image. Thus, the chromaticity or hue of the target image should not change during application of the restoration map to the image. Single-channel coding can preserve the existing chromaticity or hue of the target image (e.g., preserve the RGB ratio) and typically requires less storage space than multi-channel coding. In some implementations, the restoration image can be coded as multi-channel values. Multi-channel coding can allow adjustment of the color (e.g., chromaticity or hue) of the target image during application of the restoration map to the target image. Multi-channel coding can compensate for color loss that occurs in LDR images, such as color roll-off at high lightness. This allows for restoration of HDR lightness as well as accurate color (e.g., resaturating the sky to a blue hue, rather than merely increasing lightness so that the sky appears white). In some implementations, the restoration map of multi-channel coding can be used to restore a larger color gamut (or compensate for color gamut loss) in the target image. For example, from an LDR image in the standard RGB (sRGB) / BT.709 color gamut, the ITU-R Recommendation BT.2020 color gamut can be recovered.

[0061] In some implementations, the restoration map values ​​may use a different bit depth, such as a bit depth greater than 8 bits. For example, in some implementations, 8-bit depth may be the minimum bit depth that allows HDR representation in combination with an 8-bit single channel gain map, and smaller bit depths may not provide enough information to provide HDR image content without artifacts such as banding. In some implementations, the restored image may be encoded as floating point values ​​rather than integers. In some implementations, the restoration element may be a data structure, algorithm, neural network, etc. that encodes the values ​​of the restoration map. For example, the restoration map may be encoded into small neural network weights that are used as the restoration element. Block 216 may be followed by block 218.

[0062] In block 218, the LDR image and the restoration element are provided to an image container that can be stored as a new HDR image format. In some implementations, the LDR image can be considered as a base image in the image container. The image container can be read by a device to display the LDR image or the HDR image. In some implementations, the image container can be a standard image container, such as an AVIF (AV1 Image File Format) image container.

[0063] In some implementations, the recovery map may be quantized to the precision of the image container in which the recovery map is placed. For example, in some implementations where the LDR image is a JPEG image (or other format where 8-bit unsigned integer values ​​are used), the values ​​of the recovery map may be quantized to an 8-bit format for storage. In some examples, each value may represent a recovery value and is stored in one pixel of the recovered image. In some examples, each encoded value may be encode(x,y)=recovery(x,y)*63.75+127.5 In this encoding, for example, each value is quantized to one of 256 possible encoded values, resulting in a representation in the range of -2.0 to 2.0. In other implementations, other ranges and quantizations may be used, for example, -1.0 to 1.0 or other ranges.

[0064] The image container may include additional information related to displaying an output HDR image based on the contents of the image container. For example, the additional information may include metadata provided in the image container, the restoration element, and / or the LDR image. The metadata may encode information regarding how to present the LDR image and / or the HDR image derived therefrom on a display device. Such metadata may include, for example, the version of the restoration map format used in the container, the range scaling factor used in the restoration map, the storage method (e.g., whether the restoration map is encoded in a bidirectional grid or other data structure or compressed by a specified compression technique), the resolution of the restoration map and / or the restored image, guide weights for bidirectional grid storage, and / or other restoration map or image characteristics.

[0065] In some examples, the LDR image may include a data structure or directory of metadata that defines the order and characteristics of items (e.g., files) in an image container, and each file in the container may have a corresponding media item in the data structure. The media item may describe the location of an associated file in the image container and basic characteristics of the associated file. For example, a container element may be encoded into metadata of the LDR image, which defines the format version and data structure of the media items in the container. In some examples, the metadata is stored according to a data model that allows devices or applications that do not support metadata or cannot read metadata to read the LDR image. For example, the metadata may be stored in the recovered image and / or the LDR image according to an Extensible Metadata Platform (XMP) data mode. Block 218 may be followed by block 220.

[0066] In block 220, the image container may be stored, for example, in a storage mechanism, for access by one or more devices. For example, the image container may be sent over a network to one or more server systems, allowing access by multiple client devices that have access to the server systems.

[0067] In some implementations, the server device can save the image container in a cloud storage mechanism (or other storage mechanism) that includes additional information such as metadata. The server can store the LDR image in any format in the container. In response to receiving a request for an image from a client device, the server device can determine image data to serve to the requesting client device based on system settings, user preferences, characteristics of the client device such as the type of client device (e.g., mobile client device, desktop computer client device), characteristics of the client device's display device such as dynamic range, resolution, etc. For example, for some client devices, the server device can serve the image container to the client device, and the client device can decode the image container to obtain an image (base image or derived image) for display on the client's display device with a suitable dynamic range. For other client devices (e.g., the server device has information describing the characteristics of the client device, such as the dynamic range of the client device's display device), the server device can extract and decode the image from the image container and serve the image to the client device, the served image having a suitable dynamic range for the client's display device as determined by the server device. In some implementations, the server device can determine a base image and a derived image from an image container and provide both images to the client device, and the client device can select which of these images to display based on the dynamic range of its display device.

[0068] In some implementations, an image container may include multiple recovery maps (e.g., recovery elements). Each recovery map may include different values ​​for providing one or more different characteristics to an output image based on the recovery map, as described herein. For example, a particular one of the multiple recovery maps may be selected and applied to provide an output image having characteristics based on the selected recovery map, and thus may be more suitable for a particular use or application than other recovery maps in the image container, for example. For example, some or all of the multiple recovery maps may be associated with specific different physical display device characteristics of a target output display device. In some examples, a user may select to use different recovery maps when displaying LDR or HDR images on a display device having a different dynamic range (e.g., different peak brightness) or different color gamut, or when using a different image decoder.

[0069] In some implementations, a respective range scaling factor may be stored in the image container for use with each of these recovery maps. In some implementations, one or more of the recovery elements may be associated with a stored indication of a particular display device characteristic of a target display device, allowing selection of one of the recovery map tracks based on the particular target display device used for display, as described with reference to FIG. 5. In some implementations, a single range scaling factor may be used with multiple recovery maps. For example, a generic range scaling factor may be used for multiple recovery maps respectively associated with multiple images stored in the image container.

[0070] In some implementations, further metadata may be stored in the image container indicating the intended use of the recovery map. In some implementations storing multiple recovery maps in an image container, such metadata may include an indication of the intended use for each of the multiple recovery maps. For example, if recovery map 1 is intended to map to an LDR image, then a map, table, or index with a key of map 1 may indicate a value that is a list of output formats specified in association with the recovery map. In some examples, the metadata may specify an indication of an output range (e.g., "LDR" or "HDR", or HDR may be specified as a multiplier of the LDR range). In some examples, the metadata may specify a bit depth, e.g., "output metadata: bit_depth X," where X may be 8, 10, 12, etc. In some implementations, the specified bit_depth may be a minimum bit depth, e.g., "LDR" may indicate 8 bits or more for output via an opto-electrical transfer function (OETF) and 12 bits or more for linear / extended range output, and "HDR" may indicate 10 bits or more for output via an OETF and 16 bits or more for linear / extended range output. This type of metadata may also be used for multi-channel recovery maps where the color space may vary and an indication of the intended output color space may be included in the metadata. For example, the metadata may be specified as "output metadata:bit_depth X,color_space," where color_space may be, for example, Rec. 709, Rec. 2020, etc.

[0071] 3 is a flow diagram illustrating an example method 300 for generating a restoration map based on an HDR image and an LDR image according to some implementations. For example, the method 300 may be performed with respect to block 214 of the method 200 of FIG. 2 or in other implementations. In some implementations, the LDR image and the (original) HDR image are obtained as described with reference to FIG.

[0072] Method 300 may begin at block 302, where a linear luminance of an LDR image may be determined. In some implementations, the original LDR image (e.g., obtained at block 212 of FIG. 2) may be a non-linear or gamma-corrected image, and a linear version of the LDR image may be generated, for example, by converting a primary image color space of the non-linear LDR image to a linear version. For example, a color space with a standard RGB (sRGB) transfer function is converted to a linear color space that preserves the sRGB primaries.

[0073] The linear luminance of the linear LDR image is determined. For example, the luminance (Y) function may be defined as: Y ldr (x,y)=primary_color_profile_to_luminance(LDR(x,y)) where Yldr is the linear luminance of the low dynamic range image defined in the range 0.0 to 1.0, and primary_color_profile_to_luminance is a function that converts the primary colors of an image (the LDR image) to a linear luminance value Y of each pixel at coordinates (x,y) of the LDR image. Block 302 may be followed by block 304.

[0074] At block 304, a linear luminance of the HDR image is determined. In some implementations, the original HDR image (e.g., obtained at block 210 of FIG. 2) may be a non-linear image and / or a three-channel encoded image (e.g., a perceptual quantizer (PQ) encoded image or a hybrid log-gamma (HLG) encoded image), and a three-channel linear version of the HDR image may be generated, for example, by converting the non-linear HDR image to a linear version. In other implementations, any other color space and color profile may be used.

[0075] The linear luminance of the linear HDR image is determined. For example, the luminance (Y) function may be defined as: Y hdr(x,y)=primary_color_profile_to_luminance(HDR(x,y)) where Yhdr is the linear luminance of the high dynamic range image, defined to range from 0.0 to a range scaling factor, and primary_color_profile_to_luminance is a function that converts the primary colors of an image (the HDR image) to a linear luminance value Y for each pixel at coordinate (x,y) in the HDR image. Block 304 may be followed by block 306.

[0076] In block 306, a pixel gain function is determined based on the linear luminance determined in blocks 302 and 304. The pixel gain function is defined as the ratio of the Yhdr function to the Yldr function. For example, pixel_gain(x,y)=Yhdr(x,y) / Yldr(x,y) and Yhdr is the linear luminance of the high dynamic range image, defined to range from 0.0 to the range scaling factor, and primary_color_profile_to_luminance is a function that converts the primary colors of an image (HDR image) to a linear luminance value Y of each pixel at coordinates (x,y) of the HDR image.

[0077] Since zero is a valid luminance value, Yhdr or Yldr can be zero, which can be problematic when determining the logarithms as in the above equation or as described below. In some implementations, cases where Yhdr and / or Yldr are zero can be addressed by, for example, defining the pixel_gain function as 1, or adjusting the calculation to avoid this case, for example, by using a small enough additional factor, represented as epsilon (ε) in the following equation (e.g., by adding ε or clamping to ε):

[0078] pixel_gain(x,y)=(Yhdr(x,y)+ε) / (Yldr(x,y)+ε) In some implementations, separate values ​​of epsilon (ε) may be used in the numerator and denominator of the above equation. Block 306 may be followed by block 308.

[0079] In block 308, a range scaling factor is determined based on the maximum luminance of the HDR image relative to the maximum luminance of the LDR image. For example, the range scaling factor may be the ratio of the maximum luminance of the HDR image to the maximum luminance of the LDR image. The range scaling factor may also be referred to herein as a range compression factor and / or a range expansion factor, which is the inverse of the range compression factor. For example, if the LDR image is determined from the HDR image, the range scaling factor may be derived from the amount of the entire HDR range that has been compressed to generate the LDR image. For example, the factor may indicate an amount to reduce the luminance of the high-light parts of the HDR image to map the HDR image to the LDR dynamic range, or an amount to increase the luminance of the dark parts of the HDR image to map the HDR image to the LDR dynamic range. The range scaling factor may be a linear value that is multiplied by the entire LDR range to obtain the entire HDR range in linear space. In some examples, if the range scaling factor is 3, the dark parts of the HDR image are tripled or the high-light parts of the HDR image are reduced by a factor of 3 (or a mixture of such enhancement of the luminance of the dark parts and reduction of the luminance of the high-light parts is performed). In some implementations, the range scaling factors may be defined in other ways (e.g., by a camera application or other application, a user, or other content generator) to provide certain visual effects that may change aspects of the output image derived from the range scaling factors. Block 308 may be followed by block 310.

[0080] In block 310, a restoration map is determined that encodes a pixel gain function scaled by a range scaling factor. Thus, the restoration map is based on two linear images containing the desired HDR image luminance (Yhdr) and LDR image luminance (Yldr) through a pixel gain function. In some implementations, the restoration map encodes the normalized pixel gain in logarithmic space and is a scalar function scaled by the range scaling factor (e.g., multiplied by the inverse of the range scaling factor). In some examples, the restoration map value is proportional to the difference between the logarithm of the HDR luminance and the logarithm of the LDR luminance divided by the logarithm of the range scaling factor. For example, the restoration map may be defined as follows:

[0081] recovery(x,y)=log(pixel_gain(x,y)) / log(range_scaling_factor) In some example implementations, recovery(x,y) may tend to fall in the range of -1 to +1. For display on an HDR display device, values ​​below zero make the pixel (of an LDR image) darker, and values ​​above zero make the pixel brighter. In some examples of the effect of the recovery map range, to enhance highlights when displaying an image on an HDR display (an example is described with reference to FIG. 5), these values ​​may typically be in the range of approximately [0..1] (e.g., applying this recovery map enhances highlights while keeping darks constant), so that the brightest areas of the image may have values ​​close to 1, and darker areas of the image may have values ​​close to 0. In some implementations, to push down (darken) dark areas when displaying on an HDR display device, these values ​​may typically be in the range of approximately [-1..0] (e.g., applying this recovery map re-darkens dark areas while keeping highlights constant). In some implementations, these values ​​can cover the full range of [-1..1] to provide a combination of highlight enhancement and dark re-darkening when displayed on an HDR display device.

[0082] In some implementations, as described above, for ease of manipulation, the image is transformed into logarithmic space before processing. In some implementations, this processing may be performed by exponentiation and exponential interpolation in linear space, which is mathematically equivalent to processing in logarithmic space. In some implementations, this processing may be performed without exponentiation, e.g., by naive interpolation / extrapolation in linear space.

[0083] When applying a restoration map to an LDR image for display, the range scaling factor generally indicates the amount to increase pixel brightness, with the map indicating that some pixels are made brighter or darker. The restoration map indicates the relative amount to scale each pixel, with the range scaling factor indicating the specific amount of scaling to be performed. The range scaling factor provides normalization and context to the relative values ​​in the restoration map. The range scaling factor allows for efficient use of all bits used to store the restoration map, regardless of the amount of scaling applied by the restoration map. This means that a greater variety of images can be accurately encoded. For example, all maps may contain values ​​ranging, say, from -1 to 1, and the values ​​are given context by the range scaling factor, such that this range of values ​​may similarly represent maps that are scalable to a larger total range (e.g., 2 or 8). For example, one image in the above format may have a range scaling factor of 2, and a second image may have a range scaling factor of 8. Without the range scaling factor, the maximum / minimum absolute gain values ​​are determined in the map. If this value is, for example, 2, then a second image with a range scale of 8 cannot be represented. If this value is, for example, 8, then 2 bits of information are wasted for every pixel in the restoration map when storing an image with a range scale of 2. Such implementations that do not use a range scaling factor may be prone to (undesirable) visual differences in the displayed output HDR image compared to the original HDR image used for encoding (e.g., severe forms of such differences may include banding).

[0084] In some implementations, when pixel_gain is 0.0, the restoration function may be defined to be -2.0, which is the maximum representable attenuation. The restoration function may be outside the range of -1.0 to +1.0 because one or more regions or locations in the image may require a larger scaling (attenuation or gain) than is represented by the range scaling factor to restore the original HDR image. The range of -2.0 to +2.0 may be sufficient to provide such a larger scaling (attenuation or gain). Block 310 may be followed by block 312.

[0085] In block 312, the recovery map may be encoded into a compressed format. This allows the storage space occupied by the recovery map to be reduced. The compression format may be any of a variety of different compressions, formats, etc. In some implementations, the recovery map is determined and encoded into a bidirectional grid, as described in more detail with reference to FIG. 4. In some implementations, the recovery map may be compressed or otherwise reduced in storage requirements based on one or more additional or alternative compression techniques. In some implementations, the recovery map may be compressed into a lower resolution recovery image having a lower resolution than the recovery image determined in block 310. This recovery image may occupy less storage space than that of the full resolution recovery map. For example, JPEG compression may be used in implementations that result in an LDR image in JPEG format, or other compression types (e.g., run-length encoding (RLE)) or other image format compression (e.g., HEIC, AVIF, PNG, etc.) may be used.

[0086] Figure 4 is a flow diagram illustrating an example method 400 for encoding a restoration map into a bidirectional grid according to some implementations. For example, method 400 may be implemented as, for example, blocks 310 and 312 of method 300 of Figure 3, where a restoration map is determined based on luminance of HDR and LDR images and stored in a compressed format, which is a bidirectional grid in the implementation described in Figure 4. The bidirectional grid is a three-dimensional data structure of grid cells that are mapped to pixels of the LDR image based on guide weights, as described below.

[0087] Method 400 may begin at block 402, where a three-dimensional data structure of grid cells is defined. The three-dimensional data structure has a width, height, and depth in a number of cells, designated as size W×H×D, where the width and height correspond to the width and height of the LDR image, and the depth corresponds to the number of layers of W×H cells. A cell of the grid is defined as a vector element of length D in (width, height) and maps to a number of LDR image pixels. Thus, the number of cells in the grid may be much less than the number of pixels in the LDR image. In some examples, the width and height may be about 3% of the LDR image size, and the depth may be 16. For example, a 1920×1080 LDR image may have a grid of approximately 64×36×16 size. Block 402 may be followed by block 404.

[0088] A set of guide weights is defined in block 404. In some implementations, the set of guide weights may be defined as a multi-element vector of floating-point values ​​and may be used for generating and decoding a bidirectional grid. The guide weights represent the weight of each element of an input pixel LDR(x,y) of an LDR image to determine the corresponding depth D of a cell that is mapped to this input pixel. For example, the set of guide weights may be defined as a five-element vector as follows:

[0089] guide_weights={weight_r,weight_g,weight_b,weight_min,weight_max} Here, weight_r, weight_g, and weight_b refer to the weights of the color components (red, green, or blue) of the respective input pixels, and weight_min and weight_max refer to the weights of the minimum component values ​​of the r, g, and b values ​​and the maximum component values ​​of the r, g, and b values ​​(e.g., min(r,g,b) and max(r,g,b)), respectively.

[0090] In one example, the LDR image is in the Rec. 709 color space, and the guide weights may be set to the following values, for example, to represent the balance of luminance of the LDR(x,y) pixel as well as the minimum and maximum component values ​​of this pixel:

[0091] guide_weights={0.1495,0.2935,0.057,0.125,0.375} In this example, these guide weight values ​​weight pixel intensities (r, g, b values) by 50%, minimum component values ​​by 12.5%, and maximum component values ​​of pixels by 37.5% in the grid depth search. In some implementations, these values ​​sum to 1.0, so that, for example, all grid cells can always be searched (unlike when the values ​​sum to less than 1.0) and the depth of a pixel cannot spill beyond its final depth level in the grid (unlike when the values ​​sum to greater than 1.0). The guide weight values ​​can be practically determined, for example, by a human visually evaluating the results. In some implementations, a different number of guide weights can be used for pixel intensities (r, g, b values), for example, only three.

[0092] Block 404 may be followed by block 406 . At block 406, the grid cells are mapped to pixels of the LDR image based on the guide weights. The guide weights are used to determine the depth D that corresponds to the cell that is mapped to the input pixel.

[0093] In some implementations, the width (x) and height (y) of a grid cell are mapped to a pixel (x,y) of the LDR image. For example, pixel x,y of the LDR image may be converted to the x,y coordinates of a grid cell as follows: grid_x=ldr_x / ldr_W*(grid_W-1) grid_y=ldr_y / ldr_H*(grid_H-1) where grid_x and grid_y are the x and y positions of the grid cell, ldr_x and ldr_y are the x and y coordinates of the pixel of the LDR image that is mapped to the grid cell, ldr_W and ldr_H are the total pixel width and height of the LDR image, and grid_W and grid_H are the total width and height cell dimensions of the bidirectional grid, respectively.

[0094] In some implementations, the following equation may be used for the depth of the grid cell that is mapped to the LDR image: z_idx(x,y)=(D-1)*(guide_weights {r,g,b,min(r,g,b),max(r,g,b)}) where D is the total cell depth dimension of the grid and z_idx(x,y) is the depth in the bidirectional grid of the cell that maps to the LDR image pixel (x,y). This formula determines the dot product of a vector of guide weights and a vector of corresponding values ​​for a particular pixel of the LDR image, and then scales the result by the size of the depth dimension of the grid to indicate the depth of the corresponding cell for that particular pixel. For example, the multiplication of the guide weights can result in a value between 0.0 and 1.0 (inclusive), and the result is scaled to the various depth levels (buckets) of the grid. For example, if D is 16, then a multiplication value of 0.0 results in a mapping to bucket 0, a value of 0.5 results in a mapping to bucket 7, and a value of 1.0 results in a mapping to bucket 15 (the last bucket). Block 406 may be followed by block 408.

[0095] At block 408, the recovery map values ​​are determined using a solution to a set of linear equations. For example, in some implementations, the inputs to the bidirectional grid are defined using the following equation: pixel_gain(x,y)=bilateral_grid_linear(x,y,z_idx(x,y)) where pixel_gain(x,y) is the value at pixel (x,y) of the pixel_gain function described above with reference to method 300 of FIG. 3. Bilateral_grid_linear is the content of the bilateral grid, representing pixel gain in linear space. The coordinates x, y, and z_idx are the coordinates of the bilateral grid, which are different from the x, y coordinates of the LDR image. Depending on z_idx, the guide weights tell each pixel of the LDR image what depth its pixel_gain is located in the bilateral grid, which tells how to set up the set of linear equations to solve in block 410. Block 408 may be followed by block 410.

[0096] In some implementations, the encoding is defined as the solution of a set of linear equations that minimizes for each grid cell:

[0097] (pixel_gain(x,y)*Yldr(x,y)-Yhdr(x,y)) 2 The Yldr and Yhdr values ​​are the luminance values ​​of the LDR and HDR images, respectively, at the (x,y) pixel location of the LDR image. The pixel_gain(x,y) value that minimizes the value of the above equation is solved based on where each pixel is searched for into the depth vector, as indicated in the above equation, which is predefined based on the guide weights.

[0098] The resulting value is defined as: For{x,y,z}over{[0,W),[0,H),[0,D)}:bilateral_grid_linear(x,y,z) This definition specifies that the bidirectional grid is defined as follows: for x, 0 to the total grid width, for y, 0 to the total grid height, and for z, 0 to the total grid depth. For example, if the bidirectional grid is 64x36x16, then the grid values ​​are defined as follows: for x, 0 to 63 (inclusive), for y, 0 to 35 (inclusive), and for z, 0 to 15 (inclusive). Block 408 may be followed by block 410.

[0099] The solution values ​​are transformed and stored in a bidirectional grid at block 410. For example, the solution values ​​may be transformed as follows:

[0100] bilateral_grid(x,y,z)=log(pixel_gain(x,y)) / log(range_scaling_factor) or bilateral_grid(x,y,z)=log(bilateral_grid_linear) / log(range_scaling_factor) Bilateral_grid is the content of a bilateral grid, representing pixel gain in logarithmic space. These values ​​are stored as an encoded restoration map. As indicated in these equations, the normalized pixel gain between the LDR and HDR images is encoded in logarithmic space and scaled by a range scaling factor, as described above with reference to block 310 of FIG. 3.

[0101] When decoding the recovery map from the bidirectional grid to display an HDR image with a larger dynamic range than the LDR image, each recovery map value is extracted from the bidirectional grid (as described in detail with reference to FIG. 5 ). In the following equation, r, g, and b are the respective component values ​​of pixel LDR(x,y) of the LDR image.

[0102] z_idx(x,y)=(D-1)*(guide_weights {r,g,b,min(r,g,b),max(r,g,b)}) recovery(x,y)=bilateral_grid(x,y,z_idx(x,y)) where recovery(x,y) is the recovery map to be extracted. In some implementations, the z_idx determined in the first equation may be rounded to the nearest integer before being input into the bilateral_grid(x,y,z) of the second equation.

[0103] FIG. 5 is a flow diagram illustrating an example method 500 for decoding and displaying an image into a forward-compatible high dynamic range image format according to some implementations. In some implementations, the method 500 may be performed, for example, on the server system 102 shown in FIG. 1. In some implementations, some or all of the method 500 may be performed in one or more client devices, such as client devices 120, 122, 124, or 126 of FIG. 1, one or more server devices, such as server device 104 of FIG. 1, and / or both server devices and client devices. In the above examples, the implementing system includes one or more digital processors or processing circuits ("processors") and one or more storage devices (e.g., databases or other storage mechanisms). In some implementations, different components of one or more servers and / or clients may perform different blocks or other portions of the method 500. In some examples, an apparatus is described as performing the blocks of the method 500. Some implementations may have one or more blocks of method 500 performed by one or more other devices (e.g., other client devices or server devices) that may send results or data to the first device.

[0104] In some implementations, method 500, or portions thereof, may be initiated automatically by the system. For example, the method (or portions thereof) may be executed periodically or based on one or more particular events or conditions, such as, for example, initiation of image application 106 by a client device, capture or receipt of a new image by a client device, upload of a new image to server system 102, expiration of a predetermined period of time since the last operation of method 500, and / or the occurrence of one or more other conditions that may be specified in settings read or received by the system executing method 500.

[0105] Method 500 may be performed by a device other than the device that generated the image container. For example, a first device may generate and store the image container in a storage location (e.g., a server device) accessible by a second device other than the first device. The second device may access the image container and display the image via method 500. In some examples, the recovery map may be applied to the base image by a hardware decoder and / or a CPU and / or GPU of the second device. In some implementations, a single device may generate the image container as described in method 200 and decode and display the image container as described in method 500. In some implementations, the first device may decode an image from the image container based on method 500 and provide the decoded image to a second device for display.

[0106] Similarly, as described above with reference to FIG. 2, in some implementations, a first device (e.g., a server device in the cloud) decodes two images (e.g., a base image and a derived image based on the base image and the recovery map) from the image container and sends the two images to a second device, which selects and displays one of the images based on a decision made by the second device, e.g., based on the characteristics of the display device of the second device.

[0107] User permission may be obtained to use user data in method 500 (blocks 510-534). For example, the user data for which permission has been obtained may include images stored on a client device (e.g., any of client devices 120-126) and / or a server device, image metadata, user data associated with use of image applications, other image-based generation, etc. The user is provided with options to selectively allow or not allow access to all or any subset of the user data. If user permission is insufficient with respect to particular user data, method 500 may be performed, for example, without the user data, but using other data (e.g., images unrelated to the user).

[0108] Method 500 may begin at block 502. At block 510, an image container including an LDR image and a recovery map is obtained. For example, the image container may be a container generated by the method 200 of FIG. 2 for generating a recovery map from an LDR image and an HDR image having a larger dynamic range than the LDR image. The recovery map may be provided to the container as a recovery element, such as a recovery image. The image container may have a standardized format supported and recognized by a device performing method 500, such as a container for JPEG, a custom JPEG marker (e.g., an APP1 segment for storing metadata in a JPEG / Exif file, or a marker for storing chunks of data), an International Organization for Standardization Base Media File Format (ISOBMFF), a container for AV1 Image File Format (AVIF), HEIF (High Efficiency Image File), a Multi-Picture Format (MPF) container, Extensible Metadata Platform (XMP) encoded data, etc. In some implementations, the recovery map may be stored as a sub-image according to a standard (e.g., AVIF, HEIF) or as a new type of sub-image (e.g., defining an AV1 gain map image item, which signals to a reading device that the image item is a gain map and thus is used to define range scaling factors, e.g., in the header). Block 510 may be followed by block 512.

[0109] At block 512, the LDR image in the image container is extracted. In some example implementations, this extraction may include decoding the LDR image from its standard file format (e.g., JPEG) to a raw image format or other format used for display and / or processing. The decoded values ​​from the encoding map may be determined as follows, where r is the range and n is the number of bits for each recovery map value:

[0110] recovery(x,y)=(encode(x,y)-(2^n-1) / 2) / ((2^n-1) / 2r) In some example implementations, with respect to the example of the quantized encoding map (encode(x,y)) described above with respect to block 218 of FIG. 2, the decoded value from the encoding map may be determined as follows:

[0111] recovery(x,y)=(encode(x,y)-127.5) / 63.75 This results in a scale and offset for the range of recovery map values ​​that are stored as 8 bits, from −2 to 2. In another example, if the values ​​stored in the recovery map are from −1 to 1, then the decoded values ​​may be determined as follows:

[0112] recovery(x,y)=(encode(x,y)-127.5) / 127.5 In other implementations, other ranges, scales, offsets, bits of storage, etc. may be used. Block 512 may be followed by block 514.

[0113] In block 514, it is determined whether the output image should be displayed as an HDR image (e.g., a derived image) derived from the extracted LDR image or as an LDR image. The output image is displayed on at least one target display device associated with or in communication with the device that accesses the image container and executes the method 500. The target display device may be any suitable output display device, such as, for example, a display screen, a touch screen, a projector, goggles or glasses with a display, etc. The displayed output HDR image may have any dynamic range higher than the LDR image in the container, up to the dynamic range of the original HDR image used to generate the image container (e.g., of block 210 of FIG. 2).

[0114] The determination of whether to display the output HDR image may be based on one or more characteristics of the target display device that displays the output image. For example, the dynamic range of the target display device may determine whether to display an HDR image or an LDR image as the output image. In some examples, if the target display device can display a dynamic range greater than the dynamic range of the LDR image in the image container, it is determined to display the HDR image. If the target display device can only display a low dynamic range of the LDR image in the container, the HDR image is not displayed.

[0115] In some implementations or cases, it is determined that the target display device will not display the HDR image even though it is capable of a larger dynamic range than the LDR image, for example, user settings or preferences may dictate displaying an image having a smaller dynamic range, or displaying the image as an LDR image for visual comparison with other LDR images, etc.

[0116] If it is determined at block 514 that the HDR image is not to be displayed, the method proceeds to block 516, where the extracted LDR image from the image container is displayed by the target display device. Typically, the LDR image is displayed by an LDR-compatible target output device that cannot display a dynamic range greater than the range of the LDR image. The LDR image may be output directly by the target display device, and any metadata regarding the recovery map and HDR image format in the image container is ignored. In some implementations, the LDR image is provided in a standard format and thus can be read and displayed by any device that can read the standard format.

[0117] If, at block 514, it is determined to display the HDR image as the output image, the method proceeds to block 518, where a restoration element in the image container is extracted. Metadata related to displaying the output HDR image based on the restoration element can also be obtained from the image container. For example, the restoration element can be a restoration image as described above, and can include and / or be accompanied by metadata as described above. Metadata can also be included in the container and / or in the LDR image from the image container, or metadata can instead be included.

[0118] In some implementations, as described above, the image container may include multiple restoration maps (e.g., restoration elements). In some of these implementations, in block 518, one of the multiple restoration elements may be selected for the following processing. In some implementations, the selected restoration element may be specified by user input or user settings, or in some implementations, may be automatically selected (e.g., without current user input) by the device performing the processing, for example, based on one or more characteristics of the target output display device or other output display component, such characteristics may be obtained using an operating system call or other available source. For example, in some implementations, one or more of the restoration elements may be associated with an indication of specific display device characteristics stored in the image container (or accessible to the device performing block 518). If the target display device has been determined to have one or more specific characteristics (e.g., dynamic range, peak brightness, color gamut, etc.), in block 518, a specific restoration element associated with those characteristics may be selected. If the multiple restoration elements are associated with another range scaling factor, this range scaling factor is also selected and retrieved. Block 518 may be followed by block 520.

[0119] At block 520, the recovery map is optionally decoded from the recovery element. For example, the recovery map may be in a compressed or other encoded format in the recovery element and may be decompressed or decoded. For example, in some implementations, the recovery map may be encoded into and decoded from a bidirectional grid as described above. Block 520 may be followed by block 522.

[0120] In block 524, it is determined whether to scale the luminance of the extracted LDR image for the output HDR image. In some implementations, this scaling is based on the display output capabilities of the target display device. For example, in some implementations or cases, the generation of the output HDR image is based in part on a particular luminance that the target display device can display. In some implementations or cases, this particular luminance may be the maximum luminance that the target display device can display, and the maximum luminance of the display device sets an upper limit for the luminance of the output HDR image. In some implementations or cases, it is determined in block 524, for example, that the generation of the output HDR image does not take into account the luminance capabilities of the target display device and does not scale the luminance of the LDR image.

[0121] If at block 524 it is determined not to scale the luminance of the output image based on the capabilities of the target display device, the method proceeds to block 526, where the luminance gain of the restoration map is applied to pixels of the LDR image and the pixels are scaled based on a range scaling factor to determine corresponding pixels of the output HDR image. For example, the luminance gain and scaling are applied to the luminance of pixels of the LDR image to determine their corresponding pixel values ​​of the output HDR image to be displayed. This allows the output HDR image to obtain the same dynamic range as the original HDR image used to generate the image container.

[0122] In some example implementations, the LDR image and range scaling factors may be transformed into logarithmic space and the following equation may be used: HDR*(x,y)=LDR*(x,y)+log(range_scaling_factor)*recovery(x,y) where HDR*(x,y) is pixel (x,y) of the recovered HDR image (output image) in log space, LDR*(x,y) is pixel (x,y) of the log space version of the extracted LDR image (e.g., log(LDR(x,y)), log(range_scaling_factor) is the range scaling factor in log space, and recovery(x,y) is the recovery map value at pixel (x,y). For example, recovery(x,y) is the normalized pixel gain in log space.

[0123] In some examples, when recovery(x,y)*range_scaling_factor and per-pixel gains are applied, an interpolation is made between the LDR and HDR versions of the image. This is essentially an interpolation between the original and range-compressed versions of the image, and the amount of interpolation is dynamic, both pixel by pixel and globally, in the sense that it depends on the range scaling factor. In some examples, this may be considered an extrapolation if the recovery map value is less than 0 or greater than 1.

[0124] In some implementations, the LDR image from the image container is nonlinearly encoded, e.g., gamma encoded, and the primary image color space of the nonlinear LDR image is converted to a linear version before the LDR image is converted to LDR*(x,y) in logarithmic space for the above formula. For example, a color space with a standard RGB (sRGB) transfer function may be converted to a linear color space that preserves the sRGB primaries, similar to the above description with respect to block 302 of FIG. 3. In some implementations, the LDR image in the container may be linearly encoded and no such conversion from a nonlinear color space is used.

[0125] Applying the luminance gain to the luminance of individual pixels of the LDR image results in pixels that correspond to the original (e.g., originally encoded) HDR image. If the LDR image is stored losslessly and given the restoration map, the output HDR image may have identical pixel values ​​to the original HDR image (e.g., if the restoration map has the same resolution as the LDR image). Otherwise, the output HDR image resulting from block 532 may be a very close approximation of the original HDR image, and a user typically does not visually notice any difference in the display of the output image.

[0126] In some implementations, the output image may be converted from logarithmic space to an output color space for display. In some implementations, for example, the output image color space may be different from the color space of the extracted LDR image from the image container when using a multi-channel recovery map. Block 526 may be followed by block 534 below.

[0127] If at block 524 it is determined to scale the luminance of the image based on the output of the target display device, the method proceeds to block 528 where the maximum luminance output of the target display device is determined. The target display device may be an HDR display device, and these types of devices are capable of displaying a maximum luminance value that may vary based on the display model, manufacturer, etc. In some implementations, the maximum luminance output of the target display device may be obtained via an operating system call or other source. Block 528 may be followed below by block 530.

[0128] A display factor is determined in block 530. The display factor is used to scale the dynamic range of the output HDR image to be displayed, as described below. In some implementations, the display factor is determined to be less than or equal to the smaller of the luminance output of the target display device and the range scaling factor. For example, this may be defined as follows:

[0129] display factor≦min(maximum display luminance,range scaling factor) This determination may cap the luminance scaling of the output image at the maximum display luminance of the target display device. This prevents scaling the output image to a luminance range (as dictated by the range scaling factor) that is present in the HDR image from which the image container was generated but that is higher than the maximum luminance output range of the target display device. In some implementations, the maximum display luminance may be a dynamic characteristic of the target display device and may be adjustable, for example, by user settings, application programs, etc.

[0130] In some implementations, the display factor may be set to a luminance greater than the range scaling factor and less than or equal to the highest display luminance. For example, this may be done if the target display device is capable of outputting a greater dynamic range than is present in the original HDR image (which may be indicated by the range scaling factor). In some examples, the user may indicate (by user setting or selection input) that the output image be brightened.

[0131] In some implementations, the display factor may be set to a luminance lower than the maximum display luminance of the target display device or the maximum luminance of the original HDR image. For example, the display factor may be set based on a particular display luminance of the display device as dictated by one or more display conditions, user input, preferences or settings, etc., resulting in an output of a lower luminance level that may be desired in some cases or applications, such as those described below. Block 530 may be followed by block 532 below.

[0132] In block 532, the luminance gain of the restoration map is applied to the LDR image, and the luminance of the LDR image is scaled based on a display factor to determine corresponding pixels of the HDR image. For example, the luminance gain encoded in the restoration image is applied to pixel luminance of the LDR image to determine respective corresponding pixel values ​​of the output image to be displayed. These pixel values ​​are scaled based on a display factor determined in block 530, such as a particular luminance output of the target display device (which may be the maximum luminance output of the target display device). In some implementations, this may increase the luminance of bright areas in the LDR image to a maximum level based on the dynamic range of the original HDR image to a level that the display device can display, or may decrease the luminance of dark areas in the LDR image to a lower limit based on the dynamic range of the original HDR image to a level that the display device can display.

[0133] In some example implementations, the LDR image and display coefficients may be transformed into logarithmic space and the following equation may be used: HDR*(x,y)=LDR*(x,y)+log(display_factor)*recovery(x,y) where HDR*(x,y) is pixel (x,y) of the recovered HDR image (output image) in log space, LDR*(x,y) is pixel (x,y) of the log space version of the extracted LDR image (e.g., log(LDR(x,y)), log(display_factor) is the display factor in log space, and recovery(x,y) is the recovery map value at pixel (x,y). For example, recovery(x,y) is the normalized pixel gain in log space.

[0134] In some examples, when per-pixel gain is applied, it may be considered to interpolate between, for example, recovery_map(x,y)*display_factor and the LDR and HDR versions of the image. This may essentially be thought of as an interpolation between the original and range-compressed versions of the image, with the amount of interpolation being dynamic, both pixel-by-pixel and globally, in the sense that it depends on the display factor. In some examples, this may be considered an extrapolation when the recovery map value is less than 0 or greater than 1.

[0135] In some implementations, the LDR image from the image container is nonlinearly encoded, e.g., gamma encoded, and the primary image color space of the nonlinear LDR image is converted to a linear version before the LDR image is converted to LDR*(x,y) in logarithmic space for the above formula. For example, a color space with a standard RGB (sRGB) transfer function may be converted to a linear color space that preserves the sRGB primaries, similar to the above description with respect to block 302 of FIG. 3. In some implementations, the LDR image in the container may be linearly encoded and no such conversion from a nonlinear color space is used.

[0136] Applying the luminance gain to the luminance of individual pixels of the LDR image results in pixels that correspond to the original (e.g., originally encoded) HDR image. If the LDR image is stored losslessly and given the restoration map, the output HDR image may have identical pixel values ​​to the original HDR image (e.g., if the restoration map has the same resolution as the LDR image). Otherwise, the output HDR image resulting from block 532 may be a very close approximation of the original HDR image, and a user typically does not visually notice any difference in the display of the output image.

[0137] In some implementations, the output image may be converted from logarithmic space to an output color space for display. In some implementations, for example, the output image color space may be different from the color space of the LDR image extracted from the image container when using a multi-channel recovery map. Block 532 may be followed by block 534.

[0138] In block 534, the output HDR image determined in block 526 or block 532 is displayed by a target display device. The output HDR image is displayed by an HDR compatible display device capable of displaying a dynamic range greater than that of the LDR image. The output HDR image generated from the image container may be identical or nearly identical to the original HDR image used to generate the image container because the restoration map stores luminance information, and is generally visually indistinguishable from the original HDR image.

[0139] In some implementations, the output HDR image (from block 526 or block 532) may be suitable for display on a target display device, for example as a linear image or other output HDR image format. For example, the output HDR image may have been processed by the display coefficients of block 532, and no further modification for display may be required, such that the output HDR image resulting from applying the restoration map is directly rendered for display.

[0140] In some implementations or cases, the output HDR image may be converted to another format for display on the target display device, such as, for example, an image format that may be more suitable than the output HDR image for display by the target display device. For example, the output HDR image from block 526 (or from block 532 in some implementations) may be converted to a luminance range that is suitable for the capabilities of the target display (this may be in addition to the use of display factors in block 532 in some implementations). In some implementations, the output HDR image may be converted to a standard image having a standard image format, such as, for example, HLG / PQ or other standard HDR format. In some of these examples, a general-purpose tone mapping technique may be applied to such a standard image for display by the target display device.

[0141] In some implementations or cases, the output HDR image processed for a target display device by the display coefficients of block 532 may result in a higher quality image display on the target device than the output HDR image so converted and processed by a generic tone mapping technique without further conversion to a standard format or processing using a generic tone mapping technique for display. For example, local tone mapping provided by a restoration map may result in, for example, a more detailed and higher quality image than generic or global tone mapping. Moreover, there may be ambiguities and differences in the conversion of an HDR image to any standard format using a global tone mapping technique based on different device specifications and implementations.

[0142] In some implementations, techniques may be used to output images to a display device in a linear format, such as an extended range format, where 0 represents black, 1 represents SDR white, and values ​​greater than 1 represent HDR brightness. Such techniques can avoid problems with global tone mapping techniques to handle the display capabilities of a target display device. One or more techniques described herein can handle the display capabilities of the display device by applying a restoration map with display coefficients as described above, so that the extended range values ​​do not exceed the display capabilities of the display device.

[0143] As mentioned above, the device can scale the pixel luminance of the LDR image based on the maximum luminance output of the HDR display device and the luminance gain stored in the restoration map. In some implementations, for example in many photo and video applications, a scaling of the LDR image can be used to generate the HDR output image, reproducing the full dynamic range of the original HDR image used to generate the image container (if the target display device can display the full range of the original HDR image). In some implementations, the scaling of the LDR image can set an arbitrary dynamic range for the output HDR image that is higher than the range of the LDR image. In some implementations, the dynamic range can also be smaller than the range of the original HDR image. For example, the output image luminance can be scaled based on the capabilities of the target display device and / or other criteria, or can be scaled lower than the best capabilities of the original HDR image and / or the target display device, for example to reduce the display brightness of the image that may be visually uncomfortable or tiring to the viewer. For example, a fatigue reduction algorithm and one or more light sensors in communication with the device implementing the algorithm may be used to sense one or more of various factors such as ambient light around the display device, time of day, etc., to determine a reduced dynamic range or reduced maximum brightness for the output image. For example, the brightness level of the output image may be reduced by scaling the recovery map values ​​and / or display coefficients used to determine the HDR image as described herein with respect to some implementations, e.g., by scaling the range_scaling_factor and / or display coefficients in the example equations shown above. In another example, the device may determine that the battery power level of the device has fallen below a threshold level and therefore should conserve battery power (e.g., by avoiding high display brightness), in which case the HDR image is displayed at less than the brightness of the original HDR image.

[0144] In some implementations, the luminance of the LDR image may be scaled to be larger (e.g., higher) than the dynamic range of the original HDR image (e.g., if the target display device has a larger dynamic range than the original HDR image). The dynamic range of the output HDR image is not constrained to the dynamic range of the original HDR image, nor is it constrained to any particular high dynamic range. In some examples, the maximum dynamic range of the output HDR image may be smaller than the maximum dynamic range of the original HDR image and / or the maximum dynamic range of the target output device.

[0145] In some implementations, the display of the output HDR image may be scaled progressively from a lower luminance level to the full (maximum) luminance level of the output HDR image (the maximum luminance level, determined as described above). For example, the progressive scaling may be performed over a certain period (user configurable in some implementations), such as 2-4 seconds. The progressive scaling may avoid user discomfort due to a sudden large increase in brightness when displaying a higher luminance HDR image after displaying a lower luminance (e.g., LDR image). As described herein, an LDR image may be scaled progressively to its HDR version, for example by scaling the restoration map values ​​and / or display coefficients used in determining the HDR image to progressively brighten the LDR image to its HDR version, for example, in multiple steps. For example, the range_scaling_factor and / or display_factor may be scaled with the above-mentioned formula. In other implementations, multiple scaling operations may be performed in succession to progressively obtain a higher dynamic range intermediate image over the period. Other HDR image formats that use global tone mapping techniques as described above to scale LDR images into HDR images rather than local tone mapping as described herein for HDR formats may result in lower display quality when performing such progressive scaling or may not support progressive scaling.

[0146] In some examples, this gradual scaling of HDR image brightness may be performed or triggered for certain display conditions. Such conditions may include, for example, displaying an output HDR image immediately after displaying an LDR image on a display device. In some examples, a grid of LDR images may be displayed and a user may select one of these images and display an HDR version of the selected image on the screen (as described above, the HDR image may be determined based on recovery elements from an associated image container). To avoid a sudden large increase in brightness when switching from an LDR image display to an HDR image display, the HDR image may be initially displayed at a lower brightness level of the LDR image and gradually updated (e.g., increasing in increments to its maximum brightness level by progressively displaying image content with a high dynamic range, e.g., fade-in, fade-out animation). In another example, a display condition for triggering the gradual scaling may include an overall screen brightness being set to a low level (e.g., based on low ambient light levels around the device), as described herein. For example, the overall screen brightness may have a brightness level below a certain threshold level for activating the progressive scaling of HDR brightness.

[0147] In some cases or conditions, for example, when one or more HDR images are displayed and replaced by an output HDR image, the output HDR image may be displayed immediately at its full brightness without progressive scaling. For example, when a second HDR image is displayed following the display of a first HDR image, the second HDR image may be displayed immediately at its full brightness because the user is already accustomed to seeing the high brightness of the first HDR image.

[0148] In some implementations, after displaying an output image (e.g., either an LDR output image or an HDR output image), the device that displayed the output image can receive user input from a user (or other source, e.g., another device) instructing the device to modify the display of the output image to increase or decrease the dynamic range of the display, e.g., up to the highest dynamic range of the display device and / or the original HDR image. For example, in response to receiving the user input to modify the display, the device can modify the display factor (described above) according to the user input, which changes the dynamic range of the displayed output image. In some examples, when the user decreases the dynamic range, the display factor can be decreased by a corresponding amount, and the output image is displayed with the resulting smaller dynamic range. This change in the display can be provided in real time in response to the user input.

[0149] In some implementations, the device that displays the output image may receive input, e.g., from a user or from another source, e.g., another device, after displaying the output image, instructing the output image to be modified or edited. In some implementations, after such modification, the restoration map may be discarded, e.g., because it may no longer be properly applied to the LDR image. In some implementations, the restoration map may be maintained after such modification. In some implementations, the restoration map may be updated based on the modification to the output image to reflect the modified version of the LDR image.

[0150] In some examples, once modifications are performed by a user of an HDR display and / or an HDR-aware editing application program, the characteristics of the modifications are known in a larger dynamic range and a restoration map can be determined based on the edited image. For example, in some implementations, at load time of the image into an image editing interface, the restoration map can be decoded into the full resolution image and presented to the editor as data associated with the alpha channel, other channel, or output image. For example, the restoration map can be displayed in the editor as an additional channel in a list of channels for the image. Both the LDR image and the HDR output image can be displayed in the editor to visualize modifications to the image in both versions. When the user edits the image using the editor's tools, corresponding edits are made to the restoration map and the LDR image (base image). In some examples, the user can decide whether to edit the RGB or RGBA (red-green-blue-alpha) of the image. In some implementations, the user can manually edit the restoration map (e.g., alpha channel) and observe changes in the displayed HDR image while the displayed LDR image remains the same.

[0151] In some implementations, a global dynamic range control may be provided in the image editing interface that allows adjustment of increasing or decreasing the range scaling factor. Editing operations such as cropping, resampling, etc. on the image may update the alpha channel as well as the RGB channels. When saving, the LDR image may be saved back to the image container. The saved LDR image may be the one edited in the interface, or it may be a new LDR image generated by local tone mapping from the edited HDR image, or generated based on the edited HDR image by scaling (e.g., dividing) at each pixel by the recovery map. The edited recovery map may be saved back to the image container. In various implementations, the recovery map may be reprocessed to a bidirectional grid if such encoding is used, or the image and the recovery map may be saved losslessly with larger storage requirements, for example, if the image is further edited.

[0152] In some implementations, an editor or other program can convert an LDR image and recovery map in the described image format into a standard HDR image, such as a 10-bit HDR AVIF image or an image in another HDR format.

[0153] In various implementations, the various blocks of methods 200, 300, 400, and / or 500 may be combined, split into multiple blocks, performed in parallel, or performed asynchronously. In some implementations, the order in which one or more blocks of these methods are performed may be the same as or different from that shown in these figures. For example, in various implementations, blocks 210 and 212 of FIG. 2 and / or blocks 302 and 304 of FIG. 3 may be performed in a different order or in parallel. Methods 200, 300, 400, and / or 500, or portions thereof, may be repeated any number of times using additional inputs. For example, in some implementations, method 200 may be performed when one or more new images are received by the device performing the method and / or stored in a user's image library.

[0154] In some implementations, the HDR image and the LDR image may have their respective roles reversed as described above. For example, the original HDR image may be stored in the image container (e.g., in block 218 as the base image) rather than the original LDR image, which has a smaller dynamic range than the HDR image. A restoration map may be determined and stored in the image container that instructs how to determine a derived output LDR image from the base HDR image for an LDR display device that can only display a lower dynamic range, smaller than the dynamic range of the HDR image. The restoration map may include gain and / or scaling factors based on the base HDR image and the original LDR image, and in some implementations, the original LDR image may become a tone mapping image (or otherwise range transform of the HDR image) in a similar manner as described above.

[0155] Some of the described implementations can enable applications that provide HDR images to obtain and output higher quality LDR images based on tone mapping of the HDR image without the constraints of current techniques (e.g., HLG / PQ transfer functions) that provide global or general tone mapping for the entire image. The local tone mapping provided by using the recovery map techniques described herein can enable higher fidelity conversion of HDR images to LDR images without applying global or general tone mapping, for applications such as image sharing or uploading images to services or applications that only support LDR images.

[0156] In some implementations, a brightness control may be provided for displaying an HDR image, such as the HDR images described herein. A user of the device may adjust the HDR brightness control to specify the maximum brightness (e.g., lightness) at which the display device will display the HDR image. In some implementations, the maximum brightness of an HDR image may be determined as the highest average brightness of the displayed pixels of the HDR image, since such an average may indicate the image lightness as perceived by the user. In other implementations, the maximum brightness may be determined as the maximum value of the brightest pixel of the HDR image, as the maximum value of one or more particular pixels of the HDR image (or the maximum value of the average of such particular pixels), and / or based on one or more other characteristics of the HDR image. When the brightness control is set to a value less than the maximum displayable brightness of the HDR image, the device reduces the maximum brightness of the output HDR image to a lower brightness level based on the brightness control. This allows a user to adjust the maximum brightness of a displayed HDR image if they prefer a lower maximum brightness level, e.g., to avoid discomfort or fatigue due to glare from the high brightness HDR image and / or to reduce the contrast between the high brightness HDR image and other content of the display screen outside of the HDR image (e.g., LDR images, user interface elements, etc., which may be displayed at a lower brightness). In some cases, reducing the maximum brightness level of an HDR image can reduce the consumption of battery power by the display device by reducing the brightness of the display.

[0157] For example, the HDR brightness control may be implemented as a global brightness control by the device (e.g., by an operating system running on the device) or a system brightness control that controls HDR image brightness for applications running on the device that display HDR images. In some examples, the brightness control may apply to all applications running on the device, all applications of a particular type running on the device (e.g., image viewing applications and image editing applications), and / or specific applications specified by the user. This allows the user to specify a single brightness adjustment for all (or many) HDR images that the device displays, for example, without requiring brightness adjustments for each HDR image displayed. Such a system brightness control may provide display consistency by all applications running on the device and may also provide the user with more control over the maximum brightness of the HDR images displayed. In some implementations, the HDR brightness control may be implemented with other system controls on the device, such as, for example, accessibility controls, user preferences, user settings, etc. The value set by the system brightness control may be used to adjust the output of the HDR image output in addition to other scaling factors used to display that image, such as, for example, display factors based on the maximum display device brightness as described above, individual brightness settings for the image, etc.

[0158] In some examples, the system brightness control may be implemented as a user-adjustable slider or value indicating, for example, the maximum brightness value or percentage of maximum brightness that the display device can display for the HDR image (e.g., the maximum of the brightest pixel or a particular pixel of the HDR image, if the maximum brightness can be determined as the maximum of the average brightness of the pixels of the HDR image, or a maximum value based on one or more other image characteristics). For example, if the brightness control is set by the user to 70% (e.g., 70% of the maximum brightness of the display device, if used to determine the maximum brightness as described above), each HDR image may be displayed at 70% of their maximum brightness. In some implementations, the system brightness control allows the user to specify different maximum HDR image brightnesses for different screen brightnesses. For example, if the entire screen is displaying content at a high screen brightness, the maximum HDR image brightness may be higher, e.g., 90% or 100%, since there is not much contrast between the overall screen brightness and the HDR image brightness. At low screen brightness, the highest HDR image brightness may be lowered, e.g., 50% or 60%, to reduce the contrast between the overall screen brightness and the HDR image brightness. In some implementations, the system brightness control may be provided as a setting that correlates to the highest SDR image brightness (e.g., by a slider or other interface control). For example, the setting may set the highest HDR brightness value to a value N times the highest SDR image brightness, where N has a maximum settable value based on the maximum brightness of the display device used to display the image. In some examples, a setting value of 1 may indicate the highest brightness of the SDR (base) image, and a setting value that is the maximum value or setting of the brightness control indicates to display at the highest brightness of the display device.

[0159] In some implementations, these variable settings for HDR image brightness may be input via a user interface control. For example, a user may define a graph with a variable maximum HDR brightness versus a current screen brightness. In some implementations, a plurality of predefined HDR brightness configurations may be provided in the user interface, and a user may select one of these configurations to use with the device. For example, each configuration may result in a separate set of maximum HDR brightness values ​​associated with a particular range of screen brightness values. In some implementations, a toggle control may be provided that, once selected by the user, removes the HDR brightness to display the HDR image at the LDR brightness. For example, once the toggle is selected by the user, the associated restoration map or restoration element may be ignored (as described herein) and the base image is displayed.

[0160] 6 is an illustration of an exemplary approximation of an image 600 having a high dynamic range (the full dynamic range of an HDR image cannot be represented by this, e.g., it is an LDR depiction of an HDR scene). For example, HDR image 600 may be captured by a camera capable of capturing HDR images. In image 600, image regions 602, 604, and 606 are shown in detail, and these regions may be depicted with a larger dynamic range because they appear natural to an observer.

[0161] 7 shows an example of a representation of an LDR image 700, which has a smaller dynamic range than HDR image 600, such as the low dynamic range of a standard JPEG image. LDR image 700 depicts the same scene as image 600. For example, image 700 may be an LDR image captured by a camera. In this example, sky image region 702 is properly exposed, but foreground image regions 704 and 706 are dark and underexposed due to a lack of range compression in the image, and therefore these regions are too dark and lose visual detail.

[0162] 8 is another example of an image 800, which, like LDR image 700, has a smaller dynamic range than HDR image 600 and depicts the same scene as images 600 and 700. For example, image 800 may be an LDR image captured by a camera. In this example, sky image region 802 is overexposed and loses detail due to a lack of range compression in the image, while foreground image regions 804 and 806 are properly exposed and contain adequate detail.

[0163] FIG. 9 is an example of a range-compressed LDR image 900. For example, image 900 may be locally tone mapped from HDR image 600. Local tone mapping may be performed using any of a variety of tone mapping techniques. For example, dark areas of image 700 may be increased in brightness while maintaining contrast at the edges of the image. Thus, image 900 has a smaller dynamic range than image 600 (which is imperceptible in these figures), but may maintain image detail beyond image 700 of FIG. 7 and image 800 of FIG. 8. A range-compressed image such as image 900 may be used as an LDR image in the image formats described herein and may be included in the described image containers as an LDR version of a given HDR image.

[0164] FIG. 10 is a block diagram of an example device 1400 that may be used to implement one or more functions described herein. In one example, device 1000 may be used to implement a client device, such as, for example, any of client devices 120-126 shown in FIG. 1. Alternatively, device 1000 may implement a server device, such as, for example, server device 104. In some implementations, device 1000 may be used to implement a client device, a server device, or both a client device and a server device. Device 1000 may be any suitable computer system, server, or other electronic or hardware device, as previously described.

[0165] One or more methods described herein can operate in a number of environments and platforms, for example, as a standalone computer program that may be executed on any type of computing device, a web application having a web page, a program running on a web browser, a mobile application ("app") running on a mobile computing device (e.g., a mobile phone, a smart phone, a tablet computer, a wearable device (watch, armband, jewelry, hat, virtual reality goggles or glasses, augmented reality goggles or glasses, head mounted display, etc.), a laptop computer, etc.). In one example, a client / server architecture may be used, e.g., a mobile computing device (as a client device) sends user input data to a server device and receives final output data for output (e.g., for display) from the server. In another example, all computations may be performed inside the mobile app (and / or other apps) on the mobile computing device. In another example, computations may be split between the mobile computing device and one or more server devices.

[0166] In some implementations, the device 1000 includes a processor 1002, a memory 1004, and an input / output (I / O) interface 1006. The processor 1002 may be one or more processors and / or processing circuits for executing program code and controlling basic operations of the device 1000. A "processor" includes any suitable hardware system, mechanism, or component that processes data, signals, or other information. Systems that may include a processor include a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multi-processor configuration), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a complex programmable logic device (CPLD), dedicated circuits for implementing functionality (e.g., one or more hardware image decoders and / or video decoders), dedicated processors for performing processing based on neural network models, neural circuits, processors optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some implementations, the processor 1002 may include one or more co-processors that perform neural network processing. In some implementations, the processor 1002 may be a processor that processes data to generate a probabilistic output, e.g., the output generated by the processor 1002 may be inaccurate or accurate within a range from an expected output. The processing need not be limited to a particular geographic location, nor may it be time-limited. For example, the functions of the processor may be performed in "real-time", "offline", "batch mode", etc. Parts of the processing may be performed at different times and different locations by separate (or the same) processing systems. A computer may be any processor in communication with a memory.

[0167] The memory 1004 is generally provided within the device 1000 for access by the processor 1002, and may be any suitable processor-readable storage medium suitable for storing instructions executed by the processor, such as random access memory (RAM), read only memory (ROM), electrically erasable read only memory (EEPROM), flash memory, etc., and may be located separately from the processor 1002 or may be incorporated into the processor 1002. Software that may be stored in the memory 1004 includes an operating system 1008, an image application 1010 (which may be, for example, the image application 106 of FIG. 1), other applications 1012, and application data 1014, and is run on the server device 1000 by the processor 1002. The other applications 1012 may include applications such as a data display engine, a web hosting engine, a map application, an image display engine, a notification engine, a social networking engine, a media display application, a communication application, a web hosting engine or application, a media sharing application, etc. In some implementations, the image application 1010 may include instructions that enable the processor 1002 to perform functions described herein, such as some or all of the methods of FIGS. 2-5 and / or 10-12. In some implementations, images stored in the formats described herein (including recovery maps, metadata, etc.) may be stored as application data 1014 or other data in the memory 1004 and / or other storage of one or more other devices in communication with the device 1000. In some examples, the image application 1010 or other applications stored in the memory 1004 may include image encoding / video encoding and container generation modules (e.g., performing the methods of FIGS. 2-4) and / or image decoding / video decoding modules (e.g., performing the methods of FIG. 5), or such modules may be combined into a few or a single module or application.

[0168] Any software in memory 1004 may alternatively be stored in other suitable storage locations or computer-readable media. Additionally, memory 1004 (and / or other connected storage devices) may store one or more messages, one or more taxonomies, electronic encyclopedias, dictionaries, digital maps, thesauri, knowledge bases, message data, grammars, user preferences, and / or other instructions and data used in the functionality described herein. Memory 1004 and other types of storage mechanisms (such as magnetic disks, optical disks, magnetic tapes, or other tangible media) may be considered "storage mechanisms" or "storage devices."

[0169] The I / O interface 1006 may provide functionality that allows the server device 1000 to interface with other systems and devices. The interfaced devices may be included as part of the device 1000 or may be separate and in communication with the device 1000. For example, network communication devices, storage devices (e.g., memory and / or databases), and input / output devices may communicate through the I / O interface 1006. In some implementations, the I / O interface may connect to interface devices such as input devices (keyboards, pointing devices, touch screens, microphones, cameras, scanners, sensors, etc.) and / or output devices (display devices, speaker devices, printers, motors, etc.).

[0170] Some examples of interfaced devices connectable to the I / O interface 1006 may include one or more display devices 1020 that may be used to display content, such as images, videos, and / or user interfaces of applications as described herein. The display devices 1020 may be any suitable display device and may be connected to the device 1000 by a local connection (e.g., a display bus) and / or a network connection. The display devices 1020 may include any suitable display device, such as an LCD, LED, or plasma display screen, a CRT, a television, a monitor, a touch screen, a 3D display screen, or other visual display device. The display devices 1020 may also act as input devices, such as a touch screen input device. For example, the display devices 1020 may be a flat display screen of a mobile device, multiple display screens of a glasses or headset device, or a monitor screen of a computing device.

[0171] The I / O interface 1006 may interface to other input and output devices. Some examples include one or more cameras that can capture images and / or detect gestures. Some implementations may provide microphones for capturing sound (e.g., as part of a captured image, voice commands, etc.), radar or other sensors for detecting gestures, audio speaker devices for outputting sound, or other input and output devices.

[0172] For ease of illustration, FIG. 10 shows one block of each of the processor 1002, memory 1004, I / O interface 1006, and software blocks 1008-1014. These blocks may represent one or more processors or processing circuits, operating systems, memories, I / O interfaces, applications, and / or software modules. In other implementations, the device 1000 may have less than all of the components shown and / or other elements, including other types of elements instead of or in addition to those shown herein. Although some components are described as performing blocks and operations in some implementations described herein, any suitable component or combination of components of the environment 100, the device 1000, a similar system, or any suitable processor associated with such a system may perform the described blocks and operations.

[0173] The methods described herein may be implemented by computer program instructions or codes that may be executed on a computer. For example, the codes may be implemented by one or more digital processors (e.g., microprocessors or other processing circuits) and may be stored in a computer program product that includes a non-transitory computer-readable medium (e.g., storage medium), such as a magnetic, optical, electromagnetic, or semiconductor storage medium, semiconductor or solid-state memory, including magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), flash memory, rigid magnetic disks, optical disks, solid-state memory drives, and the like. The program instructions may also be contained in and provided as electronic signals, for example, in the form of software as a service (SaaS) delivered from a server (e.g., distributed systems and / or cloud computing systems). Alternatively, one or more methods may be implemented in hardware (e.g., logic gates), or a combination of hardware and software. Exemplary hardware may be a programmable processor (e.g., field programmable gate array (FPGA), complex programmable logic device), general-purpose processor, graphics processor, application-specific integrated circuit (ASIC), and the like. One or more of the methods may be implemented as part of or a component of an application running on the system, or as an application or software running in conjunction with other applications and the operating system.

[0174] Although described with reference to specific implementations, these specific implementations are merely illustrative and not limiting, and concepts illustrated in the examples may be applied to other examples and implementations.

[0175] In addition to the above, the user may be provided with controls that allow the system, program, or functionality described herein to select both whether and when to enable collection of user information (e.g., information related to the user's social network, social activities, or business, profession, the user's preferences, or the user's current location) and whether the user is sent content or communications from the server. In addition, certain data may be processed in one or more ways such that personal information is removed before being stored or used. For example, the user's identification information may be processed such that personal information about the user cannot be determined, or if location information (such as to the city, zip code, or state level) is obtained, the user's geographic location may be generalized such that the user's specific location cannot be determined. Thus, the user may control what information is collected about the user, how that information is used, and what information is provided to the user.

[0176] It should be noted that the functional blocks, operations, features, methods, devices, and systems described in this disclosure may be integrated or divided into various combinations of systems, devices, and functional blocks known to those skilled in the art. Any suitable programming language and programming techniques may be used to implement the routines of a particular implementation. Various programming techniques, for example, procedural or object-oriented, may be employed. The routines may be executed on a single processing device or multiple processors. Although steps, operations, or computations may be presented in a particular order, in another particular implementation, the order may be changed. In some implementations, multiple steps or operations shown herein as being performed sequentially may be performed simultaneously.

Claims

1. 1. A computer-implemented method comprising: acquiring a first image having a first dynamic range and depicting a particular scene; acquiring a second image depicting the particular scene having a second dynamic range different from the first dynamic range; generating a recovery map based on the first image and the second image, the recovery map encoding a difference in luminance between a portion of the first image and a corresponding portion of the second image, the difference being scaled by a range scaling factor comprising a ratio of a maximum luminance of the second image to a maximum luminance of the first image; The method comprises: providing the first image and the restoration map to an image container, the image container being readable to display a derived image based on applying the restoration map to the first image, the derived image having a dynamic range different from the first dynamic range. method.

2. The method of claim 1 , wherein the second dynamic range is greater than the first dynamic range, and the dynamic range of the derived image is greater than the first dynamic range.

3. The image container comprises: displaying the first image by a first display device capable of displaying the first dynamic range; Displaying the derived image by a second display device capable of displaying a dynamic range greater than the first dynamic range. The method of claim 2 , wherein the first and second bits are readable as follows:

4. The method of claim 2 , wherein obtaining the first image comprises performing range compression on the second image.

5. 2. The method of claim 1, wherein generating the restoration map comprises encoding a luminance gain such that applying a luminance gain to a luminance of an individual pixel of the first image results in a corresponding pixel in the second image.

6. 6. The method of claim 5, wherein generating the restoration map comprises encoding the luminance gain in a logarithmic space, the restoration map values ​​being proportional to the logarithmic difference of the luminance divided by the logarithm of the range scaling factor.

7. The step of generating the restitution map is based on the following equation: recovery(x,y)=log(pixel_gain(x,y)) / log(range_scaling_factor) recovery(x,y) is the recovery map of a pixel location (x,y) of the second image, and pixel_gain(x,y) is the ratio of the luminance of the second image to the luminance of the first image at location (x,y); The method according to claim 5.

8. 2. The method of claim 1 , wherein generating the recovery map comprises encoding the recovery map into a bidirectional grid, the encoding comprising determining a three-dimensional data structure of grid cells, each grid cell being a vector element that maps to a number of pixels of the first image.

9. 2. The method of claim 1 , further comprising the step of encoding the recovery map into a recovered image, wherein providing the recovery map to the image container comprises providing the recovered image to the image container, and wherein encoding the recovery map into the recovered image comprises encoding the range scaling factor into the recovered image as metadata.

10. The method of claim 9 , wherein the recovered image has an aspect ratio that is the same as an aspect ratio of the first image.

11. obtaining the image container; determining to display the second image by a display device; scaling a plurality of pixel intensities of the first image in the image container based on a particular luminance output of the display device and the restoration map to obtain the derived image; after said scaling step, displaying said derived image as an output image having a different dynamic range than said first image; The method of claim 1 further comprising:

12. 12. The method of claim 11, further comprising determining a maximum brightness display capability of the display device, and wherein scaling the pixel brightnesses comprises increasing the brightness of high-brightness areas in the first image to a brightness level that is equal to or less than the maximum brightness display capability.

13. The method of claim 1 , wherein the second dynamic range is less than the first dynamic range and the dynamic range of the derived image is less than the first dynamic range.

14. The method of claim 13 , wherein obtaining the second image comprises performing range compression on the first image.

15. The image container comprises: displaying the first image by a first display device capable of displaying the first dynamic range; Displaying the derived image on a second display device that is capable of displaying a dynamic range smaller than the first dynamic range. The method of claim 13, wherein the

16. 1. A computer-implemented method comprising the steps of obtaining an image container, The image container comprises: a first image having a first dynamic range; and a restoration map encoding a luminance gain of pixels of the first image scaled by a range scaling factor comprising a ratio of a maximum luminance of a second image to a maximum luminance of the first image, the second image corresponding to the first image of a depicted subject and having a second dynamic range different from the first dynamic range; The method comprises: determining whether to display one of the first image or a derived image having a different dynamic range than the first dynamic range; In response to determining to display the first image, causing a display device to display the first image; In response to determining to display the derived image, applying the luminance gains of the restoration map to luminances of pixels of the first image to determine corresponding pixel values ​​in each of the derived images; causing said display device to display said derived image; A method comprising:

17. The method of claim 16 , wherein the second dynamic range is greater than the first dynamic range, and the dynamic range of the derived image is greater than the first dynamic range.

18. 20. The method of claim 17, further comprising determining a maximum brightness display capability of the display device, and deciding to display the first image in response to determining that the display device is only capable of displaying a display dynamic range equal to or less than the first dynamic range.

19. 20. The method of claim 17, further comprising determining a maximum brightness display capability of the display device, and deciding to display the derived image in response to determining that the display device is capable of displaying a display dynamic range greater than the first dynamic range.

20. 20. The method of claim 17, wherein the display device is capable of displaying a display dynamic range greater than the first dynamic range, and applying the luminance gain of the restoration map comprises adapting the luminance of the pixel values ​​of the derived image to a display dynamic range of the display device.

21. 20. The method of claim 17, wherein the maximum of the dynamic range of the derived image is less than the maximum luminance of the display device and the maximum luminance of the dynamic range of the second image.

22. applying the luminance gains of the restoration map to the luminances of the pixels of the first image, 20. The method of claim 17, comprising scaling the luminance of the first image based on a particular luminance output of the display device and the restoration map.

23. The scaling of the luminance of the first image is performed according to the formula: derived_image(x,y)=first_image(x,y)+log(display_factor)*recovery(x,y) derived_image(x,y) is the log-space version of the derived image; where first_image(x,y) is the log-space version of the first image recovered from the image container, display_factor is the smaller of the range scaling factor and the maximum display luminance of the display device, recovery(x,y) is the recovery map of pixel locations (x,y) of the first image, and the range scaling factor is the ratio of the maximum luminance of the second image to the maximum luminance of the first image.

20. The method of claim 17.

24. The method of claim 16 , further comprising the step of decoding the range scaling factor from the image container in response to determining to display the derived image.

25. The method of claim 16 , further comprising the step of decoding the recovery map from a recovered image included in the image container.

26. The method of claim 16 , further comprising extracting the restitution map from a bidirectional grid stored in the image container.

27. a server device determining whether to display one of the first image or the derived image based on a display dynamic range of a display device included in or coupled to the client device; the step of displaying the first image on the display device includes a step of transmitting the first image from the server device to the client device so that the client device displays the first image; the step of displaying the derived image on the display device includes a step of transmitting the derived image from the server device to the client device such that the client device displays the derived image; 17. The method of claim 16.

28. 17. The method of claim 16, wherein the step of displaying the derived image by the display device includes reducing a maximum brightness level of the displayed derived image based on a system setting of the device on which the derived image is displayed, the system setting being selected by a user.

29. The step of displaying the derived image by the display device includes: displaying the derived image at a first maximum luminance level that is now less than a second maximum luminance level determined for the pixel values ​​of the derived image by applying the luminance gain; and gradually increasing the first maximum brightness level of the derived image to the second maximum brightness level over a certain period of time.

17. The method of claim 16.

30. A processor; a memory storing instructions and coupled to the processor, the processor executing the instructions to: a first image having a first dynamic range; and a restoration map encoding the luminance gain of pixels of the first image; obtaining an image container including determining to display the derived image on a display device; the derived image corresponds to the first image of the depicted subject matter and has a different dynamic range than the first dynamic range; The operation includes: in response to determining to display the derived image on the display device; applying the luminance gain of the restoration map to luminances of pixels of the first image to determine corresponding pixel values ​​of each of the derived images, wherein applying the luminance gain includes scaling the luminance of the first image based on a particular luminance output of the display device and the restoration map; causing the derived image to be displayed by the display device; system.

31. The dynamic range of the derived image is greater than the first dynamic range, and the instructions cause the processor to: performing further operations including determining a maximum brightness display capability of the display device; determining to display the derived image in response to determining that the display device is capable of displaying a display dynamic range greater than a first dynamic range; 31. The system of claim 30.

32. The maximum value of the dynamic range of the derived image is the maximum brightness of the display device; and the maximum luminance of the dynamic range of the original image used to generate the restoration map, whichever is smaller.

32. The system of claim 31.

33. 31. The system of claim 30, wherein the restoration map encodes the luminance gain of the pixels of the first image scaled by a range scaling factor that comprises a ratio of a maximum luminance of a second image to a maximum luminance of the first image, the second image corresponding to the first image of a depicted subject and having a second dynamic range different from the first dynamic range.

Citation Information

Patent Citations

  • Apparatus and method for converting the dynamic range of an image

    JP2014531821A

  • Image encoder and decoder and control method of these, and image processing system

    JP2015019286A

  • Image processing apparatus, image processing method and program

    JP2016092576A

  • Systems and methods for backward compatible high dynamic range / wide color gamut video coding and rendering

    US20150237322A1