High dynamic range image format compatible with low dynamic range

Abstract

Implementations relate to providing a low dynamic range compatible HDR image format. In some implementations, a computer-implemented method includes acquiring a first image depicting a scene with a first dynamic range, acquiring a second image depicting the scene with a second dynamic range different from the first dynamic range, and generating a restoration map based on the first image and the second image. The restoration map encodes a difference in luminance between corresponding portions of the first image and the second image, the difference being scaled by a range scaling factor including a ratio of a maximum luminance of the second image to a maximum luminance of the first image. The first image and the restoration map are provided to an image container, the image container being readable for display of a derived image, the derived image having a dynamic range different from the first dynamic range based on applying the restoration map to the first image.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 421,155, titled HIGH DYNAMIC RANGE IMAGE FORMAT WITH LOW DYNAMIC RANGE COMPATIBILITY, filed on October 31, 2022, and U.S. Provisional Patent Application No. 63 / 439,271, titled HIGH DYNAMIC RANGE IMAGE AND VIDEO FORMATS WITH LOW DYNAMIC RANGE COMPATIBILITY, filed on January 16, 2023, the entire contents of both applications being incorporated herein by reference.

Background Art

[0002] [[ID=十三]] Background Users of devices such as smartphones or other digital camera devices capture and store a large number of photos in an image library. Some cameras can capture high - dynamic range (HDR) images that present a larger dynamic range and a more realistic image quality than the low - dynamic range (LDR) images captured by many other (e.g., older) cameras. If viewing an HDR image, it is best to have a display device that can display the full dynamic range of the HDR image.

[0003] The description of the background of the invention provided herein is for the purpose of presenting the overall context of the present disclosure. The achievements of the inventors within the scope described in this background section of the invention, as well as aspects of the description that are not eligible as prior art at the time of filing, are not recognized as prior art to the present disclosure, either explicitly or implicitly.

Summary of the Invention

[0004] Summary The implementations described herein relate to methods, apparatus, and computer-readable media for providing an HDR image format compatible with a low dynamic range. In some implementations, a computer-based method includes the steps of: acquiring a first image depicting a particular scene having a first dynamic range; acquiring a second image depicting the particular scene having a second dynamic range different from the first dynamic range; and generating a recovery map based on the first and second images. The recovery map encodes the difference in luminance between a portion of the first image and a corresponding portion of the second image, the difference being scaled by a range scaling coefficient that includes the ratio of the highest luminance of the second image to the highest luminance of the first image. The first image and the recovery map are fed into a readable image container for displaying a derived image based on applying the recovery map to the first image, the derived image having a dynamic range different from the first dynamic range.

[0005] Various implementations of the method are described. In some implementations, the second dynamic range is greater than the first dynamic range, and the dynamic range of the derived image is greater than the first dynamic range. In some implementations, the image container is readable to display the first image on a first display device capable of displaying the first dynamic range, and to display the derived image on a second display device capable of displaying a dynamic range greater than the first dynamic range. In some implementations, the step of acquiring the first image includes the step of performing range compression on the second image. In some implementations, the step of generating a recovery map includes the step of encoding the luminance gain such that applying the luminance gain to the luminance of individual pixels in the first image yields the pixels in the second image. In some implementations, the step of generating a recovery map may include the step of encoding the luminance gain in logarithmic space, where the recovery map value is proportional to the difference in the logarithms of the luminances divided by the logarithm of the range scaling coefficient. In some implementations, the step of generating the recovery map is based on the formula recovery(x,y)=log(pixel_gain(x,y)) / log(range_scaling_factor), where recovery(x,y) is the recovery map of the pixel position (x,y) of the second image, and pixel_gain(x,y) is the ratio of the brightness of the second image to the brightness of the first image at position (x,y).

[0006] In some implementations of the method, the step of generating a recovery map includes a step of encoding the recovery map into a bidirectional grid, which in some examples includes a step of determining the three-dimensional data structure of the grid cells, where each grid cell is a vector element mapped to multiple pixels of the first image. In some implementations, the method further includes a step of encoding the recovery map into a recovery image supplied to an image container. In some implementations, range scaling factors are encoded into the recovery image as metadata. In some implementations, the recovery image has the same aspect ratio as the first image.

[0007] In some implementations, the method further includes the steps of acquiring an image container, determining to display a second image using a display device, scaling the luminance of multiple pixels in the first image in the image container based on a specific luminance output and recovery map of the display device in order to acquire a derived image, and, after the scaling step, having the display device display the derived image as an output image having a larger dynamic range than the first image. In some implementations, the method includes the step of determining the maximum luminance display capability of the display device, and the pixel luminance scaling includes increasing the luminance of the bright areas in the first image to a luminance level below the maximum luminance display capability. In some implementations, the luminance of the derived image is increased to the maximum luminance of the second image. In some implementations, the step of scaling the luminance of multiple pixels includes increasing the range of the dark areas in the first image to a level that the display device can display.

[0008] In some implementations of the method, the second dynamic range is smaller than the first dynamic range, and the dynamic range of the derived image is smaller than the first dynamic range. In some implementations, the step of acquiring the second image includes the step of performing range compression on the first image. In some implementations, the image container is readable so that the first image is displayed on a first display device capable of displaying the first dynamic range, and the derived image is displayed on a second display device capable of displaying a dynamic range smaller than the first dynamic range.

[0009] In some implementations, the method performed by the computer includes the step of obtaining an image container containing a first image having a first dynamic range and a recovery map. The recovery map encodes the luminance gain of pixels in the first image, scaled by a range scaling factor that includes the ratio of the highest luminance of the second image to the highest luminance of the first image. The second image has a second dynamic range different from the first dynamic range and corresponds to the first image of the subject to be depicted. The method determines whether to display the first image or one of the derived images having a larger dynamic range than the first dynamic range. In response to determining to display the first image, the first image is displayed on the display device. In response to determining to display the derived image, the luminance gain of the recovery map is applied to the luminance of pixels in the first image to determine the corresponding pixel values ​​of the derived image, and the derived image is displayed on the display device.

[0010] Various implementations of the method are described. In some implementations, the second dynamic range is greater than the first dynamic range, and the dynamic range of the derived image is greater than the first dynamic range. In some implementations, the method includes a step of determining the maximum brightness display capability of the display device. In some implementations, in response to determining that the display device can only display a display dynamic range less than or equal to the first dynamic range, it decides to display the first image. In some implementations, in response to determining that the display device can display a display dynamic range greater than the first dynamic range, it decides to display the derived image. In some implementations, the display device can display a display dynamic range greater than the first dynamic range, and the step of applying the gain of the recovery map includes a step of adapting the brightness of the pixel values ​​of the derived image to the display dynamic range of the display device.

[0011] In some implementations, the display device can display a display dynamic range larger than the first dynamic range, and the step of applying the gain of the recovery map includes the step of adapting the brightness of the pixel values ​​of the derived image to the display dynamic range of the display device. In some implementations, the maximum value of the dynamic range of the derived image is lower than the maximum brightness of the display device or the maximum brightness of the dynamic range of the second image.

[0012] In some implementations, the step of applying the gain of the recovery map includes a step of scaling the brightness of the first image based on a specific brightness output of the display device and the recovery map. In some implementations, the step of scaling the brightness of the first image is performed based on the following equation: derived_image(x,y)=first_image(x,y)+log(display_factor)*recovery(x,y), where derived_image(x,y) is the logarithmic version of the derived image, first_image(x,y) is the logarithmic version of the first image recovered from the image container, display_factor is the smaller of the range scaling factor and the highest (or desired) display brightness of the display device, recovery(x,y) is the recovery map of the pixel positions (x,y) of the first image, and the range scaling factor is the ratio of the highest brightness of the second image to the highest brightness of the first image.

[0013] In some implementations, the method further includes the step of decoding a range scaling factor from an image container in response to a decision to display a derived image. In some implementations, the method further includes the step of decoding a recovery map from a recovery image contained in the image container. In some implementations, the method further includes the step of extracting a recovery map from a bidirectional grid stored in the image container.

[0014] In some implementations, the server device performs a step of determining whether to display one of the first image or a derived image based on the display dynamic range of a display device included in or coupled to the client device; the step of the display device displaying the first image includes sending the first image from the server device to the client device so that the client device displays the first image; and the step of the display device displaying a derived image includes sending the derived image from the server device to the client device so that the client device displays the derived image.

[0015] In some implementations, the step of a display device displaying a derived image includes a step of lowering the maximum brightness level of the derived image to be displayed, based on the system settings of the device displaying the derived image, the system settings being selected by the user. In some implementations, the step of a display device displaying a derived image includes a step of displaying the derived image at a first maximum brightness level that is less than a second maximum brightness level determined for the pixel values ​​of the derived image by applying a brightness gain, and a step of gradually increasing the first maximum brightness level of the derived image up to the second maximum brightness level over a specific period of time.

[0016] In some implementations, the system includes a processor and memory coupled to the processor that stores instructions, and the processor performs an operation that includes the step of obtaining an image container, which includes a first image having a first dynamic range and a recovery map encoding the luminance gains of the pixels of the first image, by executing an instruction. The method decides to display on the display a derived image on a display device that corresponds to a first image of the subject to be depicted and has a different dynamic range from the first image. In response to the decision to display the derived image on the display device, the gains of the recovery map are applied to the luminance of the pixels of the first image to determine the corresponding pixel values ​​of the derived image, and the luminance of the first image is scaled based on a specific luminance output of the display device and the recovery map. The derived image is displayed on the display device.

[0017] Various implementations of the system are described. In some implementations, the dynamic range of the derived image is greater than that of the first dynamic range, and the processor performs an operation that, by instruction, further includes the step of determining the maximum brightness display capability of the display device, and in response to determining that the display device can display a display dynamic range greater than that of the first dynamic range, decides to display the derived image. In some implementations, the maximum value of the dynamic range of the derived image is the smaller of the maximum brightness of the display device and the maximum brightness of the dynamic range of the original image used to generate the recovery map. In some implementations, the recovery map encodes the brightness gain of pixels in the first image, scaled by a range scaling coefficient that includes the ratio of the maximum brightness of the second image to the maximum brightness of the first image, and the second image has a second dynamic range greater than that of the first dynamic range and corresponds to the first image of the subject being depicted. In some implementations, the processor further performs an operation that includes the step of extracting the recovery map from a bidirectional grid stored in an image container. In some implementations, the system includes one or more of the operations and / or functions of the methods described above.

[0018] Some implementations may include a computing device that includes a processor and memory coupled to the processor. The memory stores instructions, and the processor performs operations, by executing instructions, which include one or more of the operations and / or functions described above.

[0019] Some implementations include a non-temporary computer-readable medium that stores instructions, and the processor performs operations that may be similar to the operations and / or functions of the systems and / or computing devices described above by executing the instructions. [Brief explanation of the drawing]

[0020] [Figure 1]A block diagram of an exemplary network environment that can be used for one or more implementations described herein. [Figure 2] A flowchart showing an exemplary method for encoding an image into a backward-compatible high dynamic range image format according to some implementations. [Figure 3] A flowchart showing an exemplary method for generating a recovery map based on HDR and LDR images according to some implementations. [Figure 4] A flowchart showing an exemplary method for encoding a recovery map into a bidirectional grid according to some implementations. [Figure 5] A flowchart showing an exemplary method for decoding an image in a backward-compatible high dynamic range image format and displaying an HDR image according to some implementations. [Figure 6] A diagram showing an exemplary image representing a high dynamic range according to some implementations. [Figure 7] A diagram showing an exemplary image representing a low dynamic range according to some implementations. [Figure 8] A diagram showing an exemplary image representing a low dynamic range according to some implementations. [Figure 9] A diagram showing an exemplary image representing a low dynamic range according to some implementations. [Figure 10] A block diagram of an exemplary computing device that can be used to implement one or more functions described herein.

Modes for Carrying Out the Invention

[0021] Detailed Description This disclosure relates to a backward-compatible HDR image format. Both low dynamic range (LDR) and high dynamic range (HDR) images can be obtained and displayed from the container provided by this image format. The image format can be used to display the LDR version of an image by an LDR display device, and can be used to display the HDR version of an image by an HDR display device.

[0022] In some implementations, an image container is generated that implements the image format. Images with a smaller dynamic range (e.g., LDR images) and corresponding images with a larger (e.g., higher) dynamic range (e.g., HDR images) are obtained. In some examples, the HDR image is captured by a camera, and the LDR image is generated from the HDR image using, for example, tone mapping or other processing. The LDR image can be in a standard image format such as JPEG or other formats. The recovery map generated based on the HDR and LDR images encodes the difference in luminance (e.g., gain) between a portion of the LDR image (e.g., a pixel or pixel value) and the corresponding portion of the HDR image (e.g., a pixel or pixel value), where the range scaling coefficient that scales the difference includes the ratio of the highest luminance of the HDR image to the highest luminance of the LDR image. In some exemplary implementations, the recovery map can be a scalar function that encodes the luminance gain in logarithmic space, and the recovery map value is proportional to the logarithmic difference in luminance divided by the logarithm of the range scaling coefficient. In some implementations, the recovery map can be encoded into a data structure or image representation that can reduce the memory space required to store the recovery map. For example, the data structure could be a bidirectional grid using a three-dimensional data structure of grid cells. The recovered and LDR images are supplied to an image container that can be stored, for example, as a novel HDR image format.

[0023] The image container can be read and processed by a device that displays an output image, which is either an LDR image or an output HDR image (e.g., a derived image). The device that reads and displays an image stored in the above image format may use an LDR display device (e.g., a display screen) capable of displaying LDR images, for example, the display dynamic range of the LDR display device may be able to display up to the low dynamic range of an LDR image but not the larger dynamic range of an HDR image. Such an LDR display device accesses only the LDR image in the image container and ignores the recovery image. Since the LDR image is a standard format, the device can easily display the LDR image. Therefore, in some implementations, the LDR image can still be displayed by a device that does not perform encoding to detect or apply the recovery image. If the accessing device can use a display device capable of displaying an HDR image (e.g., a display device that can display a luminance region larger than the highest luminance of the LDR image in its output), it accesses the LDR image and recovery map in the image container and applies the luminance gain encoded in the recovery map to the pixels of the LDR image to determine the corresponding pixel values ​​of the output HDR image to be displayed. In some examples, the device scales the pixel brightness of an LDR image based on the luminance output of the HDR display device (e.g., the maximum luminance output of the display device) as well as the luminance gain stored in the recovery map. The output HDR image is displayed on an HDR display device with a high dynamic range. In other implementations, the HDR image and recovery map may be stored in an image container, and the recovery map may be applied to the HDR image to obtain a low dynamic range LDR image with high visual quality, which is locally tone-mapped by the recovery map.

[0024] The above features have resulted in several technical advantages, including enabling efficient storage and high-quality display of high-dynamic-range (HDR) or low-dynamic-range (e.g., standard) images by a wide range of devices and by using a single image format. For example, an image format is provided that allows any device to display an image with a dynamic range appropriate to its display capabilities. Furthermore, images displayed from these formats do not suffer any loss of visual quality (e.g., particularly in their detailed local contrast) from converting from one dynamic range to another. In various implementations, the output HDR image generated from the image container can be a lossless, exact version of the original HDR image used to generate the image container, or this output HDR image can be a version that is (visually) very similar to the original HDR image because the recovery map stores the luminance information.

[0025] The above functionality may include obtaining an output HDR image using a recovery map derived from an LDR image. The recovery map may include values ​​based on a luminance gain ratio scaled by a range scaling coefficient, which is the ratio of the highest luminance of the original HDR image to the highest luminance of the LDR image. For example, the recovery map specifies the relative amount to scale each pixel, and the range scaling coefficient specifies the scaling of a particular amount to be performed, which gives context to the relative values ​​in the recovery map (for example, extending the range of the amount so that pixels can be scaled with respect to a particular image). The range scaling coefficient allows for an efficient, accurate, and concise specification of the recovery map values ​​within a normalized range, which is advantageous in that it can then be efficiently adjusted for a particular output dynamic range using the range scaling coefficient. Using a range scaling coefficient allows for efficient use of all bits for storing the recovery map, and therefore enables accurate encoding of a wider variety of images.

[0026] Furthermore, in some implementations or cases, when displaying an output HDR image, the pixel brightness of the LDR image is also scaled based on a display coefficient. The display coefficient can be obtained based on the brightness output of the HDR display device that displays the output image. This makes it possible to generate an HDR image with any convenient dynamic range higher than LDR from the container, based on the display capabilities of the display device. The dynamic range of the output HDR image is not constrained by any standard high dynamic range or the dynamic range of the original HDR image. Since HDR display devices can vary considerably in the brightness at which they can display an image, the benefit of this feature is that the image can be displayed with high quality on any dynamic range display device. Moreover, the above feature allows for real-time changes in the dynamic range of the output HDR image (e.g., higher than LDR) based on user input or other conditions, for example, to reduce user visual stress or fatigue, or for other purposes. This display scaling allows the output HDR image to have any dynamic range greater than the range of the LDR image.

[0027] In contrast, conventional techniques may encode HDR images for displays with a specific and unchanging dynamic range or maximum brightness. Such techniques cannot utilize HDR display devices with a greater dynamic range or brightness than the encoded dynamic range. In addition, if the HDR display device cannot display a dynamic range as high as that encoded in the original HDR image, such conventional techniques may result in lower quality images. For example, techniques such as roll-off curves may be needed to reduce the dynamic range of the HDR image for display on this HDR display device, which can, for example, reduce local contrast in the image in an undesirable way and often degrade the visual quality of the image.

[0028] The above functionality provides a shareable and backward-compatible mechanism for generating and rendering images containing high dynamic range (HDR) content beyond current formats. This image format enables next-generation consumer HDR content and, in some implementations, can support specialized image workflows without requiring dedicated hardware codecs for decoding / encoding. One or more of the above implementations of the HDR format address the problems associated with current HDR transfer functions that assume global tone mapping. In contrast, the above functionality enables local tone mapping, which is better at preserving detail in scenes with different levels of brightness. Therefore, the above HDR format can produce higher fidelity and higher quality HDR-rendered content compared to conventional HDR formats.

[0029] In addition, the above features make it possible to reduce the amount of memory required for the image format. For example, with this image format, a single LDR image can be stored with a recovery map that requires less memory, thus saving considerable memory space compared to a format that stores multiple full images. For example, some implementations involve compressing the recovery map, and in some of these examples, the recovery map can be encoded into a bidirectional grid. Such a grid allows the recovery map to be stored with significantly less memory space required, and from the recovery map, it is possible to provide an output HDR image with little loss of visual quality compared to the original HDR image.

[0030] A technical effect of one or more of the above implementations is that, compared to conventional systems, the device can display an image having a dynamic range that better corresponds to the dynamic range of a particular output display device used to display the image. For example, an image that such a conventional system may provide does not have a dynamic range that corresponds to the dynamic range of the display device, resulting in inferior image display quality. The functions described herein can mitigate such disadvantages by, for example, providing a recovery map in the image format and / or scaling the image output to set the dynamic range of the image to better suit a particular display device. Furthermore, a technical effect of one or more of the above implementations may be that the device consumes fewer computing resources to obtain the result. For example, a technical effect of the above techniques may be that the consumption of system processing resources and / or memory resources is reduced compared to a conventional system that does not provide one or more of the above techniques or functions. For example, a conventional system may need to store and / or supply full LDR and full HDR images in order to display an image with a dynamic range suitable for a particular display device, which requires additional memory and communication bandwidth resources compared to the above techniques. In another example, conventional systems only store HDR images and then rely on tone mapping those HDR images to a smaller dynamic range for LDR displays, but such tone mapping can degrade visual quality when displaying a wide variety of HDR images on an LDR display.

[0031] As referred to herein, dynamic range is related to the ratio of the brightest to the darkest parts of a scene. The dynamic range of an LDR format typically does not exceed a certain low range, such as a standard dynamic range (SDR) file of a low-range color space / color profile. Similarly, the displayed dynamic range output of an LDR display device is low. As referred to herein, an image with a larger (or higher) dynamic range than an LDR image (e.g., JPEG) is considered an HDR image, and a display device capable of displaying that larger dynamic range is considered an HDR display device. HDR images can store pixel values ​​that extend over a larger tonal range than LDR. In some examples, an HDR image may have a lower dynamic range that can display the dynamic range of a real scene, more accurately and / or is larger than the dynamic range of an LDR image (e.g., an HDR image may generally have more than 8 bits per color channel).

[0032] In this explanation, the use of "log" refers to a logarithm of a specific base; for example, all logarithms herein can have any base, such as a logarithm with base 2 or a logarithm with base 10.

[0033] In addition to the descriptions herein, users are provided with controls that allow them to choose whether, and if so, to allow the systems, programs, or functions described herein to collect user information (for example, images from the user's library, social networks, social activities, or business, occupation, user preferences, user's current location, user messages, or characteristics of the user's device), and whether, and if so, to receive content or communications from the server. In addition, certain data may be processed in one or more ways such that personal information is removed before it is stored or used. For example, user identification information may be processed so that personal information about the user cannot be determined, or if location information (such as city, zip code, or state level) is obtained, the user's geographical location may be generalized so that the user's specific location cannot be determined. Thus, users can control what information is collected about them, how that information is used, and what information is provided to them.

[0034] Figure 1 shows a block diagram of an exemplary network environment 100 that may be used in some implementations described herein. In some implementations, the network environment 100 includes one or more server systems, such as server system 102 in the example of Figure 1, and a number of client devices, such as client devices 120-126, each associated with a respective user, such as users U1-U4. Each of the server system 102 and client devices 120-126 may be configured to communicate with the network 130.

[0035] The server system 102 may include a server device 104 and an image database 110. In some implementations, the server device 104 may provide an image application 106a. In Figure 1 and the remaining figures, the letters following a reference number (e.g., "106a") represent a reference to the element having that particular reference number. A reference number without following letters in the text, such as "106", represents a general reference to embodiments of the element having that reference number.

[0036] The image database 110 may be stored in a storage device that is part of the server system 102. In some implementations, the image database 110 may be implemented using a relational database, a key-value structure, or other types of database structures. In some implementations, the image database 110 may contain multiple divisions, each corresponding to an image library for each of the users 1-4. For example, as seen in Figure 1, the image database 110 may contain a first image library for user 1 (image library 1, 108a) and other image libraries for various other users (image library 2, ..., image library n). Although Figure 1 shows a single image database 110, it will be understood that the image database 110 may be implemented as a distributed database across multiple database servers, for example. Furthermore, although Figure 1 shows multiple divisions, one for each user, in some implementations, each image library may be implemented as a separate database.

[0037] The image library 108a may store multiple images (including videos) associated with user 1, metadata associated with the multiple images, and one or more other database fields stored in relation to the multiple images. Access rights to the image library 108a may be restricted so that user 1 can control how, for example, the image application 106, other applications, and / or one or more other users access the images and other data in the image library 108a. The server system 102 may be configured to enforce access rights so that image data belonging to a particular user cannot be accessed unless permitted by that user.

[0038] Images referenced herein may include digital images having pixels with one or more pixel values ​​(e.g., color values, lightness values, etc.). Images may be still images (e.g., still photographs, images having a single frame, etc.), moving images (e.g., animations, animated GIFs, cinemagraphs where some parts of the image are moving and others are still images, etc.), or video (e.g., a series of images or frames of images which may optionally include sound). Images used herein may be understood to be any of the above. For example, the implementations described herein may be used with still images (e.g., photographs, or other images), video, or moving images.

[0039] The network environment 100 may include one or more client devices, such as client devices 120, 122, 124, and 126, which may communicate with each other and / or with the server system 102 through the network 130. The network 130 can be any type of communication network, including one or more of the Internet, a local area network (LAN), a wireless network, a switch, or a hub connection. In some implementations, the network 130 may include peer-to-peer communication between devices using, for example, a peer-to-peer wireless protocol (e.g., Bluetooth®, Wi-Fi Direct). An example of peer-to-peer communication between two client devices 120 and 122 is shown by arrow 132.

[0040] In various implementations, users 1, 2, 3, and 4 may communicate with and / or with the server system 102 using their respective client devices 120, 122, 124, and 126. In some examples, users 1, 2, 3, and 4 may interact with each other through applications running on their respective client devices and / or with the server system 102 through network services such as social networking services or other types of network services implemented on the server system 102. For example, each client device 120, 122, 124, and 126 may communicate data with one or more server systems, such as the server system 102.

[0041] In some implementations, the server system 102 may supply appropriate data to client devices so that each client device can receive communication or shared content uploaded to the server system 102 and / or network services. In some examples, users 1-4 can interact by image sharing, audio or video conferencing, audio, video, or text chat, or other communication modes or applications.

[0042] The network services provided by the server system 102 may include a system that enables users to perform various communications, form links and relationships, upload and post shared content such as images, text, audio, and other types of content, and / or perform other functions. For example, a client device may display incoming data, such as content posted, transmitted, or streamed to the client device from separate client devices, the server system, and / or network services, via the server and / or network services (or directly from separate client devices). In some implementations, client devices may communicate directly with each other using peer-to-peer communication between client devices, as described above, for example. In some implementations, “user” may include one or more programs or virtual entities, as well as any person interfaced with the system or network.

[0043] In some implementations, any of the client devices 120, 122, 124, and / or 126 may provide one or more applications. For example, as shown in Figure 1, client device 120 may provide image application 106b. Client devices 122-126 may also provide similar applications. Image application 106a may be implemented using the hardware and / or software of client device 120. In various implementations, image application 106a may be a standalone client application run on any of the client devices 120-124, for example, or it may work in conjunction with image application 106b provided on server system 102.

[0044] The image application 106 may provide various image-related functions (including video) that are performed with user permission. For example, such functions may include one or more of the following: capturing images using a camera; modifying images; determining image quality (based on factors such as surface size, dirt, number of surfaces, image composition, lighting, exposure, etc.); storing images in an image library 108; encoding and decoding images and video into any of the various image and video formats (including the formats described herein); and providing a user interface for viewing displayed images or generating or collecting image-based data.

[0045] The client device 120 may include user 1's image library 108b, which may be a standalone image library. In some implementations, image library 108b may be available in combination with image library 108a of the server system 102. For example, with user permission, image library 108a and image library 108b may be synchronized via the network 130. In some implementations, image library 108 may include multiple images related to user 1, such as images captured by the user (e.g., using the camera of the client device 120 or other devices), images shared with user 1 (e.g., from the respective image libraries of other users 2-4), images downloaded by user 1 (e.g., from websites, messaging applications, etc.), screenshots, and other images. In some implementations, image library 108b on the client device 120 may include a subset of images in image library 108a on the server system 102. For example, such an implementation may be advantageous when the amount of available storage space on the client device 120 is limited.

[0046] In various implementations, the client device 120 and / or the server system 102 may include other applications (not shown) that may provide various types of functionality. User interfaces on client devices 120, 122, 124, and / or 126 may enable the display of user content and other content, including images, image-based generation, data, and other content, as well as communications, privacy settings, notifications, and other data. Such user interfaces may be displayed using a combination of software on the client devices, software on the server devices, and / or client software, and server software running on the server device 104, such as application software or client software that communicates with the server system 102. The user interface may be displayed by a display device on the client device or server device, such as a touchscreen, other display screen, or projector. In some implementations, an application program running on the server system may communicate with the client devices to receive user input on the client devices or output data such as visible data and audio data to the client devices.

[0047] For ease of explanation, Figure 1 shows one block of server system 102, server device 104, image database 110, and four blocks of client devices 120, 122, 124, and 126. Server blocks 102, 104, and 110 may represent multiple systems, server devices, and network databases, and the blocks may be configured differently from those shown. For example, server system 102 may represent multiple server systems that can communicate with other server systems through network 130. In some implementations, server system 102 may include, for example, a cloud hosting server. In some examples, the image database 110 may be stored in a storage device provided in a separate server system block from server device 104, which can communicate with server device 104 and other server systems through network 130.

[0048] Furthermore, there may be any number of client devices. Each client device could be any type of electronic device, such as a desktop computer, laptop computer, portable device or mobile device, mobile phone, smartphone, tablet computer, television, set-top box or entertainment device for television, wearable device (e.g., glasses or goggles with a display, wristwatch, headset, armband, jewelry, etc.), personal digital information processing terminal (PDA), media player, game console, etc. In some implementations, the network environment 100 may not have all of the components shown and / or may have other elements, including other types of elements, instead of or in addition to those described herein.

[0049] Other implementations of the functions described herein may use any type of system and / or service. For example, other networked (e.g., internet-connected) services may be used instead of, or in addition to, social networking services. Any type of electronic device may utilize the functions described herein. Some implementations may provide one or more of the functions described herein on one or more client or server devices that are isolated from or intermittently connected to a computer network. In some examples, a client device including or connected to a display device may display content posts that have been previously received, for example, over a communication network and stored in local storage for the client device.

[0050] Figure 2 is a flowchart illustrating an exemplary method 200 for encoding an image to a backward-compatible high dynamic range image format, such as an HDR image format with low dynamic range compatibility, in several implementations. In some implementations, method 200 may be executed on, for example, a server system 102 shown in Figure 1. In some implementations, some or all of method 200 may be implemented on one or more client devices, such as client devices 120, 122, 124, or 126 in Figure 1, one or more server devices, such as server device 104 in Figure 1, and / or both server and client devices. In the above examples, the implementing system includes one or more digital processors or processing circuits ("processors") and one or more storage devices (e.g., databases or other storage mechanisms). In some implementations, one or more separate components of a server and / or client may execute separate blocks or parts of method 200. In some examples, the devices are described as executing blocks of method 200. Some implementations may have one or more blocks of method 200 executed by one or more other devices (e.g., other client devices or server devices) that can send results or data to the first device.

[0051] In some implementations, Method 200 or part thereof may be initiated automatically by the system. For example, Method (or part thereof) may be executed periodically, or may be executed based on one or more specific events or conditions, such as the start of the image application 106 by the client device, the acquisition of a new image by the image acquisition device of the client device, the receipt of an image by the device over the network, the uploading of a new image to the server system 102, the expiration of a predetermined period since the last operation of Method 200, and / or the occurrence of one or more other states that may be specified in the settings read by Method 200.

[0052] User permission may be obtained in order to use user data in Method 200 (blocks 210-220). For example, user data for which permission may be obtained may include images stored in the client device (e.g., any of client devices 120-126) and / or the server device, image metadata, user data related to the use of image applications, and other image-based generation. The user is given the option to selectively grant or deny access to all or any subset of the user data. If user permission is insufficient with respect to particular user data, Method 200 may be executed using other data (e.g., images unrelated to the user) without using that user data.

[0053] Method 200 may begin with block 210, in which a high dynamic range (HDR) image depicting a specific scene ("original HDR image") is acquired by the device. In some implementations, the HDR image is supplied in any of several standard formats for high dynamic range images. In some examples, the HDR image may be acquired by an image acquisition device, such as a camera in a client device or another device. In further examples, the HDR image may be received from another device via a network or acquired from a storage mechanism accessible by the device. In some implementations, the HDR image may be generated based on any of several image generation techniques, such as ray tracing.

[0054] In block 212, the device acquires a low dynamic range (LDR) image (e.g., the "original LDR image"), which depicts the same scene and / or subject as the HDR image, for example, corresponding to the HDR image of the subject being depicted. The LDR image has a smaller dynamic range than the HDR image. In some implementations, the LDR image may have the dynamic range of a standard LDR image format, or a somewhat smaller dynamic range than the HDR image. For example, the LDR image may be supplied in a standard image format such as JPEG, AV1 Image File Format (AVIF), or TIFF. In some examples, the LDR image has a dynamic range given by 8 bits per channel per pixel, while the HDR image has a higher dynamic range than the LDR image (e.g., generally 10 bits or more in bit depth, a wide color space, and an HDR-oriented transfer function).

[0055] In various implementations, the HDR image in block 210 may be acquired before or after the LDR image acquisition. For example, in some implementations, the HDR image is acquired and the LDR image is derived from the HDR image. In some examples, the LDR image may be generated based on the HDR image using local tone mapping or other processing. Local tone mapping, for example, modifies each pixel according to its local characteristics in the image (rather than following a global tone mapping function that modifies each pixel in the same way). Any of the various local tone mapping techniques may be used to reduce the tonal values ​​of the HDR image to a level suitable for the LDR image, which has a smaller dynamic range. For example, local Laplacian filtering and / or other techniques may be used in the tone mapping process. In some implementations, different global tone curves (or digital gains) may be used in different regions of the image. In some implementations, different tone mapping techniques may be used for different types of image data, for example, different techniques may be used for still images and video images. In other examples, exposure fusion techniques may be used to generate an LDR image from an HDR image and / or multiple captured or generated images, for example, by combining parts of multiple images captured or generated at different exposure levels to produce a combined LDR image containing these parts. In some exemplary implementations, a combination of tone mapping and exposure fusion techniques may be used to generate an LDR image (for example, using a Laplacian tone mapping pyramid to obtain a weighted blend of pyramids from separate images with different exposure levels and then collapse the blended pyramids).

[0056] In other examples, LDR images may be acquired from other sources. For example, HDR images may be captured by a camera device, and LDR images may be captured by the same camera device, for example, immediately before or immediately after, or at least partially simultaneously with, the capture of HDR images. In some examples, the camera device may capture HDR and LDR images based on camera settings, for example, using a dynamic range indicated by camera settings or user-selected settings. Block 212 may be followed by Block 214.

[0057] In block 214, a recovery map is generated based on the HDR and LDR images. The recovery map encodes the difference in luminance between a portion of the HDR image (e.g., a pixel) and the corresponding portion of the LDR image (e.g., a pixel). The recovery map is used to convert the luminance of the LDR image to the luminance of the HDR image. For example, luminance gain and range scaling coefficients can be encoded in the recovery map so that applying a luminance gain to the luminance value of an individual pixel in the LDR image yields the corresponding pixel in the HDR image. An example of generating a recovery map is described in more detail below with reference to Figure 3. Block 214 may be followed by block 216.

[0058] In block 216, the recovery map can be encoded into a recovery element. In various implementations, the recovery element may be an image, data structure, algorithm, neural network, etc., that contains or supplies recovery map information. For example, the recovery element may be a recovery image compressed into a standard format, having the same aspect ratio as the LDR image. The recovery image may have any resolution, whether the same as or different from the LDR image. For example, the recovery image may have a lower resolution than the LDR image, such as one-quarter of the LDR image's resolution (for example, the recovery image may be 480×270 compared to the LDR image's 1920×1080).

[0059] In some examples, the recovered image may be encoded as a single-channel, 8-bit unsigned integer value, where each value represents a recovered value and is stored in one pixel of the recovered image. In some exemplary implementations, the encoding can result in a continuous representation from -2.0 to 2.0, which can be compressed, for example, by JPEG compression. In some implementations, the encoding may include one bit indicating the positive / negative sign of the map value, with the remaining bits interpreted as the magnitude of the recovered map value (for example, in the range of -1 to +1). In some implementations, such as this example, the magnitude representation may exceed 1.0 or -1.0, as some pixels may require greater attenuation or gain than what is represented by the range scaling factor in order to accurately recover or represent the luminance regions of the original HDR image.

[0060] In single-channel coding, a channel can specify a brightness-based adjustment to the target image. Therefore, the chromaticity or hue of the target image should not change while the recovery map is applied to that image. Single-channel coding can preserve the existing chromaticity or hue of the target image (e.g., the RGB ratio) and typically requires less memory space than multi-channel coding. In some implementations, the recovery image can be coded as a multi-channel value. Multi-channel coding can allow adjustment of the target image's color (e.g., chromaticity or hue) while the recovery map is applied to the target image. Multi-channel coding can compensate for color loss that occurs in LDR images, such as color roll-off at high brightness. This allows for the recovery of not only accurate color but also HDR brightness (e.g., resaturating the sky to a blue hue instead of just increasing brightness to make the sky appear white). In some implementations, the recovery map in multi-channel coding can be used to recover a wider color gamut (or compensate for color gamut loss) in the target image. For example, the ITU-R Recommendation BT.2020 color gamut can be recovered from a standard RGB (sRGB) / BT.709 color gamut LDR image.

[0061] In some implementations, the values ​​of the recovery map may use a different bit depth, such as a bit depth greater than 8 bits. For example, in some implementations, an 8-bit depth may be the minimum bit depth that enables HDR representation when combined with an 8-bit single-channel gain map, and a smaller bit depth may not provide enough information to deliver HDR image content without artifacts such as banding. In some implementations, the recovery image may be encoded as a floating-point value rather than an integer. In some implementations, the recovery element may be a data structure, algorithm, or neural network that encodes the values ​​of the recovery map. For example, the recovery map may be encoded into the weights of a small neural network used as the recovery element. Block 216 may be followed by Block 218.

[0062] In block 218, the LDR image and recovery elements are supplied to an image container that can be stored as a new HDR image format. In some implementations, the LDR image may be considered the base image in the image container. The image container can be read by the device to display the LDR image or the HDR image. In some implementations, the image container may be a standard image container, such as an AVIF (AV1 image file format) image container.

[0063] In some implementations, the recovery map may be quantized to the precision of the image container in which the recovery map is placed. For example, in some implementations where the LDR image is a JPEG image (or other format using 8-bit unsigned integer values), the values ​​of the recovery map may be quantized to an 8-bit format for storage. In some examples, each value may represent a recovery value and is stored in one pixel of the recovery image. In some examples, each encoded value is: encode(x,y)=recovery(x,y)*63.75+127.5 This can be defined as follows: In this encoding, for example, each value is quantized to one of 256 possible encoded values, resulting in a representation in the range of -2.0 to 2.0. Other implementations may use other ranges and quantizations, such as -1.0 to 1.0 or other ranges.

[0064] An image container may include additional information related to the display of an output HDR image based on the contents of the image container. For example, the additional information may include metadata supplied in the image container, recovery elements, and / or LDR image. The metadata can encode information about how the LDR image and / or the HDR image derived therefrom are presented to the display device. Such metadata may include, for example, the version of the recovery map format used in the container, the range scaling factor used in the recovery map, the storage method (e.g., whether the recovery map is encoded in a bidirectional grid or other data structure, or compressed by a specified compression technique), the resolution of the recovery map and / or recovery image, guide weights for bidirectional grid storage, and / or other recovery map or image properties.

[0065] In some examples, an LDR image may contain a metadata data structure or directory that defines the order and characteristics of items (e.g., files) within the image container, and each file within the container may have a corresponding media item within the data structure. The media item can describe the location of the associated file within the image container and the basic characteristics of the associated file. For example, a container element may be encoded into metadata for the LDR image, where the format version and data structure of the media items within the container are defined. In some examples, the metadata is stored according to a data model that enables devices or applications that do not support or cannot read metadata to read the LDR image. For example, metadata may be stored in the recovered image and / or LDR image according to the Extensible Metadata Platform (XMP) data mode. Block 218 may be followed by Block 220.

[0066] In block 220, the image container may be stored in a storage mechanism, for example, so that it can be accessed by one or more devices. For example, the image container may be sent to one or more server systems via a network, making it accessible by multiple client devices that can access the server systems.

[0067] In some implementations, the server device can store image containers in a cloud storage mechanism (or other storage mechanism) that includes additional information such as metadata. The server can store LDR images in any format within the container. In response to receiving an image request from a client device, the server device can determine which image data to supply to the requesting client device based on system settings, user preferences, client device characteristics such as the type of client device (e.g., mobile client device, desktop computer client device), and client device display characteristics such as dynamic range and resolution. For example, for some client devices, the server device can supply an image container to the client device, and the client device can decode the image container to obtain an image (base image or derived image) for display on the client's display device with an appropriate dynamic range. For other client devices (for example, the server device has information describing the characteristics of this client device, such as the dynamic range of the client device's display device), the server device can extract and decode an image from the image container and supply the image to the client device, the supplied image having an appropriate dynamic range for the client's display device as determined by the server device. In some implementations, the server device can determine the base image and derived image from the image container and supply both images to the client device, which can then select which of these images to display based on the dynamic range of its display device.

[0068] In some implementations, an image container may contain multiple recovery maps (e.g., recovery elements). Each recovery map may contain different values ​​to supply one or more different characteristics to the output image based on the recovery map, as described herein. For example, one particular recovery map may be selected from among several and applied to supply an output image having characteristics based on the selected recovery map, and thus may be more suitable for a particular use or application than, for example, other recovery maps in the image container. For example, some or all of the multiple recovery maps may be associated with specific different physical display characteristics of the target output display device. In some examples, a user may choose to use different recovery maps when displaying LDR or HDR images on a display device with a different dynamic range (e.g., different peak brightness) or a different color gamut, or when using different image decoders.

[0069] In some implementations, each range scaling factor may be stored in the image container for use with each of these recovery maps. In some implementations, as illustrated with reference to Figure 5, one or more recovery elements may be associated with stored instructions for specific display characteristics of a target display device, allowing for the selection of one of the recovery map tracks based on the specific target display device used for display. In some implementations, a single range scaling factor may be used with multiple recovery maps. For example, a generic range scaling factor may be used for multiple recovery maps, each associated with multiple images stored in the image container.

[0070] In some implementations, additional metadata indicating the intended use of a recovery map may be stored in the image container. In some implementations that store multiple recovery maps in the image container, such metadata may include instructions for the intended use for each of the multiple recovery maps. For example, if recovery map 1 is intended to map to an LDR image, a map, table, or index with the key of map 1 may indicate a value that is a list of output formats specified in relation to the recovery map. In some examples, the metadata may specify instructions for the output range (e.g., "LDR" or "HDR", or HDR may be specified as a multiplier for the LDR range). In some examples, the metadata may specify the bit depth, for example, "output metadata:bit_depth X," where X could be 8, 10, 12, etc. In some implementations, the specified bit_depth can be the minimum bit depth; for example, "LDR" can indicate 8 bits or more for output via photoelectric transfer function (OETF) and 12 bits or more for linear / extended range output, while "HDR" can indicate 10 bits or more for output via OETF and 16 bits or more for linear / extended range output. This type of metadata can also be used for multi-channel recovery maps where the color space may change and the intended output color space can be specified in the metadata. For example, the metadata may be specified as "output metadata:bit_depth X,color_space," where color_space could be, for example, Rec.709, Rec.2020, etc.

[0071] Figure 3 is a flowchart illustrating an exemplary method 300 for generating a recovery map based on HDR and LDR images in several implementations. For example, method 300 may be performed with respect to block 214 of method 200 in Figure 2, or in other implementations. In some implementations, the LDR image and the (original) HDR image are acquired as described with reference to Figure 2.

[0072] Method 300 may begin with block 302, where the linear luminance of the LDR image may be determined. In some implementations, the original LDR image (e.g., acquired in block 212 in Figure 2) may be a nonlinear or gamma-corrected image, and a linear version of the LDR image may be generated, for example, by converting the primary image color space of the nonlinear LDR image to a linear version. For example, a color space with a standard RGB (sRGB) transfer function is converted to a linear color space that preserves the sRGB primary colors.

[0073] The linear luminance of a linear LDR image is determined. For example, the luminance (Y) function can be defined by the following equation: Y ldr (x,y)=primary_color_profile_to_luminance(LDR(x,y)) Here, Yldr is the linear luminance of a low dynamic range image defined in the range of 0.0 to 1.0, and primary_color_profile_to_luminance is a function that converts the primary colors of an image (LDR image) to the linear luminance value Y of each pixel at coordinates (x,y) of the LDR image. Block 302 may be followed by Block 304.

[0074] In block 304, the linear luminance of the HDR image is determined. In some implementations, the original HDR image (e.g., acquired in block 210 in Figure 2) may be a nonlinear image and / or a 3-channel coded image (e.g., a perceptual quantizer (PQ) coded image or a hybrid log-gamma (HLG) coded image), and a 3-channel linear version of the HDR image may be generated, for example, by converting the nonlinear HDR image to a linear version. In other implementations, any other color space and color profile may be used.

[0075] The linear luminance of a linear HDR image is determined. For example, the luminance (Y) function can be defined by the following equation: Y hdr(x,y)=primary_color_profile_to_luminance(HDR(x,y)) Here, Yhdr is the linear luminance of a high dynamic range image, defined in the range from 0.0 to the range scaling coefficient, and primary_color_profile_to_luminance is a function that converts the primary colors of an image (HDR image) to the linear luminance value Y of each pixel at coordinates (x,y) of the HDR image. Block 304 may be followed by Block 306.

[0076] In block 306, the pixel gain function is determined based on the linear luminance determined in blocks 302 and 304. The pixel gain function is defined as the ratio of the Yhdr function to the Yldr function. For example, pixel_gain(x,y)=Yhdr(x,y) / Yldr(x,y) And, Yhdr is the linear luminance of a high dynamic range image, defined in the range from 0.0 to the range scaling coefficient, and primary_color_profile_to_luminance is a function that converts the primary colors of an image (HDR image) to the linear luminance value Y of each pixel at coordinates (x,y) of the HDR image.

[0077] Since zero is a valid luminance value, Yhdr or Yldr can be zero, which can be problematic when determining the logarithm as described in the formula above or below. In some implementations, the case where Yhdr and / or Yldr are zero can be addressed by defining the pixel_gain function to 1, for example, or by adjusting the calculation to avoid this case by using a sufficiently small additional element represented as epsilon (ε) in the following formula (for example, by adding ε or clamping to ε).

[0078] pixel_gain(x,y)=(Yhdr(x,y)+ε) / (Yldr(x,y)+ε) In some implementations, different values ​​of epsilon (ε) may be used in the numerator and denominator of the above equation. Block 306 may be followed by Block 308.

[0079] In block 308, the range scaling factor is determined based on the highest brightness of the HDR image relative to the highest brightness of the LDR image. For example, the range scaling factor may be the ratio of the highest brightness of the HDR image to the highest brightness of the LDR image. The range scaling factor may also be referred to herein as the range compression factor and / or the range expansion factor, which is the reciprocal of the range compression factor. For example, if the LDR image is determined from the HDR image, the range scaling factor may be derived from the amount of the total HDR range that is compressed to produce the LDR image. For example, this factor can indicate the amount by which the brightness of the high-brightness areas of the HDR image is reduced to map the HDR image to the LDR dynamic range, or the amount by which the brightness of the dark areas of the HDR image is increased to map the HDR image to the LDR dynamic range. The range scaling factor may be a linear value multiplied by the total LDR range to obtain the total HDR range in linear space. In some examples, if the range scaling factor is 3, the dark areas of the HDR image are tripled, or the high-brightness areas of the HDR image are reduced to 1 / 3 (or a mixture of such dark area enhancement and high-brightness reduction is performed). In some implementations, the range scaling factor may be defined in other ways (for example, by the camera application or other applications, users, or other content generators) to produce specific visual effects that can alter the appearance of the output image derived from the range scaling factor. Block 308 may be followed by Block 310.

[0080] In block 310, a recovery map is determined that encodes the pixel gain function scaled by a range scaling coefficient. Thus, the recovery map is based on two linear images containing the desired HDR image luminance (Yhdr) and LDR image luminance (Yldr) via the pixel gain function. In some implementations, the recovery map is a scalar function that encodes the normalized pixel gain in logarithmic space and is scaled by a range scaling coefficient (e.g., multiplied by the reciprocal of the range scaling coefficient). In some examples, the recovery map value is proportional to the difference between the logarithm of the HDR luminance and the logarithm of the LDR luminance, divided by the logarithm of the range scaling coefficient. For example, the recovery map may be defined by the following equation:

[0081] recovery(x,y)=log(pixel_gain(x,y)) / log(range_scaling_factor) In some example implementations, recovery(x,y) may tend to fall within the range of -1 to +1. For HDR display devices, values ​​less than zero darken pixels (of an LDR image), while values ​​greater than zero brighten pixels. In some examples of the effect of the recovery map range, to enhance bright areas when displaying an image on an HDR display (examples are illustrated with reference to Figure 5), these values ​​can typically be in the range of approximately [0..1] (for example, applying this recovery map enhances bright areas while keeping dark areas constant), so the brightest areas of the image may have values ​​close to 1, and the darker areas of the image may typically have values ​​close to 0. In some implementations, to push down (darken) dark areas when displaying on an HDR display device, these values ​​can typically be in the range of approximately [-1..0] (for example, applying this recovery map re-darkens dark areas while keeping bright areas constant). In some implementations, when displayed on an HDR display device, these values ​​can encompass the entire range of [-1..1] to achieve a combination of enhanced brightness in bright areas and darker black areas in dark areas.

[0082] In some implementations, for ease of operation, the image is converted to logarithmic space before processing, as described above. In some implementations, this process can be performed by indexing and exponential interpolation in linear space, which is mathematically equivalent to the logarithmic space process. In some implementations, this process can be performed without indexing, for example, by naive interpolation / extrapolation in linear space.

[0083] When applying a recovery map to an LDR image for display, the range scaling coefficient generally indicates the amount to increase the brightness of pixels, and the map indicates that some pixels will be made brighter or darker. The recovery map indicates the relative amount to scale each pixel, and the range scaling coefficient indicates the specific amount of scaling to be performed. The range scaling coefficient provides normalization and context to the relative values ​​in the recovery map. The range scaling coefficient enables efficient use of all bits used to store the recovery map, regardless of the amount of scaling applied by the recovery map. This means that a wider variety of images can be accurately encoded. For example, all maps may contain values ​​in the range of, for example, -1 to 1, and maps that can scale to a larger range (e.g., 2 or 8) can be similarly represented, with values ​​contexted by the range scaling coefficient. For example, one image in the above format may have a range scaling coefficient of 2, and a second image may have a range scaling coefficient of 8. Without a range scaling coefficient, the maximum / minimum absolute gain values ​​in the map are determined. If this value is, for example, 2, a second image with a range scale of 8 cannot be represented. If this value is, for example, 8, when storing an image with a range scale of 2, 2 bits of information are wasted for every pixel in the recovery map. Such implementations that do not use a range scaling factor are prone to (undesirable) visual differences in the displayed output HDR image compared to the original HDR image used for encoding (for example, a severe form of such difference may be banding).

[0084] In some implementations, when pixel_gain is 0.0, the recovery function may be defined as -2.0, which is the maximum attenuation that can be represented. The recovery function may be outside the range of -1.0 to +1.0 because one or more regions or locations in the image may require a larger scaling (attenuation or gain) than can be represented by the range scaling factor to recover the original HDR image. The range of -2.0 to +2.0 may be sufficient to produce such larger scaling (attenuation or gain). Block 310 may be followed by Block 312.

[0085] In block 312, the recovery map can be encoded into a compressed format. This makes it possible to reduce the memory space occupied by the recovery map. The compression format can be any of various different compressions, formats, etc. In some implementations, the recovery map is determined and encoded into a bidirectional grid, as will be described in more detail with reference to Figure 4. In some implementations, the recovery map may be compressed, or otherwise the required memory size may be reduced based on one or more additional or alternative compression techniques. In some implementations, the recovery map may be compressed into a lower-resolution recovery image having a lower resolution than the recovery image determined in block 310. The memory space occupied by this recovery image may be smaller than that of the full-resolution recovery map. For example, in implementations that result in an LDR image in JPEG format, JPEG compression may be used, or other compression types (e.g., run-length encoding (RLE)) or other image format compression (e.g., HEIC, AVIF, PNG, etc.) may be used.

[0086] Figure 4 is a flowchart illustrating an exemplary method 400 for encoding a recovery map into a bidirectional grid, in several implementation forms. For example, method 400 may be implemented as blocks 310 and 312 of method 300 in Figure 3, where the recovery map is determined based on the luminance of HDR and LDR images and stored in a compressed format, which is a bidirectional grid in the implementation form described in Figure 4. The bidirectional grid is a three-dimensional data structure of grid cells mapped to pixels of an LDR image based on guide weights, as described below.

[0087] Method 400 may begin with block 402, where a three-dimensional data structure for grid cells is defined. The three-dimensional data structure has width, height, and depth in multiple cells, indicated by size W×H×D, where the width and height correspond to the width and height of the LDR image, and the depth corresponds to the number of layers of cells of W×H. A cell in the grid is defined as a vector element of length D in (width, height) and is mapped to multiple LDR image pixels. Thus, the number of cells in the grid can be much less than the number of pixels in the LDR image. In some examples, the width and height may be about 3% of the LDR image size, and the depth may be 16. For example, a 1920×1080 LDR image may have a grid of approximately 64×36×16 size. Block 402 may be followed by block 404.

[0088] In block 404, a set of guide weights is defined. In some implementations, the set of guide weights may be defined as a multi-element vector of floating-point values ​​and may be used for generating and decoding bidirectional grids. The guide weights represent the weight of each element of an input pixel LDR(x,y) in an LDR image, for determining the corresponding depth D of the cell mapped to that input pixel LDR(x,y). For example, the set of guide weights may be defined as a five-element vector, as shown in the following equation:

[0089] guide_weights={weight_r,weight_g,weight_b,weight_min,weight_max} Here, weight_r, weight_g, and weight_b refer to the weights of the color components (red, green, or blue) of each input pixel, while weight_min and weight_max refer to the weights of the minimum component values ​​of r, g, and b, and the weights of the maximum component values ​​of r, g, and b, respectively (e.g., min(r,g,b) and max(r,g,b)).

[0090] In one example, the LDR image is in the Rec.709 color space, and the guide weights may be set to the following values ​​to represent, for example, the brightness balance of an LDR(x,y) pixel as well as the minimum and maximum component values ​​of that pixel.

[0091] guide_weights={0.1495,0.2935,0.057,0.125,0.375} In this example, these guide weights weight the pixel luminance (r, g, b values) 50%, the minimum component value 12.5%, and the maximum component value of the pixel 37.5% in the grid depth search. In some implementations, the sum of these values ​​is 1.0, so, for example, all grid cells can always be searched (unlike when the sum of values ​​is less than 1.0), and (unlike when the sum of values ​​is greater than 1.0) the pixel depth can never overflow beyond the final depth level in the grid. The guide weights can be practically determined, for example, by a person visually evaluating the results. In some implementations, a different number of guide weights may be used for the pixel luminance (r, g, b values), for example, as few as three.

[0092] Block 404 may be followed by Block 406. In block 406, grid cells are mapped to pixels in the LDR image based on guide weights. The guide weights are used to determine the depth D corresponding to the cell mapped to the input pixel.

[0093] In some implementations, the width (x) and height (y) of a grid cell are mapped to the pixels (x,y) of an LDR image. For example, the x and y pixels of an LDR image can be transformed into the x and y coordinates of a grid cell as follows: grid_x = ldr_x / ldr_W * (grid_W - 1) grid_y=ldr_y / ldr_H*(grid_H-1) Here, grid_x and grid_y are the x and y positions of the grid cells, ldr_x and ldr_y are the x and y coordinates of the pixels in the LDR image mapped to the grid cells, ldr_W and ldr_H are the total pixel width and pixel height of the LDR image, and grid_W and grid_H are the cell dimensions of the total width and height of the bidirectional grid, respectively.

[0094] In some implementations, the following formula can be used for the depth of the grid cells mapped to the LDR image: z_idx(x,y)=(D-1)*(guide_weights {r,g,b,min(r,g,b),max(r,g,b)}) Here, D is the total cell depth dimension of the grid, and z_idx(x,y) is the depth of the cell in the bidirectional grid that maps to the LDR image pixel (x,y). This formula determines the dot product of the guide weight vector and the vector of corresponding values ​​for a particular pixel in the LDR image, and then scales the result by the size of the grid depth dimension to indicate the depth of the corresponding cell for that particular pixel. For example, the result of multiplying the guide weights can be a value between 0.0 and 1.0 (inclusive), and this result is scaled to various depth levels (buckets) of the grid. For example, if D is 16, a multiplication value of 0.0 results in a mapping to bucket 0, a value of 0.5 results in a mapping to bucket 7, and a value of 1.0 results in a mapping to bucket 15 (the last bucket). Block 406 may be followed by block 408.

[0095] In block 408, the recovery map value is determined using the solution to a set of linear equations. For example, in some implementations, the input to a bidirectional grid is defined using the following equation: pixel_gain(x,y)=bilateral_grid_linear(x,y,z_idx(x,y)) Here, pixel_gain(x,y) is the value of the pixel_gain function at pixel(x,y) as described above, with reference to method 300 in Figure 3. Bilateral_grid_linear is the content of the bilateral grid, representing the pixel gain in linear space. The coordinates x, y, and z_idx are the coordinates of the bilateral grid, which are different from the x, y coordinates of the LDR image. z_idx tells the guide weights how the pixel_gain of each pixel in the LDR image is located in the bilateral grid, and how this tells us how to set up the set of linear equations to be solved in block 410. Block 408 may be followed by block 410.

[0096] In some implementations, encoding is defined as the solution to a set of linear equations that minimize the following for each grid cell:

[0097] (pixel_gain(x,y)*Yldr(x,y)-Yhdr(x,y)) 2 The Yldr and Yhdr values ​​are the luminance values ​​of the LDR and HDR images at the (x,y) pixel position, respectively. The pixel_gain(x,y) value that minimizes the value of the above equation is solved based on where each pixel is looked up into the depth vector, as indicated in the above equation, and this is predetermined based on guide weights.

[0098] The resulting value is defined by the following formula. For{x,y,z}over{[0,W),[0,H),[0,D)}:bilateral_grid_linear(x,y,z) This definition specifies that a bidirectional grid is defined as having x from 0 to the total grid width, y from 0 to the total grid height, and z from 0 to the total grid depth. For example, if the bidirectional grid is 64 × 36 × 16, the grid values ​​are defined as x from 0 to 63 (including both ends), y from 0 to 35 (including both ends), and z from 0 to 15 (including both ends). Block 408 may be followed by Block 410.

[0099] In block 410, the solution values ​​are transformed and stored in the bidirectional grid. For example, the solution values ​​may be transformed using the following formula:

[0100] bilateral_grid(x,y,z)=log(pixel_gain(x,y)) / log(range_scaling_factor) or bilateral_grid(x,y,z)=log(bilateral_grid_linear) / log(range_scaling_factor) Bilateral_grid is the content of the bidirectional grid, representing the pixel gain in logarithmic space. These values ​​are stored as an encoded recovery map. As indicated by these equations, the normalized pixel gain between the LDR and HDR images is encoded in logarithmic space and scaled by a range scaling factor, as described above with reference to block 310 in Figure 3.

[0101] To display an HDR image with a larger dynamic range than an LDR image, when decoding the recovery map from a bidirectional grid, each recovery map value is extracted from the bidirectional grid (as described in detail with reference to Figure 5). In the following equation, r, g, and b are the respective component values ​​of the pixel LDR(x,y) of the LDR image.

[0102] z_idx(x,y)=(D-1)*(guide_weights {r,g,b,min(r,g,b),max(r,g,b)}) recovery(x,y)=bilateral_grid(x,y,z_idx(x,y)) Here, recovery(x,y) is the recovery map to be extracted. In some implementations, z_idx, determined in the first equation, may be rounded to the nearest integer before being input into bilateral_grid(x,y,z) in the second equation.

[0103] Figure 5 is a flowchart illustrating an exemplary method 500 for decoding and displaying an image in a backward-compatible high dynamic range image format, in several implementations. In some implementations, method 500 may be performed on, for example, a server system 102 shown in Figure 1. In some implementations, some or all of method 500 may be performed on one or more client devices, such as client devices 120, 122, 124, or 126 in Figure 1, one or more server devices, such as server device 104 in Figure 1, and / or both server and client devices. In the above examples, the implementing system includes one or more digital processors or processing circuits ("processors") and one or more storage devices (e.g., databases or other storage mechanisms). In some implementations, different components of one or more servers and / or clients may perform different blocks or parts of method 500. In some examples, devices are described as performing blocks of method 500. Some implementations may have one or more blocks of method 500 executed by one or more other devices (e.g., other client devices or server devices) that can send results or data to the first device.

[0104] In some implementations, Method 500 or part thereof may be initiated automatically by the system. For example, Method (or part thereof) may be executed periodically, or may be executed based on one or more specific events or conditions, such as the initiation of the image application 106 by the client device, the capture or receipt of new images by the client device, the uploading of new images to the server system 102, the expiration of a predetermined period since the last operation of Method 500, and / or the occurrence of one or more other states that may be specified in a setting read or received by the system executing Method 500.

[0105] Method 500 may be performed by a device other than the one that generated the image container. For example, the first device may generate an image container and store it in a storage location (e.g., a server device) accessible by a second device other than the first device. The second device can access the image container and display the image using Method 500. In some examples, a recovery map may be applied to the base image by a hardware decoder and / or the CPU and / or GPU of the second device. In some implementations, a single device may generate an image container as described in Method 200 and decode and display the image container as described in Method 500. In some implementations, the first device may decode an image from the image container based on Method 500 and supply the decoded image to the second device for display.

[0106] Similarly, as described above with reference to Figure 2, in some implementations, the first device (for example, a server device in the cloud) decodes two images (for example, a base image and a derived image based on the base image and a recovery map) from the image container, sends the two images to the second device, and the second device determines, for example, based on the characteristics of the display device of the second device, to select and display one of the images.

[0107] User permission may be obtained in order to use user data in Method 500 (blocks 510-534). For example, authorized user data may include images stored in the client device (e.g., any of client devices 120-126) and / or the server device, image metadata, user data related to the use of image applications, and other image-based generation. The user is given the option to selectively grant or deny access to all or any subset of the user data. If user permission is insufficient with respect to particular user data, Method 500 may be executed using other data (e.g., images unrelated to the user) without using that user data.

[0108] Method 500 may begin in block 502. In block 510, an image container containing an LDR image and a recovery map is acquired. For example, the image container may be a container generated by Method 200, which generates a recovery map from an LDR image and an HDR image having a larger dynamic range than the LDR image, as shown in Figure 2. The recovery map may be supplied to the container as a recovery element, such as a recovery image. The image container may have a standardized format that is supported and recognized by the device performing Method 500, such as a container for JPEG, a custom JPEG marker (e.g., an APP1 segment for storing metadata in a JPEG / Exif file, or a marker for storing chunks of data), an International Organization for Standardization-Based Media File Format (ISOBMFF), a container for AV1 Picture File Format (AVIF), HEIF (High Efficiency Picture File), a Multi-Picture Format (MPF) container, or Extensible Metadata Platform (XMP) encoded data. In some implementations, the recovery map may be stored as a sub-image conforming to a standard (e.g., AVIF, HEIF) or as a novel type of sub-image (e.g., defining an AV1 gain-map image item that signals to the reader that the image item is a gain map and is therefore used to define the range scaling coefficient in the header, for example). Block 510 may be followed by Block 512.

[0109] In block 512, the LDR image is extracted from the image container. In some exemplary implementations, this extraction may include the step of decoding the LDR image from its standard file format (e.g., JPEG) to the raw image format or other format to be used for display and / or processing. The decoded value from the encoding map may be determined by the following equation, where in each recovered map value, r is the range and n is the number of bits.

[0110] recovery(x,y)=(encode(x,y)-(2^n-1) / 2) / ((2^n-1) / 2r) In some example implementations, with respect to block 218 in Figure 2, the decoded value from the encoded map may be determined by the following equation, in relation to the example of the quantized encoded map (encode(x,y)) described above.

[0111] recovery(x,y)=(encode(x,y)-127.5) / 63.75 This results in a scale and offset for the range of the recovery map value, which is -2 to 2 and stored as 8 bits. In another example, if the value stored in the recovery map is -1 to 1, the decoded value can be determined by the following formula:

[0112] recovery(x,y)=(encode(x,y)-127.5) / 127.5 Other implementations may use different ranges, scales, offsets, and number of bits to store. Block 512 may be followed by Block 514.

[0113] In block 514, it is determined whether to display the output image, which is an HDR image (e.g., a derived image) derived from the extracted LDR image, or the output image, which is an LDR image. The output image is displayed on at least one target display device associated with or communicating with a device that accesses the image container and performs method 500. The target display device can be any suitable output display device, such as a display screen, touchscreen, projector, goggles or glasses with a display. The displayed output HDR image may have any dynamic range higher than the LDR image in the container, up to the dynamic range of the original HDR image used to generate the image container (e.g., block 210 in Figure 2).

[0114] The decision of whether to display an output HDR image may be based on one or more characteristics of the target display device that will display the output image. For example, the dynamic range of the target display device can be used to determine whether to display an HDR or LDR image as the output image. In some examples, if the target display device can display a dynamic range greater than that of the LDR image in the image container, it is determined to display the HDR image. If the target display device can only display the low dynamic range of the LDR image in the container, the HDR image will not be displayed.

[0115] In some implementations or cases, the system may decide not to display an HDR image even if the target display device is capable of a larger dynamic range than an LDR image. For example, user settings or preferences may instruct the system to display an image with a smaller dynamic range, or to display an image as an LDR image for visual comparison with other LDR images.

[0116] If it is determined in block 514 that the HDR image should not be displayed, the method proceeds to block 516, where the extracted LDR image from the image container is displayed by the target display device. Generally, the LDR image is displayed by an LDR-compatible target output device that cannot display a dynamic range larger than the range of the LDR image. The LDR image may be output directly by the target display device, and any metadata regarding the recovery map and HDR image format in the image container is ignored. In some implementations, the LDR image is provided in a standard format and is therefore readable and displayable by any device that can read the standard format.

[0117] If, in block 514, it is determined that an HDR image should be displayed as the output image, the method proceeds to block 518, where the recovery element is extracted from the image container. Metadata related to displaying the output HDR image based on the recovery element can also be obtained from the image container. For example, the recovery element may be a recovery image as described above, and may include and / or be accompanied by the metadata as described above. The container and / or the LDR image from the image container may also contain metadata, or may contain metadata instead.

[0118] In some implementations, as described above, the image container may contain multiple recovery maps (e.g., recovery elements). In some of these implementations, in block 518, one of the multiple recovery elements may be selected for the following processing. In some implementations, the selected recovery element may be specified by user input or user setting, or in some implementations, it may be automatically selected by the device performing the processing based on one or more characteristics of, for example, the target output display device or other output display components (e.g., without current user input), and such characteristics may be obtained using operating system calls or other available sources. For example, in some implementations, one or more recovery elements may be associated with an indication of a specific display device characteristic stored in the image container (or accessible by the device performing block 518). If the target display device is determined to have one or more specific characteristics (e.g., dynamic range, peak brightness, color gamut, etc.), then in block 518, a specific recovery element associated with those characteristics may be selected. If multiple recovery elements are associated with another range scaling coefficient, this range scaling coefficient is also selected and retrieved. Block 518 may be followed by block 520.

[0119] In block 520, the recovery map is optionally decoded from the recovery elements. For example, the recovery map may be in a compressed or other encoded format in the recovery elements and can be decompressed or decoded. For example, in some implementations, the recovery map may be encoded into a bidirectional grid as described above and decoded from the bidirectional grid. Block 520 may be followed by block 522.

[0120] In block 524, it is determined whether to scale the brightness of the extracted LDR image for the output HDR image. In some implementations, this scaling is based on the display output capability of the target display device. For example, in some implementations or cases, the generation of the output HDR image is partially based on a specific brightness that the target display device can display. In some implementations or cases, this specific brightness may be the highest brightness that the target display device can display, and the highest brightness of the display device determines the upper limit of the brightness of the output HDR image. In some implementations or cases, in block 524, it is determined, for example, that the generation of the output HDR image does not take into account the brightness capability of the target display device and does not scale the brightness of the LDR image.

[0121] In block 524, if it is determined that the brightness of the output image should not be scaled based on the capabilities of the target display device, the method proceeds to block 526, where the brightness gain of the recovery map is applied to the pixels of the LDR image, and the pixels are scaled based on the range scaling coefficient to determine the corresponding pixels in the output HDR image. For example, the brightness gain and scaling are applied to the brightness of the pixels of the LDR image to determine the corresponding pixel values ​​of the displayed output HDR image. This ensures that the output HDR image has the same dynamic range as the original HDR image used when it generated the image container.

[0122] In some exemplary implementations, the LDR image and range scaling coefficients are transformed into logarithmic space, and the following equation may be used: HDR*(x,y)=LDR*(x,y)+log(range_scaling_factor)*recovery(x,y) Here, HDR*(x,y) is the pixel(x,y) of the recovered HDR image (output image) in logarithmic space, LDR*(x,y) is the pixel(x,y) of the logarithmic version of the extracted LDR image (e.g., log(LDR(x,y))), log(range_scaling_factor) is the range scaling factor in logarithmic space, and recovery(x,y) is the recovery map value at pixel(x,y). For example, recovery(x,y) is the normalized pixel gain in logarithmic space.

[0123] In some cases, when a pixel-by-pixel gain is applied, such as recovery(x,y)*range_scaling_factor, interpolation occurs between the LDR and HDR versions of an image. This is essentially interpolation between the original version of the image and the range-compressed version, and the amount of interpolation is dynamic, both pixel-by-pixel and globally, in the sense that it depends on the range scaling factor. In some cases, this can be considered extrapolation if the recovery map value is less than 0 or greater than 1.

[0124] In some implementations, the LDR image from the image container is non-linearly encoded, for example, gamma-coded, and the primary image color space of the non-linear LDR image is converted to a linear version before the LDR image is converted to LDR*(x,y) in logarithmic space for the above equation. For example, a color space with a standard RGB (sRGB) transfer function can be converted to a linear color space that preserves the sRGB primary colors, as described above with respect to block 302 in Figure 3. In some implementations, the LDR image in the container may be linearly encoded, and no such conversion from a non-linear color space is used.

[0125] Applying luminance gain to the luminance of individual pixels in an LDR image yields pixels corresponding to the original (e.g., originally encoded) HDR image. If the LDR image is stored losslessly and a recovery map is provided, the output HDR image may have the same pixel values ​​as the original HDR image (e.g., if the recovery map has the same resolution as the LDR image). Otherwise, the output HDR image resulting from block 532 may be a very close approximation of the original HDR image, and the user will usually not visually notice any difference in the display of the output image.

[0126] In some implementations, the output image may be converted from logarithmic space to the output color space for display. In some implementations, for example, the output image color space may differ from the color space of the LDR image extracted from the image container when using a multi-channel recovery map. Block 526 may be followed by the following block 534.

[0127] If it is determined in block 524 to scale the image brightness based on the output of the target display device, the method proceeds to block 528, where the maximum brightness output of the target display device is determined. The target display device may be an HDR display device, and these types of devices may display a maximum brightness value that may vary depending on the model, manufacturer, etc. of the display device. In some implementations, the maximum brightness output of the target display device may be obtained via an operating system call or other source. Block 528 may be followed by block 530.

[0128] In block 530, the display factor is determined. The display factor is used to scale the dynamic range of the displayed output HDR image, as described below. In some implementations, the display factor is determined to be less than or equal to the smaller of the luminance output and range scaling factor of the target display device. For example, this can be defined as follows:

[0129] display factor≦min(maximum display luminance,range scaling factor) This decision allows the upper limit of the output image's brightness scaling to be set to the target display's highest display brightness. This prevents scaling the output image to a brightness range higher than the target display's highest brightness output range, even if present in the HDR image from which the image container was generated (as indicated by the range scaling factor). In some implementations, the highest display brightness may be a dynamic characteristic of the target display and may be adjustable, for example, by user settings or application programs.

[0130] In some implementations, the display factor may be set to a brightness greater than or equal to the maximum display brightness, and greater than or equal to the maximum display brightness. For example, this may be done if the target display device can output a dynamic range greater than the dynamic range present in the original HDR image (which may be indicated by the range scaling factor). In some examples, the user may be able to instruct the output image to be brighter (by user settings or selected input).

[0131] In some implementations, the display factor may be set to a brightness lower than the maximum display brightness of the target display device or the maximum brightness of the original HDR image. For example, the display factor may be set based on a specific display brightness of the display device, indicated by one or more display conditions, user input, preferences, or settings, resulting in a lower brightness level output that may be desired in some cases or applications, such as those described below. Block 530 may be followed by the following block 532.

[0132] In block 532, the luminance gain of the recovery map is applied to the LDR image, and the luminance of the LDR image is scaled based on the display coefficient to determine the corresponding pixels in the HDR image. For example, the luminance gain encoded in the recovery image is applied to the pixel luminance of the LDR image to determine the corresponding pixel value of each displayed output image. These pixel values ​​are scaled based on the display coefficient determined in block 530, for example, a specific luminance output of the target display device (which may be the highest luminance output of the target display device). In some implementations, this can increase the luminance of the bright areas in the LDR image to a maximum level based on the dynamic range of the original HDR image, to a level that the display device can display, or decrease the luminance of the dark areas in the LDR image to a lower limit based on the dynamic range of the original HDR image, to a level that the display device can display.

[0133] In some exemplary implementations, the LDR image and display coefficients are transformed into logarithmic space, and the following equation may be used: HDR*(x,y)=LDR*(x,y)+log(display_factor)*recovery(x,y) Here, HDR*(x,y) is the pixel(x,y) of the recovered HDR image (output image) in logarithmic space, LDR*(x,y) is the pixel(x,y) of the logarithmic version of the extracted LDR image (e.g., log(LDR(x,y))), log(display_factor) is the display factor in logarithmic space, and recovery(x,y) is the recovery map value at pixel(x,y). For example, recovery(x,y) is the normalized pixel gain in logarithmic space.

[0134] In some cases, when a pixel-by-pixel gain is applied, for example, `recovery_map(x,y)*display_factor`, interpolation between the LDR and HDR versions of an image may be considered. This can essentially be thought of as interpolation between the original version of the image and the range-compressed version, and the interpolation amount is dynamic, both pixel-by-pixel and globally, in the sense that it depends on the display factor. In some cases, this can be considered extrapolation if the recovery map value is less than 0 or greater than 1.

[0135] In some implementations, the LDR image from the image container is non-linearly encoded, for example, gamma-coded, and the primary image color space of the non-linear LDR image is converted to a linear version before the LDR image is converted to LDR*(x,y) in logarithmic space for the above equation. For example, a color space with a standard RGB (sRGB) transfer function can be converted to a linear color space that preserves the sRGB primary colors, as described above with respect to block 302 in Figure 3. In some implementations, the LDR image in the container may be linearly encoded, and no such conversion from a non-linear color space is used.

[0136] Applying luminance gain to the luminance of individual pixels in an LDR image yields pixels corresponding to the original (e.g., originally encoded) HDR image. If the LDR image is stored losslessly and a recovery map is provided, the output HDR image may have the same pixel values ​​as the original HDR image (e.g., if the recovery map has the same resolution as the LDR image). Otherwise, the output HDR image resulting from block 532 may be a very close approximation of the original HDR image, and the user will usually not visually notice any difference in the display of the output image.

[0137] In some implementations, the output image may be converted from logarithmic space to the output color space for display. In some implementations, for example, the output image color space may differ from the color space of the LDR image extracted from the image container when a multi-channel recovery map is used. Block 532 may be followed by block 534.

[0138] In block 534, the output HDR image determined in block 526 or block 532 is displayed by the target display device. The output HDR image is displayed by an HDR-compatible display device capable of displaying a wider dynamic range than that of the LDR image. The output HDR image generated from the image container may be identical or nearly identical to the original HDR image used to generate the image container, as the recovery map stores the luminance information, and is generally indistinguishable from the original HDR image.

[0139] In some implementations, the output HDR image (from block 526 or block 532) may be suitable for display on a target display device as, for example, a linear image or other output HDR image format. For example, the output HDR image may have been processed by the display coefficients of block 532, and no further modifications for display may be necessary so that the output HDR image resulting from applying a recovery map is rendered directly for display.

[0140] In some implementations or cases, the output HDR image may be converted to another format for display on a target display device, such as an image format that may be more suitable for display by the target display device than the output HDR image. For example, the output HDR image from block 526 (or from block 532 in some implementations) may be converted to a brightness range suitable for the capabilities of the target display (this may be added in some implementations to the use of display coefficients in block 532). In some implementations, the output HDR image may be converted to a standard image having a standard image format, such as HLG / PQ or other standard HDR formats. In some of these examples, a general-purpose tone mapping technique may be applied to such a standard image for display by the target display device.

[0141] In some implementations or cases, output HDR images processed for a target display device using the display coefficients in block 532 may result in higher quality image display on the target device than output HDR images processed in the same way using generic tone mapping techniques, without further conversion to a standard format or processing using generic tone mapping techniques for display. For example, local tone mapping provided by recovery maps may result in more detailed and higher quality images than generic or global tone mapping. Furthermore, conversion from HDR images to any standard format using global tone mapping techniques based on different device specifications and implementations may have ambiguities and discrepancies.

[0142] In some implementations, techniques may be used to output images in a linear format to the display device, such as an extended range format, where 0 represents black, 1 represents SDR white, and values ​​greater than 1 represent HDR brightness. Such techniques can avoid the problems associated with global tone mapping techniques for handling the display capabilities of the target display device. One or more techniques described herein can handle the display capabilities of the display device by applying a recovery map with display coefficients as described above, so that the extended range values ​​do not exceed the display capabilities of the display device.

[0143] As mentioned above, the device can scale the pixel brightness of an LDR image based on the maximum brightness output of the HDR display device and the brightness gain stored in the recovery map. In some implementations, for example in many photographic and video applications, scaling of the LDR image may be used to reproduce the full dynamic range of the original HDR image used to generate the image container (if the target display device can display the entire range of the original HDR image) in order to generate an HDR output image. In some implementations, LDR image scaling can set an arbitrary dynamic range higher than the range of the LDR image for the output HDR image. In some implementations, the dynamic range can also be smaller than the range of the original HDR image. For example, the output image brightness may be scaled based on the capabilities of the target display device and / or other criteria, or it may be scaled lower than the original HDR image and / or the best capabilities of the target display device in order to reduce the display brightness of the image, for example, so that it does not cause visual discomfort or fatigue to the viewer. For example, a fatigue reduction algorithm and one or more light sensors communicating with the device implementing this algorithm may be used to detect one or more of various factors such as ambient light around the display device and time of day to determine the reduced dynamic range or reduced maximum brightness for the output image. For example, the brightness level of the output image may be reduced by scaling the recovery map value and / or display factor used to determine the HDR image, as described herein with respect to several implementations, for example by scaling the range_scaling_factor and / or display factor in the illustrative formula shown above. In another example, the device may determine that it should conserve battery power (for example by avoiding high display brightness) because the device's battery power level has fallen below a threshold level, and in that case, it displays the HDR image at a brightness lower than that of the original HDR image.

[0144] In some implementations, the brightness of an LDR image can be scaled to a greater (e.g., higher) degree than the dynamic range of the original HDR image (for example, if the target display device has a larger dynamic range than the original HDR image). The dynamic range of the output HDR image is not constrained by the dynamic range of the original HDR image, nor by any particular high dynamic range. In some examples, the maximum dynamic range of the output HDR image may be smaller than the maximum dynamic range of the original HDR image and / or the maximum dynamic range of the target output device.

[0145] In some implementations, the display of the output HDR image may be progressively scaled from a lower luminance level to the full (maximum) luminance level of the output HDR image (the maximum luminance level determined as described above). For example, the scaling may be progressively scaled over a specific period, such as 2 to 4 seconds (user-configurable in some implementations). Progressive scaling avoids user discomfort caused by a sudden, large increase in brightness when displaying a higher-luminance HDR image after displaying a lower-luminance image (e.g., an LDR image). As described herein, an LDR image may be progressively scaled to its HDR version by scaling, for example, the recovery map values ​​and / or display factors used in determining the HDR image, thereby progressively brightening the LDR image to its HDR version, for example, in multiple steps. For example, range_scaling_factor and / or display_factor may be scaled using the formulas described above. In other implementations, multiple scaling operations may be performed sequentially over that period to progressively acquire intermediate images with a higher dynamic range. Other HDR image formats that scale LDR images to HDR images using global tone mapping techniques, as described above, rather than local tone mapping as described herein, may have lower display quality or may not support progressive scaling when performing such progressive scaling.

[0146] In some examples, this incremental scaling of HDR image brightness may be performed or triggered in relation to specific display conditions. Such conditions may include, for example, displaying an output HDR image immediately after displaying an LDR image on the display device. In some examples, a grid of LDR images may be displayed, and the user selects one of these images to display the HDR version of this selected image on the screen (as mentioned above, the HDR image may be determined from the associated image container based on the recovery element). When switching from LDR image display to HDR image display, in order to avoid a sudden large increase in brightness, the HDR image may be displayed initially at the lower brightness level of the LDR image and updated incrementally (for example, by progressively displaying image content with a high dynamic range, increasing in increments to its maximum brightness level, e.g., fade-in, fade-out animation). In another example, the display condition for triggering incremental scaling may include an overall screen brightness set to a low level (for example, based on a low ambient light level around the device), as described herein. For example, the overall screen brightness may have a brightness level below a certain threshold level to trigger progressive scaling of HDR brightness.

[0147] In some cases or conditions, for example, when one or more HDR images are displayed and replaced by an output HDR image, the output HDR image may be displayed immediately at its highest brightness without progressive scaling. For example, when a second HDR image is displayed following the display of a first HDR image, the second HDR image may be displayed immediately at its highest brightness because the user is already accustomed to seeing the high brightness of the first HDR image.

[0148] In some implementations, after displaying an output image (e.g., either an LDR or HDR output image), the device displaying the output image may receive user input from the user (or another source, such as another device) instructing it to modify the display of the output image, increasing or decreasing the dynamic range of the display to, for example, the best dynamic range of the display device and / or the original HDR image. For example, in response to receiving user input to modify the display, the device may modify the display coefficient (as described above) according to the user input, which changes the dynamic range of the displayed output image. In some examples, if the user reduces the dynamic range, the display coefficient may be reduced by a corresponding amount, and the output image will be displayed with the resulting smaller dynamic range. This change in display may be provided in real time in response to user input.

[0149] In some implementations, the device displaying the output image may, after displaying the output image, receive input from another source, such as the user or another device, instructing it to modify or edit the output image. In some implementations, after such modification, the recovery map may be discarded, for example, because it can no longer be properly applied to the LDR image. In some implementations, the recovery map may be maintained after such modification. In some implementations, the recovery map may be updated based on the modifications to the output image to reflect the modified version of the LDR image.

[0150] In some cases, when modifications are made by the user of an HDR display and / or an HDR-aware editing application program, the characteristics of the modifications are known in a larger dynamic range, and a recovery map may be determined based on the edited image. For example, in some implementations, during the loading time of the image into the image editing interface, the recovery map may be decoded to the full-resolution image and presented to the editor as data associated with the alpha channel, other channels, or output image. For example, the recovery map may appear in the editor as an additional channel in the list of channels for the image. Both the LDR image and the HDR output image may be displayed in the editor, making the modifications to the image in both versions visible. When the user edits the image using the editor's tools, the corresponding edits are made to the recovery map and the LDR image (base image). In some cases, the user can decide whether to edit the RGB or RGBA (red, green, blue, alpha) of the image. In some implementations, the user can manually edit the recovery map (e.g., the alpha channel) to observe the changes in the displayed HDR image while the displayed LDR image remains identical.

[0151] In some implementations, overall dynamic range control may be provided in an image editing interface that allows adjustment of the range scaling coefficient. When editing operations on an image, such as cropping or resampling, the alpha channel may be updated in the same way as the RGB channels. When saving an LDR image, it may be saved again in the image container. The saved LDR image may be one edited in the interface, or a new LDR image generated from an edited HDR image by local tone mapping, or generated based on an edited HDR image by scaling (e.g., subdividing) each pixel using a recovery map. The edited recovery map may also be saved again in the image container. In various implementations, the recovery map may be reprocessed into a bidirectional grid if such encoding is used, or the image and recovery map may be saved losslessly with a larger required memory, for example, if the image is further edited.

[0152] In some implementations, an editor or other program can convert LDR images and recovery maps of the described image format to standard HDR images, such as 10-bit HDR AVIF images or images in another HDR format.

[0153] In various implementations, the various blocks of methods 200, 300, 400, and / or 500 may be combined, divided into multiple blocks, executed in parallel, or executed asynchronously. In some implementations, the order in which one or more blocks of these methods are executed may be the same as or different from those shown in these figures. For example, in various implementations, blocks 210 and 212 in Figure 2 and / or blocks 302 and 304 in Figure 3 may be executed in different orders or in parallel. Methods 200, 300, 400, and / or 500, or parts thereof, may be repeated any number of times using further inputs. For example, in some implementations, method 200 may be executed when one or more new images are received by the device performing the method and / or stored in the user's image library.

[0154] In some implementations, the roles of the HDR image and the LDR image can be reversed as described above. For example, the original HDR image may be stored in the image container (as the base image, for example, in block 218) rather than the original LDR image which has a smaller dynamic range than the HDR image. A recovery map may be stored in the image container that instructs how to determine the derived output LDR image from the base HDR image for LDR display devices that can only display a low dynamic range which is smaller than the dynamic range of the HDR image. The recovery map may include gain and / or scaling factors based on the base HDR image and the original LDR image, and in some implementations, the original LDR image may be a tone-mapping image (or a range-transformed HDR image) as described above.

[0155] Some of the implementations described can enable applications that produce HDR images for obtaining and outputting higher-quality LDR images based on tone mapping of HDR images, without the constraints of current techniques (e.g., HLG / PQ transfer functions) that provide global or generic tone mapping for the entire image. The local tone mapping provided by using the recovery map technique described herein can enable higher-fidelity conversion from HDR images to LDR images without applying global or generic tone mapping, for example, for applications such as image sharing or uploading images to services or applications that only support LDR images.

[0156] In some implementations, brightness control may be provided for the display of HDR images, such as the HDR images described herein. The user of the device can adjust the HDR brightness control to specify the maximum brightness (e.g., lightness) at which the display device will display the HDR image. In some implementations, the maximum brightness of an HDR image may be determined as the highest average brightness of the pixels displayed in the HDR image, since such an average may represent the image brightness perceived by the user. In other implementations, the maximum brightness may be determined as the maximum value of the brightest pixels in the HDR image, as the maximum value of one or more specific pixels in the HDR image (or the average maximum value of each such specific pixel), and / or based on one or more other characteristics of the HDR image. When the brightness control is set to a value less than the maximum displayable brightness of the HDR image, the device reduces the maximum brightness of the output HDR image to a lower brightness level based on the brightness control. This allows users to adjust the maximum brightness of displayed HDR images, for example, to avoid discomfort or fatigue caused by the glare of high-brightness HDR images, and / or to reduce the contrast between high-brightness HDR images and other content on the display screen outside of the HDR image (such as LDR images, user interface elements, etc., which may be displayed at lower brightness). In some cases, reducing the maximum brightness level of HDR images can reduce battery power consumption by the display device by reducing the brightness of the display.

[0157] For example, HDR brightness control can be implemented as overall brightness control or system brightness control by the device (e.g., by the operating system running on the device) that controls the HDR image brightness for applications running on the device that displays HDR images. In some examples, brightness control may apply to all applications running on the device, all applications of a particular type running on the device (e.g., image viewing applications and image editing applications), and / or specific applications specified by the user. This allows the user to specify a single brightness adjustment for all (or many) HDR images displayed by the device without, for example, needing to adjust the brightness for each individual HDR image displayed. Such system brightness control can result in display consistency across all applications running on the device and can also provide the user with further control over the maximum brightness of the displayed HDR images. In some implementations, HDR brightness control may be implemented using other system controls on the device, such as accessibility control, user preferences, and user settings. The values ​​set by system brightness control may be used to adjust the output of the HDR image output, in addition to other scaling factors used to display the image, such as the maximum display device brightness as described above, and display coefficients based on individual brightness settings for the image.

[0158] In some implementations, system brightness control may be implemented as a user-adjustable slider or value that indicates, for example, the highest brightness value or percentage of the highest brightness that the display device can display with respect to an HDR image (for example, the highest brightness may be determined as the maximum value of the average brightness of the pixels in the HDR image, or the maximum value of the brightest pixel or a specific pixel in the HDR image, or a maximum value based on one or more other image characteristics). For example, if the brightness control is set by the user to 70% (for example, 70% of the maximum brightness of the display device when used to determine the highest brightness as described above), then each HDR image may be displayed at 70% of their highest brightness. In some implementations, the user can specify different highest HDR image brightness values ​​with respect to different screen brightness values ​​through system brightness control. For example, if the overall screen is displaying content at a high screen brightness, there is not much contrast between the overall screen brightness and the HDR image brightness, so the highest HDR image brightness may be higher, for example, 90% or 100%. At low screen brightness, the maximum HDR image brightness can be lower, for example, 50% or 60%, to reduce the contrast between the overall screen brightness and the HDR image brightness. In some implementations, system brightness control may be given as a setting that correlates with the maximum SDR image brightness (e.g., by a slider or other interface control unit). For example, the setting may be set to a maximum HDR brightness value that is N times the maximum SDR image brightness, where N is the maximum configurable value based on the maximum brightness of the display device used to display the image. In some examples, setting value 1 may indicate the maximum brightness of the SDR (base) image, and setting value, which is the maximum value or setting of the brightness control, instructs to display at the maximum brightness of the display device.

[0159] In some implementations, these variable settings for HDR image brightness can be input via user interface controls. For example, the user can define a graph with a variable maximum HDR brightness relative to the current screen brightness. In some implementations, a number of predetermined HDR brightness configurations may be provided in the user interface, and the user can select one of these configurations to use with the device. For example, each configuration may result in a separate set of maximum HDR brightness values ​​related to a particular range of screen brightness values. In some implementations, a toggle control may be provided that, once selected by the user, removes HDR brightness to display the HDR image with LDR brightness. For example, when the user selects the toggle, the associated recovery map or recovery element may be ignored (as described herein) and the base image is displayed.

[0160] Figure 6 shows a specific example of an exemplary approximation of image 600 having a high dynamic range (the full dynamic range of an HDR image cannot be represented by this; for example, this is an LDR depiction of an HDR scene). For example, HDR image 600 may be captured by a camera capable of capturing HDR images. In image 600, image regions 602, 604, and 606 are shown in detail, and these regions appear natural to the observer and can therefore be depicted with a larger dynamic range.

[0161] Figure 7 shows an example of how an LDR image 700 can be represented, which has a smaller dynamic range than the HDR image 600, such as the low dynamic range of a standard JPEG image. The LDR image 700 depicts the same scene as image 600. For example, image 700 could be an LDR image captured by a camera. In this example, the empty image area 702 is properly exposed, but the foreground image areas 704 and 706 are dark areas that are underexposed due to insufficient range compression in the image, and therefore these areas are too dark and lose visual detail.

[0162] Figure 8 shows another example of image 800, which, like LDR image 700, has a smaller dynamic range than HDR image 600, and depicts the same scene as images 600 and 700. For example, image 800 could be an LDR image captured by a camera. In this example, the empty image region 802 is overexposed and loses detail due to insufficient range compression in the image, while the foreground image regions 804 and 806 are properly exposed and contain appropriate detail.

[0163] Figure 9 shows an example of a range-compressed LDR image 900. For example, image 900 may be locally tone-mapped from HDR image 600. Local tone mapping can be performed using any of the various tone mapping techniques. For example, the dark areas of image 700 may have their brightness increased while maintaining the contrast at the edges of the image. Thus, image 900 has a smaller dynamic range than image 600 (this is not perceptible in these figures), but can maintain more detail than image 700 in Figure 7 and image 800 in Figure 8. Range-compressed images such as image 900 may be used as LDR images in the image formats described herein and may be contained in the image containers described as LDR versions of a given HDR image.

[0164] Figure 10 is a block diagram of an exemplary apparatus 1400 that may be used to perform one or more functions described herein. In one example, apparatus 1000 may be used to perform a client device, such as any of the client devices 120-126 shown in Figure 1. Alternatively, apparatus 1000 may perform a server device, such as server device 104. In some implementations, apparatus 1000 may be used to perform a client device, a server device, or both a client device and a server device. As previously stated, apparatus 1000 may be any suitable computer system, server, or other electronic or hardware device.

[0165] One or more methods described herein can operate in several environments and platforms, for example, as a standalone computer program that can run on any type of computing device, a web application having a web page, a program that runs on a web browser, or a mobile application ("app") that runs on a mobile computing device (e.g., a mobile phone, smartphone, tablet computer, wearable device (such as a watch, armband, jewelry, hat, virtual reality goggles or glasses, augmented reality goggles or glasses, head-mounted display, etc.), laptop computer, etc.). In one example, a client / server architecture may be used, for example, a mobile computing device (as a client device) sends user input data to a server device and receives final output data for output (e.g., for display) from the server. In another example, all computations may be performed within a mobile app (and / or other app) on a mobile computing device. In yet another example, computations may be divided between a mobile computing device and one or more server devices.

[0166] In some implementations, the device 1000 includes a processor 1002, memory 1004, and an input / output (I / O) interface 1006. The processor 1002 may be one or more processors and / or processing circuits for executing program code to control the basic operation of the device 1000. "Processor" includes any suitable hardware system, mechanism, or component for processing data, signals, or other information. A system that a processor may include is a general-purpose central processing unit (CPU) having one or more cores (e.g., single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), dedicated circuitry for implementing functionality (e.g., one or more hardware image decoders and / or video decoders), a dedicated processor for performing processing based on a neural network model, a neural circuit, a processor optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some implementations, processor 1002 may include one or more coprocessors that perform neural network processing. In some implementations, processor 1002 may be a processor that processes data and produces a probabilistic output, for example, the output produced by processor 1002 may be inaccurate or accurate to some extent from the expected output. Processing does not need to be limited to a specific geographical location, nor does it have to be time-limited. For example, the processor's functions may be performed in "real-time," "offline," or "batch mode." Parts of the processing may be performed by separate (or identical) processing systems at different times and locations. The computer may be any processor that communicates with memory.

[0167] Memory 1004 is generally provided within the device 1000 to be accessed by the processor 1002 and may be any suitable processor-readable storage medium such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), or flash memory, suitable for storing instructions executed by the processor, and may be located separately from the processor 1002 or may be incorporated into the processor 1002. Software that may be stored in memory 1004 includes an operating system 1008, an image application 1010 (which may be, for example, the image application 106 in Figure 1), other applications 1012, and application data 1014, which are run on the server device 1000 by the processor 1002. Other applications 1012 may include applications such as a data display engine, a web hosting engine, a map application, an image display engine, a notification engine, a social networking engine, a media display application, a communication application, a web hosting engine or application, or a media sharing application. In some implementations, the image application 1010 may include instructions that enable the processor 1002 to perform some or all of the functions described herein, such as the methods shown in Figures 2–5 and / or Figures 10–12. In some implementations, images stored in the format described herein (including recovery maps, metadata, etc.) may be stored as application data 1014 or other data in memory 1004 and / or other storage devices of one or more other devices communicating with device 1000. In some examples, the image application 1010 or other applications stored in memory 1004 may include image coding / video coding and container generation modules (for example, performing the methods shown in Figures 2–4) and / or image decoding / video decoding modules (for example, performing the method shown in Figure 5), or such modules may be incorporated into a few or a single module or application.

[0168] Any software in memory 1004 may instead be stored in another suitable storage location or computer-readable medium. In addition, memory 1004 (and / or other connected storage devices) may store one or more messages, one or more classifications, electronic encyclopedias, dictionaries, digital maps, thesauruses, knowledge bases, message data, grammars, user preferences, and / or other instructions and data used in the functions described herein. Memory 1004 and other types of storage mechanisms (magnetic disks, optical disks, magnetic tapes, or other tangible media) may be considered “storage mechanisms” or “storage devices.”

[0169] The I / O interface 1006 may provide the functionality to enable the server device 1000 to interface with other systems and devices. The interfaced devices may be included as part of device 1000 or separate and able to communicate with device 1000. For example, network communication devices, storage devices (e.g., memory and / or databases), and input / output devices can communicate via the I / O interface 1006. In some implementations, the I / O interface may be connected to interface devices such as input devices (keyboards, pointing devices, touchscreens, microphones, cameras, scanners, sensors, etc.) and / or output devices (display devices, speaker devices, printers, motors, etc.).

[0170] Some examples of interfaced devices that can be connected to the I / O interface 1006 may include one or more display devices 1020 that can be used to display content such as images, videos, and / or user interfaces for applications as described herein. The display device 1020 can be any suitable display device and may be connected to device 1000 by local connections (e.g., display buses) and / or network connections. The display device 1020 may include any suitable display device such as an LCD, LED, or plasma display screen, a CRT, television, monitor, touchscreen, 3D display screen, or other visual display device. The display device 1020 may also function as an input device, such as a touchscreen input device. For example, the display device 1020 may be a flat display screen of a mobile device, multiple display screens of a glasses or headset device, or a monitor screen of a computer device.

[0171] The I / O interface 1006 can interface with other input and output devices. Some examples include one or more cameras capable of capturing images and / or detecting gestures. Some implementations may include a microphone for capturing sound (e.g., as part of captured images, voice commands, etc.), a radar or other sensor for detecting gestures, and an audio speaker or other input or output device for outputting sound.

[0172] For ease of illustration, Figure 10 shows one block each of the processor 1002, memory 1004, I / O interface 1006, and software blocks 1008-1014. These blocks may represent one or more processors or processing circuits, operating systems, memory, I / O interfaces, applications, and / or software modules. In other implementations, the device 1000 may not have all of the components shown and / or may have other components, including other types of elements, instead of or in addition to those shown herein. While some components are described as performing blocks and operations in some implementations described herein, any suitable component or combination of components of any suitable processor associated with the environment 100, the device 1000, a similar system, or any suitable processor associated with such a system may perform the described blocks and operations.

[0173] The methods described herein may be implemented by computer program instructions or code that can be executed on a computer. For example, code may be implemented by one or more digital processors (e.g., microprocessors or other processing circuits) and stored in computer program products including non-temporary computer-readable media (e.g., storage media) such as magnetic storage media, optical storage media, electromagnetic storage media, or semiconductor storage media, semiconductor memory, i.e., solid-state memory, including magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), flash memory, rigid magnetic disks, optical disks, solid-state memory drives, etc. Program instructions may also be contained in electronic signals and supplied as electronic signals, for example, in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more methods may be implemented by hardware (e.g., logic gates), or a combination of hardware and software. Exemplary hardware may include programmable processors (e.g., field-programmable gate arrays (FPGAs), complex programmable logic devices), general-purpose processors, graphics processors, application-specific integrated circuits (ASICs), etc. One or more methods may be executed as part or component of an application running on the system, or as an application or software that operates in relation to other applications and the operating system.

[0174] While the explanation has been given with reference to specific implementations, these specific implementations are merely examples and not limiting. The concepts shown in the examples may be applicable to other examples and implementations.

[0175] In addition to the above description, users may be provided with controls that allow them to choose whether, and if so, the systems, programs, or functions described herein enable the collection of user information (for example, information about the user's social network, social activities, or business, occupation, user preferences, or user's current location) and whether the user receives content or communications from the server. In addition, certain data may be processed in one or more ways such that personal information is removed before it is stored or used. For example, user identification information may be processed so that personal information about the user cannot be determined, or if location information (such as city, zip code, or state level) is obtained, the user's geographical location may be generalized so that the user's specific location cannot be determined. Thus, users can control what information is collected about them, how that information is used, and what information is provided to them.

[0176] It should be noted that the functional blocks, operations, functions, methods, apparatus, and systems described herein may be integrated or separated into various combinations of systems, apparatus, and functional blocks known to those skilled in the art. Any suitable programming language and programming technique may be used to perform routines in a particular implementation. Various programming techniques, such as procedural or object-oriented, may be employed. Routines may be executed on a single processing unit or on multiple processors. Steps, operations, or calculations may be presented in a specific order, but in another particular implementation, the order may be changed. In some implementations, multiple steps or operations shown herein as being performed sequentially may be performed simultaneously.

Claims

1. A method by which a computer performs an action. A step of obtaining a first image that depicts a specific scene having a first dynamic range, The steps include: acquiring a second image that depicts a specific scene having a second dynamic range smaller than the first dynamic range; The process includes the steps of generating a recovery map based on the first image and the second image, wherein the recovery map encodes the luminance gain between a portion of the first image and a corresponding portion of the second image, and the gain encoded in the recovery map is scaled by a range scaling coefficient determined based on the ratio of the highest luminance of the second image to the highest luminance of the first image and normalized within a specific range. The aforementioned method, The process includes the step of supplying the first image and the recovery map to an image container, wherein the image container is readable for displaying a derived image based on applying the recovery map to the first image, and the derived image has a dynamic range smaller than the first dynamic range. method.

2. The method according to claim 1, wherein the step of generating the recovery map includes the step of encoding the gain such that applying the gain to the brightness of individual pixels of the first image results in pixels corresponding to the second image.

3. The method according to claim 2, wherein the step of generating the recovery map includes the step of encoding the gain in logarithmic space, and the recovery map value is proportional to the value obtained by dividing the logarithmic difference of the luminance by the logarithm of the range scaling coefficient.

4. The method according to claim 2, wherein the step of generating the recovery map includes the step of encoding the recovery map into a bidirectional grid, the encoding step includes the step of determining a three-dimensional data structure of grid cells, each grid cell being a vector element mapped to a plurality of pixels of the first image.

5. The steps include obtaining the aforementioned image container, The steps include deciding to display the second image using a display device, To obtain the derived image, the steps include scaling the luminance of multiple pixels of the first image in the image container based on a specific luminance output of the display device and the recovery map, After the scaling step, the display device displays the derived image as an output image having a different dynamic range from the first image. The method according to claim 1, further comprising:

6. The method according to claim 1, wherein the step of acquiring the second image includes the step of performing range compression on the first image.

7. The aforementioned image container is The first image is displayed by a first display device capable of displaying the first dynamic range. The derived image is displayed using a second display device that can only display a dynamic range smaller than the first dynamic range. The method according to claim 1, which is readable in this manner.

8. A method performed by a computer, which includes the step of obtaining an image container, The aforementioned image container is A first image having a first dynamic range, and A recovery map has a second image which encodes the luminance gain of pixels in the first image, which is scaled by a range scaling coefficient determined based on the ratio of the highest luminance of the second image to the highest luminance of the first image and normalized within a specific range, wherein the second image corresponds to the first image of the subject being depicted and has a second dynamic range smaller than the first dynamic range. The aforementioned method, A step of determining whether to display the first image or a derived image having a dynamic range smaller than the first dynamic range, In response to determining that the first image is to be displayed, The steps include: causing the first image to be displayed on the display device, In response to determining that the aforementioned derived image should be displayed, The steps include: applying the luminance gain of the recovery map to the luminance of the pixels in the first image to determine the corresponding pixel values ​​of the derived image; The steps of causing the derived image to be displayed on the display device, Methods that include...

9. The method according to claim 8, further comprising the step of determining the maximum brightness display capability of the display device, wherein in response to determining that the display device can only display a display dynamic range greater than or equal to the first dynamic range, it is decided to display the first image.

10. The method according to claim 8, further comprising the step of determining the maximum brightness display capability of the display device, wherein the display device determines to display the derived image in response to determining that it can display a display dynamic range smaller than the first dynamic range.

11. The method according to claim 8, wherein the display device is capable of displaying a display dynamic range smaller than the first dynamic range, and the step of applying the luminance gain of the recovery map includes the step of adjusting the luminance of the pixels of the derived image to the display dynamic range of the display device.

12. The step of applying the luminance gain of the recovery map to the luminance of the pixels in the first image is: The method according to claim 8, further comprising the step of scaling the brightness of the first image based on a specific brightness output of the display device and the recovery map.

13. The scaling of the brightness of the first image is performed based on the following equation: derived_image(x,y)=first_image(x,y)+log(display_factor)*recovery(x,y) derived_image(x,y) is the logarithmic space version of the derived image, first_image(x,y) is the logarithmic version of the first image recovered from the image container, display_factor is the smaller of the range scaling factor and the highest display brightness of the display device, recovery(x,y) is the recovery map of the pixel positions (x,y) of the first image, and the range scaling factor is the ratio of the highest brightness of the second image to the highest brightness of the first image. The method according to claim 8.

14. The method according to claim 8, further comprising the step of extracting the recovery map from a bidirectional grid stored in the image container.

15. The method according to claim 8, wherein the step of the display device displaying the derived image includes a step of lowering the maximum brightness level of the derived image to be displayed, based on the system settings of the device that displays the derived image, the system settings being selected by the user.

16. The step of displaying the derived image is: The steps include: displaying the derived image at a first maximum brightness level that is less than the second maximum brightness level determined for the pixel values ​​of the derived image by applying the brightness gain; The step of gradually increasing the first maximum brightness level of the derived image up to the second maximum brightness level over a specific period of time, The method according to claim 8.

17. Processor and A system comprising a memory that stores instructions and is coupled to the processor, wherein the processor executes the instructions, A first image having a first dynamic range, and A recovery map that encodes the luminance gain of the pixels in the first image. Perform an operation that includes the step of obtaining an image container containing, The recovery map encodes the luminance gain of the pixels of the first image, which is scaled by a range scaling coefficient that includes the ratio of the highest luminance of the second image to the highest luminance of the first image and normalized within a specific range, wherein the second image corresponds to the first image of the subject being depicted and has a second dynamic range different from the first dynamic range. The operation includes the step of deciding to display a derived image on a display device, The derived image corresponds to the first image of the subject being depicted and has a dynamic range smaller than the first dynamic range. The aforementioned operation is, In response to deciding to display the derived image on the display device, The process includes the step of applying the luminance gain of the recovery map to the luminance of the pixels in the first image to determine the corresponding pixel values ​​of the derived image, wherein applying the luminance gain includes scaling the luminance of the first image based on a specific luminance output of the display device and the recovery map. The step includes causing the derived image to be displayed on the display device, system.

18. The aforementioned instruction is given to the processor, Further operations are performed, including the step of determining the maximum brightness display capability of the display device. In response to determining that the display device can display a display dynamic range smaller than the first dynamic range, it decides to display the derived image. The system according to claim 17.

19. A program that, when executed by a computer processor, causes the computer to perform the method described in any one of claims 1 to 16.

Images