Multi-intent image and video encoding and decoding with metadata

Abstract

A system and method for encoding and decoding multi-intent images and videos using metadata. When encoding an image as a multi-intent image, at least one appearance adjustment may be made to the image. Metadata characterizing the at least one appearance adjustment may be included in or transmitted with the encoded multi-intent image. When decoding the multi-intent image, the system may obtain a selection of a desired rendering intent and, based on the selection, render the multi-intent image with the applied appearance adjustments or use the metadata to reverse the appearance adjustments and restore the image before the appearance adjustments.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 251,427, filed October 1, 2021, and European Patent Application No. 21208445.3, filed November 16, 2021, all of which are incorporated herein by reference in their entireties. Field of Disclosure This application relates generally to systems and methods for image encoding and decoding. [Background technology]

[0002] Non-Patent Document 1 proposes color mapping side information in the SET message to ensure smooth color space transitions for future HDTV and multiphase UHDTV service deployments. It is stated that the proposed mapping helps preserve the artistic intent of studio-generated content while maintaining differentiation among TV set manufacturers. This idea was first published in JCTVC-N0180. The intent of the proposed SET message was clarified in JCTVC-00363. Furthermore, in JCTVC-P0126, complexity concerns are addressed by simplifying the color mapping model. Finally, editing issues and synchronization aspects are addressed in this proposal. Software is provided to identify the proposed model parameters. Implementation is provided in HM-13.0+RExt-6.0 encoders and decoders. When the proposed color mapping information SEI message is present, the color mapping is applied to the decoded output picture. Patent Document 1 discloses an image processing device and method that can easily improve encoding efficiency. The device and method include a setting unit that sets additional information including packing information related to packing processing that rearranges each pixel data of RAW data, which is image data before demosaic processing, according to the degree of correlation, and an encoding unit that encodes the packed RAW data and generates a bitstream that includes the resulting encoded data and the additional information set by the setting unit.

[0003] Non-Patent Document 2 proposes a definition for High Dynamic Range (HDR) and related technologies, describes current gaps in the ecosystem for creating, distributing, and displaying HDR-related content, identifies existing standards that may be affected by the HDR ecosystem, including Wide Color Gamut (WGC), and identifies areas where implementation issues may require further investigation. This report focuses on professional applications and does not explicitly discuss home delivery.

[0004] Patent Document 2 discloses a method and system for generating and applying scene-stable metadata for a video data stream. The video data stream may be divided or segmented into scenes, and a first set of metadata may be generated for a given scene of the video data. The first set of metadata may be any known metadata, such as a desired feature of the video content (e.g., luminance). The first set of metadata may be generated for each frame. Scene-stable metadata is generated, which may differ from the first set of metadata for the scene. The scene-stable metadata is generated by monitoring the desired feature along with the scene and is used to keep the desired feature within an acceptable range of values. This may help avoid noticeable and potentially unpleasant visual artifacts when rendering the video data.

[0005] Patent Document 3 discloses an apparatus and method for providing a solution to the problem of preserving the original creative intent for video playback on a target. A video bitstream contains metadata with flags indicating the creative intent for the target display. This metadata includes multiple fields representing characteristics such as content type, content subtype, intended white point, whether the video will be used in reference mode, intended sharpness, intended noise reduction, intended MPEG noise reduction, intended frame rate conversion, intended average picture level, and intended color. This metadata is designed to facilitate content creators' tagging of content. Metadata can be added to video content at multiple points, and the status of the flags is set to TRUE or FALSE to indicate whether the metadata was added by the content creator or a third party. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] US Patent Application Publication No. 2016 / 261889 [Patent Document 2] US Patent Application Publication No. 2016 / 254028 [Patent Document 3] International Publication No. 2020 / 264409A1 [Non-patent literature]

[0007] [Non-Patent Document 1] Pierre Andrivon et al., "SEI message for Color Mapping Information", Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 17th Meeting: Valencia, ES, 27 March-4 April 2014, no. JCTVC-Q0074, 2 April 2014, XP030239839 [Non-patent document 2] "Study Group Report High-Dynamic-Range (HDR) Imaging Ecosystem", SMPTE Technical Committee (TC) 10E SG, 19 September 2015, XP055250336 Summary of the Invention [Means for solving the problem]

[0008] The invention is defined by the independent claims. Dependent claims relate to optional features of some embodiments of the invention. When encoding images of a scene captured using a digital device, it is common practice to adjust the captured image by, for example, adapting the image for viewing in a reference viewing environment and applying aesthetic adjustments such as enhanced contrast and color saturation. It would be desirable to be able to transmit an original captured or preprocessed image representing the "reality" captured by the imaging sensor and then apply these operations during playback. This allows for multiple rendering intents. That is, during playback, the device can present the original captured "real" image, or alternatively, the device can create a modified "pleasant" image from the original captured "real" image. Thus, techniques have been developed for encoding and decoding multi-intent images.

[0009] Various aspects of the present disclosure relate to devices, systems, and methods for encoding and decoding one or more multi-intent images.

[0010] In one exemplary aspect of the present disclosure, a method for encoding a multi-intent image is provided, the method including obtaining an image for encoding as a multi-intent image, applying at least one appearance adjustment to the image, generating metadata characterizing the at least one appearance adjustment, and encoding the image and the metadata as the multi-intent image.

[0011] In another exemplary aspect of the present disclosure, a method for decoding a multiple-intent image is provided, the method including obtaining a multiple-intent image along with metadata characterizing at least one appearance adjustment between the multiple-intent image and an alternative version of the multiple-intent image, obtaining a selection of the alternative version of the multiple-intent image, and applying an inverse of the at least one appearance adjustment to the multiple-intent image using the metadata to restore the alternative version of the multiple-intent image.

[0012] In another exemplary aspect of the present disclosure, a method for providing a multiple-intent image is provided, the method including obtaining an original image for encoding as the multiple-intent image, generating metadata characterizing at least one appearance adjustment to the original image, encoding the original image and the metadata as the multiple-intent image, and providing the multiple-intent image.

[0013] In another exemplary aspect of the present disclosure, a non-transitory computer-readable medium is provided having stored thereon instructions that, when executed by a processor, cause the processor to perform operations including obtaining an image for encoding as a multi-intent image, applying at least one appearance adjustment to the image, generating metadata characterizing the at least one appearance adjustment, and encoding the image and metadata as the multi-intent image.

[0014] In another exemplary aspect of the present disclosure, a non-transitory computer-readable medium is provided having stored thereon instructions that, when executed by a processor, cause the processor to perform operations including obtaining a multi-intent image along with metadata characterizing at least one appearance adjustment between the multi-intent image and an alternative version of the multi-intent image, obtaining a selection of the alternative version of the multi-intent image, and applying an inverse of the at least one appearance adjustment to the multi-intent image using the metadata to restore the alternative version of the multi-intent image.

[0015] In another exemplary aspect of the present disclosure, a non-transitory computer-readable medium is provided having stored thereon instructions that, when executed by a processor, cause the processor to perform operations including obtaining an original image for encoding as a multi-intent image, generating metadata characterizing at least one appearance adjustment to the original image, encoding the original image and the metadata as the multi-intent image, and providing the multi-intent image.

[0016] In this manner, various aspects of the present disclosure provide for multi-intent image and video encoding, decoding and provisioning, resulting in improvements in at least the fields of image encoding, image decoding, image projection, image display, holography, signal processing, and the like. [Brief explanation of the drawings]

[0017] These and other more detailed and specific features of the various embodiments are more fully disclosed in the following description, which refers to the accompanying drawings.

[0018] [Figure 1] An exemplary process for an image encoding and decoding pipeline is shown.

[0019] [Figure 2] 1 illustrates an exemplary process for encoding and decoding multi-intent images and videos.

[0020] [Figure 3] 1 illustrates an exemplary process for encoding multi-intent images and videos.

[0021] [Figure 4] 1 illustrates an exemplary process for decoding multi-intent images and videos. DETAILED DESCRIPTION OF THE INVENTION

[0022] The present disclosure and aspects thereof may be embodied in various forms, including hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces and application programming interfaces, as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. The foregoing is intended only to give a general idea of ​​various aspects of the present disclosure and is not intended to limit the scope of the disclosure in any way.

[0023] In the following description, numerous details are set forth, such as optical device configurations, timing, operation, etc., to provide an understanding of one or more aspects of the present disclosure. It will be readily apparent to those skilled in the art that these specific details are merely examples and are not intended to limit the scope of the present application.

[0024] FIG. 1 illustrates an exemplary process of an image delivery pipeline (100), showing various stages from image capture to image content display. Images (102), which may include a sequence of video frames (102), are captured or generated using an image generation block (105). The images (102) may be captured digitally (e.g., by a digital camera) or computer-generated (e.g., using computer animation) to provide image data (107). Alternatively, the images (102) may be captured on film by a film camera. The film is converted to a digital format to provide the image data (107). In a production phase (110), the image data (107) is edited to provide an image production stream (112).

[0025] The image data in the production stream (112) is then provided to a processor (or one or more processors, such as a central processing unit (CPU)) in block (115) for post-production editing. The post-production editing in block (115) may include adjusting or modifying color or brightness in specific areas of the image to improve image quality or achieve a particular appearance of the image according to the image creator's creative intent. This is sometimes referred to as "color timing" or "color grading." The methods described herein may be performed by the processor in block (115). Other editing (e.g., scene selection and alignment, image cropping, addition of computer-generated visual special effects, etc.) may be performed in block (115) to provide a final version of the production (117) for distribution. During post-production editing (115), the images or video images are viewed on a reference display (125). The reference display (125) may be a consumer-level display or projector, if desired.

[0026] Following post-production (115), the image data of the final production (117) may be delivered to an encoding block (120) for downstream delivery to a decoding and playback device, such as a computer monitor, television set, set-top box, or movie theater. In some embodiments, the encoding block (120) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other delivery formats, to generate an encoded bitstream (122). At the receiver, the encoded bitstream (122) is decoded by a decoding unit (130) to generate a decoded signal (132) that represents the same or a close approximation of the signal (117). The receiver may be attached to a target display (140), which may have characteristics completely different from the reference display (125). In that case, a display management block (135) may be used to map the dynamic range of the decoded signal (132) to the characteristics of the target display (140) by generating a display-mapped signal (137). Additional methods described herein may be performed by the decode unit (130) or the display management block (135). Both the decode unit (130) and the display management block (135) may include their own processors or may be integrated into a single processing unit. While this disclosure refers to a target display (140), it will be understood that this is merely an example. It will be further understood that the target display (140) may include any device configured to display or project light, such as a computer display, a television, an OLED display, an LCD display, a quantum dot display, cinema, consumer, and other commercial projection systems, a head-up display, a virtual reality display, etc.

[0027] When capturing a scene using a digital device, realistic scene-referred radiometry is rarely directly transferred to produce an image. Instead, it is common practice for the device's original equipment manufacturer (OEM) or software application designer to adjust the image by adapting it for viewing in a reference viewing environment, such as dim ambient light and D65 lighting, and applying aesthetic adjustments such as increased contrast and color saturation. These and other adjustments produce a pleasing rendering of reality that consumers find pleasing.

[0028] Currently, these operations are lossy in two ways: first, the parameters used to apply the operations are not transmitted, and second, pixel operations can be lossy due to nonlinear clipping and quantization, lossy operations, unknown algorithms, or unknown operation orders.

[0029] Instead, it is desirable to be able to transmit the original captured / pre-processed image that represents the "reality" captured by the imaging sensor, and then apply these operations during playback. This allows for multiple rendering intents; during playback, the device can present the original captured "reality" image, or alternatively, the device can create a modified "pleasant" image from the original captured "reality" image.

[0030] It would also be desirable to allow such content to be transmitted in a backwards-compatible manner, where modifications to create a "pleasant" image could be applied at the time of capture and appropriate parameters could be transmitted to a playback device, allowing the playback device to reverse the modifications and thus restore the originally captured "real" image.

[0031] 2 provides a method 200 for allowing encoding and decoding of images with multiple intents using metadata. The method 200 may be performed by a processor, for example, as part of blocks 115 and / or 120 for encoding and as part of blocks 130 and / or 135 for decoding.

[0032] In step (202), an image is captured. In a digital capture device, the exposed scene is converted into raw sensor values ​​in a one-channel representation. Through a process known as demosaicing, the one-channel image representation is expanded into a three-color representation with three channels, e.g., red, green, and blue (RGB). There are many techniques for demosaicing, any of which will suffice in the embodiments disclosed herein.

[0033] To fully capture the colorimetry of a scene, the spectral sensitivities of the capture device should match the spectral sensitivities of the viewer. In practice, these often do not match exactly and are instead approximated using a 3×3 matrix transformation to convert the sensor sensitivities to some set of desired RGB primaries. Traditionally, the camera spectral sensitivities are not transmitted with the content during this step, making the process irreversible. In one embodiment of the present invention, the applied camera spectral sensitivities and the 3×3 matrix transformation are transmitted with the content, allowing the playback device to either apply or invert the transformation from the sensor output to the specified RGB primaries. Step (202) may include, by way of non-limiting example, reading single-channel values ​​from the sensor, applying a demosaicing scheme to create a three-color-channel (e.g., RGB) image, and optionally applying a 3×3 transformation to match the image sensitivities to the desired three-color (e.g., RGB) primaries. Step (202) may also include measuring the capture ambient luminance (e.g., the level of ambient light in the capture environment).

[0034] Once the desired RGB configuration of a captured image is determined, the values ​​can be matched to a specified reference white point. The image can be matched to one of the standardized white points (D50, D65, etc.) through a Von Kries adaptive transform. This process involves (a) estimating the ambient illuminance and white point of the capture environment and (b) applying corrections to the image to achieve a color match for an observer in a specified reference viewing environment (e.g., an environment with a known white point and ambient illumination). Methods used to adjust images to match the adaptation state of an observer in a chromatic ambient environment are outlined in PCT Application Nos. PCT / US2021 / 027826, filed April 16, 2021, and PCT / US2021 / 029476, filed April 27, 2021, each of which is incorporated by reference in its entirety for all purposes. In step (204), one or more optional source appearance adjustments can be applied to the captured image, including, but not limited to, white balance adjustments, color correction adjustments, and optical-optical transfer function (OOTF) adjustments. Step (204) can include calculating a nonlinear optical-optical transfer function (OOTF) to map from the measured capture ambient luminance to a reference review environment. The white point adjustment and the order of the 3×3 matrix can be varied. Calculating and applying the optical-optical transfer function (OOTF) can establish the intended rendering of the image on a standard display device. In practice, the OOTF is applied to map an image from the viewing environment at the time of capture for display in the reference viewing environment. The application of the OOTF today is an irreversible operation, making it difficult to invert the OOTF during playback. Similar to the white point adjustment, in a first step (a), the ambient illumination of the capture environment can be estimated, and in a second step (b), the image can be corrected to achieve a match for an observer in the reference environment.

[0035] In step (206), one or more optional source preference adjustments may be applied to the captured image, including, but not limited to, contrast adjustments, saturation adjustments, including global and / or individual saturation adjustments, tone curve slope-offset-power-Tmid adjustments, and other tone curve trims and adjustments. As used herein, "mid" refers to the average of the image's maxRGB values ​​in a perceptually quantized (PQ) encoded image, where each pixel has its own maxRGB value equal to the pixel's maximum color component value (R, G, or B). In other words, whichever color component of a pixel has the largest value is the maxRGB value for that pixel, and the average of the maxRGB values ​​across the PQ encoded image is the image's "mid." "T-mid" may refer to a "target mid," which may be the "mid" value a user or content creator desires in the final image. In some embodiments, the individual color saturation adjustments may include saturation adjustments in six different colors, which may be referred to as a "six-vector adjustment."

[0036] Steps (206) and (208) can include receiving a selection of intent from the user in step (208), the selection of intent specifying what source appearance and source preference adjustments are to be made, the coefficients of such adjustments, to which portions of the image the adjustments are to be applied, etc.

[0037] It is common practice for OEMs or software applications to apply source preference adjustments to captured images. These changes are purely aesthetic and are typically introduced to render images with higher levels of contrast and saturation. In various embodiments of the present disclosure, these preference changes determined by the OEM are transmitted as metadata with the content and applied during playback in the same manner as source appearance metadata. In each case, there is a first step of (a) calculating or specifying the desired amount of correction to apply, and (b) applying the correction using a parameterized function. Both (a) and (b) are transmitted as metadata, allowing playback devices full flexibility to render either "pleasant" or "real" images, and allowing capture devices full flexibility to transmit either "pleasant" or "real" images.

[0038] As described herein, one advantage of the various embodiments disclosed herein is that any adjustments to the three-channel image may be encoded as metadata and sent along with the content to the playback device for application. In one embodiment, the OEM or encoding device may decide not to apply adjustments for both appearance and preference to produce a "real-world" image.

[0039] Step (210) may encode the image modified in steps (206) and (208). Step (210) may include encoding the image for downstream delivery to a decoding and playback device, such as a computer monitor, television set, set-top box, movie theater, etc. In some embodiments, encoding step (210) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other delivery formats, generating coded bitstreams. In addition to encoding the image, step (210) may include creating and / or encoding metadata characterizing the source appearance adjustments applied in step (204) and the source preference adjustments applied in step (206). The metadata may include metadata related to source appearance adjustments, such as the scene white point specified in x,y coordinates (or some other system), the scene ambient luminance specified in lux (or some other system) (e.g., information about the estimated capture environment), the coefficients of the applied white point adjustment matrix, the coefficients of the applied 3x3 color matrix, the coefficients of the applied parameterized OOTF, the spectral sensitivity of the sensor used to calculate the 3x3 matrix, and coefficients or other information for other enhancements applied in step (204). Additionally, the metadata includes metadata related to source preference adjustments, such as coefficients for contrast enhancement, e.g., slope-offset-power-Tmid contrast adjustment, coefficients for saturation enhancement, coefficients for individual color saturation adjustment, coefficients for tone curve trim, and coefficients for other enhancements applied in step (206).

[0040] In step (212), the encoded image and metadata may be decoded. In step (214), a desired rendering intent selection may be obtained. As a first example, a selection may be obtained to render the image modified by the source appearance adjustments and source preference adjustments of step (204). As second and third examples, a selection may be obtained to render the image as if it had been modified by the source appearance adjustments of step (204) but not by the source preference adjustments of step (206) (or vice versa). As a fourth example, a selection may be obtained to render the image as if it had not been modified by the source appearance adjustments of step (204) or the source preference adjustments of step (206). In the fourth example, the image captured in step (202) may be partially or fully restored. In some embodiments, the rendering intent selection obtained in step (214) may be based on a user selection at the playback device. In some embodiments, a default rendering intent may be specified during the encoding process, and in the absence of user input to the contrary, the default rendering intent may be selected. In some embodiments, the default rendering intent may include rendering the image with the source appearance adjustments of step (204) and the source preference adjustments of step (206) applied.

[0041] In optional step 216, the metadata can be used to calculate inverted source preference adjustments. When applied, the inverted source preference adjustments of step 216 can undo some or all of the source preference adjustments of step 206, with the user selection and default rendering intent identifying which of the source preference adjustments are inverted.

[0042] In optional step 218, the metadata may be used to calculate inverted source appearance adjustments. When applied, the inverted source appearance adjustments of step 218 may undo some or all of the source appearance adjustments of step 204, with the user selection and default rendering intent identifying which of the source appearance adjustments are inverted.

[0043] In an optional step (220), a target appearance adjustment may be calculated and applied. The target appearance adjustment may include, by way of non-limiting example, measuring the display ambient luminance (e.g., the ambient light level in the display environment) and then calculating and applying a nonlinear optical-to-optical transfer function (OOTF) to map from the reference viewing environment to the measured display ambient luminance (e.g., the actual viewing environment).

[0044] In optional step 222, target preference adjustments may be calculated and applied, which may include, by way of non-limiting example, contrast adjustments, color saturation adjustments, slope-offset-power-Tmid adjustments, individual color saturation adjustments, and tone curve trims.

[0045] In step 224, the image may be rendered. For example, the image may be projected, displayed, saved to a storage device, transmitted to another device, or otherwise utilized.

[0046] In some embodiments, the inversion of the source adjustment and the application of the target adjustment are combined into a single processing step, and the adjustments are calculated accordingly. In other words, some or all of steps 216, 218, 220, and 222 may be combined.

[0047] In some embodiments, the rendering intent selected in step (208) is for a "real" image, and steps (204 and 206) are essentially bypassed. This corresponds to delivery of a "real" image. The metadata in such embodiments indicates that no source appearance adjustments and no source preference adjustments were made.

[0048] In some other embodiments, several source appearance adjustments and source preference adjustments are applied (e.g., in steps (204) and 206) to produce a “pleasant” image. The metadata in such embodiments may indicate the amount and type of source appearance and source preference adjustments applied. The metadata may include multiple values, each corresponding to a parameter controlling a particular function applied as a source appearance and / or preference adjustment. These functions can be reversed (or approximately reversed) by a playback device by knowing the exact functions applied, the order in which they were applied, and the parameters controlling the strength of the functions. The metadata may be configured to include the information needed by a playback device to reverse (or approximately reverse) these functions.

[0049] If desired, the metadata created in step (210) may be used to transmit a "desired rendering intent" for the content, specifying a default value for how images are to be processed during playback (whether a "realistic" image or a "pleasant" image is displayed). This may be a Boolean value or a continuously varying scale between the two. The playback device interprets this metadata as a "desired rendering intent," inverts source appearance and preference adjustments according to the source adjustment metadata, and applies target appearance adjustments according to the viewing environment. If desired, the "desired rendering intent" specified in the metadata may be overridden upon receipt of user input.

[0050] 3 provides a method 300 for allowing encoding of images with multiple intents using metadata. The method 300 may be performed by a processor, for example, as part of block 115 and / or block 120 for encoding.

[0051] In step (302), an image is captured by exposing a scene to a sensor. In step (304), raw sensor values ​​for each color channel are collected. In step (306), a demosaicing algorithm or process can be used to convert the raw sensor values ​​from each color channel into a multi-channel color image (e.g., a three-channel color image having three primary colors). In step (308), a 3x3 matrix transform can be applied to the multi-channel color image to convert the raw sensor values ​​into a desired set of primary colors, such as RGB primaries. The 3x3 matrix transform in step (308) can function to account for differences in sensor sensitivity between different color channels. In step (310), the image can be matched to a reference white point using one or more white balance adjustments, color correction adjustments, etc. In step (312), an optical-to-optical transfer function (OOTF) can be applied to map, by way of example, the ambient luminance in the capture environment to the luminance of a reference review environment. In step (314), one or more source preference adjustments may be applied, including, but not limited to, contrast adjustments, saturation adjustments, slope-offset-power-Tmid adjustments, individual saturation adjustments, and tone curve trims. Following step (314), the image is encoded and metadata is generated, allowing for the potential reversal of any source preference and source appearance adjustments made during method (300).

[0052] 4 provides a method 400 for allowing decoding of images with multiple intents using metadata. The method 400 may be performed by a processor, for example, as part of block 130 and / or block 135 for decoding.

[0053] In step (402), the multi-intention image and its corresponding metadata are decoded.

[0054] After decoding the image and metadata on the playback device, there are multiple options regarding the rendering intent of the displayed image. In one embodiment, the selected (or preferred) intent is present in the metadata as a flag or profile that guides the operation of the target / receiving device to accept the desired adjustments in both the appearance and preference domains. In another embodiment, the final rendered image may not involve accepting appearance or preference adjustments. Another embodiment involves the rendered image undergoing adaptations for appearance phenomena but not for preference (or vice versa). These intents do not need to be binary, as partial application of determined adjustments for appearance and preference phenomena is possible.

[0055] In step (404), the desired rendering intent is obtained, for example, from defaults specified in metadata, from user input, etc.

[0056] Once intent is established for the target device, source image-based adjustments may need to be reversed. Both appearance and preference adjustments made to images on the source side of the pipeline are decoded from accompanying metadata files. Based on the applied adjustments known from the metadata, the inverse can be determined, if necessary. In embodiments where the OEM decides not to apply image adjustments, there is no need to calculate the inverse of the source and the target can be applied directly. For all other embodiments, if it is desired not to apply source image-based adjustments (e.g., if it is desired to reverse the source image-based adjustments), the inverse adjustments can be calculated.

[0057] In step (406), inverted source preferences and appearance adjustments are calculated based on, for example, metadata.

[0058] Because source preference adjustments are applied last before encoding, they may need to be inverted first after decoding. The inverse preference adjustment undoes any additional image processing performed for aesthetic purposes specified by the metadata (e.g., in one embodiment, modifying image contrast and saturation). Following this, the source appearance adjustments are inverted via metadata describing the source-display OOTF, as well as any adjustments made to compensate for the presence of ambient and / or chromatic light.

[0059] Once the source adjustments are inverted, target adjustments can be applied. Similar to source appearance adjustments, target appearance adjustments utilize information about the target viewing environment and the adaptation state of a typical observer to modify the image white point, luminance, and saturation to provide an appropriate rendition of the image. The viewer's proximity to the screen determines the extent to which the screen influences the environment (an exemplary technique is described in PCT Patent Application No. PCT / US2021 / 027826, filed April 16, 2021, which is incorporated herein in its entirety for all purposes). Alternatively, the viewing distance recommended by a standard can be used to calculate the impact of screen size on adaptation. In some embodiments, additional adjustments can be applied to personalize appearance phenomena for individual viewers. These adjustments include correcting for individual contrast sensitivity functions, considerations from metamerism, and the degree of potential color blindness. Further image enhancements can be applied at the target edge to accommodate OEM preferences.

[0060] In step (408), target appearance and preference adjustments are calculated based on, for example, desired rendering intent, information about the target display environment such as ambient luminance, and the like.

[0061] In step (410), the inverted source preferences and appearance adjustments are applied to the decoded image, for example, undoing the source preferences and appearance adjustments made during method (300).

[0062] In step (412), the target appearance and preference adjustments are applied to the decoded image.

[0063] In step (414), the decoded image with the target appearance and preference adjustments applied is displayed, saved to disk, communicated to another device or party, or otherwise utilized.

[0064] The encoding system, decoding system, and method described above may provide for encoding and decoding multi-intent images and videos using metadata. The systems, methods, and devices according to the present disclosure may have any one or more of the following configurations:

[0065] (1) A method for encoding a multiple-intention image, the method including: obtaining an image for encoding as the multiple-intention image; applying at least one appearance adjustment to the image; generating metadata characterizing the at least one appearance adjustment; and encoding the image and metadata as the multiple-intention image.

[0066] (2) The method of (1), wherein the metadata characterizes the at least one appearance adjustment to a sufficient extent that the metadata can be used to reverse the at least one appearance adjustment.

[0067] (3) The method of (1) or (2), wherein applying the at least one appearance adjustment includes converting sensor values ​​to color values.

[0068] (4) A method according to any one of (1) to (3), wherein applying the at least one appearance adjustment includes converting sensor values ​​to color values ​​using a 3x3 matrix, and the metadata includes coefficients of the 3x3 matrix.

[0069] (5) A method according to any one of (1) to (4), wherein applying at least one appearance adjustment includes estimating a capture environment ambient luminance and white point and applying a white point correction based on the estimated capture environment ambient luminance and white point.

[0070] (6) The method of (5), wherein the metadata includes estimated capture environment ambient luminance and white point.

[0071] (7) A method according to any one of (1) to (4), wherein applying the at least one appearance adjustment includes estimating a capture environment ambient luminance and applying an optical-to-optical transfer function (OOTF) that prepares the image for rendering on a reference display device based in part on the estimated capture environment ambient luminance.

[0072] (8) The method of (7), wherein the metadata includes the estimated captured environment ambient luminance.

[0073] (9) The method of (7) or (8), wherein the metadata includes coefficients of the optical-to-optical transfer function.

[0074] (10) A method according to any one of (1) to (9), wherein applying at least one appearance adjustment includes applying a saturation increase, and the metadata includes a coefficient for the saturation increase.

[0075] (11) A method according to any one of (1) to (10), wherein applying at least one appearance adjustment includes applying contrast enhancement, and the metadata includes a coefficient of the contrast enhancement.

[0076] (12) A method according to any one of (1) to (11), wherein applying at least one appearance adjustment includes applying an individual saturation adjustment, and the metadata includes coefficients of the individual saturation adjustments.

[0077] (13) A method according to any one of (1) to (12), wherein applying at least one appearance adjustment includes applying a slope-offset-power-Tmid enhancement, and the metadata includes coefficients for the slope-offset-power-Tmid enhancement.

[0078] (14) A method according to any one of (1) to (13), wherein applying at least one appearance adjustment includes applying an enhancement, and the metadata includes a coefficient of the enhancement.

[0079] (15) The method according to any one of (1) to (14), wherein applying the at least one appearance adjustment includes applying a tone curve trim, and the metadata includes coefficients of the tone curve trim.

[0080] (16) A method according to any one of (1) to (15), wherein the plurality of intended images include video frames in a video.

[0081] (17) A method for decoding a multiple-intention image, the method including: obtaining the multiple-intention image along with metadata characterizing at least one appearance adjustment between the multiple-intention image and an alternative version of the multiple-intention image; obtaining a selection of the alternative version of the multiple-intention image; and applying an inverse of the at least one appearance adjustment to the multiple-intention image using the metadata to restore the alternative version of the multiple-intention image.

[0082] (18) A method, the method including: obtaining an original image for encoding as a multiple-intention image; generating metadata characterizing at least one appearance adjustment to the original image; encoding the original image and metadata as a multiple-intention image; and providing the multiple-intention image.

[0083] (19) The method described in (18), further comprising: receiving the multi-intention image in a decoder; obtaining a first rendering intent selection in the decoder; decoding the multi-intention image by applying the at least one appearance adjustment to the original image based on the first rendering intent selection; and providing the original image with the at least one appearance adjustment applied.

[0084] (20) The method described in (18) or (19), further comprising, in the decoder, a step of obtaining a selection of a second rendering intent, a step of decoding the multi-intent image without applying the at least one appearance adjustment to the original image based on the selection of the second rendering intent, and a step of providing the original image without the at least one appearance adjustment applied.

[0085] (21) The method of (18), wherein the metadata characterizes the at least one appearance adjustment to a sufficient extent that the metadata can be used to reverse the at least one appearance adjustment.

[0086] (22) The method according to any one of (18) to (21), wherein the at least one appearance adjustment includes converting a sensor value to a color value.

[0087] (23) A method according to any one of (18) to (22), wherein the at least one appearance adjustment includes converting sensor values ​​to color values ​​using a 3x3 matrix, and the metadata includes coefficients of the 3x3 matrix.

[0088] (24) A method according to any one of (18) to (19), wherein the at least one appearance adjustment includes estimating a capture environment ambient luminance and white point and applying a white point correction based on the estimated capture environment ambient luminance and white point.

[0089] (25) The method of (24), wherein the metadata includes estimated capture environment ambient luminance and white point.

[0090] (26) A method according to any one of (18) to (23), wherein the at least one appearance adjustment includes estimating a capture environment ambient luminance and applying an optical-to-optical transfer function (OOTF) to prepare the image for rendering on a reference display device based in part on the estimated capture environment ambient luminance.

[0091] (27) The method of (26), wherein the metadata includes estimated captured environment ambient luminance.

[0092] (28) The method of (26) or (27), wherein the metadata includes coefficients of the optical-to-optical transfer function.

[0093] (29) The method according to any one of (18) to (28), wherein the at least one appearance adjustment includes applying a saturation boost, and the metadata includes a coefficient for the saturation boost.

[0094] (30) The method according to any one of (18) to (29), wherein the at least one appearance adjustment includes applying a contrast enhancement, and the metadata includes a coefficient of the contrast enhancement.

[0095] (31) A method according to any one of (18) to (30), wherein the at least one appearance adjustment includes applying an individual saturation adjustment, and the metadata includes a coefficient of the individual saturation adjustment.

[0096] (32) A method according to any one of (18) to (31), wherein the at least one appearance adjustment includes applying a slope-offset-power-Tmid enhancement, and the metadata includes coefficients for the slope-offset-power-Tmid enhancement.

[0097] (33) The method according to any one of (18) to (32), wherein the at least one appearance adjustment includes applying an enhancement, and the metadata includes a coefficient of the enhancement.

[0098] (34) A method according to any one of (18) to (33), wherein the at least one appearance adjustment includes applying a tone curve trim, and the metadata includes coefficients of the tone curve trim.

[0099] (35) A method according to any one of (18) to (34), wherein the plurality of intended images include video frames within a video.

[0100] (36) A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform the operations described in any one of (1) to (35).

[0101] (37) An image delivery system for delivering multiple-intention images, comprising a processor configured to encode the multiple-intention images described in any one of (1) to (16) and (18) to (35).

[0102] (38) An image decoding system for receiving and decoding a multiple-intention image, the image decoding system comprising a processor configured to encode the multiple-intention image described in (17).

[0103] With respect to the processes, systems, methods, heuristics, etc. described herein, although steps of such processes, etc. are described as occurring according to a certain ordered sequence, it should be understood that such processes may be practiced with the described steps occurring in an order other than the order described herein. Furthermore, it should be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the process descriptions herein are provided for the purpose of illustrating certain embodiments and should not be construed as limiting the scope of the claims in any way.

[0104] Thus, it should be understood that the above description is intended to be illustrative, and not restrictive. Many embodiments and applications other than the examples provided will be apparent from reading the above description. The scope should not be determined with reference to the above description, but instead should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments in the technology described herein will occur, and that the disclosed systems and methods will be incorporated into such future embodiments. In short, it should be understood that this application is capable of modification and variation.

[0105] All terms used in the claims are intended to be given their broadest reasonable interpretation and their ordinary meaning as understood by one skilled in the art described herein, unless expressly indicated to the contrary. In particular, the use of singular articles such as "a," "the," "said," etc. should be read as describing one or more of the indicated elements unless the claim describes an express limitation to the contrary.

[0106] The Abstract of the present disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Additionally, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Accordingly, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as separately claimed subject matter.

Claims

1. 1. A method for decoding a multiple-intent image, the multiple-intent image including a representation of the image in a reference viewing environment and metadata for transforming the included representation into an alternative version of the image, the method comprising: acquiring the multiple-intent images along with metadata characterizing at least one appearance adjustment between the representation of the image in the reference viewing environment and an alternative version of the image, the metadata indicating ambient luminance and white point in the capture environment at the time the image was captured by an image sensor; obtaining a selection of desired rendering intents corresponding to the alternative versions of the multi-intent image, the desired rendering intents being of the image captured by the image sensor; and applying an inverse of the at least one appearance adjustment to the representation of the image in the reference viewing environment using the metadata to restore the alternate version of the multi-intention image based on the obtained selection. method.

2. Applying an inverse of the at least one appearance adjustment to the representation of the image in the reference viewing environment to restore the alternate version of the multi-intention image based on the obtained selection includes: mapping the image from the white point in the reference viewing environment to a white point in the capture environment; applying an optical-to-optical transfer function to the image to map from ambient luminance in the reference viewing environment to ambient luminance in the capture environment. The method of claim 1.

3. the metadata further indicates the spectral sensitivity of the image sensor that captured the image and the coefficients of a 3x3 matrix transformation that is applied to raw sensor values ​​from the image sensor to correct for differences in the spectral sensitivity of the image sensor between color channels; applying an inverse of the at least one appearance adjustment to the representation of the image in the reference viewing environment to restore the alternate version of the multi-intention image based on the obtained selection further comprises applying an inverse of the 3x3 matrix transform to the image to obtain raw sensor values. The method of claim 2.

4. The method of claim 1 , wherein the metadata includes the desired rendering intent, the desired rendering intent specifying default values ​​for how the image should be processed upon playback.

5. 1. A method for encoding a multiple-intention image, the multiple-intention image including a representation of the image in a reference viewing environment and metadata for transforming the reference representation into an alternative version of the image, the method comprising: obtaining images for encoding as the multiple-intention images, capturing a multi-channel color image by exposing a scene in a capture environment to an image sensor and collecting raw sensor values ​​from the image sensor for each color channel; determining an ambient luminance and a white point in the capture environment; Stages and; applying at least one appearance adjustment to the image to transform the captured image into the representation of the image in the reference viewing environment, mapping the image from the determined white point in the capture environment to a preferred white point in the reference viewing environment; applying an optical-to-optical transfer function to the image to map from ambient luminance in the capture environment to a preferred ambient luminance of the reference viewing environment. Stages and; generating metadata characterizing the at least one appearance adjustment, the metadata indicating a determined ambient luminance and white point in the capture environment; and encoding the transformed image and metadata as the multiple-intention image. method.

6. applying at least one appearance adjustment to the image further comprises applying a 3x3 matrix transform to the captured multi-channel color image to convert collected raw sensor values ​​to a set of desired primary colors, the 3x3 matrix transform taking into account differences in spectral sensitivity of the image sensor between color channels; the metadata further indicates the spectral sensitivity of the image sensor that captured the image and coefficients of the 3x3 matrix transformation to correct for differences in the spectral sensitivity of the image sensor between color channels, so that the metadata can transform the reference representation into an image that approximates the captured image. The method of claim 5.

7. The method of claim 5 , wherein applying the at least one appearance adjustment comprises applying an individual color saturation adjustment, and the metadata comprises a coefficient for the individual color saturation adjustment.

8. 6. The method of claim 5, wherein applying the at least one appearance adjustment comprises applying a slope-offset-power-Tmid adjustment, and the metadata includes coefficients for the slope-offset-power-Tmid adjustment.

9. The method of claim 5 , wherein applying at least one appearance adjustment comprises applying a tone curve adjustment, and the metadata includes coefficients of the tone curve adjustment.

10. The method of claim 5 , wherein the multiple intent images include video frames in a video.

11. The method of claim 5 , wherein the metadata characterizes the at least one appearance adjustment to a sufficient extent that the metadata can be used to reverse the at least one appearance adjustment.

12. 4. A decoder for decoding a multiple-intention image, the multiple-intention image including a representation of the image in a reference viewing environment and metadata for transforming the included representation into an alternative version of the image, the decoder having a processor configured to decode the multiple-intention image of any one of claims 1 to 3.

13. 12. An image delivery system for delivering a multiple-intent image, the multiple-intent image including a representation of the image in a reference viewing environment and metadata for transforming the reference representation into an alternative version of the image, the image delivery system having a processor configured to encode the multiple-intent image of any one of claims 5 to 11.

14. A non-transitory computer readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform the operations of any one of claims 1 to 4.

15. A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform the operations described in any one of claims 5 to 11.

Images