Scalable system for controlling color management including various levels of metadata

Abstract

To provide a decoder device that, in the use and application of image metadata, either a single level or several levels of metadata, improves the rendering of images and / or videos and the viewer experience to achieve video quality with improved image fidelity.SOLUTION: In a metadata-enabled image / video pipeline (200), the color management processing of source image data to be displayed on a target display (236) is altered according to various levels of metadata (204).SELECTED DRAWING: Figure 2A

Description

[Technical field]

[0001] The present invention relates to image processing, and more particularly to the encoding and decoding of image and video signals using metadata, especially at several layers of metadata. [Background technology]

[0002] Well-known scalable video encoding and decoding techniques allow for the expansion or compression of video quality depending on the capabilities of the target video display and the quality of the source video data. Summary of the Invention [Problem to be solved by the invention]

[0003] However, the use and application of image metadata, either a single level or several levels of metadata, enhances the rendering of images and / or videos and the viewer's experience. [Means for solving the problem]

[0004] Several aspects of scalable image processing systems and methods are disclosed herein whereby color management processing of source image data displayed on a target display is altered according to various levels of metadata.

[0005] In one aspect, a method is disclosed for processing and rendering image data on a target display with a set of levels of metadata, where the metadata is associated with image content, comprising: inputting image data, determining a set of levels of metadata associated with the image data, performing at least one of a set of image processing steps including switching to default values ​​and adaptively calculating parameter values ​​if no metadata is associated with the image data, and calculating color management algorithm parameters according to the set of levels of metadata associated with the image data if metadata is associated with the image data.

[0006] In yet another aspect, a system for decoding and rendering image data on a target display with a set of levels of metadata is disclosed, the system including: a video decoder that receives input image data and outputs intermediate image data, a metadata decoder that receives the input image data and is capable of detecting a set of levels of metadata associated with the input image data and outputting the intermediate metadata, a color management module that receives the intermediate metadata from the metadata decoder and the intermediate image data from the video decoder and performs image processing on the intermediate image data based on the intermediate metadata, and a target display that receives and displays the image data from the color management module. [Brief description of the drawings]

[0007] [Figure 1A] FIG. 1A shows one embodiment of a conventional video pipeline, from video signal generation to distribution and processing. [Figure 1B] FIG. 1B shows one embodiment of a conventional video pipeline, from video signal generation to distribution and processing. [Figure 1C]FIG. 1C shows one embodiment of a conventional video pipeline, from video signal generation to distribution and processing. [Figure 2A] FIG. 2A illustrates one embodiment of a video pipeline including a metadata pipeline in accordance with the teachings of the present application. [Figure 2B] FIG. 2B illustrates one embodiment of a metadata prediction block. [Diagram 3] FIG. 3 illustrates one embodiment of a sigmoid curve with level 1 metadata. [Figure 4] FIG. 4 illustrates one embodiment of a sigmoid curve with level 2 metadata. [Diagram 5] Figure 5 shows one embodiment of a histogram plot based on image / scene analysis that can be used to adjust the image / video mapping to a target display. [Figure 6] FIG. 6 illustrates an embodiment of image / video mapping adjusted based on Level 3 metadata including grading with a second reference display of image / video data. [Figure 7] FIG. 7 illustrates one embodiment of a linear mapping that may occur when a target display closely matches a second reference display used to color grade the image / video data. [Figure 8] FIG. 8 illustrates one embodiment of a video / metadata pipeline constructed in accordance with the principles of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Other features and advantages of the present system are presented in the following detailed description, which is to be read in conjunction with the drawings presented in this application. Exemplary embodiments are illustrated in the referenced figures in the drawings. The embodiments and drawings disclosed herein are to be considered illustrative and not limiting.

[0009] Throughout the following description, specific details are set forth to provide a more thorough understanding to those skilled in the art. However, well-known elements may not be shown or described in detail so as not to unnecessarily obscure the disclosure. Thus, the description and drawings should be regarded in an illustrative sense rather than a restrictive sense.

[0010] [overview] One aspect of video quality relates to rendering an image or video on a target display with the same or nearly the same fidelity as intended by the image or video creator. It is desirable to have a Color Management (CM) scheme that attempts to preserve the original look of the video content on displays of varying capabilities. To accomplish this task, it is desirable for such a CM algorithm to be able to predict how the video will have appeared to a viewer in the post-production environment where the video is finalized.

[0011] To illustrate the issues at stake in the present application and system, one embodiment of a conventional video pipeline 100 that proceeds from generation to distribution to processing of a video signal is shown in FIGS. 1A, 1B, and 1C.

[0012] The video signal generation 102 may involve color grading 104 of the video signal by a color grader 106 that may grade the signal for various image characteristics, such as brightness, contrast, and color rendering of the input video signal. The color grader 106 may grade the signal to generate an image / video mapping 108, such grading being done to a reference display device 110 that may have, for example, a gamma response curve 112.

[0013] Once the signal has been graded, the video signal can be sent out via distribution 114, which may suitably be considered to be broad. For example, distribution may be via the Internet, DVD, cinema screenings, etc. In this example, distribution is shown in FIG. 1A as feeding the signal to a target display 120 with a maximum luminance of 100 nits and a gamma response curve 124. Assuming that the reference display 110 has approximately the same maximum luminance and approximately the same response curve as the target display, then the mapping applied to the video signal can be as simple as a 1:1 mapping 122, e.g., made in compliance with the Rec709 standard for color management 118. Keeping all other factors the same (e.g., ambient lighting conditions at the target display), then what you see on the reference display is essentially what you will see on the target display.

[0014] This situation may be modified, for example, as shown in FIG. 1B, where the target display 130 differs from the reference display 110 in some respects, such as the maximum luminance (500 nits, as opposed to 100 nits for the reference display). In this case, the mapping 132 may be a 1:5 mapping for rendering on the target display. In such a case, the mapping is a linear stretching with Rec709 CM blocks. The distortion that may result from the representation of the reference display to the representation of the target display may or may not be unpleasant to the viewer, depending on the individual discrimination level. For example, dark and mid-tones may be tolerable even when stretched. Also, MPEG blocking artifacts may become more noticeable.

[0015] FIG. 1C shows a more extreme example. In this example, the target display 140 may have a more noticeable difference from the reference display. For example, the maximum luminance of the target display 140 is 1000 nits, compared to 100 nits for the reference display. If the same linear stretch mapping 142 were applied to the video signal fed to the target display, a much more noticeable and objectionable distortion may be observed by the viewer. For example, the video content may be displayed at a significantly higher luminance level (1:10 ratio). Dark and mid-tones may be stretched to the point where camera noise from the original capture becomes noticeable, and banding becomes more noticeable in dark areas of the image. Additionally, MPEG blocking artifacts may become more noticeable.

[0016] Rather than exhaustively considering all possible examples of how objectionable artifacts may appear to a viewer, it may be useful to consider a few more: For example, assume that the maximum luminance of the reference display (e.g., 600 nits) is higher than the target display (e.g., 100 nits). In this case, the mapping is still a 6:1 linear stretch, which may cause the content to be displayed entirely at a lower luminance level, the image to appear darker, and the dark details of the image may contain noticeable crush.

[0017] In yet another example, assume that the maximum luminance of the reference display (e.g., 600 nits) is different from that of the target display (e.g., 1000 nits). When linear stretching is applied, the difference in maximum luminance may be large and uncomfortable, even if the ratio difference is only slight (i.e., close to 1:2). The difference in luminance may cause the image to be too bright and painful to view. Midtones may be unnaturally stretched and may appear washed out. Also, both camera noise and compression noise may be noticeable and uncomfortable. In yet another example, assume that the reference display has a color gamut equivalent to P3 and the target display has a color gamut smaller than Rec709. The content is color graded on the reference display, but the rendered content has a color gamut equivalent to the target display. In this case, mapping the content from the gamut of the reference display to the target gamut may unnecessarily compress the content and cause it to appear desaturated.

[0018] Without some intelligent (or at least more accurate) model of the image rendering on the target display, slight distortions or unpleasant artifacts will likely be visible to the viewer of the image / video. In fact, what the viewer experiences is likely not what the image / video producer intended. Although this discussion focuses on luminance, it turns out that similar concerns apply to color. In fact, color distortions will be noticeable artifacts as well if there are differences between the color spaces of the source and target displays, if these differences are not properly taken into account. The same idea applies to ambient differences between the source and target displays.

[0019] [Use Metadata] As these examples show, it is desirable to know the characteristics and capabilities of the reference display, the target display, and the source content in order to produce video that is as faithful as possible to the original intent. There is other data, known as "metadata", that describes and conveys various aspects of the source image data and is useful for such faithful rendering.

[0020] Although tone and gamut mappers usually work well for about 80-95% of images processed for a particular display, there are problems with using such general-purpose solutions to process images. Such methods usually do not guarantee that the image displayed on the screen will match the director's or original creator's intentions. It has also been noted that several different tone or gamut mappings may work better with various different types of images or better preserve the atmosphere of the image. It has also been noted that various tone and gamut mappings may result in clipping and loss of detail, or color or hue shifts.

[0021] When tone mapping a color graded image sequence, color grading parameters such as the minimum black level and maximum white level of the content may be desirable parameters to drive tone mapping of the color graded content to a particular display. The color grader has already given the content its desired look (on a per image and even time basis). It may be desirable to preserve the perceived viewing experience of the image sequence when converting it for a different display. It should be recognized that by adding additional levels of metadata, it may be possible to improve the preservation of such look.

[0022] For example, assume a sunrise scene is shot and professionally color graded on a 1000 nits reference display. In this example, the content is mapped for display on a 200 nits display. Images before the sun rise may not use the full range of the reference display (e.g., up to 200 nits). As soon as the sun rises, the image sequence will use the full 1000 nits range, which is the upper limit of the content. Without metadata, many tone mappings use a maximum value (e.g., luminance) as a guideline for how to map the content. In this case, the tone curve applied to the image before sunrise (1:1 mapping) may be different from the tone curve applied to the image after sunrise (5x tone compression). As a result, the image displayed on the target display may have the same peak luminance before and after sunrise, which is a distortion of the creative intent. The image intended by the author to be darker before sunrise and brighter during sunrise, as created on the reference display. In such scenes, metadata can be defined that fully describes the dynamic range of the scene, and its use can ensure that artistic effects are preserved, and can also be used to minimize temporal issues with luminance from scene to scene.

[0023] As yet another example, consider the reverse case. Assume scene 1 is graded for 350 nits and is shot outdoors in natural light. If scene 2 is shot in a dark room and displayed in the same range, scene 2 will appear too dark. In this case, metadata can be used to define the appropriate tone curve to ensure scene 2 appears appropriate. In yet another example, assume the reference display has a color gamut equivalent to P3 and the target display has a color gamut smaller than Rec709. The content was color graded on the reference display, but the rendered content has a color gamut equivalent to the target display. By using metadata that defines the color gamut of the content and the color gamut of the source display, the mapping can make an intelligent decision to map the color gamut of the content 1:1. This can ensure that the color saturation of the content remains intact.

[0024] In some embodiments of the system, tone and gamut need not be treated as separate entities or terms in a set of images / videos. "Memory colors" are colors in an image that, if adjusted incorrectly, will appear wrong, even though the viewer may not know the original intent. Skin, sky, and grass are good examples of memory colors, whose hues may be altered during tone mapping in a way that makes them appear wrong. In one embodiment, the gamut mapper has knowledge (as metadata) of the protected colors in the image, which ensures that their hues are maintained in the tone mapping process. The use of this metadata can specify and highlight the protected colors in the image, which ensures accurate handling of memory colors. The ability to define local tone mapping and gamut mapping parameters is an example where metadata is not necessarily generated solely from the parameters of the reference and / or target displays.

[0025] [One embodiment of robust color management] In some embodiments of the present application, systems and methods are disclosed for providing a robust color management scheme, which employs several sources of metadata to provide higher image / video fidelity consistent with the original intent of the content creator. As described in more detail herein, in one embodiment, various sources of metadata can be added to the process depending on the availability of certain metadata.

[0026] FIG. 2A illustrates, by way of example only, a high-level block diagram of an image / video pipeline 200 with metadata. Image generation and post-production can occur in block 202. A video source 208 is input to a video encoder 210. As the video source is ingested, metadata 204 is provided to a metadata encoder 206. Examples of metadata 204 have been discussed above and may include items such as the color gamut boundaries and other parameters of the source and / or reference display, the environment of the reference display, and other encoding parameters. In one embodiment, the metadata accompanies the video signal as a subset of the metadata and is co-located temporally and spatially with the video signal that is to be rendered at the time. The metadata encoder 206 and the video encoder 210 together may be considered a source image encoder.

[0027] The video signals and metadata are then distributed by distribution 212 in any suitable manner, such as, for example, multiplexed, serial, parallel, or any other known manner. It should be recognized that distribution 212 must be considered broad for purposes of this application. Suitable distribution methods may include the Internet, DVD, cable, satellite, wireless, wired, etc.

[0028] The video signal and metadata thus delivered are provided to a target display environment 220. A metadata decoder 222 and a video decoder 224 each receive the respective data streams and provide decoding appropriate to the characteristics of the target display, among other factors. The metadata at this point can preferably be sent to either a third-party color management (CM) block 220 and / or one of the embodiments of a CM module 228 of the present application. If the video and metadata are processed by the CM block 228, a CM parameter generator 232 can take the metadata from the metadata decoder 222 as input, along with the metadata from the metadata prediction block 230.

[0029] The metadata prediction block 230 can make some predictions of higher fidelity rendering based on knowledge of previous images or video scenes. The metadata prediction block collects statistics from the input video stream to estimate the metadata parameters. One possible embodiment of the metadata prediction block 230 is shown in FIG. 2B. In this embodiment, a histogram 262 of the logarithm of the image luminance can be calculated for each frame. An optional low-pass filter 260 can precede the histogram to (a) reduce the sensitivity of the histogram to noise and / or (b) partially account for natural blurring in the human visual system (e.g., humans perceive dither patterns as monochromatic patches). Then, a minimum point 266, a maximum point 274 are obtained. Furthermore, a toe point 268 and a shoulder point 272 can be obtained based on percentile settings (such as 5% and 95%). Furthermore, a geometric mean 270 (logarithmic mean) can be calculated and used as a midpoint. These values ​​can be filtered in time, for example, so that they do not change too quickly. The values ​​can also be reset, if desired, upon a scene change. A scene change can be detected by the insertion of a black frame, or a very abrupt change in the histogram, or other such techniques. As will be apparent, the scene change detector 264 can detect scene changes from the histogram data, as shown, or directly from the video data.

[0030] In yet another embodiment, the system may calculate the average of image intensity values ​​(luminance), which may be scaled by a perceptual weighting such as logarithm, power function, or look-up table (LUT). The system may then estimate highlight and shadow regions (e.g., headroom and footroom in FIG. 5) from predefined percentile values ​​of the image histogram (e.g., 10% and 90%). Alternatively, the system may estimate highlight and shadow regions when the slope of the histogram is above or below a certain threshold. Many variations are possible, for example, the system may calculate the maximum and minimum values ​​of the input image, or from predefined percentile values ​​(e.g., 1% and 99%).

[0031] In other embodiments, the values ​​may be stabilized over time (e.g., from frame to frame), such as by fixed increase and decrease rates. The values ​​may not be stabilized over time, since sudden changes may indicate a scene change. For example, if the change is below a certain threshold, the system may limit the rate of change, otherwise proceed with the new value. Alternatively, the system may disallow certain values ​​(such as letterbox or zero values) from affecting the shape of the histogram.

[0032] Additionally, the CM parameter generator 232 can employ other (i.e., not necessarily content creation based) metadata, such as user selection of display parameters, display ambience, and factors into the color management of the image / video data. As will be apparent, the CM parameter generator 232 can make the display parameters available through a standard interface, such as Extended Display Identification Data (EDID), through an interface (such as a Display Data Channel (DDC) serial interface, High-Definition Multimedia Interface (HDMI), Digital Visual Interface (DVI), etc.). Additionally, the display ambience data can be provided by an ambient light sensor (not shown) that measures the ambient light conditions and their reflection from the target display.

[0033] Once the CM parameter generator 232 has received any appropriate metadata, it can set parameters in downstream CM algorithms 234 that can handle the final mapping of the image / video data on the target display 236. It should be appreciated that the illustrated branch of functionality between the CM parameter generator 232 and the CM algorithms 234 is not required; indeed, in some embodiments, these functions may be combined into one block.

[0034] As will be appreciated, the various blocks of Figures 2A and 2B are similarly optional from the perspective of the present embodiment, and numerous other embodiments may be constructed with or without these enumerated blocks and are within the scope of this application. Also, CM processing may be performed at a variety of different points in the image pipeline 200 and not necessarily as shown in Figure 2A. For example, the CM for the target display may be located and included within the target display itself, or such processing may be performed at the set-top box. Alternatively, the CM for the target display may be performed at the point of distribution or post-production, depending on what level of metadata processing is available or deemed appropriate.

[0035] [Scalable color management using multiple levels of metadata] In some embodiments of the present application, systems and methods are disclosed for providing a scalable color management scheme whereby several metadata sources can deliver a series of different levels of metadata to provide images / video with greater fidelity to the original intent of the content creator. As described in more detail herein, in one embodiment, depending on the availability of some metadata, different levels of metadata can be added to the process.

[0036] In many embodiments of the system, appropriate metadata algorithms can take into account many pieces of information, such as: (1) Encoded video content (2) A method for converting encoded content into linear light. (3) The color gamut boundaries of the source content (both luminance and chrominance) (4) Information about the post-production environment A method for converting to linear light may be desirable so that the actual image appearance (brightness, gamut, etc.) observed by the content creator can be calculated. Gamut boundaries are useful for pre-specifying the outermost colors so that they can be mapped to the target display without clipping or leaving too much overhead. Information about the post-production environment is desirable so that external factors that may affect the appearance on the display can be modeled.

[0037] In current video delivery mechanisms, only the encoded video content is provided to the target display. It is assumed that the content was produced in a reference studio environment with a reference display that is compliant with Rec601 / 709 and various SMPTE (Society of Motion Picture and Television Engineers) standards. The target display system is generally assumed to be Rec601 / 709 compliant, and the target display environment is largely ignored. With the basic assumption that both the post-production display and the target display are Rec601 / 709 compliant, neither display may be upgraded without introducing some degree of image distortion. In reality, some distortion may already be introduced since Rec601 and Rec709 have slightly different primary color selections.

[0038] An embodiment of a scalable system of metadata levels is disclosed herein that allows the use of reference and target displays with a wider and more sophisticated range of capabilities. Several different metadata levels allow the CM algorithm to tailor source content for a given target display with increased accuracy. The following sections describe several proposed levels of metadata.

[0039] [Level 0] Level 0 metadata is the default case and effectively means zero metadata. Metadata may be absent for a number of reasons, including:

[0040] (1) The content creator did not include metadata (or the metadata was lost at some point in the post-production pipeline). (2) The display switches content (i.e., channel surfing or commercial breaks).

[0041] (3) Corruption or loss of data. In one embodiment, it may be desirable for CM processing to address level 0 (i.e., the absence of metadata) by either inferring metadata based on video analysis or by assuming default values.

[0042] In such an embodiment, the color management algorithm may be allowed to operate in at least two different ways in the absence of metadata: Switch to the default value.

[0043] In this case, the display will behave much like a current delivery system where the characteristics of a post-production reference display are assumed. The assumed reference display can be different depending on the video encoding format. For example, for 8-bit RGB data, a Rec601 / 709 display can be assumed. For higher bit depth RGB data or LogYuv encoded data, a P3 or Rec709 gamut can be assumed if it is color graded on a professional monitor in 600 nits mode (such as a ProMonitor). This can work well when there is only one standard or de facto standard for higher dynamic range content. On the other hand, if the higher dynamic range content is created under custom conditions, the results may not be much improved and may be poor.

[0044] Parameter values ​​are adaptively calculated. In this case, the CM algorithm starts with some default assumptions, which can be modified based on information gained from analyzing the source content. Typically, this may involve analyzing the histogram of the video frame to determine how best to adjust the luminance of the input source, possibly by calculating parameter values ​​for the CM algorithm. There is then a risk that an "auto-exposed" type of looking image may be created, where each scene or frame is balanced to the same luminance level. Also, depending on the format, some other challenges may be presented; for example, if the source content is in RGB format, there is currently no automated method to determine its color gamut.

[0045] In other embodiments, a combination of the two approaches can be implemented: for example, color gamut and encoding parameters (such as gamma) can assume standard default values, and the histogram can be used to adjust the luminance levels.

[0046] [Level 1] In this embodiment, Level 1 metadata provides information that describes how the source content was created and packaged. This data allows the CM process to predict how the video content actually appeared to the content creator. Level 1 metadata parameters can be categorized into three areas:

[0047] (1) Video encoding parameters (2) Source and display parameters (3) Color gamut parameters of the source content (4) Environmental parameters Video Encoding Parameters Since many color management algorithms work at least partially in a linear light space, it may be desirable to have a way to convert encoded video into a linear (but relative) (X,Y,Z) representation that is either inherent to the encoding scheme or provided as metadata itself. For example, encoding schemes such as LogYuv, OpenEXR, LogYxy, or LogLuv TIFF inherently contain the information necessary to convert to a linear light format. However, for many RGB or YCbCr formats, additional information such as gamma and primaries may be desired. As an example, the following information may be provided to process a YCbCr or RGB input:

[0048] (1) The coordinates of the primary colors and white point used to encode the source content, which can be used to generate an RGB to XYZ color space transformation matrix, the (x,y) coordinates of red, green, blue, and white.

[0049] (2) Minimum and maximum code values ​​(e.g., "normal" or "full" range), which can be used to convert the code values ​​to normalized input values. (3) A response curve (e.g., "gamma"), either overall or per channel for each primary color, that can be used to linearize the intensity values ​​by undoing any nonlinear responses that may have been applied by the interface or reference display.

[0050] Color Gamut Parameters of the Source Display It may be useful for a color management algorithm to know the color gamut of the source display. These values ​​correspond to the capabilities of the reference display used to grade the content. The color gamut parameters of the source display, preferably measured in a completely dark environment, can include:

[0051] (1) Color primaries given as CIE xy chromaticity coordinates or XYZ specified along with maximum luminance. (2) Tristimulus values ​​of white and black, such as CIE XYZ.

[0052] Color Gamut Parameters of the Source Content It may be useful for a color management algorithm to know the boundaries of the color gamut used to create the source content. Generally, those values ​​correspond to the capabilities of the reference display used to grade the content. However, they may differ depending on the software settings, or if only a part of the display's capabilities was used. In some cases, the color gamut of the source content may not match the color gamut of the encoded video data. For example, the video data may be encoded in LogYuv (or other encoding) that covers the entire visible spectrum. Source gamut parameters may include:

[0053] (1) Color primaries given as CIE xy chromaticity coordinates or XYZ specified along with maximum luminance. (2) Tristimulus values ​​of white and black, such as CIE XYZ.

[0054] Environmental parameters In certain situations, simply knowing the luminance levels of a reference display may not be enough to determine how the source content "saw" to the viewer in post-production. Information about the light levels caused by the surrounding environment can also be useful. The combination of both the display and the environment light is the signal to the human eye and creates a "look". It may be desirable to maintain this look throughout the video pipeline. Environmental parameters, preferably measured in a typical color grading environment, can include:

[0055] (1) The ambient color of a reference monitor given as absolute XYZ values ​​that can be used to estimate the viewer's level of adaptation to his or her own environment. (2) The absolute XYZ values ​​of the black level of a reference monitor in a typical color grading environment, which can be used to determine the effect of ambient lighting on the black level.

[0056] (3) The color temperature of the ambient light, given as absolute XYZ values ​​of a white reflective sample (such as paper) in front of the screen. This value can be used to estimate the viewer's adaptation to the white point. As mentioned above, Level 1 metadata can provide color gamut, encoding, and environmental parameters of the source content. This can enable CM solutions to predict how the source content will have looked when it was approved. However, it does not provide much guidance on how to best adjust color and luminance for the target display.

[0057] In one embodiment, applying a single sigmoidal curve globally to the video frames in RGB space may be a simple and stable way to map between different source and target dynamic ranges. Moreover, a single sigmoidal curve can be used to modify each channel (R, G, B) independently. Such a curve can also be sigmoidal in some perceptual space, such as logarithmic or power function. An example curve 300 is shown in FIG. 3. Obviously, other mapping curves are suitable, such as linear mapping (as shown in FIGS. 3, 4, 6) or other mappings such as gamma.

[0058] In this case, the minimum and maximum points on the curve are known from Level 1 metadata and information about the target display. The exact shape of the curve can be static and be what has been found to work well on average based on the input and output ranges, or it can be adaptively modified based on the source content.

[0059] [Level 2] Level 2 metadata provides additional information about the characteristics of the source video content. In one embodiment, Level 2 metadata may divide the luminance range of the source content into specific luminance regions. More specifically, in one embodiment, the luminance range of the source content may be divided into five regions, where the regions may be defined by a number of points along the luminance range. Such ranges and regions may be defined in an image, in a series of images, in a video scene, or in multiple video scenes.

[0060] For illustrative purposes, one embodiment using level 2 metadata is shown in Figures 4 and 5. Figure 4 shows a mapping 400 of input luminance to output luminance on a target display. In this case, the mapping 400 is shown as an approximately sigmoidal curve with a series of breakpoints along the curve. The points can correspond to image processing related values, in this figure, min in , foot in , mid in , head in , max in It displays:

[0061] In this embodiment, min in and max in can correspond to the minimum and maximum luminance values ​​of the scene. The third point, min in can be taken as the median value corresponding to the perceptual "mean" luminance value or "median tone". The last two points, foot in and head in may be a footroom value and a headroom value. The area between the footroom and headroom values ​​may define a significant portion of the dynamic range of the scene. It may be desirable to ensure that the content between these points is preserved as much as possible. The content below the footroom may be compressed if necessary. The content above the headroom represents the highlights and may be clipped if necessary. It should be recognized that these points tend to define the curve itself, and thus other embodiments may result in a curve that best fits these points. Also, such a curve may assume a linear, gamma, sigmoid, or other suitable and / or desirable shape.

[0062] Further to this embodiment, FIG. 5 illustrates the min, footroom, mid, headroom, and max points as they would appear on a histogram plot 500. Such a histogram may be created per image, per video scene, or even per series of video scenes, depending on how granular the histogram analysis is desired to help maintain content fidelity. In one embodiment, these five points may be specified by code values ​​of the same encoded representation as the video data. It should be noted that, typically, but not always, the min and max may correspond to the same values ​​in the range of the video signal.

[0063] Depending on the granularity and frequency of such histogram plots, histogram analysis can be used to dynamically redefine the points along the luminance map of FIG. 4, thereby changing the curve over time. This can also be used to improve the fidelity of the content presented to the viewer on the target display. For example, in one embodiment, periodically communicating the histogram allows the decoder to potentially get more information than just min, max, etc. The encoder can also include a new histogram only when there is a significant change. This saves the decoder from having to calculate it on the fly for every frame. In yet another embodiment, the histogram is used to estimate metadata, either to replace missing metadata or to supplement existing metadata.

[0064] [Level 3] In one embodiment, the level 3 metadata may employ parameters from the level 1 and level 2 metadata for a second reference grading of the source content. For example, the primary grading of the source content may have been performed on a reference monitor (e.g., a ProMonitor) using a P3 color gamut at a luminance of 600 nits. The level 3 metadata may similarly provide information about a secondary grading that may be performed, for example, on a CRT reference. In this case, the additional information would indicate Rec601 or Rec709 primaries and a lower luminance, such as 120 nits. The corresponding min, foot, mid, head, and max levels are also fed to the CM algorithm.

[0065] Level 3 metadata allows adding additional data such as color gamut, environment, color primaries, and luminance level information of a second reference grading of the source content. This additional information can then be used together to define a sigmoid curve 600 (as shown in Figure 6) that maps the primary input to the range of the reference display. Figure 6 shows an example of how the input and reference display (output) levels can be combined to form a suitable mapping curve.

[0066] If the capabilities of the target display closely match those of the secondary reference display, then this curve can be used directly to map the primary source content. On the other hand, if the capabilities of the target display are intermediate between those of the primary and secondary reference displays, then the mapping curve for the secondary display can be used as a lower bound. The curve used for the actual target display can then be an interpolation between no reduction (e.g., the linear mapping 700 as shown in FIG. 7) and the full range reduction curve generated using the reference levels.

[0067] [Level 4] Level 4 metadata is the same as Level 3 metadata, except that the metadata for the second reference grading is adjusted to match the actual target display.

[0068] Level 4 metadata can also be implemented in Over-The-Top (OTT) scenarios (i.e., Netflix, mobile streaming, or other video-on-demand (VOD) services) where the actual target display sends its characteristics to the content provider and the content is delivered with the best available curve. In one such embodiment, the target display can communicate with a video streaming service, a VOD service, or the like, and the target display can provide information such as its EDID data or other suitable metadata available to the data streaming service. Such a communication path is shown in FIG. 2A as dotted path 240 to either a video encoder and / or a metadata encoder (210 and 206, respectively), as known in the art for services such as Netflix. Typically, Netflix and other such VOD services monitor the amount of data and data throughput speed to the target device, and the metadata is not necessarily for color management. However, for purposes of this embodiment, it is sufficient that metadata is transmitted from the target data via delivery 212 or other method (in real-time or in advance) to a creation or post-production department to modify the color, tone, or other characteristics of the image data delivered to the target display.

[0069] The reference luminance levels provided by Level 4 metadata are dedicated to that target display. In this case, a sigmoid curve can be constructed as shown in Figure 6 and can be used directly without interpolation or adjustment.

[0070] [Level 5] Level 5 metadata extends Level 3 or Level 4 by identifying distinctive features such as:

[0071] (1) Protected Colors: Colors in an image that are identified as common memory colors that should not be processed, such as skin tones, sky colors, grass colors, etc. Areas of an image with such protected colors may have image data supplied to the target display without modification.

[0072] (2) Prominent Highlights: Shows light sources, maximum emission and specular highlights. (3) Out of Gamut: Features in an image that have been intentionally color graded outside the color gamut of the source content.

[0073] In some embodiments, objects shown this way can be artificially mapped to the display maximum if the target display can accommodate higher luminance, or they can be clipped to the display maximum without recreating detail if the target display can accommodate lower luminance, in which case the objects are ignored and a defined mapping curve is applied to the remaining content to preserve more detail.

[0074] It should be recognized that in some embodiments, such as when attempting to map a VDR down to a lower dynamic range display, knowing the light sources and highlights may be useful since it may be possible to clip them without too much impact. As an example, a brightly lit face that is backlit (i.e., not at all a light source) is not a feature that is desirable to clip. Alternatively, such features may be compressed more gradually. In yet another embodiment, if the target display is capable of a wider color gamut, those content objects may be stretched to extend to the full capabilities of the display. Also, in another embodiment, the system may ignore mapping curves defined to ensure highly saturated colors.

[0075] It should be recognized that in some embodiments of the present application, the levels themselves do not form a strict hierarchy of metadata processing. For example, level 5 can be applied to either level 3 or level 4 data. Also, some lower numbered levels may not be present and the system can process higher numbered levels if they are present.

[0076] [One embodiment of a system using multiple metadata levels] As discussed above, various metadata levels provide increasing information about the source material, which allows the CM algorithm to provide increasingly accurate mappings for the target display. One embodiment employing such scalable levels of metadata is shown in FIG.

[0077] The illustrated system 800 shows an overall video / metadata pipeline through five blocks: creation 802, container 808, encoding / distribution 814, decoding 822, and consumption 834. As will be apparent, numerous variations of several different implementations are possible, some with more blocks and some with fewer blocks. The scope of this application should not be limited to the description of the embodiments herein, and indeed the scope of this application encompasses these various implementations and embodiments.

[0078] The generator 802 generally takes image / video content 804 and processes it as described above by color grading tools 806. The processed video and metadata are placed in a suitable container 810, e.g., in any suitable format or data structure for subsequent distribution as known in the art. As an example, the video may be stored and served as VDR color graded video and the metadata as VDR XML formatted metadata. This metadata is separated into several levels as described above, as shown at 812. The container block may embed data in the formatted metadata that encodes up to which level metadata is available and associated with the image / video data. It should be appreciated that not all levels of metadata need be associated with the image / video data, and whatever metadata and levels are associated, downstream decoding and rendering may be able to see such available metadata and process it accordingly.

[0079] Encoding can proceed by obtaining metadata and providing it to an algorithm parameter determination block 816, while the video can be provided to an AVCVDR encoder 818, which includes a CM block for processing the video prior to distribution 820.

[0080] Once delivered (e.g., by internet, DVD, cable, satellite, wireless, wired, etc., which may be considered broad), the decoding of the video data / metadata can proceed to the AVCVDR decoder 824 (or optionally to a legacy decoder 826 if the target display is not VDR-enabled). Both the video data and the metadata are restored by the decoding (as blocks 830, 828, respectively, and possibly as block 832 if the target display is legacy). The decoder 824 takes the input image / video data and restores the input image data and / or may separate it into an image / video data stream to be further processed and rendered, and a metadata stream for computing parameters for later CM algorithm processing on the rendered image / video data stream. The metadata stream further includes information on whether there is metadata associated with that image / video data stream. If there is no associated metadata, the system can proceed with level 0 processing as described above. If not, the system can proceed with further processing with all metadata associated with the image / video data stream according to a series of different levels of metadata as described above.

[0081] As will be appreciated, the presence or absence of metadata associated with the image / video data being rendered can be determined in real-time. For example, some sections of a video stream may have no metadata associated with them (either due to data corruption or due to the content creator's intent of no metadata), while other sections may have metadata available, and possibly many metadata at various levels of sequence associated with other sections of the video stream. This may be the intent of the content creator, but in at least one embodiment of the present application, it should be possible to make a determination in real-time, or substantially dynamically, as to the presence or absence or what level of metadata is associated with such a video stream.

[0082] In the processing block, an algorithmic parameter determination block 836 can restore previous parameters that may have been processed prior to delivery or recalculate parameters based on metadata from the target display and / or the target environment (which may be, for example, from standard interfaces such as EDID or modern VDR interfaces, as described above in the context of the embodiments of Figures 2A and / or 2B, or even input from viewers or sensors in the target environment). Once the parameters are calculated or restored, they can be sent to one or more of the CM systems (838, 840, and / or 842) for eventual mapping of source and intermediate image / video data to the target display 844 according to some embodiments disclosed herein.

[0083] In other embodiments, the implementation blocks of Figure 8 need not be subdivided: for example, generally speaking, the processes involving algorithm parameter determination and the color management algorithm itself do not necessarily need to be branched as shown in Figure 8, but can assume and / or be implemented as a color management module.

[0084] Additionally, although this specification describes a series of various levels of metadata for use in a video / image pipeline, it should be recognized that in practice the system need not process the image / video data in the order of the numbering of the levels of metadata. In fact, some levels of metadata may be available during rendering while others may not. For example, second reference color grading may or may not be performed, and level 3 metadata may or may not be present during rendering. A system configured in accordance with this application will continue to process the best metadata available at the time, taking into account the presence or absence of the various levels of metadata.

[0085] The foregoing provides a detailed description of one or more embodiments of the present invention, read in conjunction with the accompanying drawings, illustrating the principles of the invention. It should be recognized that while the invention has been described in connection with such embodiments, the invention is not limited to any of the embodiments. The scope of the present invention is limited only by the claims, and the present invention encompasses numerous alternatives, modifications, and equivalents. Various specific details have been described herein to provide a thorough understanding of the present invention. These details are provided for illustrative purposes, and the present invention may be practiced according to the claims without some or all of these specific details. For purposes of clarity, technical matters known in the art related to the present invention have not been described in detail so as not to unnecessarily obscure the invention.

Claims

1. 1. A decoder device, comprising: A receiver that receives image data as a bit stream, the encoded content in the bitstream is generated on a physical reference display device, and a plurality of environmental parameters of the reference display device are characterized in the bitstream by a set of one or more parameters; the image data being associated with corresponding image metadata; a decoder for decoding the bitstream; A determination unit, determining whether the image metadata includes a first set of metadata associated with the image data, where the first set of metadata includes parameters for ambient lighting conditions at the reference display device; and determining whether the image metadata includes a second set of metadata describing tone mapping parameters for displaying the image data on a target display.

2. 2. The decoder apparatus of claim 1, wherein the first set of metadata and the second set of metadata are received within the bitstream.

3. 3. The decoder apparatus of claim 2, wherein the first set of metadata and the second set of metadata are separated separately in the bitstream.

4. 3. The decoder apparatus of claim 2, wherein the parameters for the ambient light conditions include ambient light color parameters for the reference display device.

5. 3. The decoder apparatus of claim 2, wherein the second set of metadata defines a sigmoidal tone mapping curve.

6. 6. The decoder apparatus of claim 5, wherein the second set of metadata for the tone mapping sigmoid curve includes a minimum signal value, a sigmoid median value, and a maximum signal value to be used in tone mapping the image data to the target display.

Images