A scalable system that controls color management, including various levels of metadata.

Abstract

PROVIDING VIDEO QUALITY WITH IMPROVED IMAGE FIDELITY SOLUTION: Some embodiments of scalable image processing systems and methods are disclosed herein in which the color management processing of source image data to be displayed on a target display (236) is varied according to different levels of metadata (204).SELECTED DRAWING: Figure 2A

Description

[Technical Field]

[0001] The present invention relates to image processing, and more specifically to the encoding and decoding of image and video signals using metadata, particularly several layers of metadata. [Background technology]

[0002] Well-known scalable video encoding and decoding techniques allow for expansion or compression of video quality according to the capabilities of the target video display and the quality of the source video data. [Overview of the project] [Problems that the invention aims to solve]

[0003] However, the use and application of image metadata, whether at a single level or several levels, improves the rendering of images and / or videos, as well as the viewer experience. [Means for solving the problem]

[0004] Several aspects of scalable image processing systems and methods are disclosed herein, wherein the color management processing of source image data displayed on a target display is modified according to various levels of metadata.

[0005] In one embodiment, a method for processing and rendering image data on a target display using a set of levels of metadata, wherein the metadata is associated with image content. The method comprises performing at least one of a set of image processing steps, including inputting image data; checking a set of levels of metadata associated with the image data; switching to default values ​​if no metadata is associated with the image data; and, if metadata is associated with the image data, calculating color management algorithm parameters according to the set of levels of metadata associated with the image data.

[0006] In yet another embodiment, a system for decoding and rendering image data on a target display using a set of levels of metadata is disclosed. The system comprises a video decoder that receives input image data and outputs intermediate image data; a metadata decoder that receives input image data and is capable of detecting a set of levels of metadata associated with the input image data and outputting intermediate metadata; a color management module that receives intermediate metadata from the metadata decoder and intermediate image data from the video decoder and performs image processing on the intermediate image data based on the intermediate metadata; and a target display that receives and displays image data from the color management module. [Brief explanation of the drawing]

[0007] [Figure 1A] Figure 1A shows one embodiment of a conventional video pipeline, from video signal generation to distribution and processing. [Figure 1B] Figure 1B shows one embodiment of a conventional video pipeline, from video signal generation to distribution and processing. [Figure 1C]Figure 1C shows one embodiment of a conventional video pipeline, from video signal generation to distribution and processing. [Figure 2A] Figure 2A shows one embodiment of a video pipeline that includes a metadata pipeline as taught in this application. [Figure 2B] Figure 2B shows one embodiment of the metadata prediction block. [Figure 3] Figure 3 shows one embodiment of a sigmoid curve using Level 1 metadata. [Figure 4] Figure 4 shows one embodiment of a sigmoid curve using Level 2 metadata. [Figure 5] Figure 5 shows one embodiment of an image / scene analysis-based histogram plot that can be used to adjust image / video mapping to a target display. [Figure 6] Figure 6 shows one embodiment of image / video mapping adjusted based on level 3 metadata, including grading of image / video data by a second reference display. [Figure 7] Figure 7 shows one embodiment of linear mapping that can occur when the target display closely matches a second reference display used for color grading of image / video data. [Figure 8] Figure 8 shows one embodiment of a video / metadata pipeline configured according to the principles of this application. [Modes for carrying out the invention]

[0008] Other features and effects of this system are presented in the following detailed description, which should be read in conjunction with the drawings presented in this application. Exemplary embodiments are shown in the reference drawings. The embodiments and drawings disclosed herein are not limiting and should be considered illustrative.

[0009] The following explanation provides specific details to give a more complete understanding to those skilled in the art. However, to avoid unnecessarily obscuring the disclosure, well-known elements may not be presented or described in detail. Therefore, the explanations and drawings should be considered illustrative rather than restrictive.

[0010] [overview] One aspect of video quality concerns rendering an image or video on a target display with the same or nearly the same fidelity as intended by the creator of the image or video. It is desirable to have a color management (CM) scheme that attempts to maintain the original appearance of the video content on displays with varying capabilities. To achieve this, it is desirable that such a CM algorithm be able to predict how the video would have appeared to the viewer in the post-production environment where the video was finalized.

[0011] To illustrate the issues deeply related to this application and the system, an embodiment of a conventional video pipeline 100, which generates and distributes video signals and then processes those video signals, is shown in Figures 1A, 1B, and 1C.

[0012] The video signal generation unit 102 may also perform color grading of the video signal (104) using a color grader 106, which can grade the signal for various image characteristics such as brightness, contrast, and color rendering of the input video signal. The color grader 106 can grade the signal to generate an image / video mapping 108, which may be performed on a reference display device 110 that has a gamma response curve 112.

[0013] Once the signal is graded, the video signal can be transmitted via distribution 114, and the distribution in this case is considered to be appropriately broad in scope. For example, the distribution may be via the Internet, DVD, theatrical screening, and the like. In the present example, the distribution is illustrated in Figure 1A as supplying the signal to a target display 120 having a gamma response curve 124 with a maximum luminance of 100 nits. Assuming that the reference display 110 has substantially the same maximum luminance and substantially the same response curve as the target display, the mapping applied to the video signal in that case can be as simple as a 1:1 mapping 122, and can be created, for example, in compliance with the Rec.709 standard for color management 118. If all other factors (such as ambient light conditions at the target display) are kept the same, what is then seen on the reference display will be substantially what will be seen on the target display.

[0014] This situation may be modified, for example, as shown in Figure 1B, where the target display 130 differs from the reference display 110 in several respects, such as a maximum luminance of 500 nits as opposed to 100 nits for the reference display. In this case, the mapping 132 may be a 1:5 mapping for rendering on the target display. In such a case, the mapping is a linear scaling by the Rec.709 CM block. Distortion that may occur from the display of the reference display to the display of the target display may or may not be uncomfortable to the viewer depending on the individual's level of discrimination. For example, dark tones and mid-tones may be acceptable even when stretched. Also, MPEG blocking artifacts may become more noticeable.

[0015] Figure 1C shows a further more extreme example. In this example, the target display 140 may have a more significant difference from the reference display. For example, relative to 100 nits of the reference display, the maximum luminance of the target display 140 is 1000 nits. If the same linear scaling mapping 142 is applied to the video signal supplied to the target display, far more significant and unpleasant distortion may be perceived by the viewer. For example, video content may be displayed at a significantly high luminance level (1:10 ratio). Dark colors and mid-tones may be stretched to a point where camera noise from the original capture becomes noticeable, and banding becomes more prominent in dark regions of the image. Furthermore, MPEG blocking artifacts may become more significant.

[0016] Rather than exhaustively reviewing every possible example of how unpleasant artifacts may appear to a viewer, it may be beneficial to discuss a few more. For example, assume that the maximum luminance of the reference display (e.g., 600 nits) is higher than that of the target display (e.g., 100 nits). In this case, the mapping is still a 6:1 linear scaling, which may cause the entire content to be displayed at a lower luminance level, the image appears dark, and dark details of the image may include noticeable crush.

[0017] In yet another example, suppose the maximum brightness of the reference display (e.g., 600 nits) is different from that of the target display (e.g., 1000 nits). When linear stretching is applied, even a slight difference in ratio (i.e., close to 1:2) can result in a significant and unpleasant difference in maximum brightness. The difference in brightness can cause the image to become too bright, making it painful to view. Midtones may be unnaturally stretched and appear faded. Both camera noise and compression noise can also become noticeable and unpleasant. In yet another example, suppose the reference display has a color gamut equivalent to P3, and the target display has a smaller color gamut than Rec709. The content is color-graded on the reference display, but the rendered content is assumed to have a color gamut equivalent to that of the target display. In this case, mapping the content from the reference display's color gamut to the target color gamut can unnecessarily compress the content and reduce its apparent saturation.

[0018] Without some intelligent (or at least more accurate) model of image rendering on the target display, it is highly likely that some distortion or unpleasant artifacts will be visible to the viewer of the image / video. In fact, what the viewer experiences is likely not what the image / video creator intended. While this discussion focuses on brightness, similar concerns apply to color. Indeed, if there is a difference between the color space of the source display and the color space of the target display, and that difference is not properly accounted for, color distortion will also become a noticeable artifact. The same thinking applies to differences in the ambient environments between the source display and the target display.

[0019] [Using Metadata] As these examples show, in order to create a video that is as faithful as possible to the original intent, it is desirable to have an understanding of the characteristics and capabilities of the reference display, target display, and source content. There is also other data that describes various aspects of the source image data and conveys that information, which is called "metadata" and is useful for such faithful rendering.

[0020] While tone and gamut mappers typically work well for about 80-95% of images processed for a particular display, there are problems with using such general-purpose solutions for image processing. Typically, this method does not guarantee that the image displayed on the screen will match the director's or original creator's intentions. It has also been noted that some different tone or gamut mapping techniques may work better with various types of images or better preserve the image's overall feel. Furthermore, it should be noted that various tone and gamut mapping techniques can result in clipping and loss of detail, or shifts in color or hue.

[0021] When tone mapping a color-graded image sequence, color grading parameters such as the content's minimum black and maximum white levels can be desirable parameters for toning the color-graded content to a particular display. The content has already been given its preferred look (image by image, and even on a time-based basis) by the color grader. When converting it for a different display, it is sometimes desirable to maintain the perceived viewing experience of the image sequence. It should be recognized that increasing the level of metadata can sometimes improve the preservation of such look.

[0022] For example, suppose a sunrise scene is filmed and color-graded by a professional on a 1000-nit reference display. In this example, the content is mapped for display on a 200-nit display. The image before sunrise may not utilize the full range of the reference display (e.g., up to 200 nits). As soon as the sun rises, the image sequence will utilize the entire 1000-nit range, which is the maximum limit of the content. Without metadata, many tone mappings use maximum values ​​(e.g., brightness) as a guideline for how to map the content. In this case, the tone curve applied to the image before sunrise (1:1 mapping) may differ from the tone curve applied to the image after sunrise (5x compression). As a result, the image displayed on the target display may have the same peak brightness before and after sunrise, which is a distortion of the creative intent. The image intended by the artist was darker before sunrise and brighter during sunrise, as it was created on the reference display. In such scenes, metadata can be defined that fully describes the scene's dynamic range, and the use of this metadata can ensure that the artistic effect is preserved. It can also be used to minimize temporal issues with brightness transitions from scene to scene.

[0023] As yet another example, consider the reverse of the above. Suppose Scene 1 is graded for 350 nits and is shot outdoors in natural light. If Scene 2 is shot in a dark room and displayed at the same range, Scene 2 will appear too dark. In this case, metadata can be used to define the appropriate tone curve and ensure that Scene 2 looks appropriate. In yet another example, assume that the reference display has a color gamut equivalent to P3 and the target display has a smaller color gamut than Rec709. Assume that the content is color graded on the reference display, but the rendered content has a color gamut equivalent to the target display. By using metadata that defines the color gamut of the content and the color gamut of the source display, mapping can be made to make intelligent decisions and enable a 1:1 mapping of the content's color gamut. This can ensure that the content's saturation remains undiminished.

[0024] In some embodiments of this system, tone and color gamut do not need to be treated as separate entities or conditions within a series of images / videos. “Memory colors” are colors in an image that, if incorrectly adjusted, will appear wrong, even if the viewer is unaware of the original intent. Skin tones, sky, and grass are good examples of memory colors, and these hues can be altered to appear wrong during tone mapping. In one embodiment, color gamut mapping has knowledge (as metadata) of protected colors in the image, thereby ensuring that their hues are preserved during the tone mapping process. The use of this metadata allows for the definition and highlighting of protected colors in the image, thereby ensuring accurate handling of memory colors. The ability to define local tone mapping and color gamut mapping parameters is an example of how metadata is not necessarily generated solely from the parameters of the reference and / or target displays.

[0025] [One embodiment of robust color management] In some embodiments of this application, systems and methods for providing a robust color management scheme are disclosed, which, by employing several sources of metadata, provide higher image / video fidelity that matches the original intent of the content creator. As will be described in more detail herein, in one embodiment, various metadata sources can be added to the process depending on the availability of some metadata.

[0026] Figure 2A shows a high-level block diagram of an image / video pipeline 200 using metadata, as just one example. Block 202 allows for image generation and post-production. The video source 208 is input to the video encoder 210. Simultaneously with the capture of the video source, metadata 204 is supplied to the metadata encoder 206. Examples of metadata 204 have already been described, but it can include items such as the color gamut boundaries and other parameters of the source and / or reference display, the environment of the reference display, and other encoding parameters. In one embodiment, the metadata is accompanied by the video signal as a subset of the metadata, and is temporally and spatially arranged with the video signal that is to be rendered at that time. The metadata encoder 206 and the video encoder 210 can be considered together as the source image encoder.

[0027] The video signal and metadata are then distributed by distribution 212 by any suitable method, such as multiplexing, serializing, parallelizing, or any other known method. It should be recognized that distribution 212 is intended to be broad for the purposes of this application. Suitable distribution methods may include the internet, DVD, cable, satellite, wireless, wired, etc.

[0028] The video signal and metadata thus delivered are supplied to the target display environment 220. The metadata decoder 222 and the video decoder 224 each receive their respective data streams and provide decoding that is particularly suited to the characteristics of the target display among many factors. The metadata at this point can preferably be sent to either a third-party color management (CM) block 220 and / or one of the embodiments of the CM module 228 of this application. If the video and metadata are processed by the CM block 228, the CM parameter generator 232 can take the metadata from the metadata decoder 222, along with the metadata from the metadata prediction block 230, as input.

[0029] The metadata prediction block 230 can make some predictions for higher-fidelity rendering based on knowledge of previous image or video scenes. The metadata prediction block collects statistics from the input video stream to estimate metadata parameters. One possible embodiment of the metadata prediction block 230 is shown in Figure 2B. In this embodiment, a logarithmic histogram 262 of image luminances can be calculated for each frame. An optional low-pass filter 260 can be placed prior to the histogram, thereby (a) reducing the histogram's sensitivity to noise and / or (b) partially accounting for natural blur in the human visual system (e.g., humans perceive dither patterns as monochromatic patches). Then, the minimum point 266 and maximum point 274 are obtained. Furthermore, toe point 268 and shoulder point 272 can be obtained based on percentile settings (such as 5% and 95%). Furthermore, a geometric mean 270 (logarithmic mean) can be calculated and used as a midpoint. These values ​​can be filtered temporally, for example, to prevent them from changing too quickly. These values ​​can also be reset during scene changes, if necessary. Scene changes can be detected by inserting black frames, extremely rapid changes in the histogram, or other such methods. As is evident, the scene change detector 264 can detect scene changes from histogram data or directly from video data, as shown in the figure.

[0030] In yet another embodiment, the system may calculate the average of image intensity values ​​(luminance). In this case, the image intensity can be scaled using perceptual weighting such as logarithmic, power-law, or lookup table (LUT). The system may then estimate highlight and shadow regions (e.g., headroom and footroom in Figure 5) from predetermined percentile values ​​(e.g., 10% and 90%) of the image histogram. Alternatively, the system may estimate highlight and shadow regions based on whether the slope of the histogram exceeds or falls below a certain threshold. Numerous variations are possible; for example, the system can calculate the maximum and minimum values ​​of the input image, or calculate them from predetermined percentile values ​​(e.g., 1% and 99%).

[0031] In other embodiments, these values ​​can be stabilized over time (e.g., from frame to frame) by fixed rates of increase and decrease. Rapid changes may indicate a scene change, and therefore, these values ​​may not be subject to stabilization over time. For example, if a change is below a certain threshold, the system may limit the rate of change; otherwise, it may proceed with the new value. Alternatively, the system may refuse to allow certain values ​​(such as letterboxing or zero values) to affect the shape of the histogram.

[0032] Furthermore, the CM parameter generator 232 can incorporate other metadata (i.e., not necessarily based on content creation), such as display parameters, the display environment, and user-selected factors, into the color management of image / video data. As is evident, the CM parameter generator 232 makes display parameters available through standard interfaces such as EDID (Extended Display Identification Data), via interfaces such as DDC (Display Data Channel) serial interface, HDMI (High-Definition Multimedia Interface), and DVI (Digital Visual Interface). Display environment data can also be supplied by an ambient light sensor (not shown) that measures ambient light conditions and their reflection from the target display.

[0033] Once the CM parameter generator 232 receives any appropriate metadata, it can set parameters in the downstream CM algorithm 234, which can handle the final mapping of image / video data on the target display 236. It should be recognized that a functional branch as shown in the illustration is not required between the CM parameter generator 232 and the CM algorithm 234. In fact, in some embodiments, these functions can be combined into a single block.

[0034] As is clear, the various blocks comprising Figures 2A and 2B are similarly optional from the viewpoint of this embodiment, and by adopting or omitting these enumerated blocks, numerous other embodiments can be constructed, which are within the scope of this application. Furthermore, CM processing can be performed at various different points in the image pipeline 200, and is not necessarily as shown in Figure 2A. For example, CM for a target display can be placed within and included in the target display itself, or such processing can be performed in a set-top box. Alternatively, depending on the level of metadata processing available or deemed appropriate, CM for a target display can be performed at the time of delivery or at the time of post-production.

[0035] [Scalable color management using various levels of metadata] In some embodiments of this application, systems and methods for providing a scalable color management scheme are disclosed, which allows several metadata sources to be delivered to a range of different levels of metadata in order to provide images / videos that are more faithful to the original intent of the content creator. As will be described in more detail herein, in one embodiment, different levels of metadata can be added to the processing depending on the availability of some metadata.

[0036] In many embodiments of this system, a suitable metadata algorithm can take into account a great deal of information, such as the following: (1) Encoded video content (2) A method for converting encoded content into linear light. (3) Color gamut boundaries of the source content (both luminance and chromaticity) (4) Information regarding the post-production environment A method for converting to linear light is sometimes desirable because it allows for the calculation of the actual appearance of the image (luminance, color gamut, etc.) as observed by the content creator. Color gamut boundaries are useful for pre-specifying the outermost colors, which allows such outermost colors to be mapped to the target display without clipping or leaving too much overhead. Information about the post-production environment is desirable because it models external factors that may affect the appearance on the display.

[0037] In current video distribution systems, only encoded video content is provided to the target display. It is assumed that the content was produced in a reference studio environment using reference displays compliant with Rec601 / 709 and various SMPTE (Society of Motion Picture and Television Engineers) standards. The target display system is generally assumed to be Rec601 / 709 compliant, and the target display environment is largely ignored. The fundamental assumption that both the post-production display and the target display are Rec601 / 709 compliant means that neither display may be upgraded without introducing some degree of image distortion. In reality, since Rec601 and Rec709 have slightly different primary color selections, some distortion may already be introduced.

[0038] This specification discloses one embodiment of a metadata-level scalable system that enables the use of reference and target displays with a broader and more advanced range of capabilities. Several different metadata levels allow the CM algorithm to tailor source content for a given target display with improved accuracy. The following sections describe some of the proposed levels of metadata.

[0039] [Level 0] Level 0 metadata is the default case and essentially means zero metadata. Metadata may be absent for several reasons, including:

[0040] (1) The content creator did not include metadata (or the metadata was lost at some point in the post-production pipeline). (2) The display switches content (i.e., channel surfing or commercial breaks).

[0041] (3) Data corruption or loss. In one embodiment, it may be desirable in CM processing to handle level 0 (i.e., no metadata) either by estimating metadata based on video analysis or by assuming default values.

[0042] In such embodiments, the color management algorithm can be enabled to operate in at least two different ways when metadata is unavailable. Switch to default values.

[0043] In this case, the display will operate much like a current distribution system where the post-production reference display characteristics are assumed. The assumed reference display may differ depending on the video encoding format. For example, for 8-bit RGB data, a Rec601 / 709 display can be assumed. If color grading is done on a professional monitor in 600-nit mode (such as a ProMonitor), a P3 or Rec709 color gamut can be assumed for higher bit depth RGB data or LogYuv encoded data. This can work well if there is only one standard or de facto standard for higher dynamic range content. On the other hand, if higher dynamic range content is created under custom conditions, the results may not improve significantly, or may even be unfavorable.

[0044] The parameter values ​​are calculated adaptively. In this case, the CM algorithm starts with several default assumptions, which can be modified based on information obtained from analyzing the source content. Typically, this involves analyzing the video frame histogram, which determines the best way to adjust the brightness of the input source, sometimes by calculating parameter values ​​for the CM algorithm. This carries the risk of producing an "auto-exposure" type image where each scene or frame is balanced to the same brightness level. Furthermore, certain formats may present other challenges; for example, if the source content is in RGB format, there is currently no automated method for determining its color gamut.

[0045] In other embodiments, these two approaches can be combined. For example, color gamut and encoding parameters (such as gamma) can be assumed to be standard default values, and luminance levels can be adjusted using a histogram.

[0046] [Level 1] In this embodiment, Level 1 metadata provides information describing how the source content was created and packaged. This data allows CM processing to predict how the video content actually appeared to the content creator. Level 1 metadata parameters can be categorized into the following three areas:

[0047] (1) Video coding parameters (2) Source display parameters (3) Color gamut parameters of the source content (4) Environmental parameters Video coding parameters Since many color management algorithms operate at least partially in linear light space, it is sometimes desirable to have a way to convert encoded video to a linear (but relative) (X,Y,Z) representation, either specific to the encoding scheme or provided as metadata itself. For example, encoding schemes such as LogYuv, OpenEXR, LogYxy, or LogLuv TIFF inherently contain the information necessary for conversion to a linear light format. However, many RGB or YCbCr formats may require additional information such as gamma and primary colors. As an example, the following information can be supplied to process a YCbCr or RGB input:

[0048] (1) The coordinates of the primary and white points used to encode the source content. These can be used to generate a color space transformation matrix from RGB to XYZ, and the (x,y) coordinates for red, green, blue, and white.

[0049] (2) The minimum and maximum code values ​​(e.g., the "Standard" or "Full" range). These can be used to convert the code values ​​into normalized input values. (3) Overall or channel-specific response curves for each primary color (e.g., "gamma"). These can be used to linearize intensity values ​​by canceling out any nonlinear responses that may have been applied by the interface or reference display.

[0050] Source display color gamut parameters It can be useful for color management algorithms to know the color gamut of the source display. These values ​​correspond to the capabilities of the reference display used for content grading. Color gamut parameters of the source display, preferably measured in a completely dark environment, may include the following:

[0051] (1) Primary colors given as CIE·xy chromaticity coordinates or XYZ, specified along with maximum luminance. (2) White and black tristimulus values, such as CIE XYZ.

[0052] Source content color gamut parameters It can be useful for color management algorithms to understand the color gamut boundaries used to create the source content. Generally, these values ​​correspond to the capabilities of the reference display used for content grading. However, they can vary depending on software settings or if only a portion of the display's capabilities are used. In some cases, the color gamut of the source content may not match the color gamut of the encoded video data. For example, video data may be encoded in LogYuv (or other encodings) that cover the entire visible spectrum. Source color gamut parameters may include the following:

[0053] (1) Primary colors given as CIE·xy chromaticity coordinates or XYZ, specified along with maximum luminance. (2) White and black tristimulus values, such as CIE XYZ.

[0054] Environmental parameters In certain situations, simply knowing the light levels from a reference display may not be sufficient to determine how the source content "appeared" to the viewer in post-production. Information about the light levels generated by the surrounding environment can also be useful. The combined light from both the display and the environment is the signal that reaches the human eye and creates the "look." It is sometimes desirable to maintain this look throughout the video pipeline. Preferably, environmental parameters measured in a typical color grading environment may include the following:

[0055] (1) Peripheral color of the reference monitor, given as absolute XYZ values. This value can be used to estimate the viewer's level of adaptation to their environment. (2) The absolute XYZ values ​​of the black level of the reference monitor in a typical color grading environment. These values ​​can be used to identify the influence of ambient lighting on the black level.

[0056] (3) The color temperature of the ambient light, given as the absolute XYZ values ​​of a white reflective sample (such as paper) in front of the screen. This value can be used to estimate the viewer's adaptation to the white point. As mentioned above, Level 1 metadata can provide parameters for the source content's color gamut, encoding, and environment. This can enable CM solutions to predict how the source content looked when it was approved. However, much guidance is not provided on how to optimally adjust color and brightness to target displays.

[0057] In one embodiment, applying a single sigmoidal curve globally to video frames in RGB space can be a simple and stable method for mapping between different dynamic ranges of source and target. Furthermore, each channel (R, G, B) can be modified independently using a single sigmoidal curve. Such a curve can also be sigmoid in some perceptual space, such as a logarithmic or power function. An example of a curve 300 is shown in Figure 3. As is clear, other mapping curves are also suitable, such as linear mapping (as shown in Figures 3, 4, and 6) or other mappings such as gamma.

[0058] In this case, the minimum and maximum points on the curve can be determined from the Level 1 metadata and information about the target display. The exact shape of the curve can be static, or it can be determined to work well on average based on the input / output range. It can also be adaptively modified based on the source content.

[0059] [Level 2] Level 2 metadata provides additional information about the characteristics of the source video content. In one embodiment, Level 2 metadata can divide the luminance range of the source content into specific luminance regions. More specifically, in one embodiment, the luminance range of the source content may be divided into five regions, in which case these regions can be defined by several points along the luminance range. Such ranges and regions can be defined in a single image, a series of images, a single video scene, or multiple video scenes.

[0060] For illustrative purposes, one embodiment using level 2 metadata is shown in Figures 4 and 5. Figure 4 is a mapping 400 of input luminance to output luminance on a target display. In this case, the mapping 400 is shown as a substantially sigmoid curve including a series of breakpoints along the curve. These points can correspond to image processing related values, and in the figure, min in , foot in , mid in , head in , max in are indicated.

[0061] In the present embodiment, min in and max in can correspond to the minimum luminance value and the maximum luminance value of the scene. The third point, min in can be a median value corresponding to a perceptual "average" luminance value or "mid-tone". The last two points, foot in and head in can be a footroom value and a headroom value. The region between the footroom value and the headroom value can define a significant portion of the dynamic range of the scene. It may be desirable to ensure that the content between these points is protected as much as possible. Content below the footroom can be compressed as needed. Content above the headroom corresponds to highlights and can be clipped as needed. It should be appreciated that these points tend to define the curve itself, and thus other embodiments may result in a curve that best fits these points. Also, such a curve can assume a linear, gamma, sigmoid, or other suitable and / or desirable shape.

[0062] Furthermore, in this embodiment, Figure 5 shows the minimum, footroom, center, headroom, and maximum points when displayed on the histogram plot 500. Such histograms can be created per image, per video scene, or even per series of video scenes, depending on the granularity of the histogram analysis required to help maintain content fidelity. In one embodiment, these five points can be specified by code values ​​of the same encoded representation as the video data. It should be noted that min and max usually correspond to the same values ​​as the range of the video signal, but this is not always the case.

[0063] Depending on the granularity and frequency of such histogram plots, histogram analysis can be used to dynamically redefine points along the luminance map in Figure 4, thereby changing the curve over time. This can also be used to improve the fidelity of the content displayed to the viewer on the target display. For example, in one embodiment, by periodically providing histograms, the decoder can potentially obtain more information than just min, max, etc. Alternatively, the encoder can include new histograms only when there are significant changes. This saves the decoder the trouble of calculating it on the fly for each frame. In yet another embodiment, the histogram is used to estimate metadata, either to replace missing metadata or to supplement existing metadata.

[0064] [Level 3] In one embodiment, the parameters of the Level 1 and Level 2 metadata can be used as Level 3 metadata for a second reference grading of the source content. For example, the primary grading of the source content may be performed on a reference monitor (e.g., ProMonitor) using the P3 color gamut at a luminance of 600 nits. The Level 3 metadata can similarly provide information about a secondary grading, which may be performed on a CRT reference, for example. In this case, the additional information would indicate primary colors of Rec601 or Rec709 and lower luminances, such as 120 nits. The corresponding min, foot, mid, head, and max levels are also supplied to the CM algorithm.

[0065] Level 3 metadata allows for the addition of additional data, such as color gamut, environment, and primary colors, as well as luminance level information for a second reference grading of the source content. This additional information can then be used to define a sigmoid curve 600 (as shown in Figure 6) that maps the primary input to the range of the reference display. Figure 6 illustrates an example of how input and reference display (output) levels can be combined to form an appropriate mapping curve.

[0066] If the capabilities of the target display closely match those of the secondary reference display, this curve can be used directly to map the primary source content. On the other hand, if the capabilities of the target display are intermediate between those of the primary and secondary reference displays, the mapping curve for the secondary display can be used as a lower bound. The curve actually used for the target display can then be an interpolation between a no-reduction curve (e.g., a linear mapping 700 as shown in Figure 7) and a full-range reduction curve generated using the reference level.

[0067] [Level 4] Level 4 metadata is the same as Level 3 metadata, except that the metadata for the second reference grading is adjusted to match the actual target display.

[0068] Level 4 metadata can also be implemented in over-the-top (OTT) scenarios (i.e., Netflix, mobile streaming, or other video-on-demand (VOD) services) where an actual target display transmits its characteristics to a content provider and content is delivered along with the best available curve. In one such embodiment, the target display can communicate with a video streaming service, VOD service, etc., and the target display can supply the data streaming service with information such as its EDID data or other appropriate metadata available. Such a communication channel is shown in Figure 2A as a dotted line path 240 to either a video encoder and / or a metadata encoder (210 and 206, respectively), as is known in the art for services such as Netflix. Generally, Netflix and other such VOD services monitor the amount of data and data throughput rate to the target device, and the metadata is not necessarily for color management. However, for the purposes of this embodiment, it is sufficient that metadata is transmitted from the target data to the creation or post-production unit (in real time or in advance) by distribution 212 or other means in order to change the color, tone, or other characteristics of the image data delivered to the target display.

[0069] The reference luminance level provided by the Level 4 metadata is specific to its target display. In this case, the sigmoid curve can be constructed as shown in Figure 6 and used directly without interpolation or adjustment.

[0070] [Level 5] Level 5 metadata extends Level 3 or Level 4 by identifying prominent features such as the following:

[0071] (1) Protected colors: Colors in an image that are identified as common memory colors that should not be processed, such as skin tones, sky colors, and grass colors. Areas of an image with such protected colors may have image data that is fed to the target display without modification.

[0072] (2) Prominent highlights: Shows light sources, maximum emission, and reflected highlights. (3) Out-of-gamut: Feature areas within an image that have been intentionally color-graded outside the color gamut of the source content.

[0073] In some embodiments, such objects can be artificially mapped to the maximum brightness of the target display if the target display is capable of handling higher brightness levels. If the target display is capable of handling lower brightness levels, these objects can be clipped to the maximum brightness of the display without any detail being preserved. In this case, these objects are ignored, and a defined mapping curve is applied to the remaining content to preserve more detail.

[0074] It should be recognized that, in some embodiments, such as when attempting to map a VDR to a display with a lower dynamic range, knowing the light source and highlights can be useful because it may be possible to clip them without significant impact. For example, a face that is brightly lit by backlighting (i.e., not by any light source) is not a feature that should be clipped. Alternatively, such feature areas may be compressed more gradually. In yet another embodiment, if the target display can support a wider color gamut, those content objects can be stretched to the full capacity of the display. In yet another embodiment, the system may ignore mapping curves defined to guarantee high-saturation colors.

[0075] It should be recognized that, in some embodiments of this application, the levels themselves do not constitute a strict hierarchy of metadata processing. For example, level 5 can be applied to either level 3 or level 4 data. Also, some lower numbered levels may not exist, and the system can process higher numbered levels if they exist.

[0076] [One embodiment of a system using multiple metadata levels] As described above, various metadata levels provide increased information about the source material, enabling the CM algorithm to provide more accurate mapping for the target display. One embodiment employing several such scalable levels of metadata is shown in Figure 8.

[0077] The illustrated system 800 represents the entire video / metadata pipeline through five blocks: a creation unit 802, a container 808, an encoding / distribution unit 814, a decoding unit 822, and a consumption unit 834. As is evident, numerous variations of several different implementations are possible, some with more blocks and some with fewer blocks. The scope of this application should not be limited to the embodiments described herein, and in fact, the scope of this application covers these various implementations and embodiments.

[0078] The generation unit 802 broadly acquires image / video content 804 and processes it using a color grading tool 806 as described above. The processed video and metadata are placed in a suitable container 810 in any appropriate format or data structure for subsequent distribution, for example, known in the art. For example, the video can be stored and supplied as VDR color-graded video, and the metadata as VDR-XML format metadata. This metadata is divided into several levels as described above, as shown in 812. The container block can embed data in the formatted metadata that encodes which levels of metadata are available and associated with the image / video data. It should be recognized that not all levels of metadata need to be associated with the image / video data; wherever metadata and levels are associated, downstream decoding and rendering can identify and process such available metadata as appropriate.

[0079] Encoding can proceed by acquiring metadata and supplying it to the algorithm parameter determination block 816, while the video can be supplied to the AVCVDR encoder 818, which includes a CM block for processing the video prior to distribution 820.

[0080] Once delivered (by a wide range of means, such as the internet, DVD, cable, satellite, wireless, wired, etc.), the video data / metadata can be decoded to the AVCVDR decoder 824 (or optionally to the legacy decoder 826 if the target display is not VDR compatible). Both the video data and metadata are restored by decoding (as blocks 830 and 828, respectively, and optionally as block 832 if the target display is legacy). The decoder 824 can take the input image / video data and restore the input image data, separating it into an image / video data stream to be further processed and rendered, and a metadata stream for calculating parameters for subsequent CM algorithm processing in the rendered image / video data stream. The metadata stream further contains information on whether there is metadata associated with that image / video data stream. If there is no associated metadata, the system can proceed with level 0 processing as described above. Otherwise, the system can proceed with further processing with all metadata associated with the image / video data stream according to a set of different levels of metadata as described above.

[0081] As is clear, the presence or absence of metadata associated with rendered image / video data can be determined in real time. For example, some sections of a video stream may lack metadata associated with them (due to data corruption or the content creator's preference for no metadata), while other sections may have metadata available, and in some cases, a range of many metadata at various levels may be associated with other sections of the video stream. This may be at the content creator's discretion, however, in at least one embodiment of this application, it must be possible to determine the presence or absence, or at what level, of metadata associated with such a video stream in real time or substantially dynamically.

[0082] In the processing block, the algorithm parameter determination block 836 can restore previous parameters that are thought to have been processed before distribution, or recalculate parameters based on metadata from the target display and / or target environment (which are thought to be from standard interfaces such as EDID or modern VDR interfaces, or even input from viewers or sensors in the target environment, as described above in the context of embodiments in Figure 2A and / or 2B). Once the parameters are calculated or restored, they can be sent to one or more of the CM systems (838, 840, and / or 842) for final mapping of source and intermediate image / video data to the target display 844 according to some embodiments disclosed herein.

[0083] In other embodiments, the implementation block in Figure 8 does not need to be subdivided further. For example, generally speaking, the process including algorithm parameter determination and the color management algorithm itself does not necessarily need to branch as shown in Figure 8, but can assume a color management module and / or be implemented as a color management module.

[0084] Furthermore, while this specification describes a series of various levels of metadata used in a video / image pipeline, it should be recognized that, in practice, the system does not need to process image / video data in the order of the numbers assigned to the metadata levels. In fact, some levels of metadata may be available during rendering while others are not. For example, second-level reference color grading may or may not be performed, and level 3 metadata may or may not be present during rendering. The system configured according to this application will take into account the presence or absence of various levels of metadata and continue with the best metadata processing at that time.

[0085] The above provides a detailed description of one or more embodiments of the present invention, to be read in conjunction with the accompanying drawings illustrating the principles of the present invention. It should be recognized that, although the present invention is described in relation to such embodiments, the present invention is not limited to any of these embodiments. The scope of the present invention is limited only by the claims, and the present invention covers a number of alternatives, variations, and equivalents. Various specific details are described herein in order to give a complete understanding of the present invention. These details are presented for illustrative purposes only, and the present invention can be carried out according to the claims even if some or all of these specific details are omitted. For the purpose of clarity, and to avoid unnecessarily obscuring the invention, technical matters known in the art related to the present invention are not described in detail.

Claims

1. A device for generating a bitstream containing encoded image data and metadata, It comprises an input section for accessing a source video signal to generate encoded image data, and a processor, The aforementioned processor, Output metadata related to the encoded image data is generated, A bitstream is generated that includes the encoded image data and the output metadata. The output metadata includes a first set of metadata and a second set of metadata, The first set of metadata includes one or more parameters associated with a portion of the image data, which describe the characteristics of a reference display device used to create the image data for color grading the source video signal. The one or more of the above parameters are, a. The white point of the reference display device and b. The three primary colors of the reference display device, c. Including at least one brightness level of the reference display device, The second set of metadata is associated with the same portion of the image data and includes at least the maximum brightness level of the image data, in the apparatus.

2. The apparatus according to claim 1, wherein the metadata of the first set and the metadata of the second set are separated separately in the bitstream.

3. The apparatus according to claim 1, further comprising distributing the bitstream.

Images