Metadata-Based Power Management

Abstract

The method and apparatus therefor include: receiving image data and power metadata, the power metadata including information regarding power consumption or expected power consumption; determining, based on the power metadata, an amount and duration of drive modification that may be performed by the target display in response to the power consumption or the expected power consumption; and performing power management of the target display based on the power metadata, based on the results of the determination, modifying the drive of at least one light-emitting element associated with the target display relative to a manufacturer-determined threshold, the power metadata including at least one of temporal luminance energy metadata, spatial luminance energy metadata, spatial-temporal variation metadata, or a combination thereof.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to the following priority applications: U.S. Provisional Application No. 63 / 004,019, filed April 2, 2020 (Docket No. D19107USP1), and European Application No. 20171001.9, filed April 23, 2020 (Docket No. D19107EP).

[0002] 1. Fields of Disclosure This application relates generally to images, and more particularly, this application relates to metadata-based power management in displays. [Background technology]

[0003] 2. Description of Related Art As used herein, the term "metadata" refers to any auxiliary information transmitted as part of an encoded bitstream that assists a decoder in rendering a decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters, as described herein.

[0004] In practice, an image includes one or more color components (e.g., RGB, luma Y, and chroma Cb and Cr), where in a quantized digital system, each color component is represented with n bits of precision per pixel (e.g., n=8). Bit depths of n≦8 (e.g., color 24-bit JPEG images) may be used with standard dynamic range (SDR) images, while bit depths of n>8 may be considered for enhanced dynamic range (EDR) images to avoid contouring and staircase artifacts. In addition to integer data types, EDR and high dynamic range (HDR) images may be stored and distributed using high-precision (e.g., 16-bit) floating-point formats, such as the OpenEXR file format developed by Industrial Light and Magic.

[0005] Many consumer desktop displays have a maximum brightness of 200-300 cd / m 2 non-EDR content, and consumer high-definition and ultra-high-definition television ("HDTV" and "UHD TV") at 300-400 nits. As such, such display output is representative of low dynamic range (LDR), also known as SDR, in the context of HDR or EDR. As EDR content becomes more available due to advances in both capture equipment (e.g., cameras) and EDR displays (e.g., the Sony Trimaster HX 31" 4K HDR Master Monitor), EDR content may be color graded and displayed on EDR displays that support a higher dynamic range (700-5000 nits or higher). In general, the systems and methods described herein relate to any dynamic range.

[0006] Regardless of dynamic range, video content includes a series of still images (frames) that can be grouped into sequences such as shots and scenes. A shot is, for example, a collection of temporally connected frames. Shots may be separated by "shot cuts" (e.g., points in time when the entire content of an image changes, not just a portion of it). A scene is, for example, a sequence of shots that describes a story-telling segment of larger content. In the particular example where the video content is an action movie, the video content may include (among other things) a chase scene, which may include a series of shots (e.g., a shot of the driver of the pursuing vehicle, a shot of the driver of the vehicle being pursued, a shot of the street where the chase takes place, etc.).

[0007] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, it should not be assumed that problems identified with one or more approaches have been recognized in any prior art based on this section unless specifically noted. Summary of the Invention [Means for solving the problem]

[0008] Various aspects of the present disclosure relate to circuits, systems, and methods for image processing, including metadata-based power management in displays.

[0009] In one exemplary aspect of the present disclosure, a method is provided that includes: receiving image data and power metadata, wherein the power metadata includes information regarding power consumption or expected power consumption; determining, based on the power metadata, an amount and duration of drive modifications that may be performed by a target display in response to the power consumption or the expected power consumption; and performing power management of the target display based on the power metadata, modifying drive of at least one light-emitting element associated with the target display relative to a manufacturer-determined threshold based on a result of the determination, wherein the power metadata includes at least one of temporal luminance energy metadata, spatial luminance energy metadata, spatial-temporal variation metadata, or a combination thereof.

[0010] In another exemplary aspect of the present disclosure, there is provided an apparatus having a display including at least one light emitting element and display management circuitry configured to perform the following steps: receive power metadata, the power metadata including information regarding power consumption or expected power consumption; determine, based on the power metadata, an amount and duration of drive modification that may be performed by the display in response to the power consumption or the expected power consumption; and perform power metadata-based power management of the display, based on results of the determination, modifying drive of the at least one light emitting element relative to a manufacturer-determined threshold, wherein the power metadata includes at least one of temporal luminance energy metadata, spatial luminance energy metadata, spatial-temporal variation metadata, or a combination thereof.

[0011] In this manner, various aspects of the present disclosure provide improvements in at least the art of image processing and display, and the related arts of image capture, encoding, and broadcast. [Brief explanation of the drawings]

[0012] These and other more detailed and specific features of the various embodiments are more fully disclosed in the following description, taken in conjunction with the accompanying drawings. [Figure 1] 1 illustrates an exemplary video delivery pipeline in accordance with various aspects of the present disclosure. [Figure 2A] 1 illustrates an exemplary metadata generation process according to various aspects of the present disclosure. [Figure 2B] 1 illustrates an exemplary metadata generation process according to various aspects of the present disclosure. [Figure 3A] 10 illustrates another exemplary metadata generation process according to various aspects of the present disclosure. [Figure 3B] 10 illustrates another exemplary metadata generation process according to various aspects of the present disclosure. [Figure 4] 1A-B illustrate exemplary data streams according to various aspects of the present disclosure. [Figure 5] 1 illustrates an exemplary metadata hierarchy according to various aspects of the present disclosure. [Figure 6] 1 illustrates an exemplary operational timeline according to various aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] The present disclosure and aspects thereof may be embodied in various forms, including hardware or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces and application programming interfaces, as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits, field programmable gate arrays, etc. The foregoing summary is intended merely to give a general idea of ​​various aspects of the disclosure and is not intended to limit the scope of the disclosure in any way.

[0014] In the following description, numerous details are set forth, such as spectrum, timing, operation, etc., to provide an understanding of one or more aspects of the present disclosure. It will be readily apparent to those skilled in the art that these specific details are merely examples and are not intended to limit the scope of the present application.

[0015] Additionally, while this disclosure primarily focuses on examples in which various elements are used in a consumer display system, it will be understood that this is but one example of implementation. It will be further understood that the disclosed systems and methods can be used in any device that needs to display image data, such as movie theaters, consumer and other commercial projection systems, smartphones and other consumer electronic devices, heads-up displays, virtual reality displays, etc.

[0016] overview Display devices contain several components, including light-emitting pixels in self-emissive display technologies such as organic light-emitting diode displays (OLEDs) or plasma display panels (PDPs), or backlights in other display technologies that use transmissive light modulators, such as liquid crystal displays (LCDs). In such devices, when various components are driven beyond their technical and physical limits, expected behavior, such as color rendition, can be impaired, increasing the failure rate of the display system. Such driving can lead to temporary or permanent component failure. To remedy this, some component manufacturers (often called original equipment manufacturers, or OEMs) may limit technological capabilities by applying operational thresholds. For example, component manufacturers may apply thresholds related to power consumption for components such as light-emitting diodes (LEDs), LED driver chips, and power supplies. Additionally or alternatively, component manufacturers may apply thresholds related to thermal properties, such as spatial heat propagation through the display chassis.

[0017] These thresholds are typically conservative to avoid potential publicity or branding issues, such as when a relatively rare failure becomes the subject of unfavorable press, and to prevent an increase in service calls to the component manufacturer's support and customer service groups, thereby increasing costs to the component manufacturer. However, thresholds may be conservative enough to avoid approaching the technical limitations of the display system in practice. Component manufacturers may choose to set conservative thresholds because, in comparative examples, content characteristics related to energy consumption are not known prior to playback. Thus, energy management parameters in display devices are often evaluated in real time; for example, signal input may be analyzed at or just before display time.

[0018] However, if the power consumption that occurs or is expected to occur during content playback is known in advance, a power management system within the display device may be able to modify the display drive (e.g., adjust the brightness rendering requirements of the content). Some non-limiting examples of adjustments include limiting brightness to conserve power (e.g., when the device is running on battery power) and / or exceeding maximum brightness output determined by manufacturer-determined safety thresholds, if the duration of the overdrive is known to cause no long-term harm to the display system or its components. These are sometimes referred to as performing "underdrive" or "overdrive." In some examples, evaluation of overdrive (or underdrive) level and duration may be performed during the content creation or content distribution process, and light-emitting elements of the display system may then be selectively overdriven (or underdriven) as a result of the evaluation.

[0019] FIG. 1 illustrates an exemplary video delivery pipeline, showing various stages from video capture to video content display. Additionally, while the following description is provided with respect to video (i.e., moving images), this disclosure is not limited thereto. In some examples, the image content may be still images or a combination of video and still images. The image content may be represented by raster (or pixel) graphics, vector graphics, or a combination of raster and vector graphics.

[0020] FIG. 1 illustrates an image generation block 101, a production block 102, a post-production block 103, an encoding block 104, a decoding block 105, and a display management block 106. The various blocks illustrated in FIG. 1 may be implemented as or via hardware, software, firmware, or a combination thereof. Additionally, various groups of the illustrated blocks may combine their respective functions and / or execute on different devices and / or at different times. Individual or groups of the illustrated blocks may be implemented via circuitry including, but not limited to, a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and combinations thereof. Operations performed by one or more of these blocks may be processed locally, remotely (e.g., cloud-based), or a combination of local and remote.

[0021] 1, the video delivery pipeline further includes a reference display 111, which may be provided to assist or monitor the operations performed in the post-production blocks, and a target display 112. For purposes of explanation, the image generation block 101, the production block 102, the post-production block 103, and the encoding block 104 may be referred to as "upstream" blocks or components, while the decoding block 105 and the display management block 106 may be referred to as "downstream" blocks or components.

[0022] 1, a series of video frames 121 are captured or generated in image generation block 101. Video frames 121 may be captured digitally (e.g., by a digital camera) or generated by a computer (e.g., using computer animation) to generate video data 122. Alternatively, video frames 121 may be captured on film by a film camera and then converted to a digital format to provide video data 122. In either case, video data 122 is provided to production block 102, where it is edited to provide production stream 123.

[0023] The video data in the production stream 112 is then provided to a processor(s) in the post-production block 103 for post-production editing. Editing performed in the post-production block 103 may include adjusting or modifying the color or brightness of specific areas of the image to improve image quality or achieve a particular look for the image according to the video creator's (or editor's) creative intent. This is sometimes referred to as "color timing" or "color grading." Other editing (e.g., scene selection and sequencing, image cropping, adding computer-generated visual special effects or overlays, etc.) can be performed in the post-production block 103 to generate the distribution stream 124. In some examples, the post-production block 103 may provide an intermediate stream 125 to the reference display 111 so that the image can be viewed on that screen, for example, to aid in the editing process. One, two, or all of the production block 102, post-production block 103, and encoding block 104 may further include processing to add metadata to the video data. This further processing may include, but is not limited to, statistical analysis of content properties, and may be performed locally or remotely (e.g., cloud-based processing).

[0024] Following post-production, distribution stream 124 may be delivered to encoding block 104 for downstream distribution to decoding and playback devices such as television sets, set-top boxes, movie theaters, laptop computers, tablet computers, etc. In some examples, encoding block 104 may include audio and video encoders, such as those defined by the Advanced Television Systems Committee (ATSC), Digital Video Broadcasting (DVB), Digital Versatile Disc (DVD), Blu-Ray, and other distribution formats, to generate encoded bitstream 126. At the receiver, encoded bitstream 126 is decoded by decoding unit 105 to generate decoded signal 127, which represents an identical or approximation of distribution stream 124. The receiver may be attached to a target display 112, which may have different characteristics than the reference display 111. If the reference display 111 and the target display 112 have different characteristics, the display management block 106 may be used to map the dynamic range or other characteristics of the decoded signal 127 to the characteristics of the target display 112 by generating a display-mapped signal 128. The display management block 106 may additionally or alternatively be used to provide power management for the target display 112.

[0025] The target display 112 generates an image using an array of pixels. The specific array structure depends on the display's architecture and resolution. For example, if the target display 112 operates on an LCD architecture, it may include a relatively low-resolution backlight array (e.g., an array of LEDs or other light-emitting elements) and a relatively high-resolution liquid crystal array and color filter array that selectively attenuates the white light from the backlight array to provide colored light (often referred to as dual-modulation display technology). If the target display 112 operates on an OLED architecture, it may include a high-resolution array of self-emitting color pixels.

[0026] The link between the upstream block and the downstream block (i.e., the path through which the encoded bitstream 126 is provided) may be embodied via live or real-time transport, e.g., wireless broadcast using electromagnetic waves, or content delivery lines such as fiber optics, twisted pair (Ethernet), and / or coaxial cable. In other examples, the link may be embodied by time-independent transport, such as recording the encoded bitstream on a physical medium (e.g., a DVD or hard disk) for physical delivery to an end-user device (e.g., a DVD player). The decoder block 105 and the display management block 106 may be incorporated into a device associated with the target display 112, e.g., a device in the form of a smart TV that includes decoding, display management, power management, and display functions. In some examples, the decoder block 105 and / or the display management block 106 may be incorporated into a device separate from the target display 112, e.g., a device in the form of a set-top box or media player.

[0027] The decoder block 105 and / or the display management block 106 may be configured to receive, parse, and act in response to metadata included or added by upstream blocks. Such metadata may then be used to provide additional control or management of the target display 112. The metadata may include imaging metadata (e.g., Dolby Vision metadata) and / or non-imaging metadata (e.g., power metadata).

[0028] Metadata Generation As mentioned above, metadata (including power metadata) may be generated in one or more upstream blocks shown in Figure 1. The metadata may then be combined with the distribution stream (e.g., in encoding block 104) for transmission as part of the coded bitstream 126. The power metadata may include temporal luminance energy metadata, spatial luminance energy metadata, spatial-temporal variation metadata, etc.

[0029] As used herein, temporal luminance energy metadata may include information related to the temporal luminance energy of a particular frame or frames of image data. For example, the temporal luminance energy metadata may provide a snapshot of the total luminance budget utilized by each content frame. This may be expressed as the sum of the luminance values ​​of all pixels in a given frame. In some examples, the above may be resampled to be resolution-independent of the target display 112 (i.e., to fit display resolutions of 1080p, 2k, 4k, and 8k). The temporal luminance energy metadata included within a given frame of the encoded bitstream 126 may include information related to future frames. In one example, the temporal luminance energy metadata included within a given frame may include temporal luminance energy information for the following 500 frames. In another example, the temporal luminance energy metadata included within a given frame may include temporal luminance energy information for a greater or lesser number of subsequent frames. Thus, transmission of the temporal luminance energy metadata need not be performed for each frame in the encoded bitstream 126, but may instead be transmitted intermittently. In some examples, the temporal luma energy metadata included in a given frame, if it includes the temporal luma energy for the following N frames, may be transmitted along with the encoded bitstream 126 at a period shorter than N (e.g., N / 2, N / 3, N / 4, etc.). The more frequently the temporal luma energy metadata is transmitted, the more robust the metadata scheme will be to latency or other data transmission errors. However, the less frequently the temporal luma energy metadata is transmitted, the less data bandwidth will be used to transmit the metadata. One example relationship between the frequency of metadata transmission and the data bandwidth used will be described in more detail below with respect to FIG. 5.

[0030] By transmitting frame-based luminance energy for future frames in advance, the display power manager (e.g., display management block 106) can determine, based on the temporal progression of luminance energy, how to most effectively map content to maintain directorial intent while maximizing hardware capabilities. This may include deciding to overdrive (or underdrive) some or all of the light-emitting elements in the end-user display (e.g., target display 112) for a particular scene or shot, deciding to reduce the luminance of selected or all pixels to conserve electrical energy (e.g., from the battery), determining periods for panel cooling after intensive use or between overdrive periods, etc.

[0031] 2A-2B illustrate an exemplary generation process for temporal luminance energy metadata. FIG. 2A shows an exemplary process flow for generating temporal luminance energy metadata, and FIG. 2B illustrates an exemplary process flow. The illustrated generation process includes, in operation 201, receiving image data for a shot of video content. The shot may include a series of frames, each frame including image data formed by pixels arranged in a two-dimensional array, in this example. In some applications, each frame may include image data for a stereoscopic display, a multi-view display, a light field display, and / or a volumetric display, in which case the image data may be in a form other than a two-dimensional array. Thereafter, in operation 202, a quantity L sum,i (i.e., a quantity representing the sum of the luminance values ​​for all pixel luminance levels in the frame) may be calculated for a given frame i (starting with i at 1 to start with the first frame in the shot) according to the following equation (1):

number

[0032] In the above equation (1), x corresponds to the x coordinate of a pixel in the array, y corresponds to the y coordinate of a pixel in the array, and L xyi represents the intensity of pixel (x, y) for frame i. In equation (1), each frame contains n×m pixels.

[0033] In operation 203, it is determined whether the shot is complete. This can be accomplished by comparing the value of the current frame i to a maximum value P, which represents the total number of frames in the shot. If it is determined that the shot is not complete, frame i is incremented by 1 in operation 204, and the process flow continues by adding the quantity L for the new frame. sum,i Return to operation 202 to calculate the quantity L sum,temporal is generated. Amount L sum,temporal corresponds to the sum of the luminance per frame for the entire shot, and for each frame i from i=1 to i=P, the quantity L sum,i It may be represented as a one-dimensional data array indicating:

[0034] Figure 2B shows this diagrammatically. As input, the process takes multiple frames 2111-2112 of image data. P As output, the process provides the temporal luminance energy metadata 212 for that shot as a one-dimensional data structure, which is plotted here. The x-axis represents the individual frames, and the y-axis represents the spatial luminance sum of the frames.

[0035] Spatial or temporal luminance energy metadata may include information about the total luminance energy of a particular pixel or pixels with specific xy coordinates in the image data across an entire scene or shot. In some display technologies, excess heat must be transported out of the display housing to prevent damage to display device components. For example, in many physical displays, the lower center of the display is most susceptible to excessive heat or heat buildup because the latent energy must pass through a large portion of the rest of the display panel before exiting the housing at the top or sides. To avoid the problem, many component manufacturers limit heat buildup by globally (temporally and / or spatially) limiting the luminance output for a comparison display system, where the comparison system's power manager has no information about the luminance requirements of future frames. In the case of spatial luminance energy metadata, by providing the spatial luminance energy metadata to the end-user display, the display power manager (e.g., display management block 106) can determine how much to drive (or even overdrive or underdrive) light-emitting elements in the end-user display (e.g., target display 112) based on pixel location and intensity or duration.

[0036] 3A-3B illustrate an exemplary generation process for spatial luminance energy metadata. FIG. 3A illustrates an exemplary process flow for generating spatial luminance energy metadata, and FIG. 3B illustrates the exemplary process flow diagrammatically. The illustrated generation process includes, in operation 301, receiving image data for a shot of video content. The shot may include a series of frames, each frame including image data corresponding to each pixel in a two-dimensional array. Thereafter, in operation 302, a quantity L sum,xy(i.e., a quantity representing the total luminance for all frames of a shot for a given pixel) may be calculated for a given pixel (x,y) according to the following equation (2) (x and y start at 1 in this example to start with the top-left pixel):

number

[0037] In operation 303, it is determined whether the row of pixels is complete. This may be accomplished by comparing the value x of the current pixel to a maximum value n, which represents the total number of rows in the array. If it is determined that the row is not complete, then in operation 304, the x coordinate of the pixel is incremented by 1, the y coordinate of the pixel is reinitialized to 1, and process flow returns to operation 302 to calculate the quantity L for the new pixel. sum,xy If the row is determined to be complete, then in operation 305 it is determined whether all rows have been analyzed. This can be accomplished by comparing the value of the current pixel, y, to a maximum value, m, which represents the total number of columns in the array. If the row is determined to be not the last row, then in operation 306 the x coordinate of the pixel is reinitialized to 1, the y coordinate of the pixel is incremented by 1, and the process flow returns to operation 302 to calculate the quantity L for the new pixel. sum,xy If the row is determined to be the last row, then in operation 307 the quantity L sum,spatial is generated. Amount L sum,spatial corresponds to the frame-by-frame luminance sum for each pixel for the entire shot, and the quantity L sum,xy The data may be represented as a two-dimensional data array indicating:

[0038] While Figure 3A illustrates an exemplary process flow in which pixels are analyzed row-by-row, starting with the top-left pixel (1,1), in practice, pixels may be analyzed in any order. In some examples, pixels are analyzed row-by-row, starting with another corner pixel, e.g., the bottom-right pixel (n,m), the top-right pixel (1,m), the bottom-left pixel (n,1), or an interior pixel. In other examples, pixels are analyzed column-by-column, starting with a corner or interior pixel.

[0039] Figure 3B illustrates the above process diagrammatically. As input, the process takes multiple frames 3111-3112 of image data. P As an output, the process provides spatial luminance energy metadata 312 for the shot as a two-dimensional data structure. In the diagrammatic illustration of FIG. 3B, dark regions, such as region 313, correspond to pixel locations where lower luminance image elements are depicted across most or all frames of the shot. This corresponds to lower luminance energy pixels (e.g., lower energy across the time interval from 1 to P). Light regions, such as region 314, correspond to pixel locations where high luminance image elements are depicted across most or all frames of the shot. This corresponds to high luminance energy pixels.

[0040] Light-emitting elements that provide illumination for bright areas (e.g., backlight LEDs in an LCD architecture, or groups of OLED pixels in an OLED architecture) tend to consume more power and / or consume power for longer periods of time when there are also bright portions of the image presented in the same portion of the display over time. Without spatial luminance energy metadata and proper management, this can result in stress on components (e.g., the light-emitting elements themselves, drivers, circuit board traces, etc.), potential heat generation that must flow upward and be removed from the housing, active dimming of pixels or the entire screen, etc. By providing the spatial luminance energy metadata of a shot to the target display 112 before rendering and displaying the shot, these problems and / or component damage can be prevented.

[0041] In addition to, or as an alternative to, calculating spatial luminance energy metadata, spatial-temporal variation metadata may be calculated. The spatial-temporal variation metadata may include information about the energy variation of a particular pixel or pixels of image data across a scene or shot. For example, pixels that remain at roughly the same luminance level throughout a scene or shot will have low energy variation, while pixels that change luminance levels (e.g., display bright, high-frequency strobe lights) will have high energy variation.

[0042] The spatial-temporal variation metadata may be calculated in a similar manner as illustrated in FIG. 3A, except that in operation 302, the quantity L sum,xy Calculation of the quantity L fluct,xy (i.e., a quantity representing the variation over all frames for a given pixel), which may be calculated for a given pixel (x,y) according to the following equation (3) (in this example, x and y start at 1, since we start from the top left pixel):

number

[0043] In equation (3), σ represents the standard deviation function. In some examples, both spatial luminance energy metadata and spatial-temporal variation metadata may be calculated in operation 302. In other examples, the process flow of FIG. 3A may be executed twice in succession, whereby a first process flow calculates spatial luminance energy metadata and a second process flow calculates spatial-temporal variation metadata (or vice versa).

[0044] In some cases, the skewness of the luminance distribution

number

number

[0045] Metadata Transmission In some implementations, the power metadata described above may be transmitted as part of the encoded bitstream 126 along with the actual image data and any additional metadata that may be present. In other implementations, the power metadata may be transmitted over a different transmission path than the actual image data ("side-loaded"); for example, the power metadata may be transmitted from the Internet or other distribution device via TCP / IP, Bluetooth, or other communications standards. FIG. 4A shows an example of a frame of image data in which power metadata is transmitted as part of the encoded bitstream 126. In this example, the frame of image data includes metadata used for image formation 401, power metadata 402, and image data 403. Image formation metadata 401 may be any metadata used to render the image on the screen (e.g., tone mapping data). Image data 403 includes the actual content (e.g., image pixels) to be displayed on the screen.

[0046] As noted above, power metadata (including temporal luminance energy metadata, spatial luminance energy metadata, spatial-temporal variation metadata, and combinations thereof) is a type of non-imaging metadata. That is, an image can be rendered without power metadata or with only a partial set of power metadata. Thus, in contrast to the imaging metadata used to accurately render an image, less than the complete set of power metadata can be encoded in each content frame. Power metadata may be embedded out of sequence or piecemeal. Furthermore, missing portions of power metadata may be interpolated from the portions of power metadata present or simply ignored without adversely affecting basic image fidelity.

[0047] In one example of the present disclosure, power metadata is segmented and transmitted as pieces or pages for each content frame (e.g., as part of the encoded bitstream 126). FIG. 4B illustrates a series of frames (here, two frames) of image data following this operation. In FIG. 4B, each frame includes imaging metadata 401 and image data 403 corresponding to that frame. However, compared to FIG. 4A, each frame does not include the entire set of power metadata 402. In this example, power metadata 402 is divided into N pieces. Thus, the first frame includes a first portion 402-1 of power metadata, the second frame includes a second portion 402-2 of power metadata, and so on, until all N portions of power metadata are transmitted. The power manager (e.g., the decode block 105 and / or the display management block 106) first determines whether power metadata is present for the current frame, scene, or shot, and then acts in response to the determination. For example, if no power metadata exists for the current frame, scene, or shot, the power manager may simply treat the frame, scene, or shot as is (i.e., no overdrive / underdrive or power consumption mapping). However, if power metadata exists for the current frame, scene, or shot, the power manager may adjust the display mapping (e.g., in the display management block 106 or the target display 112) and / or the power consumption and / or mapping behavior of the display hardware. The power manager may also store some additional power metadata (e.g., power metadata for future frames) in a buffer or other memory to derive a preferred mapping strategy. An example could be power metadata that is pre-submitted before the actual image frame is rendered and displayed. During playback, the power manager can apply any pre-buffered power metadata to improve rendering behavior.

[0048] The amount of frames (i.e., N) budgeted for transporting power metadata 402 is based on the size of the payload and bandwidth allocation for this particular metadata type. Each piece of power metadata 402 may not have the same length (i.e., total bytes) as a frame interval of the content, and thus the rate (bytes / frame) for power metadata 402 may not be the same as the rate for imaging metadata 401. Furthermore, in instances where temporal luminance energy metadata, spatial luminance energy metadata, and spatial-temporal variation metadata are all implemented, some types of power metadata may be calculated or derived from other types of power metadata.

[0049] FIG. 5 illustrates an exemplary metadata hierarchy according to various aspects of the present disclosure. The metadata hierarchy generally has a pyramidal shape, with higher layers of the pyramid corresponding to coarser metadata (and thus have smaller data payloads and / or cover longer time intervals of the content) and lower layers of the pyramid corresponding to finer metadata (and thus have larger data payloads and typically cover shorter time intervals of the content). At the top of the pyramid is total luminance metadata 501. Total luminance metadata 501 contains information about luminance energy for the full content (i.e., for many scenes and shots). Because total luminance metadata 501 describes the entire content, its data payload is relatively small. In some examples, total luminance metadata 501 is a single number representing the sum of all energy levels across all pixels, frames, shots, and scenes. Below total luminance metadata 501 is shot luminance metadata 502. Shot luminance metadata 502 contains information about luminance energy for each full shot. The data payload of shot luminance metadata 502 is larger than the data payload of total luminance metadata, but is still smaller in absolute value. In some examples, shot luminance metadata 502 is a one-dimensional data array, with each value in the array describing the total luminance for an entire shot. In this example, if the content contains N shots, then shot luminance metadata 502 is a one-dimensional data array of length N. The next level up is temporal luminance energy metadata 503.

[0050] The temporal luma energy metadata 503 includes information about luma energy for each frame in a shot. Thus, each block of the temporal luma energy 503 may correspond to the temporal luma energy metadata 212 described above with respect to FIG. 2B. The data payload of the temporal luma energy metadata 503 is larger than the data payload of the shot luma metadata 502, and much larger than the data payload of the total luma metadata 501.

[0051] The lowest layer is spatial luma energy metadata 504. Spatial luma energy metadata 504 includes information about the luma energy of each pixel over the duration of an individual shot. Thus, each block of spatial luma energy metadata may correspond to spatial luma energy metadata 312 described above with respect to FIG. 3B. Of all the metadata categories shown in FIG. 5, spatial luma energy metadata 504 has the largest payload. In some examples, spatial luma energy metadata 504 may be split into pieces (e.g., in the manner shown in FIG. 4B).

[0052] For a given type of metadata, there may be an inverse relationship between data payload and transmission frequency. Furthermore, there may be an inverse relationship between data payload and proximity to the actual image data described by a given type of metadata. For example, because total luminance metadata 501 has a very small data payload (e.g., a single number), it may be repeated very frequently in the encoded bitstream 126 and may not be transmitted very close to the image frames described therein. Because shot luminance metadata 502 has a small data payload, it may be repeated frequently in the encoded bitstream 126, but less frequently than total luminance metadata 501 and similarly may not be transmitted very close to the image frames described therein. Furthermore, in some examples, shot luminance metadata 502 may describe only a subset of the total number of shots, with shot luminance metadata 502 corresponding to earlier shots being transmitted before shot luminance metadata 502 corresponding to later shots.

[0053] In some examples, only some types of metadata are calculated directly, while other types of metadata are derived therefrom. For example, temporal luma energy metadata 503 may be calculated (e.g., as described above with respect to FIG. 3A ). Shot luma metadata 502 may then be derived from temporal luma energy metadata 503, e.g., by adding each frame luma value across all frames in the shot. In some examples, total luma metadata 501 may then be derived from shot luma metadata 502, e.g., by summing each shot luma value across all shots in the content. These derivations may be performed in upstream blocks shown in FIG. 1 and transmitted as part of encoded bitstream 126, or may be performed in downstream blocks shown in FIG. 1.

[0054] As an alternative to, or in addition to, repeating important power metadata in a predetermined order and / or at predetermined intervals, other transmission orders may be implemented. For example, if content is submitted as a 1:1 stream, power metadata may be dynamically added to the content stream and dynamically adjusted by a playback server (e.g., one or more of the upstream blocks shown in FIG. 1). In this configuration, it may be possible to transmit more relevant portions of power metadata earlier or more frequently, which may provide additional robustness against transmission errors and may facilitate display if the end user chooses to jump through or start the content midway. This may also be used to coordinate the power consumption of a group of related target devices, for example, to maintain a given maximum power budget when several target displays are powered from a common power source.

[0055] power management Upon receiving the encoded bitstream 126, downstream blocks shown in FIG. 1 can perform power management based on the received power metadata. To facilitate power management, certain metadata flags may be included and frame synchronized to pre-signal power management events. For example, if the power metadata indicates backlight (or pixel) overdrive, the power manager may receive timed advance notification of an upcoming boostable event. FIG. 6 shows an exemplary operational timeline for performing such power management. As will be understood and appreciated by those skilled in the art, such examples may also be applied, by analogy or in a similar manner, to power management that underdrives some (or all) of the backlight (or pixels).

[0056] In the example shown in FIG. 6, the content includes three shots. The first shot contains no significant highlights and has a duration of 15 frames, the second shot contains boostable highlights and has a duration of 7 frames, and the third shot contains no significant highlights and has a duration of 8 frames. The source metadata (e.g., power metadata received by the power manager as part of encoded bitstream 126) includes first flag data indicating a frame countdown to the next overdrive (OD) request and second flag data indicating the frame duration of the overdrive request. As shown in FIG. 6, the first flag data begins at frame 6, indicating that the next overdrive request will begin at frame 16, and the second flag data also begins at frame 6, indicating that the next overdrive request will last for 7 frames.

[0057] In some examples, the power receiver continually outputs target metadata (e.g., power metadata received and used by the target display 112). The target metadata may include first target flag data indicating a maximum scaled luminance for a given frame (where 1 indicates no overdrive) and second target flag data indicating an absolute maximum luminance at the average picture level (APL) of the shot. The maximum scaled luminance and absolute maximum luminance are the same in the particular example shown in FIG. 6, although this disclosure is not limited thereto. In FIG. 6, the first and second target flag data indicate no overdrive for frames 1 through 15 (i.e., for shot 1), 50% overdrive for frames 16 through 22 (i.e., for shot 2), and no overdrive for frames 23 through 30 (i.e., for shot 3).

[0058] The power receiver may further output data regarding the charge state of a supercapacitor or other fast-discharging energy storage device if the target display 112 implements a supercapacitor or other such device to overdrive (or underdrive) one or more light-emitting elements. If the energy storage device is a supercapacitor, this data instructs the target display 112 to begin charging the supercapacitor at a specific time so that the supercapacitor is sufficiently charged when overdrive is scheduled to begin. In some examples, the data may instead instruct the target display 112 to charge the supercapacitor well before the overdrive request and maintain the charged state until a discharge request is received, indicating that the light-emitting element will be overdriven. In some examples, the target display 112 itself may determine how far in advance to begin charging the supercapacitor. As will be understood and appreciated by those skilled in the art, the above examples of overdriving one or more light-emitting elements (e.g., by charging the supercapacitor well in advance) may also be applied by analogy or similarly to underdriving the one or more light-emitting elements by, for example, discharging a supercapacitor.

[0059] Power metadata (e.g., the source metadata and / or target metadata described above) may be stored in a buffer or other memory associated with one or more of the downstream blocks shown in FIG. 1 . For example, the power metadata may be stored in a buffer or other memory provided at the target display 112 itself. This allows for an ordering scheme in which portions of the power metadata are received out of order and / or in advance, and the power manager is configured to subsequently reorder or reassemble the portions of the power metadata. When used in conjunction with a transmission scheme that repeats the transmission of certain portions of the power metadata, this may provide additional robustness against data loss. Thus, even if power metadata is available for only a portion of the complete content, power management, including overdrive (or underdrive), may still be applied. In some implementations, the power metadata may be stored outside the target display 112, for example, on a set-top box or in the cloud.

[0060] The buffer may also store a configuration file describing various configuration parameters specific to the target display 112 and its hardware characteristics. For example, the configuration file may include information about one or more of the following: power consumption specifications, including maximum loads for power supply units, driver chips, light-emitting elements, etc.; cool-down times for light-emitting elements or power electronics (e.g., LED drivers); spatial heat transfer as a function of local heat generation within the display housing; maximum overdrive duration for the display, which may be a function of overdrive level; the presence of supercapacitors and, if any, their capacitance, depletion rate, and charge rate; etc. The configuration file may also be updatable, in whole or in part, to, for example, implement a usage counter and thereby provide information regarding the age or wear level of the display. In some examples, one or more ambient condition sensors (e.g., temperature sensor, humidity sensor, ambient light sensor, etc.) may be provided to detect corresponding ambient conditions, and information detected by the one or more ambient condition sensors may be stored within or alongside the configuration file to facilitate determining the wear level of the display. This real-time sensor information may also be used to influence the display power management system (e.g., to influence overdrive or underdrive) to avoid image fidelity artifacts. One example is to avoid underdriving pixels during high ambient light levels.

[0061] Uses and effects The various approaches, systems, methods, and devices described herein can implement power metadata to affect the behavior of a target display in the manners described above, without limitation. That is, various aspects of the present disclosure may be used to affect display management mapping behavior (e.g., limit luminance output, deviate from baseline mapping, etc.); overdrive the backlight unit or (in self-emissive display technologies) the pixels themselves, thereby increasing the maximum luminance of individual pixels, pixel groups, or the entire panel beyond overly conservative manufacturer-set limits while avoiding overtaxing the power supply unit; increase granularity for display power management systems, for example, to manage thermal panel or backlight characteristics based on spatial and / or temporal power and energy expectations; provide trim-path-like behavior to manage power in multi-display systems, representing luminance levels after the signal has been tone-mapped by the target device; intelligently limit display power usage for regulatory (e.g., Energy Star compliance) purposes or for power conservation (e.g., battery-operated devices); etc.

[0062] A trim pass is a feature that facilitates human override of mapping parameters that would otherwise be determined by a computer algorithm (e.g., the algorithm that generates one or more portions of the power metadata). In some examples, the override may be performed during the color grading process to ensure a certain look is provided or preserved after determining whether the results of the computer algorithm cover the video or content creator's intent for a particular target display's dynamic range bracketing (e.g., at a display max of 400 nits). Thus, the power metadata may be updated to include information that allows the target display to modify or override the algorithm's recommendations for one or more shots or scenes.

[0063] To achieve this, trim-path-like behavior may be achieved by configuring the target display system to utilize power metadata according to its current playback luminance bracket. If the display is mapped to a non-default target luminance bracket, the display power management system may be configured to determine the trim path accordingly. For example, if the display transitions from a default mapping to a boost mode mapping (e.g., Overdrive), the display power management system may switch from a lower luminance energy trim path to a higher one.

[0064] In one specific example, during generation of the power metadata, the algorithm may indicate that underdrive should be performed for a particular shot. However, underdrive for that particular shot may be inadvisable for narrative or other reasons. Thus, a color grader (human or otherwise) may modify or supplement the power metadata, thereby causing the display power management system to drive (rather than underdrive) the target display despite the initial output of the algorithm.

[0065] Systems and devices according to the present disclosure may take any one or more of the following configurations: A method comprising: (1) receiving image data and power metadata, the power metadata including information regarding power consumption or expected power consumption; determining, based on the power metadata, an amount and duration of drive modifications that may be performed by a target display in response to the power consumption or the expected power consumption; and performing power management of the target display based on the power metadata, modifying drive of at least one light-emitting element associated with the target display relative to a manufacturer-determined threshold based on the results of the determination, wherein the power metadata includes at least one of temporal luminance energy metadata, spatial luminance energy metadata, spatial-temporal variation metadata, or a combination thereof. (2) The method of (1), wherein the step of determining the amount and duration of the drive modification that can be performed by the target display includes determining the amount and duration of overdrive that can be performed by the target display without damaging the at least one light-emitting element, and the step of performing power management of the target display includes selectively overdriving the at least one light-emitting element beyond a manufacturer-determined threshold. (3) The method of (1) or (2), wherein the step of determining the amount and duration of drive modification that can be performed by the target display includes determining the amount and duration of underdrive that can be performed by the target display in response to the power consumption or the expected power consumption, and the step of performing power management of the target display includes reducing the brightness of the at least one light-emitting element. (4) A method according to any one of (1) to (3), wherein the image data and the power metadata are received together as an encoded bitstream. (5) The method of (4), further comprising: receiving a first portion of the power metadata in a first frame of the encoded bitstream; and storing the first portion of the power metadata in a buffer. (6) retrieving the first portion of the power metadata from the buffer; and performing power management of the target display for the image data corresponding to a second frame of the encoded bitstream based on the first portion of the power metadata, the second frame being a later image frame compared to the first frame. (5) The method described in (5). (7) A method according to any one of (1) to (6), wherein the image data and the power metadata are received via different transmission paths. (8) The power metadata includes the temporal luma energy metadata, and the method further includes: deriving shot luma metadata from the temporal luma energy metadata, the shot luma metadata including information about luma energy of a shot of the encoded bitstream. A method according to any one of (1) to (7). (9) A method according to any one of (1) to (8), further comprising generating target metadata based on the power metadata, wherein the target metadata includes at least one of first flag data indicating a frame countdown to an overdrive request or second flag data indicating a frame duration of the overdrive request. (10) A method according to any one of (1) to (9), wherein the step of performing power management of the target display includes causing the target display to charge at least one energy storage device associated with the target display. (11) A method according to any one of (1) to (10), wherein the step of performing power management of the target display includes causing the target display to discharge at least one energy storage device associated with the target display. (12) A method according to any one of (1) to (11), further comprising the steps of: receiving image formation metadata; and controlling the target display to display the image data based on the image formation metadata. (13) A non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer to perform operations including the method described in any one of (1) to (12). (14) An apparatus having a display including at least one light emitting element; and a display management circuit, the display management circuit configured to perform the steps of: receiving power metadata, the power metadata including information regarding power consumption or expected power consumption; determining, based on the power metadata, an amount and duration of drive modifications that may be performed by the display in response to the power consumption or the expected power consumption; and performing power management of the display based on the power metadata, modifying drive of the at least one light emitting element relative to a manufacturer-determined threshold based on results of the determination, wherein the power metadata includes at least one of temporal luminance energy metadata, spatial luminance energy metadata, spatial-temporal variation metadata, or a combination thereof. (15) The device of (14), further comprising a memory configured to store a predetermined configuration file, the predetermined configuration file including information related to at least one setting parameter of the display. (16) The device described in (15), wherein the configuration file includes information regarding at least one of the power consumption specifications of the display, the cooling time of the at least one light-emitting element, the spatial heat transfer of the display, the maximum overdrive duration of the display, or the presence of a supercapacitor in the display. (17) The device of (15) or (16), wherein the configuration file includes a usage counter that indicates information regarding at least one of the age of the display or the level of wear on the display. (18) The device according to any one of (15) to (17), further comprising an ambient condition sensor configured to detect ambient conditions, and the memory configured to store information about the ambient conditions. (19) The device described in any one of (14) to (18), further comprising a decoder configured to receive an encoded bitstream including image data and the power metadata and provide the power metadata to the display management circuit. (20) The device described in (19), wherein the encoded bitstream further includes image formation metadata, and the display management circuitry is configured to control the display to modify the display of the image data based on the image formation metadata.

[0066] With respect to the processes, systems, methods, heuristics, etc. described herein, although the steps of such processes, etc. are described as occurring according to a certain ordered sequence, it should be understood that such processes may be implemented with the described steps performed in an order other than the order described herein. Furthermore, it should be understood that certain steps may be performed simultaneously, that other steps may be added, or that certain steps described herein may be omitted. In other words, the process descriptions herein are provided for the purpose of illustrating certain embodiments and should not be construed as limiting the claims in any way.

[0067] Thus, it should be understood that the foregoing description is illustrative and not restrictive. Many embodiments and applications other than the examples provided will become apparent upon reading the above description. The scope should be determined not with reference to the above description, but with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technology discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In short, it should be understood that this application is capable of modification and alteration.

[0068] All terms used in the claims are intended to be given their broadest reasonable interpretation and their ordinary meaning as understood by one skilled in the art described herein, unless expressly indicated otherwise in the specification. In particular, the use of singular articles such as "a," "the," "said," etc. should be read as describing one or more of the indicated elements unless the claim describes an express limitation to the contrary.

[0069] The Abstract of the present disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Moreover, in the foregoing Detailed Description, it will be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments include more features than are expressly recited in each claim. Rather, as the claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Accordingly, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as separately claimed subject matter.

Claims

1. receiving image data and power metadata, the power metadata including information regarding expected power consumption; determining, based on the power metadata, an amount and duration of drive modifications that may be performed by the target display in response to the expected power consumption; and performing power management of the target display based on the power metadata, modifying the driving of at least one light element associated with the target display relative to a manufacturer-determined threshold based on the result of the determination; the power metadata includes at least one of: temporal luminance energy metadata indicating temporal luminance energy determined from at least one content frame in a plurality of content frames; spatial luminance energy metadata indicating spatial luminance energy determined from a pixel across the plurality of content frames; spatial-temporal variation metadata indicating energy variation determined from the pixel across the plurality of content frames; or a combination thereof; method.

2. The method of claim 1 , wherein the power metadata included in a frame further includes power metadata for future frames.

3. determining an amount and duration of the drive modification that can be performed by the target display includes determining an amount and duration of overdrive that can be performed by the target display without damaging the at least one light-emitting element; performing power management of the target display includes selectively overdriving the at least one light emitting element above a manufacturer-determined threshold; The method according to claim 1 or 2.

4. determining an amount and duration of drive modification that may be performed by the target display includes determining an amount and duration of underdrive that may be performed by the target display in response to the expected power consumption; performing power management of the target display includes reducing the brightness of the at least one light-emitting element; 4. The method according to any one of claims 1 to 3.

5. The method of claim 1 , wherein the image data and the power metadata are received together as an encoded bitstream.

6. receiving a first portion of the power metadata in a first frame of the encoded bitstream; storing the first portion of the power metadata in a buffer. The method of claim 5.

7. Retrieving the first portion of the power metadata from the buffer; performing power management of the target display for the image data corresponding to a second frame of the encoded bitstream based on the first portion of the power metadata; The second frame is a later image frame compared to the first frame. The method of claim 6.

8. The method of claim 1 , wherein the image data and the power metadata are received via different transmission paths.

9. The power metadata includes the temporal luminance energy metadata, and the method comprises: and deriving shot luma metadata from the temporal luma energy metadata, the shot luma metadata including information about the luma energy of a shot of the encoded bitstream.

8. The method according to any one of claims 5 to 7.

10. 10. The method of claim 1, further comprising generating target metadata based on the power metadata, wherein the target metadata includes at least one of first flag data indicating a frame countdown to an overdrive request or second flag data indicating a frame duration of the overdrive request.

11. 11. The method of claim 1, wherein performing power management for the target display comprises causing the target display to charge or discharge at least one energy storage device associated with the target display.

12. 12. The method of claim 1, further comprising the steps of: receiving imaging metadata; and controlling the target display to display the image data based on the imaging metadata.

13. A non-transitory computer readable medium storing instructions that, when executed by a processor of a computer, cause the computer to perform operations including the method of any one of claims 1 to 12.

14. a display including at least one light emitting element; and a display management circuit, The display management circuitry: receiving power metadata, the power metadata including information regarding expected power consumption; determining, based on the power metadata, an amount and duration of driving modifications that may be performed by the display in response to the expected power consumption; and performing power management of the display based on the power metadata, modifying the driving of the at least one light element relative to a manufacturer-determined threshold based on the result of the determination; the power metadata includes at least one of: temporal luminance energy metadata indicating temporal luminance energy determined from at least one content frame in a plurality of content frames; spatial luminance energy metadata indicating spatial luminance energy determined from a pixel across the plurality of content frames; spatial-temporal variation metadata indicating energy variation determined from the pixel across the plurality of content frames; or a combination thereof; Device.

15. The apparatus of claim 14 , wherein the power metadata included in a frame further includes power metadata for future frames.

16. 16. The apparatus of claim 14 or claim 15, further comprising a memory configured to store a predetermined configuration file, the predetermined configuration file including information relating to at least one setting parameter of the display.

17. 17. The device of claim 16, wherein the configuration file includes information regarding at least one of a power consumption specification of the display, a cool-down time of the at least one light-emitting element, a spatial heat transfer of the display, a maximum overdrive duration of the display, or the presence of a supercapacitor in the display.

18. 18. The device of claim 16 or 17, wherein the configuration file includes a usage counter that indicates information about at least one of the age of the display or the level of wear on the display.

19. further comprising an ambient condition sensor configured to detect an ambient condition; the memory is configured to store information about the ambient conditions; 19. Apparatus according to any one of claims 16 to 18.

20. a decoder configured to receive an encoded bitstream including image data and the power metadata and to provide the power metadata to the display management circuit; 20. Apparatus according to any one of claims 14 to 19.

21. 21. The apparatus of claim 20, wherein the encoded bitstream further includes imaging metadata, and wherein the display management circuitry is configured to control the display to modify the display of the image data based on the imaging metadata.

Images