DASH signaling of display attenuation maps in adaptive streaming services

Abstract

Systems and methods for flexible signaling of display attenuation maps and relative attenuation map information within an Adaptive Streaming Protocol Manifest or Media Presentation Descriptor (MPD) file include associating attenuation maps in the MPD with related video data. In some implementations, a client device may receive a manifest that includes multiple representations of an item of streaming media content and the identification of one or more display attenuation maps. The client device may retrieve media segments of the representation of the media content; retrieve media segments of the display attenuation map corresponding to the representation; and display the media segments of the representation adjusted based on the media segments of the display attenuation map. The selection of the representation and / or display attenuation map may be performed by the client based on network and power conditions and parameters within the manifest.

Description

Technology Field

[0001] Cross-reference regarding related applications

[0002] This application claims priority to European provisional patent application No. 24305102.6 filed on January 16, 2024, the contents of which are incorporated herein by reference.

[0003] Technology field

[0004] The embodiments generally relate to systems and methods for multimedia processing and distribution, in particular systems and methods for streaming multimedia. Background Technology

[0005] Reducing the energy consumption of electronic devices has become not only a requirement for electronic device manufacturers but also a requirement to limit environmental impact as much as possible and contribute to the emergence of a sustainable display industry. The introduction of high dynamic range imaging, as well as the increase in display resolution from SD to HD, 4K, and soon beyond 8K, has led to a corresponding increase in the energy requirements of display devices. This does not align with the global need to reduce energy consumption. In fact, displays are the primary source of energy consumption, whether for battery-powered devices (e.g., smartphones) or within the global video distribution chain.

[0006] In some embodiments, the present disclosure relates to implementations of systems and methods for flexible signaling of display attenuation maps and relative attenuation map information within an adaptive streaming protocol manifest or media presentation descriptor (MPD) file—which includes associating attenuation maps in the MPD with related video data. Such implementations may enable a streaming client to identify a set of available display attenuation maps for a video being streamed and to efficiently select the most appropriate attenuation map(s) to optimize display energy consumption of playback devices in streaming services. Brief explanation of the drawing

[0007] A more detailed understanding can be obtained from the following description, which is provided as an example in relation to the attached drawings, and identical reference numbers within the drawings indicate identical elements. Figure 1 is a block diagram of a dynamic adaptive streaming (DASH) delivery process model over the hypertext transfer protocol (HTTP) according to some implementations. Figure 2 is an example of a DASH Media Presentation Description (MPD) manifest according to some implementations. Figures 3a–3c are representations of exemplary XML schemas for attenuation map descriptors according to some implementations. Figures 4a and 4b are representations of exemplary MPD manifests according to some implementations. Figures 5a and 5b are representations of other exemplary MPD manifests according to some implementations. Figures 6a and 6b are representations of another exemplary MPD manifest according to some implementations. Figures 7a–7c are representations of another exemplary MPD manifest according to some implementations. FIG. 8 is a flowchart of a method for processing streaming video having an associated display attenuation map according to some implementations. FIG. 9 illustrates a block diagram of an embodiment of a video encoder in which various aspects of the embodiments can be implemented. FIG. 10 illustrates a block diagram of an embodiment of a video encoder in which various aspects of the embodiments can be implemented. FIG. 11 illustrates a block diagram of a system in which aspects of the embodiments can be implemented. Specific details for implementing the invention

[0008] The embodiments described herein describe principles relating to specific drafts of Versatile Video Coding (VVC) or High Efficiency Video Coding (HEVC) standards, but are not limited to VVC or HEVC and may be applied, for example, to other standards and recommendations, whether existing or future to be developed, and to any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, the embodiments described in this application may be used individually or in combination.

[0009] The present disclosure relates to signaling mechanisms for video and multimedia streaming that enable a device to control the energy consumption of its display and the quality of the rendered video on the screen of an internet-connected desktop or laptop computer, smartphone, tablet, set-top box, or television in the context of an adaptive streaming protocol (e.g., MPEG-DASH specified in ISO / IEC 23009-1; HLS (HTTP Live Streaming); or similar protocols). Although such devices may consume a significant amount of energy, by providing brightness or dimming metadata or light attenuation maps as part of the streamed video or multimedia signal, the decoder of the display may adjust one or more backlight or pixel illumination levels to reduce energy consumption without significant degradation of image quality.

[0010] For example, Organic Light Emitting Diode (OLED) displays are becoming increasingly popular due to their numerous advantages over non-emissive displays, such as Thin-Film Transistor Liquid Crystal Displays (TFT-LCDs). Rather than using a uniform backlight, OLED displays consist of LEDs as image pixels. Therefore, the power consumption of OLEDs is highly correlated with the image content and can be easily estimated by considering the brightness levels of the image pixels being displayed. Although OLED displays consume energy in a more controllable and efficient manner than many other display technologies, they remain the primary source of energy consumption in the video transmission chain.

[0011] The ISO / IEC 23001-11 standard ("Energy-Efficient Media Consumption: Green Metadata"), published by the International Organization for Standardization in 2023 and incorporated herein by reference, defines metadata referred to as green metadata to facilitate the reduction of energy usage during media consumption (i.e., decoding and display operations), and in particular to reduce display power consumption. The metadata described in the standard is particularly well suited to display technologies based on non-emissive pixels, such as LCDs, and embedded backlight illumination, because the metadata is designed to achieve display energy reduction by utilizing display adaptation techniques that dynamically generate some RGB-component statistics and quality indicator metrics regarding the consumed video content at the emitter side. The metadata can be used to perform RGB picture component rescaling to allow voltage reduction and thus energy consumption reduction by establishing the best trade-off between backlight / voltage reduction and picture quality. However, this metadata is difficult to consider optimal because it conveys global information and does not provide any information that aids in the use of pixel-level attenuation maps, since such maps are not used for displays of non-emissive pixel types.

[0012] In some implementations, display green metadata may be provided as a timed metadata stream, such as an MPEG-DASH timed metadata representation (e.g., having the @codecs attribute set to "dipi" as described in ISO / IEC 23001-11). The stream associated with this representation may be retrieved and used by a decoder to perform post-processing on all available and associated media representations and to reduce energy consumption.

[0013] A display attenuation map may include video generated by a video content analyzer through a pixel-level energy-aware frame algorithm. Such maps are typically 2D, but in some implementations, 3D attenuation maps may be used for, for example, 3D environments, pixel clouds, augmented reality, or virtual reality. The frame algorithm may have smoothness and scalability characteristics that can be dynamically adjusted in some implementations. The associated attenuation map information includes the characteristics of the attenuation map (e.g., display target model, width, length, energy attenuation rate, etc.) and the actions to be applied to the original video before rendering the video through the display. Applying information conveyed from the display attenuation map video to the analyzed video intelligently and locally reduces the brightness of the frames to optimize the trade-off between Quality of Experience (QoE) and energy attenuation.

[0014] A attenuation map is designed to generate a modified image that requires less energy for display than the input image when applied to the input image. One implementation is to reduce the luminance according to a selected energy reduction rate. More complex implementations consider other parameters, such as the similarity between the modified image and the input image, the contrast sensitivity function of human vision, or smoothness characteristics that allow the attenuation map to be downscaled without introducing heavy artifacts when upscaling it on the decoder side.

[0015] Applying an attenuation map to an image may involve combining them according to the selected operation type. The values ​​of the samples in the attenuation map and the type of operation are closely related. Specifically, when the operation is addition, the attenuation map contains sample values ​​with negative values; when the operation is subtraction, the attenuation map contains sample values ​​with negative values; and when the operation is multiplication, the attenuation map contains floating-point sample values ​​within the range of 0 (pixel becomes black) and 1 (no attenuation). The sample values ​​may be referred to in various ways as attenuation values, attenuation factors, attenuation vectors, or by any other similar description. Attenuation may be applied to the lumina values ​​of pixels, the chroma values ​​of pixels, or both the lumina and chroma values ​​of pixels.

[0016] To convey attenuation map information, in some implementations, supplemental enhancement information (SEI) messages may be used, but their generality makes them inefficient for many use cases. In other implementations, more use-specific SEI messages may be used, such as those described in PCT application no. PCT / EP2023 / 083362 dated November 28, 2023, incorporated herein by reference. These messages, conveyed within codec signaling, aim to guide the use of pixel-level display attenuation maps. Dimming maps may be transmitted as auxiliary data to the receiver side. In other implementations, display attenuation maps may be carried in ISO-based media file format (ISOBMFF) media containers. However, many of these implementations may require changes to core syntax structures or underlying protocols. Instead, implementations of the systems and methods discussed herein provide signaling of display attenuation map information in adaptation or representation sets within manifest files.

[0017] Specifically, the present disclosure provides implementation details of systems and methods for enhancing dynamic adaptive streaming protocols, such as Dynamic Adaptive Streaming (DASH) or HTTP Live Streaming (HLS) signaling over HTTP, in order to enable the transmission of display attenuation map-coded video and relative information from standard HTTP servers to HTTP clients. The additional signaling enables a media player to select one or more complementary display attenuation maps (sometimes referred to as dimming maps) containing video media representations and relative attenuation map information, thereby modifying the video and reducing energy consumption while displaying the video on the screen of a receiver device or a screen connected thereto. A display attenuation map representation may be associated with one or more video media representations (e.g., at different bit rates or bandwidths, or with variable resolution, color depth, frame rate, etc.). In various implementations, these systems and methods may include signaling of a codec used for adaptation sets carrying display attenuation map information as a restricted video codec, definition and use of a new DASH descriptor for signaling attenuation map information at adaptation set or representation levels, and instructions for grouping adaptation sets carrying display attenuation maps with adaptation sets of video using presets to define some experiences.

[0018] The Dynamic Adaptive Streaming over HTTP (i.e., DASH, ISO / IEC 23009-1) protocol enables the continuous delivery of media content from standard HTTP servers to HTTP clients and allows for the caching of content by standard HTTP caches. This allows for the highest visual quality while taking into account various factors such as network bandwidth, display resolution, and viewing conditions.

[0019] FIG. 1 is a block diagram of an embodiment of a DASH delivery process model (100). A DASH media presentation preparation function (110) can interact with a media presentation description (MPD) delivery function (120) and a DASH segment delivery function (130). Each function (110-130) may be provided by the same or different computing devices (or virtual devices such as cloud computing devices, media provider clouds, etc.). For example, the functions (110-130) may be provided by an HTTP server running on a computing device or implemented on that server. An HTTP cache (140) may store media segments and / or MPD metadata for delivery to a DASH client (150), such as a desktop computer, laptop computer, smart television, appliance, tablet computer, smartphone, or other such computing device or media consumption device. The HTTP cache (140) may be part of such a computing device or media consumption device, or may be implemented in a gateway, router, access point, network appliance, or other such device.

[0020] The DASH media presentation preparation function (110) can be configured to prepare a media presentation for the DASH client (150) to watch simultaneously (e.g., in the case of live content) or at an earlier time (e.g., in the case of on-demand video or pre-recorded content). For example, the DASH media presentation preparation function (310) can receive data regarding media content from a content delivery network and can prepare an MPD, sometimes referred to as a manifest, to describe the media content.

[0021] Referring briefly to FIG. 2, an example of a DASH manifest or MPD (200) is illustrated. The MPD (200) may contain one or more URLs for keys, initialization vectors, ciphers, segments, and / or message authentication codes. The MPD (200) may list the URLs as static addresses or as functions that can be used to determine associated URLs. The MPD (200) may be any suitable format, such as Extensible Markup Language (XML). The MPD (200) may contain information for one or more periods (210). Periods may include timing data and may represent a period of content in which a consistent set of encoded versions of media content (e.g., a set of unchanging available bitrates, languages, captions, subtitles, etc.) is available. Each period description (210) may include one or more adaptive set descriptions (220). An adaptation set may represent a set of interchangeable encoded versions of one or more media content components. For example, a first adaptation set may include a major video component, a second adaptation set may include a major audio component, and a third adaptation set may include captions, etc. An adaptation set may also include multiplexed content, such as combined video and audio. An adaptation set description (220) may define one or more representations (230). A representation may describe one or more transferable encoded versions of media content components, such as an ISO-based media file format (ISO-BMFF) version, an MPEG (Moving Picture Experts Group) version 2 transport system (MPEG-2 TS) version, etc. A representation may describe, for example, any necessary codecs, encryption, and / or other data required to present the media content.The DASH client (150) may dynamically switch between representations based on network conditions, device capabilities, user choices, etc., which may be referred to as adaptive streaming (e.g., switching to a more compressed or lower resolution version as a result of network congestion). Each set of representations (230) may include one or more segments (240), sometimes referred to as chunks. Each segment (240) may include media content data, may be associated with a URL, and may be retrieved by the DASH client (150) using standard methods such as an HTTP GET request. Each segment (240) may include a predefined byte size (e.g., 1,000 bytes) and / or an interval of playback time of the media content (e.g., 2, 5, or 15 seconds) (segments do not need to be of the same length or duration). A segment may include the smallest individually addressable units of data that can be downloaded using URLs advertised via MPD. Periods, adaptation sets, expressions, and / or segments may be described in terms of attributes and elements that can be modified to affect the presentation of media content by the DASH client (150). When the MPD is prepared, the DASH media presentation preparation function (110) can deliver the MPD to the MPD delivery function (120).

[0022] Returning to FIG. 1, the client (150) may request an MPD from the MPD delivery function (120), for example, via an HTTP GET request. The MPD delivery function (120) may respond with the MPD directly or via an HTTP cache (140) (e.g., an edge cache or other device that stores a copy of the MPD). Based on the address data within the MPD, the DASH client (150) may request appropriate segments (e.g., segments of a specific resolution, encoding type, bitrate, etc.) from the DASH segment delivery function (130). Although only one MPD delivery function, DASH segment delivery function, and cache are exemplified, in many implementations, these functions may be redundant or provided by many different computing devices (including virtual computing devices or cloud devices). Thus, segments and / or MPDs may be retrieved from multiple different devices and / or multiple URLs and / or physical locations. The DASH client (150) can present the segments found based on the commands in the MPD.

[0023] The systems and methods discussed herein, in one embodiment, relate to signaling display attenuation map information via a DASH MPD manifest or a similar adaptive streaming protocol manifest. Display attenuation map representations are mapped to relevant video streams within the MPD, enabling streaming clients to efficiently select the most appropriate attenuation map based on network and battery conditions at any given time. In some implementations, display attenuation map video frames may be stored in ISO-based media files as media samples of media tracks using extensions to ISO / IEC 14496-12. In some implementations, display attenuation map tracks may be signaled in the MPD or manifest file and may be associated with one or more representations of the target video, so that the decoder can apply the map to these representations before displaying or rendering them on the display. By using MPD or manifest values ​​to signal the display attenuation map, service providers can avoid the need to use SEI messages or auxiliary data within the codec bitstream. This may reduce bandwidth requirements for the stream itself and may also allow caching of display attenuation maps along with the manifest (for example, at geographically close network edge caches, this may reduce latency and total network bandwidth requirements).

[0024] In some implementations, display attenuation maps may be signaled in MPD as representations of an adaptation set. An adaptation set containing representations of display attenuation maps is referred to herein as the Display Attenuation Map Adaptation Set. In some implementations, the Display Attenuation Map Adaptation Set may include a role descriptor as defined in ISO / IEC 23009-1, having an @schemeIdUri attribute set to "urn:mpeg:dash:role:2011" and an @value attribute set to "ami".

[0025] A display attenuation map adaptation set may include one or more representations of a coded display attenuation map sequence. Each representation may be encoded using a different coding configuration and / or may result in different energy attenuation levels when applied to the associated video. The association between the representations of the display attenuation map adaptation set and the representations within the associated video adaptation set may be signaled by including the @associationId identifier and the @associationType attribute in the representations of the display attenuation map adaptation set. The @associationId attribute may be assigned the same value as the value of the @id attribute for the corresponding representation within the video adaptation set, and the @associationType attribute may be set to the value "amit" (indicating that the adaptation set is for the display attenuation map for the associated video adaptation set).

[0026] The @codecs attribute for the display attenuation map adaptation set, or the representations of this adaptation set if @codecs is not signaled for the AdaptationSet element, can be set based on each codec used to encode the attenuation map. The value of @codecs can be set to 'resv.gmat.XXXX', where XXXX corresponds to the 4-character code (4CC) of the video codec from the original_format field in the RestrictedSchemeInfoBox of the sample entry (e.g., 'avc1' or 'hvc1') of the corresponding ISOBMFF track.

[0027] In another embodiment, a plurality of display attenuation map adaptation sets may be associated with the same video representation. For example, each display attenuation map adaptation set may be associated with a specific area within video frames. This area may be spatial (e.g., a coordinate range from 0,0 to 250,400, a window within the frame such as the top-left, etc.), temporal (e.g., from 0:00 to 5:30, for 120 seconds, etc.), or both spatial and temporal (e.g., having a specified coordinate range and time range). In these embodiments, the display attenuation map adaptation sets may be grouped with the associated video adaptation sets using a green video preset.

[0028] Green video preselection is signaled using preselection elements as defined in ISO / IEC 23009-1, and the list of identifiers for the @preselectionComponents attribute includes the identifier of the primary adaptation set (video adaptation set) for volumetric media and the identifiers of the subsequent associated display attenuation map adaptation sets. The @codecs attribute for preselection can be set based on the codec used by the primary adaptation set (i.e., video adaptation set).

[0029] An AttenuationMap descriptor may be used to signal the characteristics of a display attenuation map carried by a display attenuation map adaptation set. This descriptor may include an EssentialProperty or SupplementalProperty descriptor having an @schemeIdUri set to a unique URI (e.g., "urn:mpeg:mpegI:green:2023:ami"). The AttenuationMap descriptor may exist at the adaptation level and / or the representation level for each representation within the display attenuation map adaptation set.

[0030] In some implementations, the @value of the AttenuationMap descriptor may be omitted. The AttenuationMap descriptor may include AMI elements that describe the display attenuation map and its characteristics, the attributes and sub-elements of which are represented by an adaptation set. XML elements and attributes for this descriptor may be defined in a separate namespace "urn:mpeg:mpegI:green:2023". The namespace specifier "green:" is used to refer to this namespace in this document. Table 1 lists exemplary elements and attributes of the AttenuationMap descriptor according to some implementations.

[0031] [Table 1]

[0032] Table 1: Elements and attributes of the AttenuationMap descriptor

[0033]

[0034]

[0035]

[0036]

[0037]

[0038]

[0039] In some implementations, data types for various elements and attributes are defined by XML schemas illustrated in FIGS. 3a–3c. In other implementations, the AttenuationMap descriptor is a SupplementalProperty descriptor having an @schemeIdUri attribute set to a unique URI (e.g., "urn:mpeg:mpegI:green:2023:ami"). In other implementations, the sizes of the defined attributes for the various elements of the attribute may differ based on the allowed range of values ​​for the attribute.

[0040] Figures 4a and 4b are representations of an exemplary MPD manifest according to some implementations. This example shows a DASH MPD having a presentation that includes a video adaptation set with three representations and a display attenuation map adaptation set with one representation associated with the first video representation. The attenuation map carried by the display attenuation map adaptation set results in a 20% reduction in energy consumption when applied to the associated video representation. The application of the display attenuation map representation also results in a 5% reduction in video quality in terms of peak signal-to-noise ratio (PSNR).

[0041] FIGS. 5a-5b are representations of other exemplary MPD manifests according to some implementations. This example describes a DASH MPD having a presentation comprising a video adaptation set with three representations and a display attenuation map adaptation set with two representations associated with the first video representation. The attenuation map carried by the first representation of the display attenuation map adaptation set results in a 20% reduction in energy consumption when applied to the video representation, whereas the second representation of the display attenuation map adaptation set results in a 40% reduction in energy consumption when applied to the same video representation.

[0042] FIGS. 6a and 6b are another exemplary representation of an MPD manifest according to some implementations. This example is similar to the example in FIGS. 5a and 5b, but uses one instance of a descriptor at the adaptation set level containing common information for all representations, and additional instances for each representation having representation-specific information.

[0043] FIGS. 7a–7c are representations of another exemplary MPD manifest according to some implementations. This example is similar to the example in FIGS. 6a–6b, but uses a preset to represent the grouping of video adaptation sets and associated display attenuation map adaptation sets. In other implementations, variations of the above exemplary manifests may be used.

[0044] As discussed above, a DASH client such as client (150) can be guided by information provided in MPD. FIG. 8 is a flowchart of a method (800) for processing streaming video having an associated display attenuation map using the signaling presented in MPD and the schema discussed above, depending on some implementations.

[0045] In 805, the client can retrieve the manifest. In some embodiments, the client first issues an HTTP request and downloads the MPD file from the content server. In other implementations, the client can retrieve the MPD file from the cache. In 810, in some implementations, the client can parse the MPD file to generate corresponding in-memory representations of the XML elements within the MPD file.

[0046] To identify display attenuation maps available over a specified period, in 815, the streaming client scans AdaptationSet elements to find adaptation sets that have role descriptors containing an AttenuationMap descriptor element and where @value is set to 'ami'. For each representation within a display attenuation map adaptation set, in 820, the client can also identify the associated representation within a video adaptation set using @associationId. In some implementations, 815–820 may be repeated for each additional adaptation set within the manifest.

[0047] In 825, in some implementations, the client begins to select one of the representations of the video adaptation set based on its capabilities and network conditions. For example, the client may monitor network conditions or perform measurements (e.g., sending short pings or requests and measuring round-trip time or packet loss rate, etc.) and select a representation where the network conditions meet minimum requirements (e.g., based on throughput).

[0048] In 830, the client can download an initialization segment for the selected representation and download initialization segments for all representations from the display attenuation map adaptation set associated with the selected video representation. In 835, the client can sequentially download media segments from the video representation and begin decoding and displaying the media segments.

[0049] During the display of media segments, in some implementations of 840, the client may continuously or periodically monitor the device's energy status and / or consumption (e.g., energy conditions such as on battery or on AC power; charge status; remaining power in the device's battery; rate of change of battery level, etc.), and based on these measurements or conditions, the remaining playback time of the media (where the media is pre-recorded) and information signaled in the AttenuationMap descriptor may determine whether to reduce energy consumption. Then, in 845, the client device may select one of the display attenuation map representations.

[0050] Returning to 835, the client can subsequently download the corresponding media segments from the display attenuation map representation along with their respective media segments from the video representation. The downloaded display attenuation map media segments are decoded, and the decoded frames are applied to the corresponding decoded frames from the video before rendering.

[0051] In some implementations, steps 835-845 may be performed repeatedly, such as when the energy consumption level is changed or restored. For example, after reducing energy consumption due to a low battery level, when the device is plugged in or the battery is fully charged, the client may select a new display attenuation map representation with higher energy consumption (lower attenuation, or higher QoE) or even an unattenuated representation.

[0052] The various methods and other embodiments described in this application may be used to modify modules of a video encoder (900) and decoder (1000) as illustrated in FIGS. 9 and 10, e.g., motion compensation (970, 1075), motion estimation (975), entropy coding, intra (960, 1060), and / or decoding modules (945, 1030). Furthermore, the embodiments described herein are not limited to WC or HEVC and may be applied, for example, to other standards and recommendations, whether existing or future to be developed, and to any extensions of such standards and recommendations (including WC and HEVC). Unless otherwise indicated or technically excluded, the embodiments described in this application may be used individually or in combination.

[0053] For example, various numerical values ​​are used in this application. Specific values ​​are for exemplary purposes and the described embodiments are not limited to these specific values.

[0054] FIG. 9 illustrates an encoder (900). Variations of this encoder (900) are considered, but for clarity, the encoder (900) is described below without describing all expected variations.

[0055] Before encoding, the video sequence may undergo pre-encoding processing (901), for example, applying a color conversion to the input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0), or performing remapping of the input picture components to obtain a signal distribution more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata is associated with the preprocessing and can be attached to the bitstream.

[0056] In the encoder (900), the picture is encoded by encoder elements as described below. The picture to be encoded is partitioned (902) and processed, for example, into units of CUs. Each unit is encoded using, for example, an intra or inter mode. When a unit is encoded in intra mode, it performs intra prediction (960). In inter mode, motion estimation (975) and compensation (970) are performed. The encoder determines whether to use intra mode or inter mode to encode the unit (905), and indicates the intra / inter decision, for example, by a prediction mode flag. Prediction residuals are calculated, for example, by subtracting the predicted block from the original image block (910).

[0057] Next, the predicted residuals are transformed (925) and quantized (930). The quantized transformation coefficients, as well as motion vectors and other syntax elements, are entropy-coded to output a bitstream (945). The encoder can skip the transformation and apply quantization directly to the un-transformed residual signal. The encoder can bypass both transformation and quantization, that is, the residuals are coded directly without the application of transformation or quantization processes.

[0058] The encoder decodes the encoded block to provide a reference for additional predictions. The quantized transform coefficients are inversely quantized (940) and inversely transformed (950) to decode the prediction residuals. By combining the decoded prediction residuals and the predicted block (955), the image block is reconstructed. In-loop filters (965) are applied to the reconstructed picture to perform deblocking / SAO (Sample Adaptive Offset) filtering to reduce encoding artifacts, for example. The filtered image is stored in a reference picture buffer (980).

[0059] FIG. 10 illustrates a block diagram of a video decoder (1000). In the decoder (1000), the bitstream is decoded by decoder elements as described below. The video decoder (1000) generally performs a decoding pass that is reciprocal to the encoding pass as described in FIG. 9. The encoder (900) also generally performs video decoding as part of encoding video data.

[0060] In particular, the input to the decoder includes a video bitstream that can be generated by a video encoder (900). The bitstream is first entropy decoded (1030) to obtain transform coefficients, motion vectors, and other coded information. Picture partition information indicates how the picture is partitioned. Thus, the decoder can partition the picture (1035) according to the decoded picture partitioning information. The transform coefficients are inversely quantized (1040) and inversely transformed (1050) to decode the prediction residuals. By combining the decoded prediction residuals and the predicted block (1055), an image block is reconstructed. The predicted block can be obtained from an intra prediction (1060) or a motion-compensated prediction (i.e., an inter prediction (1075)) (1070). In-loop filters (1065) are applied to the reconstructed image. The filtered image is stored in the reference picture buffer (1080).

[0061] The decoded picture may additionally undergo post-decoding processing (1085), for example, an inverse color transform (e.g., a conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which performs the reverse of the remapping process performed in the pre-encoding process (901). The post-decoding processing may use metadata derived from the pre-encoding process and signaled in the bitstream.

[0062] FIG. 11 illustrates a block diagram of an example of a system in which various aspects and embodiments may be implemented. The system (1100) may be implemented as a device comprising various components described below and configured to perform one or more of the aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home devices, and servers. The elements of the system (1100) may be implemented, alone or in combination, as a single integrated circuit, multiple ICs, and / or individual components. For example, in at least one embodiment, the processing and encoder / decoder elements of the system (1100) are distributed across multiple ICs and / or individual components. In various embodiments, the system (1100) is coupled to other systems or other electronic devices, for example, via a communication bus or via dedicated input and / or output ports. In various embodiments, the system (1100) is configured to implement one or more of the embodiments described in this application.

[0063] The system (1100) includes, for example, at least one processor (1110) configured to execute instructions loaded therein to implement the various embodiments described in this application. The processor (1110) may include embedded memory, an input / output interface, and various other circuit components known in the art. The system (1100) includes at least one memory (1120) (e.g., a volatile memory device and / or a non-volatile memory device). The system (1100) includes a storage device (1140), which may include non-volatile memory and / or volatile memory, which includes, but is not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drive, and / or optical disk drive. The storage device (1140) may include, as non-limiting examples, an internal storage device, an attached storage device, and / or a network-accessible storage device.

[0064] The system (1100) includes, for example, an encoder / decoder module (1130) configured to process data to provide an encoded video / 3D object or a decoded video / 3D object, and the encoder / decoder module (1130) may include its own processor and memory. The encoder / decoder module (1130) represents a module(s) that may be included in a device to perform encoding and / or decoding functions. As is known, the device may include one or both of the encoding and decoding modules. Additionally, the encoder / decoder module (1130) may be implemented as a separate element of the system (1100) or may be integrated within the processor (1110) as a combination of hardware and software, as is known to a person skilled in the art.

[0065] Program code to be loaded into a processor (1110) or an encoder / decoder (1130) to perform the various embodiments described in this application may be stored in a storage device (1140) and subsequently loaded into memory (1120) for execution by the processor (1110). According to various embodiments, one or more of the processor (1110), memory (1120), storage device (1140), and encoder / decoder module (1130) may store one or more of various items during the execution of the processes described in this application. These stored items may include, but are not limited to, input video / 3D objects, decoded video / 3D objects or parts of decoded video / 3D objects, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, actions, and action logic.

[0066] In some embodiments, memory within the processor (1110) and / or encoder / decoder module (1130) is used to store instructions and to provide working memory for processing required during encoding or decoding. However, in other embodiments, memory outside the processing device (e.g., the processing device may be either the processor (1110) or the encoder / decoder module (1130)) is used for one or more of these functions. The external memory may be memory (1120) and / or storage device (1140), e.g., dynamic volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, high-speed external dynamic volatile memory, such as RAM, is used as working memory for coding and decoding operations, e.g., MPEG-2, HEVC, or VVC.

[0067] Inputs to the elements of the system (1100) may be provided through various input devices as shown in block (1105). These input devices include, but are not limited to, (i) an RF part that receives, for example, an RF signal transmitted over the air by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0068] In various embodiments, the input devices of block (1105) are associated with individual input processing elements as known in the art. For example, the RF portion may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting the signal to a band of frequencies), (ii) down-converting the selected signal, (iii) band-limiting it again to a narrower band of frequencies to select a signal frequency band that may be referred to as a channel in certain embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a stream of desired data packets. The RF portion of various embodiments includes one or more elements for performing these functions, e.g., frequency selectors, signal selectors, band-limiters, channel selectors, filters, down-converters, demodulators, error correctors, and demultiplexers. The RF section may include a tuner that performs various of these functions, for example, including down-converting the received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing element receive an RF signal transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the elements described above (and others), remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, for example, inserting amplifiers and analog-to-digital converters.In various embodiments, the RF portion includes an antenna.

[0069] Additionally, USB and / or HDMI terminals may include individual interface processors for connecting the system (1100) to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented as needed, for example, within a separate input processing IC or within the processor (1110). Similarly, aspects of USB or HDMI interface processing may be implemented as needed within separate interface ICs or within the processor (1110). The demodulated, error-corrected, and demultiplexed stream is provided to various processing elements, including an encoder / decoder (1130) and a processor (1110) that operate in combination with memory and storage elements to process the data stream as needed for presentation on an output device, for example.

[0070] Various elements of the system (1100) may be provided within an integrated housing. Within the integrated housing, the various elements may be interconnected using internal buses known in the art, including suitable connection arrangements (1115), e.g., I2C buses, wiring, and printed circuit boards, and may transmit data between them.

[0071] The system (1100) includes a communication interface (1150) that enables communication with other devices through a communication channel (1190). The communication interface (1150) may include, but is not limited to, a transceiver configured to transmit and receive data through the communication channel (1190). The communication interface (1150) may include, but is not limited to, a modem or a network card, and the communication channel (1190) may be implemented, for example, within a wired and / or wireless medium.

[0072] In various embodiments, data is streamed to the system (1100) using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signals of these embodiments are received through a communication channel (1190) and a communication interface (1150) adapted for Wi-Fi communications. The communication channel (1190) of these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments provide the streamed data to the system (1100) using a set-top box that transmits data via an HDMI connection of the input block (1105). Still other embodiments provide the streamed data to the system (1100) using an RF connection of the input block (1105).

[0073] The system (1100) may provide output signals to various output devices, including a display (1165), speakers (1175), and other peripheral devices (1185). The other peripheral devices (1185) include, in various examples of the embodiments, one or more of a standalone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functions based on the output of the system (1100). In various embodiments, control signals are communicated between the system (1100) and the display (1165), speakers (1175), or other peripheral devices (1185) using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. The output devices may be communicateably coupled to the system (1100) through dedicated connections via individual interfaces (1160, 1170, and 1180). Alternatively, output devices may be connected to the system (1100) using a communication channel (1190) via a communication interface (1150). The display (1165) and speakers (1175) may be integrated into a single unit with other components of the system (1100) in an electronic device, e.g., a television. In various embodiments, the display interface (1160) includes a display driver, e.g., a timing controller (T Con) chip.

[0074] The display (1165) and speaker (1175) may alternatively be separated from one or more of the other components, for example, if the RF portion of the input (1105) is part of a separate set-top box. In various embodiments where the display (1165) and speakers (1175) are external components, the output signal may be provided through dedicated output connections, for example, including HDMI ports, USB ports, or COMP outputs.

[0075] Various methods are described herein, and each of the methods comprises one or more steps or actions to achieve the described method. Unless a specific order of steps or actions is required for the proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Furthermore, terms such as “first,” “second,” etc., may be used in various embodiments to modify elements, components, steps, actions, etc., such as “first decoding” and “second decoding.” The use of these terms does not imply a specific order for the modified actions unless specifically required. Accordingly, in this example, the first decoding does not need to be performed before the second decoding, and may occur, for example, before, during, or within an overlapping period of the second decoding.

[0076] Furthermore, the embodiments described herein are not limited to DASH, HLS, or other streaming protocols, but may be applied, for example, to other standards and recommendations, and any extensions of such standards and recommendations. Unless otherwise indicated or technically excluded, the embodiments described herein may be used individually or in combination. Various numerical values ​​are used in this application. Specific values ​​are for exemplary purposes and the described embodiments are not limited to these specific values.

[0077] As discussed above, the embodiments described herein describe principles related to specific drafts of Versatile Video Coding (VVC) or High Efficiency Video Coding (HEVC) standards, but are not limited to VVC or HEVC and may be applied, for example, to other standards and recommendations, whether existing or future to be developed, and to any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, the embodiments described in this application may be used individually or in combination.

[0078] Various implementations include decoding. As used in this application, “decoding” may include all or part of the processes performed on a received encoded sequence to generate a final output suitable for a display, for example. In various embodiments, these processes include one or more of the processes typically performed by a decoder, for example, entropy decoding, inverse quantization, inverse transform, and differential decoding. Whether the phrase “decoding process” is intended to refer specifically to a subset of operations or to refer generally to a broader decoding process will be clear based on the context of the specific descriptions and is believed to be well understood by a person skilled in the art.

[0079] Various implementations include encoding. In a manner similar to the above discussion regarding "decoding," the term "encoding" as used in this application may include, for example, all or part of the processes performed on an input video sequence to generate an encoded bitstream.

[0080] In this application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "encoded" or "coded" may be used interchangeably, the terms "pixel" or "sample" may be used interchangeably, and the terms "image," "picture," and "frame" may be used interchangeably. Generally, though not necessarily, the term "reconstructed" is used on the encoder side, while "decoded" is used on the decoder side.

[0081] The implementations and aspects described herein may be implemented, for example, as methods or processes, devices, software programs, data streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), implementations of the discussed features may also be implemented in other forms (e.g., devices or programs). Devices may be implemented, for example, in appropriate hardware, software, and firmware. Methods may be implemented, for example, in devices, for example, processors, which generally refer to processing devices, including, for example, computers, microprocessors, integrated circuits, or programmable logic devices. Processors also include communication devices, such as, for example, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate information communication between end-users.

[0082] References to "one embodiment," "an embodiment," "one implementation," or "an implementation," as well as other variations thereof, imply that the specific features, structures, characteristics, etc. described in relation to the embodiment are included in at least one embodiment. Accordingly, the appearance of phrases such as "in one embodiment," "in an embodiment," "in one implementation," or "in an implementation," as well as any other variations, appearing at various locations throughout this application, do not necessarily all refer to the same embodiment. Additionally, this application may refer to "determining" various pieces of information. Determining information may include, for example, estimating information, calculating information, predicting information, or retrieving information from memory.

[0083] Furthermore, this application may refer to "accessing" various pieces of information. Accessing information may include, for example, receiving information, retrieving information (for example, from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0084] Additionally, this application may refer to various pieces of information as "receiving." Receiving is intended in a broad sense, just as "accessing." Receiving information may include, for example, accessing information, or retrieving information (for example, from memory). Furthermore, "receiving" is typically involved in some way during operations, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0085] For example, in the cases of "A / B", "A and / or B" and "at least one of A and B", it should be understood that using any of " / ", "and / or", and "at least one of" is intended to include the selection of only the first listed option (A), the selection of only the second listed option (B), or the selection of both options (A and B). As an additional example, in the cases of "A, B, and / or C" and "at least one of A, B, and C," this construction is intended to include the selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended to as many items as listed, as will be apparent to a person skilled in the art and related technical fields.

[0086] Additionally, as used herein, the word "signal" refers, among other things, to indicating something to the corresponding decoder. For example, in certain embodiments, the encoder signals a quantization matrix for inverse quantization. In this way, in the embodiments, the same parameter is used on both the encoder side and the decoder side. Thus, for example, the encoder may transmit a specific parameter to the decoder (explicit signaling) so that the decoder may use the same specific parameter. Conversely, if the decoder already has other parameters in addition to the specific parameter, signaling may be used without transmission (implicit signaling) so that the decoder simply knows and selects the specific parameter. By avoiding the transmission of any actual functions, bit savings are realized in various embodiments. It should be understood that signaling can be achieved in various ways.

[0087] For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder. Although the foregoing relates to the verb form of the word "signal," the word "signal" may also be used as a noun in this specification.

[0088] As is apparent to a person skilled in the art, implementations may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over various different wired or wireless links. The signal may be stored on a processor-readable medium.

Claims

Claim 1 A method for using attenuation maps for adaptive streaming, comprising: receiving an adaptive streaming manifest including identification of one or more adaptive sets by a client device including a video decoder and a display, wherein a first adaptive set among the one or more adaptive sets includes a plurality of representations of an item of streaming media content and identifications of one or more display attenuation maps; searching for one or more media segments of an item of streaming media content corresponding to one of the plurality of representations by the client device; searching for one or more media segments of a first display attenuation map among the one or more display attenuation maps corresponding to one of the plurality of representations by the client device; and displaying one or more media segments of an item of streaming media content adjusted based on the one or more media segments of the first display attenuation map by the client device through the display. Claim 2 A method according to claim 1, wherein the one or more media segments of the first display attenuation map include pixel attenuation factors. Claim 3 A method according to claim 1, wherein the step of displaying one or more media segments of an item of streaming media content further comprises the step of, for each pixel of a media segment of an item of streaming media content, combining the luminance value, the chroma value, or both the luminance and chroma values ​​of the pixel with the corresponding pixel attenuation factor of the corresponding media segment of the display attenuation map. Claim 4 The method of claim 1, wherein the first adaptation set comprises identifications of a plurality of display attenuation maps, and the method further comprises the step of selecting the first display attenuation map by the client device based on one or more of a device power condition, a battery charge level, and the remaining time of an item of the streaming media content. Claim 5 A method according to claim 1, wherein the client device receives the adaptive streaming manifest from the first server device. Claim 6 A method according to claim 1, wherein the client device retrieves the one or more media segments of the items of the streaming media content from a plurality of server devices. Claim 7 A method according to claim 1, further comprising the step of selecting a first expression among a plurality of expressions based on network conditions by the client device. Claim 8 In claim 7, a method wherein one of the plurality of expressions comprises the selected first expression. Claim 9 A method according to claim 1, further comprising the step of determining to reduce the energy usage of the display of one or more media segments by the client device. Claim 10 In claim 9, the step of deciding to reduce the energy usage of the display of one or more media segments is based on one or more of the device power condition, the battery charge level, and the remaining time of the item of the streaming media content. Claim 11 A client device utilizing attenuation maps for adaptive streaming, comprising: a video decoder; and a display operably coupled to the video decoder, wherein the client device is configured to receive an adaptive streaming manifest including identification of one or more adaptive sets, wherein a first adaptive set among the one or more adaptive sets includes a plurality of representations of an item of streaming media content and identifications of one or more display attenuation maps; wherein the client device is further configured to retrieve one or more media segments of an item of streaming media content corresponding to one of the plurality of representations, wherein the client device is further configured to retrieve one or more media segments of a first display attenuation map among the one or more display attenuation maps corresponding to one of the plurality of representations; and wherein the client device is further configured to display one or more media segments of an item of streaming media content adjusted based on the one or more media segments of the first display attenuation map. Claim 12 In claim 11, the client device wherein the one or more media segments of the first display attenuation map include pixel attenuation factors. Claim 13 A client device according to claim 11, wherein displaying one or more media segments of an item of streaming media content further comprises, for each pixel of a media segment of an item of streaming media content, combining the luminance value, chroma value, or both luminance and chroma values ​​of the pixel with the corresponding pixel attenuation factor of the corresponding media segment of the display attenuation map. Claim 14 A client device according to claim 11, wherein the first adaptation set comprises identifications of a plurality of display attenuation maps, and further comprises selecting the first display attenuation map by the client device based on one or more of a device power condition, a battery charge level, and the remaining time of an item of the streaming media content. Claim 15 In paragraph 11, the client device is a client device that receives the adaptive streaming manifest from the first server device. Claim 16 In paragraph 11, the client device retrieves one or more media segments of an item of the streaming media content from a plurality of server devices. Claim 17 A client device according to claim 11, further comprising selecting a first expression among the plurality of expressions based on network conditions by the client device. Claim 18 A client device according to claim 11, further comprising determining to reduce the energy usage of the display of one or more media segments by the client device. Claim 19 A client device utilizing attenuation maps for adaptive streaming, comprising: a video decoder; and an input / output device, wherein the client device is configured to receive an adaptive streaming manifest comprising identifications of one or more adaptive sets, wherein a first adaptive set among the one or more adaptive sets comprises identifications of a plurality of representations of an item of streaming media content and one or more display attenuation maps; wherein the client device is further configured to retrieve one or more media segments of an item of streaming media content corresponding to one of the plurality of representations, wherein the client device is further configured to retrieve one or more media segments of a first display attenuation map among the one or more display attenuation maps corresponding to one of the plurality of representations; and wherein the client device is further configured to cause one or more media segments of an item of streaming media content, adjusted based on the one or more media segments of the first display attenuation map, to be displayed. Claim 20 In claim 11, the client device transmits the one or more media segments of the item of the streaming media content to be displayed to an external display, thereby causing the one or more media segments of the item of the streaming media content to be displayed.

Images