Workload Allocation and Processing in Cloud-Based Encoding of HDR Video

The method optimizes workload allocation in cloud-based HDR video encoding by distributing scenes evenly across nodes, reducing metadata updates and maintaining video quality, addressing the overhead issues in existing technologies.

JP7577827B2Active Publication Date: 2024-11-05DOLBY LABORATORIES LICENSING CORP
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
JP2023500024
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-07-09
Filing Date
2021-07-08
Publication Date
2024-11-05
Estimated Expiration
2041-07-08

AI Technical Summary

Technical Problem

In cloud-based environments for HDR video encoding, existing workload allocation methods result in unacceptable overhead due to frequent updates of reshaping-related metadata, especially at low bit rates, compromising video quality and computational efficiency.

Method used

A method for workload allocation and node-based processing in cloud-based video encoding, involving scene-to-segment assignment and iterative refinement to distribute scenes evenly across computational nodes, reducing the need for frequent metadata updates by using scene-based forward and backward reshaping functions.

Benefits of technology

This approach reduces metadata overhead while maintaining video quality by optimizing workload distribution, ensuring uniform processing loads and minimizing the frequency of reshaping metadata updates, particularly effective in low-bitrate scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007577827000024
    Figure 0007577827000024
  • Figure 0007577827000025
    Figure 0007577827000025
  • Figure 0007577827000026
    Figure 0007577827000026
Patent Text Reader

Abstract

In a cloud-based system for encoding high dynamic range (HDR) video, a compute node is assigned to be an orderer node, partitioning an input video into scenes and generating scene-to-segment assignments for use by other compute nodes. The scene-to-segment assignment process involves one or more iterative steps that include an initial random assignment of scenes to compute nodes, followed by a refined assignment based on optimizing assignment costs across all compute nodes. A method for generating scene-based forward and backward reshaping functions to optimize video coding and improve the coding efficiency of reshaping-related metadata is also discussed.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority from U.S. Patent Application No. 63 / 049,673, and European Patent Application No. EP20184883.5, both filed on July 9, 2020, each of which is incorporated by reference in its entirety.

[0002] technology The present disclosure relates generally to imaging. More particularly, certain embodiments of the present disclosure relate to workload allocation and processing in cloud-based encoding of high dynamic range (HDR) video. [Background technology]

[0003] As used herein, the term "dynamic range" (DR) may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image, e.g., from darkest gray (black) to brightest white (highlight). In this sense, DR relates to "scene-referred" intensity. DR may also relate to the ability of a display device to adequately or approximately render a particular width of intensity range. In this sense, DR relates to "display-referred" intensity. Unless a particular meaning is expressly specified as being particularly significant at any point in the description, it should be presumed that the term may be used in either sense, e.g., interchangeably.

[0004] As used herein, the term high dynamic range (HDR) refers to a DR width that spans 14-15 orders of magnitude of the human visual system (HVS). In practice, the DR where humans can simultaneously perceive a wide range of intensity ranges may be somewhat truncated for HDR. As used herein, the terms "visual dynamic range" (VDR) or "enhanced dynamic range" (EDR), individually or interchangeably, refer to the DR perceivable in a scene or image by the human visual system (HVS) that includes eye movements and allows for some light adaptation changes across the scene or image. As used herein, VDR may refer to a DR that spans 5-6 orders of magnitude. Thus, although it may be somewhat narrower than a true scene-based HDR, VDR or EDR still represent a wide DR width and may still be referred to as HDR.

[0005] In practice, an image has one or more color components (e.g. luma Y and chroma Cb and Cr), where each color component is represented with a precision of n bits per pixel (e.g. n=8). For example, using gamma-luminance encoding, images with n≦8 (e.g. color 24-bit JPEG images) may be considered standard dynamic range images, and images with n≧10 may be considered enhanced dynamic range images. HDR images may be stored and distributed using high precision (e.g. 16-bit) floating point formats, such as the OpenEXR file format developed by Industrial Light & Magic.

[0006] Most consumer desktop displays currently have a resolution of 200 to 300 cd / m 2 or nits luminance. Most consumer HDTVs are in the 300 to 500 nits range, with newer models supporting 1000 nits (cd / m 2). Thus, such conventional displays are representative of lower dynamic range (LDR), also called standard dynamic range (SDR), as opposed to HDR. As HDR content becomes more available due to advances in both filming equipment (e.g., cameras) and HDR displays (e.g., the PRM-4200 Professional Reference Monitor from Dolby Laboratories), HDR content may be color graded and displayed on HDR displays that support a higher dynamic range (e.g., 1000 nits to 5000 nits or more). As used herein, the term "forward reshaping" refers to the process of sample-to-sample or codeword-to-codeword mapping of a digital image from an original bit depth and an original codeword distribution or representation (e.g., gamma, PQ, HLG, etc.) to the same or a different bit depth and a different codeword distribution or representation. The reshaping allows for improved compressibility or improved image quality at a fixed bit rate. For example, but not by way of limitation, reshaping may be applied to 10-bit or 12-bit PQ coded HDR video to improve coding efficiency in a 10-bit video coding architecture. At the receiver, after decompressing the received signal (which may or may not have been reshaped), the receiver may apply an "inverse (or backward) reshaping function" to restore the signal to the original codeword distribution and / or achieve a higher dynamic range. Each of the following references is incorporated by reference in its entirety. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] U.S. Patent No. 10,575,028, H. Kadu et al., "Coding of high-dynamic range video using segment-based reshaping" [Patent Document 2] U.S. Patent No. 8,811,490, GM. Su et al., "Multiple color channel multiple regression predictor" [Patent Document 3] WO 2019 / 217751, Q. Song et al., PCT Patent Application No. PCT / US2019 / 031620, "High-fidelity full reference and high-efficiency reduced reference encoding in end-to-end single-layer backward compatible encoding pipeline", filed May 9, 2019 [Patent Document 4] U.S. Patent No. 10,264,287, B. Wen et al., "Inverse luma / chroma mappings with histogram transfer and approximation" [Patent Document 5] U.S. Patent No. 10,397,576, H. Kadu and GM. Su, "Reshaping curve optimization in HDR coding" Summary of the Invention [Problem to be solved by the invention]

[0008] In many video delivery scenarios, HDR video may be encoded in a multi-processor environment, typically referred to as a "cloud computing server." In such environments, the tradeoff between ease of computation, workload balance among computing nodes, and video quality may force reshaping-related metadata to be updated every frame, which may result in unacceptable overhead, especially when transmitting video at low bitrates. As recognized by the inventors, improved techniques for workload allocation and node-based processing are desired to improve the quality of encoded video in cloud-based environments while minimizing the overhead of reshaping-related metadata.

[0009] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise noted, it should not be assumed that any of the approaches described in this section qualify as prior art by virtue of their inclusion in this section. Similarly, unless otherwise noted, it should not be assumed that problems identified with one or more approaches have been recognized in the prior art based on this section. [Brief description of the drawings]

[0010] Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numerals refer to similar elements and in which:

[0011] [Figure 1A] 1 illustrates an exemplary single-layer encoder for HDR data using a reshaping function according to the prior art.

[0012] [Figure 1B] 1 shows an exemplary HDR decoder corresponding to the encoder of FIG. 1A according to the prior art.

[0013] [Diagram 2]1 illustrates an exemplary architecture for cloud-based encoding of HDR video, according to an embodiment.

[0014] [Figure 3A] 1 illustrates an exemplary process for scene-to-segment assignment, according to one embodiment.

[0015] [Figure 3B] FIG. 13 illustrates an example of a refinement assignment process within the scene-to-segment assignment process, according to one embodiment.

[0016] [Figure 4] 1 illustrates an exemplary encoder for scene-based encoding using reshaping, in accordance with one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Methods for workload allocation and node-based processing in cloud-based video encoding of HDR video are described herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and apparatuses have not been described in exhaustive detail in order to avoid unnecessarily obscuring, obscuring, or obscuring the present invention.

[0018] overview Exemplary embodiments described herein relate to cloud-based reshaping and coding for HDR images. In one embodiment, in a cloud-based system for encoding HDR video, a node is assigned to be an orderer node that segments an input video into scenes and generates a scene-to-segment assignment that is used by other computational nodes. A processor in the orderer node receives a sequence of scenes, each scene containing one or more video frames, and then the processor: receiving a sequence of scenes, each scene including one or more video frames; and performing one or more allocation iterations to generate an optimal output allocation, said performing one or more allocation iterations comprising: For an iteration in the one or more allocation iterations: generating an initial random assignment (305) of a sequence of scenes to M computational nodes based on a random seed selection for the assignment iterations, where M>1; performing a refinement allocation step (310) based on the initial random allocation to generate a refined allocation of the sequence of scenes to the M computational nodes and a refined allocation cost; and updating a best allocation cost and a best output allocation (315) based on the refined allocation and the refined allocation cost.

[0019] In another embodiment, for a node among the M computational nodes, a processor within the node performs the steps of accessing scenes assigned to the node according to a scene-to-segment assignment, the scenes including a sequence of high dynamic range (HDR) frames and a corresponding sequence of standard dynamic range (SDR) frames, and generating an output bitstream and corresponding reshaping metadata using a scene-based forward reshaping function and a scene-based backward reshaping function.

[0020] Exemplary HDR Encoding System 1A and 1B show an exemplary single-layer backward-compatible codec framework with image reshaping according to the prior art. More specifically, FIG. 1A shows an exemplary encoder architecture that may be implemented in one or more computing processors in an upstream video encoder. FIG. 1B shows an exemplary decoder architecture that may also be implemented in one or more computing processors in one or more downstream video decoders.

[0021] Under this framework, given a reference HDR content (120) and a corresponding reference SDR content (125) (i.e., content that represents the same image as the HDR content, but color graded and rendered in standard dynamic range), a reshaped HDR content (134) is encoded and transmitted as SDR content in a single layer of the coded video signal (144) by an upstream encoding device implementing an encoder architecture. The received SDR content is received and decoded in a single layer of the video signal by a downstream decoding device implementing a decoder architecture. The backward reshaping metadata (152) is also encoded and transmitted in the video signal together with the reshaped content, so that an HDR display device can reconstruct the HDR content based on the (reshaped) SDR content and the backward reshaping metadata. Without loss of generality, in some embodiments, such as in a non-backward compatible system, the reshaped SDR content may not be viewable by itself, but must be viewed in combination with a backward reshaping function that generates a viewable SDR or HDR content. In other embodiments that support backward compatibility, legacy SDR decoders can still play received SDR content without using the backward reshaping function.

[0022] As shown in FIG. 1A, after generating a forward reshaping function (132) in step 130 given an HDR image (120) and a target dynamic range, a forward reshaping mapping step (132) is applied to the HDR image (120) given the forward reshaping function to generate a reshaped SDR base layer (134). A compression block 142 (e.g., an encoder implemented according to any known video encoding algorithm such as AVC, HEVC, AV1, etc.) compresses / encodes the SDR image (134) in a single layer (144) of the video signal. Furthermore, a backward reshaping function generator (150) can generate a backward reshaping function that can be transmitted to a decoder as metadata (152). In some embodiments, the metadata (152) can represent the forward reshaping function (130), and it is then up to the decoder to generate the backward reshaping function (not shown).

[0023] Examples of backward reshaping metadata that represent / specify optimal backward reshaping functions may include, but are not limited to: an inverse tone mapping function, an inverse luma mapping function, an inverse chroma mapping function, a look-up table (LUT), a polynomial, an inverse display management coefficients / parameters, etc. In various embodiments, the luma backward reshaping function and the chroma backward reshaping function may be derived / optimized jointly or separately, and may be derived using a variety of techniques, such as, for example, without limitation, as described later in this disclosure.

[0024] The backward reshaping metadata (152) generated by the backward reshaping function generator (150) based on the reshaped SDR image (134) and the target HDR image (120) may be multiplexed as part of the video signal 144, for example as supplemental enhancement information (SEI) messaging.

[0025] In some embodiments, the rear reshaping metadata (152) is carried in the video signal as part of the overall image metadata, which is carried in the video signal separate from the single layer in which the SDR image is encoded in the video signal. For example, the rear reshaping metadata (152) may be encoded in a component stream in the encoded bitstream, which may or may not be separate from the single layer (of the encoded bitstream) in which the SDR image (134) is encoded.

[0026] In this manner, the backward reshaping metadata (152) can be generated or pre-generated at the encoder side to take advantage of the powerful computational resources and offline encoding flows available at the encoder side, including, but not limited to, content adaptive multiple passes, look-ahead operations, inverse luma mapping, inverse chroma mapping, CDF-based histogram approximation and / or transfer, etc.

[0027] The encoder architecture of FIG. 1A can be used to avoid directly encoding the target HDR image (120) into the coded / compressed HDR image in the video signal; instead, using the backward reshaping metadata (152) in the video signal, the downstream decoding device can backward reshape the SDR image (134) (encoded in the video signal) into a reconstructed image that is identical to the reference HDR image (120) or that closely / optimally approximates it.

[0028] In some embodiments, as shown in FIG. 1B, a video signal encoded with a reshaped SDR image in a single layer (144) and backward reshaping metadata (152) as part of the overall image metadata are received as input at the decoder side of the codec framework. A decompression block (154) decompresses / decodes the compressed video data in the single layer (144) of the video signal into a decoded SDR image (156). Decompression 154 typically corresponds to the inverse of compression 142. The decoded SDR image (156) may be the same as the SDR image (134) subject to quantization errors in the compression block (142) and decompression block (154), which may be optimized for an SDR display device. In a backward compatible system, the decoded SDR image (156) may be output (through an HDMI interface, through a video link, etc.) in an output SDR video signal to be rendered on an SDR display device.

[0029] Optionally, alternatively or additionally, in the same or another embodiment, the rear reshaping block 158 extracts rear (or front) reshaping metadata (152) from the input video signal, constructs a rear reshaping function based on the reshaping metadata (152), and performs a rear reshaping operation on the decoded SDR image (156) based on the optimal rear reshaping function to generate a rear reshaped image (160) (or a reconstructed HDR image). In some embodiments, the rear reshaped image represents a production-quality or near-production-quality HDR image that is identical to or closely / optimally approximates the reference HDR image (120). The rear reshaped image (160) may be output in an output HDR video signal (e.g., through an HDMI interface, through a video link, etc.) to be rendered on an HDR display device.

[0030] In some embodiments, display management operations specific to an HDR display device may be performed on the rear reshaped image (160) as part of an HDR image rendering operation that renders the rear reshaped image (160) on an HDR display device.

[0031] Cloud-based coding Existing reshaping techniques may be frame-based, i.e. new reshaping metadata is sent for each new frame, or scene-based, i.e. new reshaping metadata is sent for each new scene. As used herein, the term "scene" in reference to a video sequence (sequence of frames / images) may refer to a series of consecutive frames in a video sequence that share similar luminance, color, and dynamic range characteristics. Scene-based methods work well in video workflow pipelines where complete scenes are accessible. However, it is not uncommon for content providers to use cloud-based multiprocessing, where after splitting a video stream into segments, each segment is processed independently by a single computational node in the cloud. As used herein, the term "segment" refers to a series of consecutive frames in a video sequence. A segment may be part of a scene or may contain one or more scenes. Thus, the processing of a scene may be split across multiple processors.

[0032] As discussed in US Pat. No. 6,399,663, in certain cloud-based applications, under certain quality constraints, segment-based processing may require generating reshaping metadata for every frame, resulting in undesirable overhead. This may be problematic for very low bitrate (e.g., less than 1 Mbit / s) applications. Figure 2 shows an example architecture of a new scene-based distributed architecture that allows for data rate reduction of the reshaping metadata without compromising the quality of the decoded video.

[0033] As shown in Figure 2, the proposed architecture includes two stages: a) an orderer stage (205), typically but not exclusively implemented on a single computational node, that assigns a scene to segments, and b) an encoding stage (210), where each node in the cloud encodes a sequence of segments.

[0034] Given a video source (202) for content distribution, typically called a mezzanine file, the first stage node fetches video metadata (e.g., from an XML file) and: (a) determines scene boundaries, step 215; and (b) determines a scene-to-segment assignment list for each worker node, step 220. The main objective for scene boundary determination is to ensure that there are no noticeable luminance or color changes, such as fade-ins, fade-outs, or dissolves, during normal playback within a scene. (A dissolve in video editing refers to a smooth transition from one image to another. A dissolve from a blank (or black) image to another image is also called a fade-in or fade-out.) The goal of the scene-to-segment assignment unit (220) is to ensure that a scene is not split and encoded on two different computation nodes, which may cause abrupt changes near segment boundaries. Furthermore, the assignment task should strive towards a uniform workload across all computation nodes (210).

[0035] In the second stage (210), each computational node receives its own list of segments from the scene (S2S list) (230) from stage 1 and its own partial mezzanine (225) for the corresponding segments from the input video (202). Each node encodes its assigned segments in parallel and outputs an encoded bitstream. Details of each processing task are discussed next.

[0036] Orderer node (dispatcher node) Depending on the workload distribution requirements at each node, there are two main scenarios of interest. In one embodiment, segments may have non-uniform length, which allows for non-uniform workload at different nodes. This is adjusted for scene-based solutions where a scene cannot be split to be encoded at multiple nodes. In another embodiment, segments have a fixed length, thus enforcing a uniform workload across all nodes. Under the proposed orderer and worker node model, the proposed architecture can handle both scenarios.

[0037] In a non-uniform segment length scenario, each worker node may receive a different workload for processing. To enable scene-based encoding, the orderer node reads an XML file (extracted from Mezzanine) and determines scene boundaries. In particular, it determines how to split or merge frames in fade-ins / fade-outs and how to dissolve scenes to new scene cut boundaries. Among these new defined scenes, the orderer determines which scenes should be encoded by which nodes. The output of this process is a Scene to Segment (S2S) list (230). The main objective is to distribute the number of frames in each node as uniformly as possible. In one embodiment, without being limited thereto, a metric for measuring uniformity at this stage is the standard deviation of the number of frames assigned at each node. A smaller standard deviation means that the workload of each node is more uniform. In one embodiment, the S2S list may be derived as the output of an optimization problem that seeks the most uniform load across all nodes under an uninterrupted scene processing constraint.

[0038] Scene cuts may be defined in the XML file of the video source (202), but typically such metadata defines color grading boundaries. For example, scene cut flags can be inserted during dissolve scenes for color grading convenience so that the display management process does not distort the colors during playback. However, these XML data do not take into account that the baseline data is reshaped and that the reshaping may affect the final appearance in the dissolve. In one embodiment, to avoid such problems, dissolves can be split into multiple single frames per scene to allow for slow transitions along the time domain. Note that this method increases the bitrate of the reshaping related metadata during these special transition effects. The same technique can be applied to fade-in and fade-out transitions.

[0039] If an XML file is not available, the purchaser must identify the scene cuts themselves using any of the known scene cut detection techniques known in the art. For example, in one embodiment, one can measure the luminance change along the time domain and see if the change is at a constant rate. Once a scene cut is detected, the entire scene cut can be split into single frames, with each frame representing a separate "scene."

[0040] In addition to the above methods, we can also consider soft transitions near scene cut boundaries to avoid false scene cut boundaries. For example, for a detected scene cut, we can add a few single-frame "scenes" before the scene cut and add a few single-frame "scenes" after the scene cut. Such methods will increase the bitrate of scene-based metadata.

[0041] Given a scene boundary determination (215), the Scene to Segment unit (220) determines which scenes should be included in which segments. This kind of allocation results in a Scene to Segment Allocation List (S2S List) (230). The orderer node outputs one S2S List for each worker node.

[0042] Consider a video sequence with J total frames grouped into K scenes. Let S be the corresponding starting frame index for the kth scene. k Let the number of frames for the kth scene be D k where k=0,1,…,K-1. Thus:

number

[0043] Let the set of scenes assigned to node m be Φ m Here, m = 0, 1, ..., M-1. Following the rules above, Φ m φ is the first scene index in m where φ mhas a value range between 0 and K - 1. In certain embodiments, to simplify the implementation, a monotonic increment rule may be imposed. That is: when m < n, φ m < φ n In certain embodiments, φ 0 = 0, and thus the first scene is always assigned to the first segment. If the number of scenes K is greater than the number of nodes M, φ m must be unique. That is, its value must not be the same at any other node. This is to ensure that the workload of no node is zero. If K < M, a simple solution is to assign one scene to each node and leave the remaining nodes with zero scenes.

[0044] Figure 3A shows an exemplary process (300) for segment assignment from scenes, according to an embodiment. As shown in Figure 3A, the process begins with an initial random assignment (305). In this step, a list of K scenes is randomly divided into M segments. The initial list is further adjusted using a sequential iterative algorithm. Step 305 initializes two sets: · The candidate set (Ω (t) ) is the original list of scene indices at the end of the t-th iteration step (for t > 0). · The selected set (Ψ (t) ) is the list of assigned scene indices at the end of the t-th iteration step.

[0045] At the beginning of the operation, i.e., when t = 0, Ω (0) includes all scenes except the first scene (i.e., Ω (0) = {φ m | m = 1,…, K - 1}) and Ψ (0) includes only the first scene φ 0 . In the t-th iteration step (t > 0), one element is randomly selected from Ω (t-1) and this element is removed from Ω (t-1)and put this selected element into the set Ψ (t) This process is (t) This is repeated M-1 times until contains M elements sorted in ascending order. (t) is the output from this stage. Table 1 shows this process in pseudocode. Table 1: Initialization steps for scene to segment allocation [Table 1]

[0046] As an example, consider a list of 10 scenes to be allocated among three nodes, each with a variable number of frames, as shown below. [Table 2] Let the output of step 305 be Ψ (2) ={0,3,8}, then after this step the scene will be assigned to nodes (or segments) as follows: Node 0: Scenes 0-2 Node 1: Scenes 3-7 Node 2: Scenes 8-9

[0047] In step 310, this initial random assignment (Ψ (M-1) ) is further refined iteratively, as shown in FIG. 3B. As shown in FIG. 3B, there are two iterative steps: a) one at the node level (for all nodes), and b) one at the total allocation cost level (until convergence). Starting at step 345, the total allocation cost is refined by iteratively iteratively, as shown in FIG. f opt = inf) to a large value that can approximate the metric. The algorithm is then iterated over each node m, m = 0, 1, 2, ... M-1. At each node iteration step, the workload of each node can be looked at and each node's workload can be adjusted (350) using three possible scenarios: (A) Remove that last scene and move it to the next node (this does not apply to node M-1) (B) Add one more scene from the previous node (this does not apply to node 0) (C) Maintain your current allocation Of these three options, step 355 measures the allocation-related cost (e.g., the standard deviation of the number of frames in each segment). A lower cost is more preferable since it means the workload is more uniform. Thus, for each node, the setting that gives the lowest allocation cost is selected in step 360. After all nodes have been processed, step 362 selects the setting with the lowest allocation cost (e.g., σ f (t) ) is the total existing allocation cost (e.g., σ f opt ) if the lowest allocation cost is deemed lower than the total allocation cost, the value of the total allocation cost is updated with the lowest allocation cost and the process returns to step 350. Otherwise, if there is no cost improvement or the improvement is deemed too small, in step 365 the refinement allocation stage 310 ends by outputting the segment allocations from the last scene and the corresponding costs (i.e. the final value of the total allocation cost). These are referred to as the refined S2S allocations and the refined allocation costs.

[0048] In another embodiment, instead of starting the node iteration (e.g., steps 350, 355, and 360) at node 0 and proceeding forward, the node iteration may begin at node M-1 and proceed backwards. Alternatively, one may attempt to iterate through all nodes in both directions and select the workload with the smallest cost between the two.

[0049] After stage 310, given the segment allocation from the refined scene, a new best overall allocation cost (and associated S2S allocation) may be calculated in step 315. In one embodiment, to avoid bad random initialization steps 305 that may lead to sub-optimal allocations, steps 305-315 are repeated L times for L different random initialization steps 305 (e.g. by using different random seed generators), each step calculating an overall allocation cost (l), l=1,2,...L (e.g. σ f,l opt ) is then given in step 315. The best overall cost (e.g., the one with the smallest standard deviation σ f,l opt ) for the allocation. Experimental results show that L=100 in combination with the refined allocation step (310) produces satisfactory results, and larger values ​​of L do not significantly improve the overall S2S allocation strategy.

[0050] Therefore, at l=1, σ f,1 opt simply represents the initial refined allocation cost (i.e., σ f * = σ f,1 opt = σ f opt ), where σ f * represents the best overall allocation cost. In subsequent iterations, σ f * <σ f,l opt If so, this iteration is ignored; otherwise, the best allocation cost is updated (e.g., σ f * = σ f,l opt ), the corresponding workload for this iterative process is considered as the best scene-to-segment assignment.

[0051] Step 320 checks whether all L iterations have been performed, if so, in step 325 the best scene-to-segment assignment, i.e. the one with the best cost among all L iterations, is output, if not the process is repeated with another initial random assignment (305).

[0052] For ease of discussion, we introduce another variable φ that indicates the end of the video sequence. M (t) =K is added. {φ m (t)}, m=0,1,...,M, the number of frames at each node in the t-th iteration can be calculated as follows:

number

number

[0053] In one embodiment, without limitation, the standard deviation of the frames used provides a good cost metric for the refinement allocation stage, but alternative cost metrics such as the following may be applied: · Workload bounds, measured as the maximum number of all allocated frames in a node minus the minimum number of all allocated frames in a node The average workload of all frames at each node (e.g., μ f (t) ) ·

number

[0054] Returning to our example, Tables 3 and 4 show the allocation of segments from the scene and the corresponding S2S parameters after the random initialization stage. m (0) The overall cost, measured by the standard deviation of the values ​​in}, is σ f (0) =6.81. Table 3: Example S2S allocations after initialization [Table 4] Table 4: Example allocation parameters after the initialization stage [Table 5]

[0055] Now consider an example of refined allocation (310). In the first iteration of this stage (t=0), for the first node m=0, three different strategies are tried, each with a standard deviation σ f (0) is measured. The results are shown in Tables 5 and 6. As shown in Table 5, under option A, node 0 is assigned only scenes 0 and 1 with a cost of 10.12, under option B, node 0 is assigned scenes 0-3 with a cost of 5.04, and under option C (unchanged from before), the cost remains the same (6.81). Thus, option B is selected as the best strategy to continue refining scene assignments in subsequent nodes. So the same process is repeated. Table 5: Example refined allocation for node 0 [Table 6] Table 6: S2S parameters for options A, B, and C for node 0 [Table 7]

[0056] At the end of t=0, in this example, the best S2S is the one with the cost σ f opt It remains as it was in 5.04; namely: Node 0: Scenes 0-3 Node 1: Scenes 4-7 Node 2: Scenes 8-9

[0057] Then, at t = 1, steps 350, 355 and 360 are repeated. In this example, at t = 1, there is no improvement in the overall cost, so the process ends.

[0058] In some embodiments, it may be preferable for all segments to have the same number of frames. For such a scenario, the number of frames for the first M-1 nodes may be assigned as follows:

number

number

[0059] Scene-Based Coding Given an allocation of segments from a scene (230), Figure 4 shows an exemplary architecture for scene-based encoding on each node (210) in the cloud. Let S be the starting frame index for the kth scene.k Recall that, given a scene k, a node is written as k , S k +1, S k +2, …, and S k+1 −1 needs to be processed. A reference HDR frame (404) and a corresponding SDR frame (402) for a scene may be stored in corresponding SDR and HDR scene buffers (not shown).

[0060] From FIG. 4, in step 405, the input SDR and HDR frames are used to generate a scene-based forward reshaping function. The parameters of such a function are used for the entire scene (rather than updated every frame), thus reducing the overhead for metadata 152. Next, in step 132, the forward reshaping is applied to the HDR scene (404) to generate a reshaped base layer 407, which is encoded by the compression unit (142) to generate the encoded bitstream 144. Finally, in step 410, the reshaped SDR data 407 and the original HDR data (404) are used to generate parameters 152 for a backward reshaping function that are sent together to the downstream decoder. These steps are described in more detail below. Without being limiting, the steps are described in the context of what is called a three-dimensional mapping table (3DMT) representation, where, to simplify operation, each frame is represented as a three-dimensional mapping table, each color component (e.g., Y, Cb, or Cr) is subdivided into "bins," and instead of representing the image using explicit pixel values, the pixel average within each bin is used. Details of the 3DMT formulation can be found in U.S. Pat. No. 6,399,433.

[0061] The scene-based generation of the forward reshaping function (405) consists of two levels of operation. First, statistics are collected for each frame. For example, for luma, SDR(h j s (b)) Frame and HDR(h jv (b)) Compute histograms for both frames and store them in the frame buffer for the jth frame, where b is the bin index. After generating the 3DMT representation for each frame, generate an “a / B” matrix representation, denoted as:

number

[0062] Given the statistics of each frame in the current scene, a scene-level algorithm can be applied to calculate the optimal forward reshaping coefficients. For example, for luma, we can calculate the SDR (h s (b)) and HDR data (h v (b) A scene-based histogram for the scene can be generated. For example, in one embodiment,

number

[0063] Having both scene-level histograms, we can apply cumulative density function (CDF) matching (Patent Documents 4-5) to generate a forward mapping function (FLUT) from HDR to SDR.

number

number

number

number

[0064] Generating a scene-based backward reshaping function (410) involves both frame-level and scene-level operations. Since the luma mapping function is a single-channel predictor, we simply invert the forward reshaping function to obtain the backward reshaping function. For chroma, we use the reshaped SDR data (407) and the original HDR data (404) to form a 3DMT representation and compute a new frame-based a / B representation as follows:

number

[0065] At the scene level, for luma, the histogram weighted BLUT construction of US Pat. No. 6,233,636 may be applied to generate the backward luma reshaping function. For chroma, the frame-based a / B representations can again be averaged to compute the scene-based a / B representation.

number

number

number

[0066] Exemplary Computer System Implementation Embodiments of the invention may be implemented using computer systems, systems comprised of electronic circuits and components, integrated circuit (IC) devices, such as microcontrollers, field programmable gate arrays (FPGAs) or other configurable or programmable logic devices (PLDs), discrete time or digital signal processors (DSPs), application specific ICs (ASICs), and / or apparatuses including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or execute instructions related to workload distribution and node-based processing in cloud-based video encoding of HDR videos as described herein. The computer and / or IC may calculate any of a variety of parameters or values ​​related to workload distribution and node-based processing in cloud-based video encoding of HDR videos as described herein. Image and video dynamic range enhancement embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0067] Certain implementations of the invention include a computer processor executing software instructions that cause the processor to perform the methods of the invention. For example, one or more processors in a display, encoder, set-top box, transcoder, etc., may implement the methods for workload distribution and node-based processing in cloud-based video encoding of HDR video as described above by executing software instructions in a program memory accessible to the processor. The invention may be provided in the form of a program product. The program product may include any non-transitory, tangible medium carrying a set of computer-readable signals that include instructions that, when executed by a data processor, cause the data processor to perform the methods of the invention. A program product according to the invention may be in any of a wide variety of non-transitory, tangible forms. The program product may include a physical medium, such as, for example, magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD-ROMs, DVDs, ROMs, electronic data storage media including flash RAMs, etc. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0068] Where a component (e.g., a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, references to the component (including references to "means") should be interpreted as including any components that perform the function of the described component (e.g., are functionally equivalent) as equivalents of that component, including components that are not structurally equivalent to the disclosed structures that perform that function in the illustrated exemplary embodiments of the invention.

[0069] Equivalents, Extensions, Substitutes and Others Enumerated example embodiments (EEE) of the present invention are defined as follows, but are not limited to the following. [EEE1] 1. A method for assigning a sequence of scenes to segments to be encoded by one or more computational nodes, the method comprising: receiving a sequence of scenes, each scene including one or more video frames; and performing one or more allocation iterations to generate an optimal output allocation, said performing one or more allocation iterations comprising: For an iteration in the one or more allocation iterations: generating an initial random assignment (305) of said sequence of scenes to M computational nodes based on a random seed selection for the assignment iterations, where M>1; performing a refinement allocation step (310) based on the initial random allocation to generate a refined allocation of the sequence of scenes to the M computational nodes and a refined allocation cost; updating a best allocation cost and the best output allocation (315) based on the refined allocation and the refined allocation cost. method. [EEE2] Executing the refinement allocation step (310) includes: Initialize a total allocation cost using a first value; for each computing node, setting a node workload according to the initial random assignment of the sequence of scenes to the M computing nodes; Until convergence: Starting with the first node, for each node in turn: removing the scene from the workload of the node and assigning the scene to the workload of the next available node, and calculating a first cost metric for the M compute nodes; adding the scene taken from the workload of the previously available node to the workload of the node and calculating a second cost metric for the M compute nodes; Calculating a third cost metric for the M compute nodes while keeping the workload of that node unchanged; generating an updated node workload based on a minimum of the first cost metric, the second cost metric, and the third cost metric (360); Calculate an iteration allocation cost based on the updated node workload; If the total allocation cost is less than the iteration allocation cost: signal convergence, output the updated node workload as the refined allocation, and output the total allocation cost as the refined allocation cost; else: continue by replacing the total allocation cost with the iteration allocation cost. The method according to claim 1, further comprising: [EEE3] The method of EEE2, wherein the first value comprises an estimate of the maximum possible standard deviation of the total number of frames allocated to each node. [EEE4] generating said initial random assignment comprising: generating a candidate set with scene indices ranging from 1 to K−1, where K indicates a total number of scenes in the sequence of scenes to be assigned to the M computational nodes; generating an assignment set with an initial element of 0; updating the allocation set according to a random selection using the random seed selection to generate an updated allocation set; sorting the updated assignment set in ascending order to generate a sorted assignment set; generating the initial random assignment according to the sorted set of assignments; Updating the assignment set includes: For t=1 to M-1: Choose a random integer p between 0 and Kt-1; Identifying the pth element in the candidate set and appending it to the assignment set; remove the pth element in the candidate set; sorting the candidate set in ascending order; The method according to any one of EEE1 to 3. [EEE5] generating the initial random assignments according to the sorted set of assignments: assigning to node m all scenes having an index between the mth element in the sorted assignment set and the m+1th element in the sorted assignment set. The method described in EEE4. [EEE6] Calculate a cost metric for every compute node based on the node assignments from the scene for each compute node: For each compute node, calculate the total number of frames assigned to that compute node based on the node assignment from the scene; calculating a standard deviation of the total number of frames assigned to each computing node; The method according to any one of EEE1 to 5. [EEE7] Removing a scene from the workload of the node and allocating the scene to the workload of the next available node comprises: identifying the last scene assigned to the node and assigning it as the first scene in the workload of the next available node; The method according to any one of EEE1 to 6. [EEE8] Adding a scene taken from the workload of a previously available node to the workload of that node: identifying the last scene assigned to the previously available node and assigning it as an initial scene to the node's workload; The method according to any one of EEE1 to 7. [EEE9] To update the best allocation cost and the best output allocation: In a first allocation iteration, setting the refined allocation as a best output allocation and setting the refined allocation cost as a best allocation cost; a subsequent allocation iteration step includes comparing the refined allocation cost with a best allocation cost; if the best allocation cost is greater than the refined allocation cost, selecting the refined allocation as a best output allocation; and selecting the refined allocation cost as the best allocation cost. The method according to any one of EEE1 to EEE8. [EEE10] For a node among the M compute nodes: accessing a sequence of high dynamic range (HDR) frames and a corresponding sequence of standard dynamic range (SDR) frames for a scene assigned to the node according to a best output assignment to that node of said sequence of scenes; generating an output bitstream for the scene assigned to that node. The method according to any one of EEE1 to 9. [EEE11] Generating the output bitstream further comprises: generating a scene-based forward reshaping function based on said sequence of HDR frames and said sequence of SDR frames; mapping the sequence of HDR frames to a sequence of reshaped SDR frames based on the scene-based forward reshaping function; generating an encoded bitstream by compressing said sequence of reshaped SDR frames; generating a scene-based backward reshaping function based on the sequence of reshaped SDR frames, the sequence of HDR frames, and the scene-based forward reshaping function; generating metadata based on parameters of said scene-based backward reshaping function; outputting the output bitstream including the encoded bitstream and the metadata. The method described in EEE10. [EEE12] A non-transitory computer-readable storage medium storing computer-executable instructions for performing the method of any one of EEE1 to EEE11 on one or more processors. [EEE13] 12. An apparatus having a processor and configured to perform any of the methods described in EEE1 to EEE11.

[0070] Thus, exemplary embodiments relating to workload allocation and node-based processing in cloud-based video encoding of HDR videos have been described. In the above specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indication of what is the invention and what is intended by the applicant to be the invention is the set of claims issued on this application in the particular form in which the claims are permitted, including any subsequent amendments. The definitions expressly set forth herein for terms contained in such claims govern the meaning of the terms used in the claims. Thus, no limitations, elements, properties, features, advantages or attributes not expressly set forth in the claims should in any way limit the scope of such claims. Thus, the specification and drawings are to be regarded in an illustrative and not restrictive sense.

Claims

1. 1. A method for allocating scenes of a sequence of scenes to be encoded by a plurality of computational nodes, the method comprising: receiving a sequence of scenes, each scene including one or more video frames; and performing one or more allocation iterations to generate a best output allocation in which each scene of the sequence of scenes is scheduled for encoding on a particular one of M computational nodes, where M>1, and each allocation iteration of the one or more allocation iterations comprises: generating, for each computational node, an initial random assignment (305) of one or more scenes of the sequence of scenes to its respective computational node based on a random seed selection for that assignment iteration; for each computing node, performing a refinement allocation step (310) based on the initial random allocation to generate a refined allocation of the one or more scenes of the sequence of scenes to its respective computing node and a refined allocation cost, the refined allocation cost reflecting the workload imposed on the respective computing node when encoding the assigned scene; and updating the best allocation cost and the best power allocation (315) based on the refined allocation such that uniformity of workload distribution across the plurality of compute nodes is maximized. method.

2. The method of claim 1 , wherein maximizing a uniformity of workload distribution across the plurality of compute nodes comprises minimizing the refined allocation cost for the M compute nodes.

3. Performing the refinement allocation step (310) comprises: initializing a total allocation cost using a first value; for each computing node m, set a node workload according to the initial random assignment of the sequence of scenes to the M computing nodes; Until convergence: For each compute node m, starting from compute node m=0 until we reach compute node m=M-1, for only compute node m<M-1, removing the scene from the workload of that node and assigning the scene to the workload of compute node m+1, and calculating a first cost metric for the M compute nodes; for only compute node m>0, add the scene taken from the workload of compute node m-1 to the workload of that node and calculate a second cost metric for the M compute nodes; keeping the workload of the nodes unchanged and calculating a third cost metric for the M compute nodes; generating (360) an updated node workload based on a minimum of the first cost metric, the second cost metric, and the third cost metric; Calculate an iteration assignment cost based on the updated node workload; If the total allocation cost is less than the iteration allocation cost: signal convergence and output the updated node workload as the refined allocation and output the total allocation cost as the refined allocation cost; else: continue by replacing the total allocation cost with the iteration allocation cost. This includes repeating The method according to claim 1 or 2.

4. Generating the initial random assignment comprises: generating a candidate set with scene indices ranging from 1 to K−1, where K indicates a total number of scenes in the sequence of scenes to be assigned to the M computational nodes; generating an assignment set having an initial element 0; updating the allocation set according to a random selection using the random seed selection to generate an updated allocation set; sorting the updated assignment set in ascending order to generate a sorted assignment set; generating the initial random assignment according to the sorted set of assignments; Updating the assignment set includes: For t=1 to M-1: Select a random integer p between 0 and K-t-1; Identifying the pth element in the candidate set and appending it to the assignment set; remove the pth element in the candidate set; sorting the candidate set in ascending order; 4. The method according to claim 1 , wherein the first and second components are selected from the group consisting of fluorouracil, fluorophenyl ether ...

5. Generating the initial random assignments according to the sorted set of assignments: assigning to computation node m all scenes having an index between the mth element in the sorted assignment set and the m+1th element in the sorted assignment set; The method according to claim 4.

6. Calculate a cost metric for all compute nodes based on the assignment of nodes from the scene to each compute node: for each computing node, calculate a total number of frames from the scene that are assigned to that computing node based on the node's assignment; calculating a standard deviation of the total number of frames assigned to each computing node; 6. The method according to any one of claims 1 to 5.

7. Removing a scene from the workload of said node and allocating the scene to the workload of compute node m+1 comprises: identifying the last scene scheduled for encoding on compute node m and assigning it as the first scene to be scheduled for encoding on compute node m+1; The method according to claim 3.

8. Adding a scene taken from the workload of compute node m-1 to its workload: identifying the last scene scheduled for encoding on compute node m-1 and assigning it as the first scene to be scheduled for encoding on compute node m; 8. The method according to any one of claims 1 to 7.

9. Updating the best allocation cost and the best power allocation: In a first allocation iteration, setting the refined allocation as a best output allocation and setting the refined allocation cost as a best allocation cost; a subsequent allocation iteration step includes comparing the refined allocation cost with a best allocation cost; if the best allocation cost is greater than the refined allocation cost, selecting the refined allocation as the best output allocation, and selecting the refined allocation cost as the best allocation cost.

9. The method according to any one of claims 1 to 8.

10. For a given computing node among the M computing nodes: accessing a sequence of high dynamic range (HDR) frames and a corresponding sequence of standard dynamic range (SDR) frames for a scene assigned to the computing node according to a best output assignment of the sequence of scenes to the computing node; generating an output bitstream for the scene assigned to that computational node.

10. The method according to any one of claims 1 to 9.

11. Generating the output bitstream further comprises: generating a scene-based forward reshaping function based on the sequence of HDR frames and the sequence of SDR frames; mapping the sequence of HDR frames to a sequence of reshaped SDR frames based on the scene-based forward reshaping function; generating an encoded bitstream by compressing said sequence of reshaped SDR frames; generating a scene-based backward reshaping function based on the sequence of reshaped SDR frames, the sequence of HDR frames, and the scene-based forward reshaping function; generating metadata based on parameters of the scene-based backward reshaping function; outputting the output bitstream including the encoded bitstream and the metadata. The method of claim 10.

12. A non-transitory computer readable storage medium storing computer executable instructions for performing the method of any one of claims 1 to 11 on one or more processors.

13. Apparatus comprising a processor and configured to perform any of the methods according to claims 1 to 11.

Citation Information

Patent Citations

  • Method and device for encoding image

    JP2002199392A

  • System and method for time-optimized encoding

    JP2010529809A

  • US10,264,287、B

  • US10,397,576、H

  • US10,575,028、H