Simulation-based evaluation technique for coding ladders used in video streaming

Abstract

In various embodiments, the simulation evaluation application generates a first streaming header based on multiple stages of a first candidate encoding ladder, each stage specifying a different encoded video resolution and bitrate. The simulation evaluation application then executes an adaptive bitrate algorithm on the first streaming header based on the network throughput trace to determine a first value for an indicator related to quality of experience. The simulation evaluation application also generates a second streaming header based on a second candidate encoding ladder. The simulation evaluation application then executes an adaptive bitrate algorithm on the second streaming header based on the network throughput trace to determine a second value for the indicator. The simulation evaluation application then compares the first value with the second value to determine that the first candidate encoding ladder should be used to stream the media title instead of the second candidate encoding ladder.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Patent Application No. 18 / 154,709, filed January 13, 2023, the contents of which are incorporated herein by reference. [Technical Field]

[0002] Various embodiments of the present disclosure relate generally to computer science and video streaming technologies, and more particularly to simulation-based evaluation techniques for coding ladders used in video streaming. [Background technology]

[0003] A typical video streaming service provides users with access to a library of media titles that can be viewed on a variety of client devices. During operation, a given client device connects to the video streaming service under a variety of connection conditions, making it susceptible to differences in network throughput. To ensure that a given media title can be streamed to a client device without interruption, the video streaming service generates an encoding ladder for that media title. Each rung of the encoding ladder specifies a different encoded video for that media title, a resolution for that encoded video, and a bitrate for that encoded video. Note that if the network throughput exceeds the bitrate of a given encoded video, that encoded video can be streamed to the client device without interruption. Furthermore, the higher the bitrate of the encoded video for a media title streamed to a client device, the better the image quality of the media title presented to the client device typically becomes.

[0004] In practice, the encoded videos specified in multiple encoding ladders corresponding to a media title library are typically delivered to client devices via a content delivery network (CDN) with limited storage resources. Therefore, generating an encoding ladder for a media title typically requires a trade-off between the streaming QoE associated with the encoding ladder and the storage footprint of the encoding ladder. As used herein, the "streaming Quality of Experience (QoE)" associated with a media title's encoding ladder refers to the overall QoE a viewer will experience when the media title is streamed to a client device using the encoding ladder. In practice, the streaming QoE typically correlates with the overall visual quality (image quality) of the media title as it is streamed and presented to a client device. Furthermore, the "storage footprint" of an encoding ladder refers to the total size of the encoded videos specified in the encoding ladder.

[0005] One method for generating an encoding ladder for a media title is to incrementally build the encoding ladder based on multiple heuristics corresponding to a set of ladder constraints. These ladder constraints are designed as a set of constraints to ensure a desired streaming QoE even when the media title is transmitted to various client devices over networks with varying throughput. To generate a given encoding ladder for a media title, multiple different encoded videos are created based on the source video for the media title, reflecting various combinations of resolution and bitrate. The encoding ladder starts empty, and a heuristic corresponding to each ladder constraint is applied sequentially to the different encoded videos. Each heuristic determines whether the encoding ladder already meets the corresponding ladder constraint. If the encoding ladder already meets the ladder constraint, the heuristic leaves the encoding ladder unchanged. If not, the heuristic adds at least one encoded video to the encoding ladder to make it conform to the ladder constraint.

[0006] However, this approach had drawbacks. Because ladder constraints were applied one by one and coded videos were not removed from the coding ladder, the relationship between streaming QoE and storage footprint represented in the coding ladder might not be optimal. More specifically, because ladder constraints were applied sequentially, the opportunity to consciously select a single coded video that could simultaneously satisfy multiple ladder constraints and achieve a better tradeoff between streaming QoE and storage footprint was lost. This configuration could result in unnecessarily low streaming QoE for a media title, i.e., the average video quality of the media title obtained using coded videos specified in the coding ladder was too low relative to the storage footprint of the coding ladder. Conversely, it could result in unnecessarily large storage footprint for a coding ladder, i.e., the storage footprint of the coding ladder was too large relative to the average video quality of the media title obtained using coded videos specified in the coding ladder. However, given these challenges, opportunities exist to generate improved coding ladders for media titles by using a single coded video to simultaneously satisfy multiple ladder constraints, thereby enabling more efficient use of CDN storage resources. The resulting improved coding ladders can provide the same or better streaming QoE while maintaining a smaller storage footprint. Summary of the Invention [Problem to be solved by the invention]

[0007] As can be seen from the above, there is a need in the art for more effective techniques for generating coding ladders for use in video streaming. [Means for solving the problem]

[0008] In one embodiment, a computer-implemented method for evaluating candidate encoding ladders for use in streaming a media title is disclosed, the method including: generating a first streaming header based on a first set of stages associated with a first candidate encoding ladder, wherein each stage in the first set of stages specifies a resolution and bitrate of a different encoded video in a set of encoded videos; running an adaptive bitrate algorithm on the first streaming header based on a first network throughput trace to determine a first indicator value for a first indicator related to quality of experience; generating a second streaming header based on a second set of stages associated with a second candidate encoding ladder; running the adaptive bitrate algorithm on the second streaming header based on the first network throughput trace to determine a second indicator value for the first indicator; and determining that the first candidate encoding ladder should be used to stream the media title rather than the second candidate encoding ladder by comparing the first indicator value and the second indicator value.

[0009] At least one technical advantage of the technology of the present disclosure over the prior art is that the technology of the present disclosure simultaneously considers multiple different ladder constraints from the beginning when generating an encoding ladder, thereby enabling the generation of an encoding ladder with the overall goal of increasing the quality level associated with the coded videos included in the encoding ladder while minimizing the storage footprint of the encoding ladder. This approach enables the generation of an encoding ladder to identify and exploit opportunities for using a single coded video that simultaneously satisfies multiple different ladder constraints, thereby improving the trade-off between the weighted average quality level associated with the coded videos included in the encoding ladder and the storage footprint of the encoding ladder. As a result, the trade-off between the streaming quality experience provided by the encoding ladder used for a given media title and the storage footprint of the encoding ladder can be significantly improved compared to the prior art. These technical advantages result in one or more technical improvements over prior art approaches. [Brief explanation of the drawings]

[0010] Having briefly summarized the inventive concept above, in order to provide a detailed understanding of the features of the various embodiments described above, the inventive concept will now be described more particularly with reference to various embodiments, some of which are also illustrated in the accompanying drawings. It should be noted that the accompanying drawings merely illustrate exemplary embodiments of the inventive concept, and therefore should not be construed as limiting the scope of the present disclosure in any way, and that there may be other embodiments that have equivalent effects. [Figure 1] FIG. 1 is a conceptual diagram of a system configured to implement one or more aspects of various embodiments. [Figure 2] FIG. 2 illustrates a more detailed view of the coding ladder application shown in FIG. 1, in accordance with various embodiments. [Figure 3] FIG. 3 illustrates one of the tier allocation engines shown in FIG. 2 in more detail, according to various embodiments. [Figure 4] FIG. 2 illustrates a more detailed view of the simulation evaluation application shown in FIG. 1, according to various embodiments. [Figure 5] FIG. 1 is a flow diagram illustrating method steps for generating candidate encoding ladders for use in streaming a media title, according to various embodiments. [Figure 6] FIG. 1 is a flow diagram illustrating method steps for determining an encoding ladder for a media title based on the results of performing adaptive streaming simulations on candidate encoding ladders generated for the media title, according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0011] In the following description, numerous specific details are set forth to provide a thorough understanding of various embodiments. However, it will be apparent to one of ordinary skill in the art that the concepts of the present invention may be practiced even if one or more of these specific details are omitted. For convenience of explanation, multiple instances of similar objects are represented by a reference number that identifies the object and, where necessary, alphanumeric characters in parentheses that identify the instance.

[0012] A video streaming service generates an encoding ladder for a given media title to ensure uninterrupted streaming of the media title to a client device. Each rung of the encoding ladder specifies a different encoded video for the media title, a resolution for the encoded video, and a bitrate for the encoded video. If available network bandwidth exceeds the bitrate for a given encoded video, the encoded video can be streamed to a client device without interruption. Additionally, the higher the bitrate of the encoded video for a media title streamed to a client device, the better the image quality of the media title presented to the client device typically becomes.

[0013] In practice, encoded video specified in multiple encoding ladders associated with a media title library is typically split into multiple encoded chunks, and these encoded chunks are delivered to a client device via a CDN. To play a particular media title, the client device executes an endpoint application. The endpoint application often implements an adaptive bitrate algorithm to select an encoded video from among the various encoded videos specified in the encoding ladder for that media title based on network throughput and, if necessary, the associated screen resolution. The endpoint application then sends a sequence of requests for the selected encoded video chunks to an edge server device within the CDN that is relatively close to the client device. The endpoint application then decodes the various received encoded chunks, upscales them as needed, and generates reconstructed chunks at the same resolution as the associated screen. The endpoint application then plays the media title on the client device by playing back these various reconstructed chunks.

[0014] However, because CDNs have limited storage resources, generating an encoding ladder for a media title typically requires a trade-off between the streaming QoE associated with that encoding ladder and the storage footprint of that encoding ladder. As used herein, the "streaming QoE" associated with an encoding ladder refers to the average QoE a viewer would experience if the encoded media title were streamed to a client device. Streaming QoE reflects both the quality level associated with the encoded chunks used for streaming and the impact of playback interruptions, such as re-buffering, on the overall quality of the viewing experience. The "storage footprint" of an encoding ladder refers to the total size of the encoded video specified in that encoding ladder.

[0015] One approach to generating an encoding ladder for a media title is to incrementally build the encoding ladder based on multiple heuristics corresponding to a set of ladder constraints. The ladder constraints are designed to ensure a desired streaming QoE even when the media title is transmitted to a wide range of client devices over networks with variable and varying throughput, allowing the media title to be viewed on screens with various resolutions. To generate the encoding ladder, different pre-encoded videos are created based on the media title's source video, reflecting various combinations of resolution and bitrate. The encoding ladder starts empty, and heuristics corresponding to each ladder constraint are applied to it in turn. If the encoding ladder already meets the ladder constraint, the heuristic leaves the encoding ladder unchanged. If not, the heuristic adds at least one pre-encoded video to the encoding ladder to make it conform to the ladder constraint.

[0016] However, this approach had drawbacks. Because ladder constraints were applied one by one and no coded videos were removed from the coding ladder, the relationship between streaming QoE and storage footprint represented in the coding ladder might not be optimal. In particular, because ladder constraints were applied sequentially, the opportunity to consciously select a single coded video that could simultaneously satisfy multiple ladder constraints and achieve a better tradeoff between streaming QoE and storage footprint was lost. This configuration resulted in the storage footprint of the coding ladder being unnecessarily large compared to the average quality of the media titles obtained using the coded videos specified in the coding ladder, resulting in wasted CDN storage resources.

[0017] In contrast, in accordance with the techniques of this disclosure, an encoding ladder application generates one or more candidate encoding ladders for a media title based on the explicit goals of improving streaming QoE and reducing one or more costs (e.g., storage footprint, etc.) while satisfying ladder constraints. In some embodiments, an encoding ladder workflow executes the encoding ladder application to generate multiple optimized candidate ladders. The encoding ladder workflow then determines a final encoding ladder for the media title using a numerical evaluation application and a simulation evaluation application.

[0018] The coding ladder application formulates the coding ladder generation problem as a parameterized constrained optimization problem for assigning bitrate-quality points to each rung of the candidate coding ladder. The coding ladder application determines the constants of this parameterized constrained optimization problem based on bitrate / quality points generated from source video associated with the media title. Each bitrate / quality point specifies a different encoded video derived from the source video and its corresponding resolution, bitrate, and image quality score. Note that because the bitrates and image quality scores associated with the encoded video are typically, but not always, on different magnitudes, the coding ladder application normalizes the bitrates and image quality scores of the bitrate / quality points as necessary to fall within the same range.

[0019] The coding ladder application defines the objective and constraints of a parameterized constrained optimization problem using a parameterized objective function and parameterized constraints. The parameterized objective function represents a weighted trade-off between a quality term that approximates the streaming QoE represented by a candidate coding ladder and a footprint term that is proportional to the storage footprint of the candidate coding ladder. The quality term is a weighted average of the normalized image quality scores for each step of the candidate coding ladder. The footprint term is the sum of the normalized bitrates for each step. The parameterized objective function includes the number of steps and weights. The parameterized constraints include both intrinsic logical constraints and operational logical constraints. The intrinsic logical constraints ensure the validity of the candidate coding ladder itself. The operational logical constraints reflect operational requirements and priorities related to client device capabilities, network capacity, CDNs, and human perception of image quality.

[0020] The coding ladder application generates multiple ladder configurations. Each ladder configuration has a different combination of the number of stages, weights, and various parameters (e.g., relative bit rate intervals) in the parameterized constraints. For each ladder configuration, the coding ladder application generates an objective function based on the parameterized objective function and also generates constraints associated with the objective function based on the parameterized constraints. The coding ladder application then uses a constrained optimization algorithm to solve each objective function under the constraints associated with the objective function, generating a different assignment matrix for each ladder configuration. Each assignment matrix specifies the bit rate / quality score assigned to each stage of the candidate coding ladder.

[0021] The numerical evaluation application uses statistical data derived from streaming session history (e.g., throughput distribution, bitrate demand distribution, etc.) to perform a numerical evaluation of the candidate coding ladders. In some embodiments, the numerical evaluation application filters out any number (including zero) of candidate coding ladders that provide a poor trade-off between streaming QoE and storage footprint.

[0022] In some other embodiments, the numerical evaluation application may estimate any number and / or type of streaming QoE metrics in any technically feasible manner. Examples of other streaming QoE metrics include time-weighted image quality, time-weighted bitrate, the percentage of image quality that falls within a predefined "high image quality," the percentage of image quality that falls within a predefined "low image quality," and rebuffering probability. In the same or other embodiments, the numerical evaluation application may filter out any number (including zero) of candidate coding ladders based on trade-offs among multiple dimensions, such as trade-offs between expected streaming QoE, storage footprint, and network bandwidth consumption.

[0023] The streaming evaluation application performs a simulation-based evaluation on the remaining coding ladder candidates (e.g., those not filtered out by the above-mentioned filtering). For the remaining coding ladder candidates, the streaming evaluation application generates a different composite streaming header for each candidate based on the coded videos specified in the coding ladder candidate. For each coded video specified in a given coding ladder candidate, the composite streaming header specifies the coded video, the resolution of the coded video, the bit rate of the coded video, the corresponding coded chunk sequence, the bit rate of each coded chunk, and the image quality score of each coded chunk.

[0024] The streaming rating application emulates the operation of the adaptive bitrate algorithm using an adaptive streaming simulator. Using each remaining encoding ladder candidate and its corresponding encoded chunk metadata, the application performs the emulation on multiple simulated streaming sessions defined by different streaming session traces. Each streaming session trace specifies the network throughput as a function of time for a different streaming session history. The result of each simulation is a request sequence for encoded chunks of the media title. For each request sequence, the streaming rating application calculates a different set of values ​​for a set of metrics related to streaming QoE. The streaming rating application selects one of the remaining encoding ladder candidates by performing any number and / or types of comparison operations between the value sets of the metrics. The streaming rating application then generates an encoding ladder for the media title based on the selected encoding ladder candidate.

[0025] At least one technical advantage of the disclosed technology over the prior art is that the disclosed technology enables the encoding ladder application to generate each encoding ladder candidate by explicitly optimizing the approximate tradeoff between streaming QoE and storage footprint while simultaneously satisfying multiple ladder constraints. Thus, unlike the prior art, the encoding ladder application automatically identifies and exploits opportunities to employ a single encoded video that simultaneously satisfies multiple different ladder constraints and achieves a better tradeoff between streaming QoE and storage footprint. As a result, the encoding ladder candidate used for a given media title can exhibit a significantly improved tradeoff between streaming QoE and storage footprint compared to the prior art. Another advantage of the disclosed technology is that the simulation evaluation application efficiently compares the performance of a much larger number of encoding ladder candidates, requiring less time, processing resources, and network resources than deploying and evaluating encoding ladder candidates on a real network. These technical advantages result in one or more technical improvements over prior art approaches.

[0026] 1 is a conceptual diagram of a system 100 configured to implement one or more aspects of various embodiments. For ease of explanation, multiple instances or versions of similar objects are indicated by a reference number identifying the object and, where necessary, an alphanumeric character in parentheses identifying the instance or version. As shown, in some embodiments, system 100 includes, but is not limited to, compute instance 110(1), compute instance 110(2), streaming session history database 104, and adaptive streaming simulator 106.

[0027] In some other embodiments, compute instance 110(1), compute instance 110(2), streaming session history database 104, adaptive streaming simulator 106, or any combination thereof, may be omitted from system 100. In the same or other embodiments, system 100 may include, without limitation, one or more other compute instances, one or more other streaming session history databases, or any combination thereof. These components of system 100 may be distributed across any number of common geographic locations and / or any number of different geographic locations, and / or may be implemented in any combination in one or more cloud computing environments (i.e., encapsulated shared resources, software, data, etc.).

[0028] As shown, compute instance 110(1) includes, but is not limited to, processor 112(1) and memory 116(1), and compute instance 110(2) includes, but is not limited to, processor 112(2) and memory 116(2). Compute instance 110(1) and compute instance 110(2) may be referred to herein individually as "compute instance 110" or collectively as "compute instances 110." Processor 112(1) and processor 112(2) may be referred to herein individually as "processor 112" or collectively as "processors 112." Additionally, memory 116(1) and memory 116(2) may be individually referred to herein as "memory 116" or collectively referred to herein as "memories 116." Each compute instance (e.g., compute instance 110) may be implemented in a cloud computing environment, as part of another distributed computing environment, or standalone.

[0029] The processor 112 may be any instruction execution system, apparatus, or device capable of executing instructions. For example, the processor 112 may comprise a central processing unit, a graphics processing unit, a controller, a microcontroller, a state machine, or any combination thereof. The memory 116 of the compute instance 110 stores content such as software applications and data used by the processor 112 of the compute instance 110. The memory 116 may be one or more local or remote digital storage devices and may take the form of readily available memory (e.g., random access memory, read-only memory, floppy disk, hard disk), etc.

[0030] In other embodiments, any number of compute instances may include any combination of any number of processors and any number of memories. In particular, compute instance 110(1), compute instance 110(2), any number of other compute instances, or any combination thereof, may provide a multiprocessing environment in any manner that is technically feasible.

[0031] In some embodiments, storage (not shown) may supplement or replace the memory 116 of the compute instance 110. The storage may comprise any number and type of external memory accessible to the processor 112 of the compute instance 110. For example, without limitation, the storage may include a secure digital card, external flash memory, portable compact disk read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0032] Generally, each compute instance (e.g., compute instance 110) is configured to implement one or more software applications. For purposes of illustration, each software application is illustrated as residing in the memory 116 of one compute instance and executing on the processor 112 of the same compute instance. However, in some embodiments, the functionality of each software application may be distributed across any number of other software applications, and these other software applications may be configured to reside in the memory of any number of compute instances and execute on the processors of any number of compute instances, in any combination. Additionally, the functionality of any number of software applications may be integrated into a single software application.

[0033] In particular, in some embodiments, production encoding pipeline 120 resides in memory 116(1) of compute instance 110(1) and executes on processor 112(1) of compute instance 110(1). As shown, production encoding pipeline 120 includes, but is not limited to, a shot-based encoding application 132 and a ladder unfolding application 190.

[0034] The shot-based coding application 132 divides the source video 102 into multiple shots (not shown). The source video 102 may include, but is not limited to, any amount and / or type of video content. Examples of video content include, but are not limited to, a feature film, an episode of a television program, or an entire music video or any portion thereof, to name just a few. Each shot typically consists of a sequence of frames with similar spatial and temporal characteristics that lasts for an uninterrupted period of time. In some embodiments, each shot is captured consecutively by a single camera or a virtual representation of a camera (e.g., in the case of computer-animated video). When combined, these shots provide uninterrupted, unoverlapping coverage of the entire source video 102.

[0035] The shot-based encoding application 132 downscales each shot to multiple different resolutions to generate a set of low-resolution shots. Then, the shot-based encoding application 132 encodes each shot and each low-resolution shot by applying different combinations of one or more parameter values ​​(value sets) for a parameter set consisting of one or more encoding parameters. This generates multiple encoded shots with different combinations of resolution and bit rate. Furthermore, the shot-based encoding application 132 calculates a bit rate and a quality score for each encoded shot.

[0036] In this specification, the bit rate of a coded frame sequence (e.g., a coded shot or a coded video) refers to the average bit rate over the entire coded frame sequence, and the quality score of a coded frame sequence refers to the quality score of a reconstructed frame sequence obtained from the coded frame sequence. Furthermore, the quality score of a reconstructed frame sequence refers to the average estimated image quality level over the entire reconstructed frame sequence.

[0037] The quality scores can be values ​​for any type of metric that can be correlated to image quality in any technically feasible way. In some embodiments, each quality score is a value for a image quality metric. Examples of image quality metric include, but are not limited to, peak signal-to-noise ratio (PSNR) and video multimethod assessment fusion (VMAF) metrics. VMAF metrics estimate human perception of video quality for reconstructed video content (e.g., reconstructed shots, reconstructed videos, etc.).

[0038] The shot-based encoding application 132 generates a convex hull (not shown) of bitrate / quality evaluation points for each resolution based on the encoded shots at each resolution. The shot-based encoding application 132 then aggregates the bitrate / quality evaluation points contained in each convex hull to obtain a union across different resolutions, and sets this as convex hull metadata 136. The convex hull metadata 136 facilitates the generation of an encoding ladder for the media title.

[0039] The shot-based coding application 132 divides the source video 102 into any number of source chunks. Each source chunk contains a shot sequence of one or more shots. Each source chunk defines a portion of a media title, and when streaming the media title, a client device requests each of these portions independently and sends the requested portion to the client device. When combined, these source chunks provide seamless, uninterrupted coverage of the entire source video 102.

[0040] Based on the source chunks and the coded shots, the shot-based coding application 132 identifies the coded chunks for each coded picture specified (via the corresponding coded picture ID) in at least one of the bitrate / quality evaluation points included in the convex hull metadata 136. Then, the shot-based coding application 132 calculates the bitrate and quality score for each coded chunk corresponding to each coded picture specified in the convex hull metadata 136, and generates coded chunk metadata 138.

[0041] The ladder expansion application 190 sends the encoded chunks specified in the media title's encoding ladder to a CDN (not shown) for subsequent distribution from any number of server devices in the CDN to any number of client devices (not shown). The ladder expansion application 190 also sends the media title's encoding ladder to a playback server (not shown) for subsequent client devices to request the encoded chunks from a nearby server in the CDN based on the encoding ladder and available network throughput, thereby enabling streaming of the media title.

[0042] As described above, in a conventional technique for generating an encoding ladder for a media title, the encoding ladder is sequentially constructed based on a plurality of coded videos having various resolution and bitrate combinations using a plurality of heuristics corresponding to a set of ladder constraints. The encoding ladder is started empty, and a heuristic corresponding to each ladder constraint is applied sequentially based on a different coded video. The heuristic determines whether the encoding ladder already meets the corresponding ladder constraint. If the encoding ladder already meets the ladder constraint, the heuristic leaves the encoding ladder unchanged. If not, the heuristic adds at least one coded video to the encoding ladder to make it meet the ladder constraint.

[0043] However, the above approach has a drawback: because ladder constraints are applied one by one and no coded videos are removed from the coding ladder, an opportunity may be missed to consciously select a single coded video that simultaneously satisfies multiple ladder constraints and thus achieves a better trade-off between streaming QoE and storage footprint as expressed in the coding ladder.However, unlike the above approach, which misses such an opportunity, an improved coding ladder can provide the same or better streaming QoE while keeping the storage footprint smaller, thereby enabling more efficient use of CDN storage resources.

[0044] Generating coding ladders using constrained optimization techniques To address the above challenges, system 100 includes, but is not limited to, encoding ladder workflow 140. Encoding ladder workflow 140 uses constrained optimization techniques to generate an encoding ladder 178 for media titles associated with source video 102. As shown, in some embodiments, encoding ladder 178 includes rungs 180(1), ..., 180(L), where L may be any positive integer. Rung 180(1) is assigned an encoded video ID 182(1), a resolution 184(1), and a bitrate 186(1). Encoded video ID 182(1) identifies an encoded video of source video 102. Resolution 184(1) and bitrate 186(1) are assigned the resolution and bitrate of the encoded video corresponding to encoded video ID 182(1).

[0045] Encoding ladder workflow 140 includes, but is not limited to, encoding ladder application 150, numerical evaluation application 160, and simulation evaluation application 170. As shown, in some embodiments, encoding ladder workflow 140, encoding ladder application 150, numerical evaluation application 160, and simulation evaluation application 170 are stored in memory 116(2) of compute instance 110(2) and execute on processor 112(2) of compute instance 110(2).

[0046] As shown, the encoding ladder application 150 generates a candidate encoding ladder set 158 ​​for a media title corresponding to the source video 102 based on the convex hull metadata 136. The candidate encoding ladder set 158 ​​includes one or more candidate encoding ladders (not shown in FIG. 1 ) for the media title. As used herein, a “candidate encoding ladder” for a media title refers to any encoding ladder that can be used as an encoding ladder for the media title. Based on the convex hull metadata 136, the encoding ladder application 150 formulates the encoding ladder generation problem as one or more distinct constrained optimization problems.

[0047] Each constrained optimization problem describes an overall objective of improving the streaming QoE of a coding ladder while minimizing the storage footprint of the coding ladder under an associated set of constraints. However, each constrained optimization problem assigns a different weight (relative importance) to the reduction in the storage footprint of the coding ladder candidate versus the QoE improvement associated with the coding ladder candidate and / or constraint set. The coding ladder application 150 independently solves each constrained optimization problem to generate different coding ladder candidates in the coding ladder candidate set 158. The coding ladder application 150 is described in more detail below in the discussion of FIG. 2.

[0048] The numerical evaluation application 160 performs any number and / or type of numerical evaluations on the coding ladder candidate set 158 ​​based on the streaming session history database 104. The streaming session history database 104 includes recorded data associated with any number of past streaming sessions and any amount (including zero) of data derived from the recorded data. Specifically, the numerical evaluation application 160 numerically evaluates each coding ladder candidate included in the coding ladder candidate set 158 ​​using the throughput distribution 122 and / or the bitrate demand distribution 124 derived from the recorded data associated with any number of past streaming sessions described in the streaming session history database 104. Furthermore, based on the results of the numerical evaluations, the numerical evaluation application 160 selects any number (including zero) of coding ladder candidates from the coding ladder candidate set 158 ​​that provide an unsatisfactory trade-off between predicted streaming QoE and storage footprint. The numerical evaluation application 160 then filters out (eg, removes) the selected coding ladder candidates from the coding ladder candidate set 158 ​​to generate a filtered coding ladder candidate set 168 .

[0049] As shown, in some embodiments, the simulation evaluation application 170 performs any number and / or types of simulation-based evaluations on the filtered encoding ladder candidate set 168 based on the encoded chunk metadata 138 and the streaming session history database 104. Then, for each encoding ladder candidate in the filtered encoding ladder candidate set 168, the simulation evaluation application 170 runs T different simulations using that encoding ladder candidate and streaming session traces 126(1), ..., 126(T), where T can be any positive integer.

[0050] For ease of explanation, streaming session traces 126(1), ..., streaming session traces 126(T) may be individually referred to herein as "streaming session traces 126" or collectively referred to herein as "streaming session traces 126." Each streaming session trace 126 includes recorded measurements of one or more network characteristics over a period of time or synthesized measurements of one or more network characteristics over a period of time.

[0051] In some embodiments, each streaming session trace 126 is a network throughput trace that indicates the throughput of the network as a function of time over the duration of the trace. In some embodiments, including the embodiment shown in FIG. 1, each streaming session trace 126 is a recorded network throughput trace that specifies recorded measurements of throughput in an actual network. However, in some alternative embodiments, each streaming session trace 126 is a synthetic network throughput trace generated by a software application (e.g., simulation evaluation application 170) in any technically feasible manner.

[0052] In some embodiments, to simulate a candidate coding ladder using a streaming session trace 126(t), the simulation evaluation application 170 sends the candidate coding ladder and the streaming session trace 126(t) to the adaptive streaming simulator 106, where t can be any integer between 1 and T. The adaptive streaming simulator 106 then runs an adaptive bitrate (ABR) algorithm based on the candidate coding ladder for the simulated streaming session defined by the streaming session trace 126(t). For the simulated streaming session, the ABR algorithm attempts to select a sequence of encoded chunks with the highest possible bitrate without exceeding the available network throughput.

[0053] For each simulation, the simulation evaluation application 170 calculates a combination of values ​​(value sets) for a set of metrics, referred to herein as a “streaming evaluation metric set.” The simulation evaluation application 170 then performs any number and / or types of evaluations and / or comparisons among these value sets to select any number of the candidate coding ladders associated with these value sets for further evaluation and deployment through the production coding pipeline 120.

[0054] In some embodiments, including the embodiment shown in Figure 1, the simulation evaluation application 170 selects a candidate coding ladder to deploy through the production encoding pipeline 120. The simulation evaluation application 170 then generates a coding ladder 178 based on the selected candidate coding ladder and sends the generated coding ladder 178 to a ladder deployment application 190 included in the production encoding pipeline 120. The simulation evaluation application 170 is described in more detail below in the discussion of Figure 4.

[0055] The encoding ladder application 150 formulates the problem of generating an encoding ladder as a constrained optimization problem. Therefore, the encoding ladder application 150 has the advantage of simultaneously considering various ladder constraints when generating encoding ladder candidates for inclusion in the encoding ladder candidate set 158. As a result, the encoding ladder application 150 can automatically identify and exploit opportunities to employ a single encoded video that simultaneously satisfies multiple different ladder constraints when generating each encoding ladder candidate. This enables the encoding ladder application 150 to generate encoding ladder candidates for media titles that provide comparable or better streaming QoE while maintaining a smaller storage footprint than encoding ladders for media titles generated using prior art techniques. Furthermore, unlike prior art techniques, each objective function explicitly describes the trade-off between streaming QoE and storage footprint. Therefore, the encoding ladder application 150 can generate encoding ladder candidates for media titles that provide a higher streaming QoE while maintaining a comparable or smaller storage footprint than encoding ladders for media titles generated using prior art techniques.

[0056] As those skilled in the art will recognize, a typical prior art streaming QoE comparison technique involves A / B testing multiple different encoding ladders for a media title under various network conditions and comparing the resulting streaming QoEs. However, A / B testing is time-consuming and consumes significant processing and network resources, so the number of encoding ladders for a media title that can be compared through A / B testing is typically limited to a relatively small number. In contrast, the simulation evaluation application 170 has the advantage of being able to efficiently compare the performance of a much larger number of candidate encoding ladders and / or encoding ladders using a much larger number of network throughput traces while requiring less time, processing resources, and network resources. Thus, the simulation evaluation application 170 can evaluate a wider and more efficient range of possible encoding ladders than conventional techniques, thereby generating encoding ladders with an improved trade-off between streaming QoE and storage footprint.

[0057] It should be noted that the various techniques described herein are illustrative only and not limiting. That is, variations and modifications to these techniques are possible without departing from the broader spirit and scope of the present invention. Those skilled in the art will recognize that many modifications and variations can be made to the functionality provided by the encoding ladder application 150, numerical evaluation application 160, simulation evaluation application 170, encoding ladder workflow 140, shot-based encoding application 132, ladder development application 190, production encoding pipeline 120, and adaptive streaming simulator 106 without departing from the scope and spirit of the embodiments described herein. In some embodiments, the inventive concepts described herein in the context of the encoding ladder application 150 can be implemented without any of the other inventive concepts described herein. In some embodiments, the inventive concepts described herein in the context of the simulation evaluation application 170 can be implemented without any of the other inventive concepts described herein. In the same or other embodiments, the simulation evaluation application 170 can use one or more ABR algorithms to evaluate any number of candidate encoding ladders for use in streaming a media title associated with the source video 102.

[0058] In some embodiments, the coding ladder application 150 can incorporate any number and / or type of objective functions. Each objective function is configured to maximize streaming QoE while minimizing one or more cost terms. As used herein, a "cost term" refers to a portion of the objective function that is targeted for reduction when solving a constrained optimization problem and is associated with (e.g., calculated based on) any number and / or type of costs. In some embodiments, the footprint term 324 corresponds to the cost term of the objective function 340. Additionally, a "cost" can refer to any characteristic, metric, etc. associated with transmitting coded video to a client device over any number and / or type of network connections. Two examples of costs are the storage footprint of the coding ladder and network bandwidth consumption.

[0059] In some alternative embodiments, any amount (including zero or all) of the convex hull metadata 136 and / or the encoded chunk metadata 138 can be derived from the encoded video, and any remaining amount (including zero or all) of the convex hull metadata 136 and / or the encoded chunk metadata 138 can be estimated for a “virtual” encoded video. Note that if this embodiment is used, the technical content described herein can be modified accordingly. As used herein, metadata estimated for a “virtual encoded video” refers to metadata estimated for encoded video that could potentially be generated based on the source video 102. Any amount and / or type of metadata can be estimated for the virtual encoded video in any technically feasible manner. For example, in some embodiments, the production encoding pipeline 120 and / or the encoding ladder application 150 estimates the metadata for the virtual encoded video based on a curve. Also, in the same or other embodiments, the production encoding pipeline 120 and / or the encoding ladder application 150 may perform any number and / or types of extrapolation and / or interpolation operations on metadata associated with one or more encoded versions of the source video 102 (encodings of the source video 102) to estimate metadata for hypothetical encoded video that could potentially be generated from the source video 102. For convenience of description, "encoded video metadata" herein may refer to either metadata derived from metadata for an encoded version of a video or metadata estimated as hypothetical encoded video metadata. For example, "encoded video bitrate and quality score" may refer to estimated values ​​for hypothetical encoded video bitrate and quality score based on the source video 102 and / or zero or more encoded versions of the source video 102.

[0060] Those skilled in the art will appreciate that many modifications and variations can be made to the arrangement, amount, and / or type of data described herein without departing from the scope and spirit of the embodiments described herein. For example, in some embodiments, each rung of the encoding ladder 178 may be specified with a quality score in addition to the bit rate and resolution of the encoded video, which is specified via the encoded video ID.

[0061] It will be appreciated that the system 100 illustrated herein is illustrative and that variations and modifications are possible. For example, the topology between the various components illustrated in FIG. 1 may be modified as desired. For example, in some embodiments, one or more of the production encoding pipeline 120, the numerical evaluation application 160, the simulation evaluation application 170, or the encoding ladder workflow 140 may be omitted from the system 100, and the encoding ladder application 150 may generate and / or obtain the encoded chunk metadata 138 in any technically feasible manner.

[0062] 2 is a more detailed diagram of the encoding ladder application 150 shown in FIG. 1 , according to various embodiments. As described above with respect to FIG. 1 , the encoding ladder application 150 generates an encoding ladder candidate set 158 ​​based on the convex hull metadata 136. The encoding ladder candidate set 158 ​​includes encoding ladder candidate 280(1), ..., encoding ladder candidate 280(C), where C may be any positive integer. Any of the encoding ladder candidate 280(1), ..., encoding ladder candidate 280(C) is a candidate that can be used as an encoding ladder for the media title associated with the convex hull metadata 136.

[0063] As shown, the convex hull metadata 136 includes, but is not limited to, bitrate / quality evaluation points 210(1), ..., bitrate / quality evaluation points 210(M), where M can be any positive integer. As discussed above with respect to FIG. 1 , each of the bitrate / quality evaluation points 210(1), ..., bitrate / quality evaluation points 210(M) corresponds to a different coded video generated based on multiple shots included in the source video 102. For ease of explanation, the bitrate / quality evaluation points 210(1), ..., bitrate / quality evaluation points 210(M) may be individually referred to as "bitrate / quality points 210" or collectively referred to as "bitrate / quality points 210."

[0064] As shown, bitrate / quality evaluation point 210(1) includes, but is not limited to, coded image ID 212(1), resolution 214(1), bitrate 216(1), and quality score 218(1). Coded image ID 212(1) is an ID that identifies coded image that has already been generated or can be generated based on shots included in source image 102. If coded image ID 212(1) identifies coded image that has already been generated, resolution 214(1), bitrate 216(1), and quality score 218(1) specify the resolution, bitrate, and quality score (e.g., VMAF index value) of the coded image. On the other hand, if the coded video ID 212(1) does not identify a coded video that has already been generated, the resolution 214(1), bit rate 216(1), and quality score 218(1) specify the resolution, estimated average bit rate, and estimated quality score of a coded video that can be generated based on the coded video ID 212(1). As shown in italics in the figure, in this specification, the resolution 214(1) may be represented as R1, the bit rate 216(1) as B1, and the quality score 218(1) as Q1.

[0065] Although not shown, if x is an integer between 2 and M, the bitrate / quality evaluation point 210(x) includes, but is not limited to, an encoded video ID 212(x), a resolution 214(x), a bitrate 216(x), and a quality score 218(x). The encoded video ID 212(x) is an ID that identifies an encoded video that has already been generated or can be generated based on shots included in the source video 102. If the encoded video ID 212(x) identifies an encoded video that has already been generated, the resolution 214(x), bitrate 216(x), and quality score 218(x) specify the resolution, average bitrate, and image quality score of the encoded video. On the other hand, if the coded video ID 212(x) does not identify a coded video that has already been generated, the resolution 214(x), bit rate 216(x), and quality score 218(x) specify the resolution, estimated average bit rate, and estimated quality score of a coded video that can be generated based on the coded video ID 212(x). For convenience of explanation, the resolution 214(x) will be referred to as R x , bit rate 216(x) x , quality score 218(x) x It can also be expressed as

[0066] In the coding ladder application 150, the problem of generating coding ladder candidates 280(1), ..., coding ladder candidate 280(C) is formulated as a parameterized constrained optimization problem. As used herein, a "parameterized" constrained optimization problem refers to a general form of a constrained optimization problem expressed using at least one parameter. Each parameter associated with a constrained optimization problem is represented by a symbol. Various constrained optimization problems can be formulated by applying various combinations of values ​​to the parameters associated with the parameterized constrained optimization problem. Meanwhile, when solving the formulated constrained optimization problem, the values ​​assigned to the parameters remain fixed. Note that, in this specification, the values ​​assigned to the parameters may be referred to as "parameter values." Techniques related to constrained optimization, constrained optimization problems, and methods for solving constrained optimization problems are well known in the art. For an overview, see https: / / en.wikipedia.org / wiki / Constrained_optimization

[0067] As shown, the coding ladder application 150 includes, but is not limited to, a normalization engine 220, coding point metadata 230, a parameterized objective function 240, a parameterized constraint set 250, a ladder configuration 260(1), ..., ladder configuration 260(C), a rung assignment engine 270(1), ..., rung assignment engine 270(C), coding ladder candidates 280(1), ..., coding ladder candidates 280(C), and a coding ladder candidate set 158.

[0068] It should be noted that, as one skilled in the art will recognize, the bitrates and quality scores associated with coded video are typically, but not always, on different scales. Thus, in some embodiments, to facilitate formulating the generation of the coding ladder as a parameterized, constrained optimization problem, normalization engine 220 normalizes bitrates 216(1), ..., bitrates 216(M) and quality scores 218(1), ..., quality scores 218(M) to fall into a common range.

[0069] More precisely, in some embodiments, normalization engine 220 performs a logarithmic transformation on bit rates 216(1), ..., bit rates 216(M) to generate logarithmically transformed bit rates (not shown). Normalization engine 220 performs normalization on the logarithmically transformed bit rates and quality scores 218(1), ..., quality scores 218(M) to generate normalized bit rate array 236 and normalized quality score array 238. The normalized bit rates included in normalized bit rate array 236 are B1, ..., B M are derived from B'1,...,B' M The normalized quality scores included in the normalized quality score array 238 are represented as Q1, ..., Q M , and in this specification, Q'1,...,Q' M It is expressed as:

[0070] In some alternative embodiments, the normalization engine 220 and / or the encoding ladder application 150 may perform any number (including zero) and / or type of normalization operations on the bit rates 216(1), ..., bit rates 216(M) and / or the quality scores 218(1), ..., quality scores 218(M) in any technically feasible manner. If this embodiment is used, the technical content described herein may be modified accordingly. For example, in some alternative embodiments, the normalization engine 220 may normalize the bit rates 216(1), ..., bit rates 216(M) but not the quality scores 218(1), ..., quality scores 218(M), or may normalize the quality scores 218(1), ..., quality scores 218(M) but not the bit rates 216(1), ..., bit rates 216(M). Furthermore, in some embodiments, the normalization engine 220 may be omitted from the encoding ladder application 150.

[0071] The coding point metadata 230 includes multiple constant arrays, consisting of constants specified in or derived from the convex hull metadata 136, that are used to formulate a parameterized constrained optimization problem. As shown, the coding point metadata 230 includes, but is not limited to, an ID array 222, a resolution array 224, a bitrate array 226, a quality score array 228, a normalized bitrate array 236, and a normalized quality score array 238. The coding ladder application 150 generates the ID array 222, the resolution array 224, the bitrate array 226, and the quality score array 228 by rearranging the convex hull metadata 136. The ID array 222 includes coded picture IDs 212(1), ..., coded picture ID 212(M). The resolution array 224 includes resolutions 214(1), ..., resolution 214(M), which are designated R1, ..., R2, respectively. M The bit rate array 226 includes bit rates 216(1), ..., bit rate 216(M), which are represented as B1, ..., BM The quality score array 228 includes quality scores 218(1), ..., quality score 218(M), which are denoted as Q1, ..., Q M where M is the total number of bitrate / quality evaluation points, i.e., the "bitrate-quality point count," and is treated as a constant in the parameterized constrained optimization problem.

[0072] The coding ladder application 150 defines a parameterized objective function 240 based on the overall objective: to determine N rung assignments for a candidate coding ladder that optimize the overall trade-off between improving streaming QoE and simultaneously minimizing storage footprint. Herein, N represents the "run count," a parameter that specifies the total number of runs in the candidate coding ladder. Note that the "run count" may also be referred to herein as the "number of runs." Each rung assignment indicates which different bitrate / quality scores 210 are assigned to which different rungs in the candidate coding ladder.

[0073] As will be explained in more detail below in the description of parameterized constraint set 250, the rank assignment is constrained such that the bit rate assigned to each rank must monotonically increase from rank 1 to rank N. For ease of explanation, rank 1 of a candidate coding ladder may be referred to as the "bottom rank" of the candidate coding ladder, and rank N may be referred to as the "top rank" of the candidate coding ladder.

[0074] In general, the "streaming QoE" of a coding ladder quantifies the average viewer's QoE when an encoded media title is streamed to a client device. A typical QoE metric reflects both the image quality level associated with the encoded chunks used for streaming and the impact of rebuffering events on the overall quality of the viewing experience. As those skilled in the art will recognize, the streaming QoE associated with a given coding ladder cannot be accurately measured until the coding ladder is actually deployed. Therefore, the coding ladder application 150 defines quality terms (not shown in FIG. 2) that correlate with the streaming QoE.

[0075] The quality term is a weighted average of the normalized quality scores assigned to the N bitrate / quality metrics assigned to the N rungs of the candidate coding ladder. This quality term has a different "rung quality" weight associated with each of the N rungs, and these N rung quality weights are parameters in the parameterized constrained optimization problem. Herein, the rung quality weight associated with rung j is denoted by w j The value of the tier quality weight may be determined in any technically feasible manner. In some embodiments, the value of the tier quality weight is a predetermined value that is set in advance. In other embodiments, the value of the tier quality weight may be derived based on empirical statistics such as network throughput or bitrate demand distribution.

[0076] As those skilled in the art will recognize, the sum of the bit rates of the coded videos included in a coding ladder is proportional to the storage footprint of that coding ladder. Thus, coding ladder application 150 defines a footprint term (not shown in FIG. 2 ) as the sum of the normalized bit rates of the N coded videos assigned to the N rungs of a candidate coding ladder.

[0077] The overall objective of maximizing the streaming QoE of a candidate coding ladder while minimizing its storage footprint is primarily expressed in a trade-off between streaming QoE and storage footprint. To flexibly and explicitly capture this trade-off between streaming QoE and storage footprint, coding ladder application 150 weights quality terms with quality term weights and footprint terms with footprint term weights. Thus, parameterized objective function 240 represents a weighted trade-off between a weighted average of a subset of normalized quality scores associated with a set of coded pictures and a sum of a subset of normalized bitrates associated with that set of coded pictures. More precisely, in some embodiments, parameterized objective function 240 represents a weighted trade-off between a weighted average of a subset of normalized quality scores array 238 corresponding to a subset of ID array 222 and a sum of a subset of normalized bitrates array 236 corresponding to the same subset of ID array 222.

[0078] The quality term weight and footprint term weight are used as parameters in a parameterized constrained optimization problem. The quality term weight value indicates the relative importance of the candidate coding ladder in maximizing streaming QoE, and the footprint term weight value indicates the relative importance of the candidate coding ladder in minimizing storage footprint.

[0079] As will be described in more detail below with reference to FIG. 3 , to enable the row assignments to be expressed by the parameterized objective function 240, the coding ladder application 150 defines an assignment matrix. This assignment matrix specifies multiple assignments (assignments for each row) between the coded video sets corresponding to the ID array 222 and the row sets included in the coding ladder candidate. More specifically, the assignment matrix is ​​an M×N Boolean matrix, denoted as X herein. Rows 1, ..., row M of X correspond to bitrate / quality evaluation points 210(1), ..., bitrate / quality evaluation point 210(M), respectively. Meanwhile, columns 1, ..., column N of X correspond to rows 1, ..., row N of the coding ladder candidate 280(1), respectively.

[0080] X i,j represents the element in the i-th row and j-th column of the assignment matrix 370(1). i,j If X is "1", it indicates that the bitrate / quality score 210(i) is assigned to the jth stage of the candidate coding ladder. i,j is "0", this indicates that the bitrate / quality evaluation point 210(i) is not assigned to stage j of the candidate coding ladder. In this specification, when the bitrate / quality evaluation point 210(i) is assigned to stage j of the candidate coding ladder, the coded video, resolution 214(i), bitrate 216(i), and quality score 218(i) corresponding to the coded video ID 212(i) are also assigned to this stage j. In addition, when the bitrate / quality evaluation point 210(i) is assigned to stage j of the candidate coding ladder, this specification states that stage j "specifies" the coded video, coded video ID 212(i), resolution 214(i), bitrate 216(i), and quality score 218(i) corresponding to the coded video ID 212(i).

[0081] Although not shown, in some embodiments, the coding ladder application 150 implements the parameterized objective function 240 as follows:

number

[0082] The coding ladder application 150 calculates the row quality weights w1 to w2 using the following formula: N Define the value of

number

number

[0083] By assigning higher values ​​to the row quality weights corresponding to higher rows in a given coding ladder, historical streaming statistics that indicate that rows with higher quality scores are typically more frequently actually selected for streaming by client devices can be emulated, according to equation (2b). Thus, the quality term in equation (1) approximates the average image quality level of multiple coded video instances, as specified in the candidate coding ladder, that are predicted to be streamed to a client device.

[0084] In some other embodiments, the value of the tier quality weight is set to 1, such that the quality term in equation (1) represents the average quality level of the coded video assigned to the coding ladder candidate. In general, in generating the objective function, the tier assignment engine can calculate any number of values ​​for any number of weights associated with a given tier set based on the ladder configuration and referenced by the objective function.

[0085] As shown, the coding ladder application 150 generates a parameterized constraint set 250. The parameterized constraint set 250 may include any number and / or type of constraints. Each constraint limits one or more stage assignments and may be associated with zero or more parameters. In some embodiments, the parameterized constraint set 250 includes intrinsic logical constraints and operational logical constraints. The intrinsic logical constraints ensure that the set of stage assignments specified by the "optimized" final assignment matrix corresponds to a valid coding ladder. Meanwhile, the operational logical constraints reflect various requirements and priorities of operational aspects of video streaming. Examples of operational aspects of video streaming include client device capabilities, network capacity, CDNs, and human perception of image quality.

[0086] In some embodiments, the intrinsic logic constraints include, but are not limited to, a parameterized stage allocation constraint, a parameterized score allocation constraint, and a parameterized monotonically increasing bitrate constraint, while the operational logic constraints include, but are not limited to, a parameterized upper footprint constraint, a parameterized required resolution constraint, a parameterized low bitrate score constraint, a parameterized high quality score constraint, a parameterized minimum quality interval constraint, a parameterized maximum quality interval constraint, and a parameterized bitrate interval constraint.

[0087] The parameterized stage allocation constraint is a constraint that requires that each stage of a candidate coding ladder be assigned only one of the bitrate / quality scores 210. The parameterized stage allocation constraint can be expressed as follows:

number

[0088] The parameterized score assignment constraint is a constraint that requires that each bitrate / quality score 210 be assigned to at most one rung in a candidate coding ladder. The parameterized score assignment constraint can be expressed as follows:

number

[0089] The parameterized monotonically increasing bitrate constraint is a constraint that requires the bitrate to monotonically increase with each successive stage. The parameterized monotonically increasing bitrate constraint can be expressed as follows:

number

[0090] Because CDN storage resources are limited, the encoding ladder application 150 implements a parameterized footprint upper limit constraint, which requires that the sum of the coded video bit rates assigned to each rung of a candidate encoding ladder does not exceed a footprint upper limit. The parameterized footprint upper limit constraint can be expressed as follows:

number

[0091] Because screen resolutions vary from client device to client device, the coding ladder application 150 implements one or more constraints associated with at least one resolution requested to be represented by the candidate coding ladder. Although not shown in FIG. 2 , the coding ladder application 150 implements a different requested resolution constraint for each of zero or more requested resolutions based on a parameterized requested resolution constraint. The parameterized requested resolution constraint is a constraint that requires that at least one of the bitrate / quality evaluation points 210 assigned to each rung of the candidate coding ladder specifies the requested resolution. The parameterized requested resolution constraint can be expressed as follows:

number

[0092] To ensure uninterrupted playback of media titles under difficult network conditions, the encoding ladder application 150 implements a parameterized low bitrate score constraint, which requires that at least one of the bitrate / quality scores 210 assigned to each rung of a candidate encoding ladder specifies a bitrate less than or equal to "low bitrate." The parameterized low bitrate score constraint can be expressed as follows:

number

[0093] To ensure that viewers experience sufficiently high image quality when streaming media titles over relatively high network capacity connections, the encoding ladder application 150 implements a parameterized high quality score constraint, which requires that at least one of the bitrate / quality scores 210 assigned to each rung of a candidate encoding ladder specifies a quality score greater than or equal to a "high quality score." The parameterized high quality score constraint can be expressed as follows:

number

[0094] A parameterized minimum quality interval constraint is a constraint that requires that the interval between quality scores assigned to successive runs of a candidate coding ladder be at least a minimum quality interval. The parameterized minimum quality interval constraint can be expressed as follows:

number

[0095] A parameterized maximum quality interval constraint is a constraint that requires that the interval between quality scores assigned to successive runs of a candidate coding ladder does not exceed a maximum quality interval. The parameterized maximum quality interval constraint can be expressed as follows:

number

[0096] A parameterized bitrate spacing constraint is a constraint that requires that the spacing between bitrates specified for successive stages of a candidate coding ladder does not exceed the relative bitrate spacing. The parameterized bitrate spacing constraint can be expressed as follows:

number

[0097] Ladder configurations 260(1), ..., 260(C) each specify a different set of values ​​for a parameter set associated with a parameterized constrained optimization problem, including, but not limited to, ladder configurations 260(1), ..., 260(C). Encoding ladder application 150 can generate ladder configurations 260(1), ..., 260(C) in any technically feasible manner. In general, encoding ladder application 150 can generate ladder configurations 260(1), ..., 260(C) by determining different sets of parameter values ​​for the union of the parameter set included in parameterized objective function 240 and the parameter set included in parameterized constraint set 250.

[0098] As shown, ladder configuration 260(1) includes, but is not limited to, total number of runs 262(1), rung quality weight set 264(1), objective parameter set 266(1), and constraint parameter set 268(1). Although not shown, if x is an integer between 2 and C, ladder configuration 260(x) includes, but is not limited to, total number of runs 262(x), rung quality weight set 264(x), objective parameter set 266(x), and constraint parameter set 268(x).

[0099] Each of the total stages 262(1), ..., 262(C) specifies a value for the total stage. Each of the stage quality weight sets 264(1), ..., 264(C) specifies a different value for each of a plurality of stage quality weights. In some embodiments, the encoding ladder application 150 calculates the stage quality weight sets 264(1), ..., 264(C) based on the total stages 262(1), ..., 262(C) according to equations (2a) and (2b).

[0100] Objective parameter sets 266(1), ..., objective parameter sets 266(C) each specify values ​​for quality term weights and footprint term weights. Constraint parameter sets 268(1), ..., constraint parameter sets 268(C) each specify zero or more required resolutions, and values ​​for footprint upper limit, high quality score, low bitrate, minimum quality interval, maximum quality interval, and relative bitrate interval.

[0101] Rung assignment engines 270(1), ..., 270(C) are different instances of a single software application referred to herein as the "rung assignment engine." As shown, coding ladder application 150 generates candidate coding ladder 280(1) by running rung assignment engine 270(1) on ladder configuration 260(1), coding point metadata 230, parameterized objective function 240, and parameterized constraint set 250. Also shown, coding ladder application 150 generates candidate coding ladder 280(C) by running rung assignment engine 270(C) on ladder configuration 260(C), coding point metadata 230, parameterized objective function 240, and parameterized constraint set 250. Although not shown, if x is an integer between 2 and C-1, the coding ladder application 150 generates a coding ladder candidate 280(x) by running a rung assignment engine 270(x) on the ladder configuration 260(x), the coding point metadata 230, the parameterized objective function 240, and the parameterized constraint set 250.

[0102] Generally, the stage allocation engine derives an objective function from parameterized objective function 240 and derives constraints from parameterized constraint set 250 using parameter values ​​specified in the ladder configuration. The stage allocation engine implements any number and / or type of constrained optimization techniques to optimize the derived objective function under these constraints. Note that, in this specification, the process of determining element values ​​of an assignment matrix to optimize the objective function under such constraints may be referred to as "solving" the constrained optimization problem defined by the objective function and constraints. The stage allocation engine 270(1) is described in more detail below in the description of FIG. 3.

[0103] Those skilled in the art will appreciate that many modifications and variations can be made to the functionality provided by the coded ladder application 150, parameterized objective function 240, parameterized constraint set 250, ladder configurations 260(1), ..., ladder configuration 260(C), and rung assignment engine described herein without departing from the scope and spirit of the embodiments described herein. In some embodiments, the coded ladder application 150 can implement any number and / or type of objective function and / or parameterized objective function instead of or in addition to the parameterized objective function 240. Additionally, in the same or other embodiments, the coded ladder application 150 can generate any number of candidate coded ladders for each of any number of objective functions and / or parameterized objective functions. Additionally, in some embodiments, the number and / or type of parameters associated with the parameterized objective function 240 and / or the parameterized constraint set 250 can be changed. Additionally, in the same or other embodiments, the number and / or type of parameterized constraints included in the parameterized constraint set 250 can be changed.

[0104] Figure 3 is a more detailed diagram of one of the rung assignment engines shown in Figure 2, according to various embodiments. More specifically, Figure 3 shows rung assignment engine 270(1) generating candidate coding ladder 280(1) based on ladder configuration 260(1), coding point metadata 230, parameterized objective function 240, and parameterized constraint set 250.

[0105] As described above with reference to Figure 2, ladder configuration 260(1) includes, but is not limited to, total number of runs 262(1), rung quality weight set 264(1), objective parameter set 266(1), and constraint parameter set 268(1). In the embodiment shown in Figure 3, total number of runs 262(1) is set to "6," and therefore rung quality weight set 264(1) specifies values ​​for six rung quality weights represented by w1 to w6.

[0106] For ease of explanation, objective parameter set 266(1) specifies a value of 0.4 for the quality term weight and a value of 0.6 for the footprint term weight, while constraint parameter set 268(1) specifies two required resolutions (e.g., 1280×720 and 1920×1080), values ​​for the footprint upper limit, high quality score, low bitrate, minimum quality interval, and maximum quality interval, and a value of 1.7 for the relative bitrate interval.

[0107] 2 above, the coding point metadata 230 is a plurality of constant arrays consisting of constants specified in or derived from the convex hull metadata 136, which are used to formulate a parameterized constrained optimization problem. More specifically, the coding point metadata 230 includes, but is not limited to, a resolution array 224, a bitrate array 226, a quality score array 228, a normalized bitrate array 236, and a normalized quality score array 238. As mentioned above, the resolution array 224 and the bitrate array 226 contain R1, ..., R M , the quality score array 228 contains B1,…,B M , the normalized bitrate array 236 contains Q1,…,Q M , the normalized quality score array 238 contains Q'1,...,Q' M Includes:

[0108] As shown, rung allocation engine 270(1) includes, but is not limited to, objective function 340, constraints 350(1), ..., constraints 350(11), constrained optimization solver 360, and assignment matrix 370. Rung allocation engine 270(1) generates objective function 340 based on total rung count 262(1) and objective parameter set 266(1). Rung allocation engine 270(1) also generates constraints 350(1), ..., constraints 350(11) based on ladder configuration 260(1) and parameterized constraint set 250. Constrained optimization solver 360 implements any number and / or type of constrained optimization techniques to determine element values ​​of assignment matrix 370 to optimize objective function 340 according to ladder configuration 260(1) and coding point metadata 230 under constraints 350(1), ..., constraints 350(11).

[0109] As shown, the allocation matrix 370(1) is an M×6 Boolean matrix, denoted as X. Rows 1, ..., and M in X correspond to bitrate / quality evaluation points 210(1), ..., and bitrate / quality evaluation points 210(M), respectively. Meanwhile, columns 1, ..., and 6 in X correspond to stages 1, ..., and 6, respectively, of the candidate coding ladder 280(1). Once the constrained optimization solver 360 determines the final values ​​for the candidate coding ladder 280(1), the stage allocation engine 270(1) generates each stage of the candidate coding ladder 280(1) based on the entries of "1" in the allocation matrix 370(1). Specifically, X i,j If the entry for is "1", the row assignment engine 270(1) generates row j of the candidate coding ladder 280(1) based on the bitrate / quality score 210(i).

[0110] As described above in the description of Figure 2, parameterized objective function 240 can be expressed by equation (1). Stage allocation engine 270(1) generates objective function 340 by replacing the parameters N, q, and b in equation (1) with total stage number 262(1) = 6, quality term weight = 0.4, and footprint term weight = 0.6, respectively. This objective function 340 can be expressed as follows:

number

[0111] As shown, the following part (13a) in equation (13) represents a quality term 322. This quality term 322 is expressed as a stage-weighted average of the normalized quality scores associated with the subset of bitrate / quality scores 210 assigned to each stage of candidate coding ladder 280(1).

number

[0112] Also, the following part (13b) in equation (13) represents the footprint term 324. This footprint term 324 is expressed as the sum of the normalized bit rates associated with the subset of bit rate / quality scores 210 assigned to each rung of the candidate coding ladder 280(1).

number

[0113] The parameter value of 0.4 for the quality term weight and the parameter value of 0.6 for the footprint term weight indicate the tradeoff between QoE and storage footprint associated with objective function 340, and thus with candidate coding ladder 280(1). More specifically, the QoE-storage footprint tradeoff represented by objective function 340 prioritizes reducing the storage footprint of candidate coding ladder 280(1), even at the expense of some improvement in the QoE of candidate coding ladder 280(1).

[0114] As discussed above in the description of FIG. 2, the parameterized constraint set 250 includes a parameterized upper footprint constraint, a parameterized required resolution constraint, a parameterized low bitrate rating point constraint, a parameterized high quality rating point constraint, a parameterized minimum quality interval constraint, a parameterized maximum quality interval constraint, and a parameterized bitrate interval constraint.

[0115] Stage allocation engine 270(1) generates constraints 350(1), ..., 350(11) based on total stage count 262(1), constraint parameter set 268(1), and parameterized constraint set 250. Constraints 350(1), ..., 350(11) correspond to a stage allocation constraint, a score allocation constraint, a monotonically increasing bitrate constraint, a footprint upper limit constraint, two required resolution constraints, a low bitrate score constraint, a high-quality score constraint, a minimum quality interval constraint, a maximum quality interval constraint, and a bitrate interval constraint, respectively. For ease of explanation, FIG. 3 shows details of constraints 350(1) and 350(11).

[0116] Constraint 350(1) is a stage assignment constraint derived by stage assignment engine 270(1) based on the parameterized stage assignment constraint described above in the description of Figure 2. Stage assignment engine 270(1) generates constraint 350(1) by replacing N in equation (3) with the total number of stages 262(1) = 6. Constraint 350(1) requires that only one of the bitrate / quality assessment points 210 be assigned to each of the six stages of candidate coding ladder 280(1). As shown, constraint 350(1) can be expressed as follows:

number

[0117] Constraint 350(11) is a bitrate interval constraint derived by stage allocation engine 270(1) based on the parameterized bitrate interval constraint described above in the description of FIG. 2. Stage allocation engine 270(1) generates constraint 350(11) by replacing (N-1) with 5 (one less than the total number of stages 262(1) = 6) and λ with 1.7 in equation (12). Constraint 350(11) requires that for two consecutive stages of candidate coding ladder 280(1), the bitrate of the higher stage is not more than twice the bitrate of the lower stage. As shown, constraint 350(11) can be expressed as follows:

number

[0118] As shown, row allocation engine 270(1) causes constrained optimization solver 360 to solve objective function 340 under constraints 350(1), ..., constraints 350(11) based on ladder configuration 260(1) and coding point metadata 230 to generate final allocation matrix 370. Also shown is matrix X, which has M rows arranged along evaluation point axis 372 and 6 columns arranged along row axis 374.

[0119] Along the evaluation point axis 372, the top row of the allocation matrix 370 corresponds to evaluation point index=1, and therefore corresponds to bitrate / quality evaluation point 210(1), coded picture ID 212(1), R1, B1, Q1, B'1, and Q'1. Similarly, the bottom row of the allocation matrix 370 corresponds to evaluation point index=M, and therefore corresponds to bitrate / quality evaluation point 210(M), coded picture ID 212(M), R1, B1, Q1, B'1, and Q'1. M , B M , Q M , B' M , and Q' M Along row axis 374, the leftmost column of assignment matrix 370 corresponds to row 1, which corresponds to the bottom row of encoding ladder candidate 280(1), and the rightmost column corresponds to row 6, which corresponds to the top row.

[0120] In some embodiments, constrained optimization solver 360 generates candidate coding ladder 280(1) by running a constrained optimization algorithm on objective function 340, constraints 350(1), ..., constraints 350(11), set of stage quality weights 264(1), and coding point metadata 230. More generally, constrained optimization solver 360 can run any number and / or type of constrained optimization algorithm to solve objective function 340 under constraints 350(1), ..., constraints 350(11) using values ​​specified in set of stage quality weights 264(1) and coding point metadata 230. As used herein, "constrained optimization algorithm" refers to any algorithm that implements any number and / or type of constrained optimization techniques known in the art.

[0121] For example, the constrained optimization solver 360 may execute a genetic algorithm that implements a search-based optimization technique. Alternatively, the constrained optimization solver 360 may execute a surrogate optimization algorithm that uses a surrogate model that approximates the objective function 340. Other examples of constrained optimization techniques that can be used to generate candidate coding ladders include branch-and-bound, cutting plane, and surrogate model techniques. Constrained optimization solvers are known in the art. For an overview of a software suite known as "Google Optimization Tools," which includes several different constrained optimization solvers, see https: / / github.com / google / or-tools#readme.

[0122] As mentioned above, X i,j If X is "1", it indicates that bitrate / quality score 210(i) is assigned to stage j. i,j is not "1", it indicates that no bitrate / quality score 210(i) is assigned to stage j. Note that constraint 350(3) (i.e., the monotonically increasing bitrate constraint) requires that the bitrate specified by the bitrate / quality score 210 assigned to stage x must be less than the bitrate specified by the bitrate / quality score 210 assigned to stage (x+1), where x is an integer between 1 and (N-1).

[0123] Upon completion of execution of constrained optimization solver 360, rung allocation engine 270(1) generates coding ladder candidate 280(1) based on the "1" entries in allocation matrix 370(1) and ignores the "0" entries in allocation matrix 370(1). More specifically, rung allocation engine 270(1) generates coding ladder candidate 280(1) based on the six "1" entries in allocation matrix 370.

[0124] In some embodiments, X i,jIf the entry for is "1", the row assignment engine 270(1) generates row j of the candidate coding ladder 280(1) and assigns to row j, including but not limited to, the coded video ID 212(i), R i , and B i , and even Q if necessary i Referring back to Figure 2, R i is resolution 214(i), B i is bit rate 216(i), Q i corresponds to the quality score 218(i) associated with the coded picture corresponding to coded picture ID 212(i).

[0125] For ease of explanation, only the entries in the first five rows and the last five rows of the assignment matrix 370 at the time of completion of execution of the constrained optimization solver 360 are shown in FIG. 4,1 and X M-2,6 are all set to "1". X 4,1 is "1", so row assignment engine 270(1) generates a row that specifies coded picture ID 212(4), R4, and B4, and optionally Q4, as row 1 of coding ladder candidate 280(1). Similarly, X M-2,6 is "1", the stage allocation engine 270(1) assigns the coded video ID 212(M-2), R M-2 , and B M-2 , and optionally Q M-2 The row specifying this is generated as row 6 of the coding ladder candidate 280(1). Although not shown in the figure, the row allocation engine 270(1) generates rows 2 to 5 of the coding ladder candidate 280(1) based on the other four entries of “1” in the allocation matrix 370.

[0126] Simulation-Based Evaluation and Selection of Candidate Encoding Ladders 4 is a diagram illustrating the simulation evaluation application 170 shown in FIG. 1 in more detail, according to various embodiments. The simulation evaluation application 170 generates an encoding ladder 178 based on the filtered encoding ladder candidate set 168, the encoded chunk metadata 138, and the streaming session trace 126. As shown, the simulation evaluation application 170 includes, but is not limited to, a streaming header synthesis engine 410, synthesized streaming headers 430(1), ..., synthesized streaming header 430(F), and a ladder evaluation and selection engine 480.

[0127] The streaming header synthesis engine 410 generates a composite streaming header 430(1), ..., a composite streaming header 430(F) based on each encoding ladder candidate included in the filtered encoding ladder candidate set 168 and the encoded chunk metadata 138. As described above with reference to FIG. 1 , the filtered encoding ladder candidate set 168 includes F encoding ladder candidates 280(1), ..., encoding ladder candidate 280(C), where C may be any positive integer and F may be any positive integer less than or equal to C. Note that the F encoding ladder candidates included in the filtered encoding ladder candidate set 168 each have a different stage set, and the number of stages included in each stage set may differ among the filtered encoding ladder candidate sets 168. Furthermore, the encoded video and resolution specified in these stage sets may also differ among the filtered encoding ladder candidate sets 168. The encoded chunk metadata 138 includes, but is not limited to, a number of encoded chunks corresponding to each encoded video specified in the convex hull metadata 136 (via the corresponding encoded video ID), the bitrates of these encoded chunks, and the quality scores of these encoded chunks.

[0128] Each of the composite streaming headers 430(1), ..., 430(F) includes a different set of streaming metadata for each rung included in the corresponding candidate encoding ladder. For ease of explanation, FIG. 4 shows the details of the composite streaming header 430(1). The streaming header synthesis engine 410 generates the composite streaming header 430(1) based on the candidate encoding ladder 280(1) (described above in connection with FIG. 3) and the encoded chunk metadata 138. More specifically, the streaming header synthesis engine 410 generates the composite streaming header 430(1) based on the rung set included in the candidate encoding ladder 280(1) and the encoded chunk metadata 138. Referring again to the ladder configuration 260(1) shown in FIG. 3, the candidate encoding ladder 280(1) has six runs, each of which specifies a different coded video ID, a corresponding coded video resolution, and a corresponding coded video bitrate.

[0129] As shown, the composite streaming header 430(1) includes streaming metadata sets 432(1), ..., 432(6), each of which describes information about the video rate and quality per chunk for a corresponding rung among rungs 1 through 6 of the encoding ladder candidate 280(1). Streaming metadata set 432(1) includes, but is not limited to, an encoded video ID 434(1), an encoded chunk ID set 436(1), a resolution 442(1), a bitrate 444(1), an encoded chunk bitrate set 446(1), and an encoded chunk quality score set 448(1). The encoded video ID 434(1) identifies the encoded video corresponding to rung 1 of the encoding ladder candidate 280(1). Resolution 442(1) and bitrate 444(1) specify the resolution and average bitrate, respectively, of the encoded video corresponding to encoded video ID 434(1). Encoded chunk IDs 436(1) identify multiple encoded chunks included in the encoded video corresponding to encoded video ID 434(1). Encoded chunk bitrates 446(1) and encoded chunk quality scores 448(1) specify the bitrates and quality scores, respectively, of the multiple encoded chunks corresponding to encoded chunk IDs 436(1). Note that only metadata information is used to generate the composite streaming header. Therefore, in some embodiments, the encoding ladder workflow 140 generates the composite streaming header without the corresponding encoded video having been generated by production encoding pipeline 120.

[0130] The ladder evaluation and selection engine 480 generates the encoding ladder 178 based on the composite streaming headers 430(1), ..., 430(F) and the streaming session traces 126(1), ..., 126(T). Each of the streaming session traces 126(1), ..., 126(T) is a network throughput trace having a duration equal to or greater than the playback time of the media title. As discussed above with respect to FIG. 1, a network throughput trace indicates the network throughput as a function of time over the duration of the trace.

[0131] In some embodiments, each of streaming session traces 126(1), ..., streaming session trace 126(T) is either a recorded network throughput trace or a synthetic network throughput trace. A recorded network throughput trace specifies recorded measurements of throughput in a real network, while a synthetic network throughput trace is synthesized by a software application (e.g., simulation evaluation application 170) in any technically feasible manner.

[0132] As shown, the ladder evaluation and selection engine 480 executes the adaptive streaming simulator 106 on simulation configurations 450(1,1), ..., 450(F,T) to generate corresponding request sequences 460(1,1), ..., 460(F,T). Thus, the ladder evaluation and selection engine 480 executes the adaptive streaming simulator 106 a total of (F*T) times. The adaptive streaming simulator 106 emulates some of the behavior of endpoint applications running on client devices during an adaptive streaming session. Specifically, the adaptive streaming simulator 106 implements an ABR algorithm 406, which attempts to optimize the perceived quality of the playback of a streaming media title while avoiding playback interruptions due to rebuffering events. In other words, the ABR algorithm 406 attempts to select encoded chunk sequences with the highest possible bitrate without exceeding the available network throughput. In some other embodiments, the adaptive streaming simulator 106 may implement any number and / or type of ABR algorithms in any technically feasible manner to optimize the perceived quality of playback based on any number and / or type of criteria.

[0133] For ease of explanation, it is assumed that indexes f and t may be any integer between 1 and F, and t may be any integer between 1 and T. Then, for the simulation configuration 450(f,t), the adaptive streaming simulator 106 executes the ABR algorithm 406 on the composite streaming header 430(f) based on the streaming session trace 126(t) to generate a request sequence 460(f,t). More precisely, the adaptive streaming simulator 106 configures the ABR algorithm 406 to sequentially generate the request sequence 460(f,t) based on the composite streaming header 430(f) in the simulated streaming session defined by the streaming session trace 126(t). Thus, the request sequence 460(f,t) is a request sequence that specifies the encoded chunks to be generated by the ABR algorithm 406 based on the composite streaming header 430(f) and the network throughput sequence included in the streaming session trace 126(t). That is, for each source chunk of source video 102, request sequence 460(f,t) specifies a corresponding encoded chunk from one of the encoded videos specified for each rung of the potential encoding ladder corresponding to composite streaming header 430(f).

[0134] The ladder evaluation and selection engine 480 generates index value sets 470(1,1), ..., index value set 470(F,T) based on the request sequence 460(1,1), ..., request sequence 460(F,T) and the composite streaming header 430(1), ..., composite streaming header 430(F). More precisely, the ladder evaluation and selection engine 480 generates index value set 470(f,t) based on the request sequence 460(f,t) and the composite streaming header 430(f). The index value sets 470(1,1), ..., index value set 470(F,T) each specify a different set of values ​​(value sets) for the streaming evaluation index set.

[0135] The set of streaming evaluation metrics can include, without limitation, any number and / or type of metrics related to streaming QoE and any number and / or type of metrics related to costs associated with streaming (e.g., storage footprint and network bandwidth consumption). In some embodiments, the set of streaming evaluation metrics includes QoE-related metrics commonly measured in production A / B testing of coding ladders. In the same or other embodiments, the set of metric values ​​470(f,t) specifies metrics related to streaming of a media title using the fth coding ladder candidate in the filtered coding ladder candidate set 168. Such metrics may include a time-weighted quality score, an average playback bitrate, a total number of rebuffering events, a total rebuffering time, a total number of stage switches, a weighted aggregation of any number of these metrics representing streaming QoE, or any combination thereof. Examples of quality scores include an average PSNR value, a time-weighted value of a VMAF metric (a “time-weighted VMAF score”), an average value of a VMAF metric (a “average VMAF score”), etc. Also, in the same or other embodiments, the set of index values ​​470(f,t) may specify at least one index value that approximates a trade-off between streaming quality of experience and storage footprint associated with the fth encoding ladder candidate in the filtered encoding ladder candidate set 168. In yet other embodiments, the set of index values ​​470(f,t) may also specify at least one index value that approximates a trade-off between streaming quality of experience associated with the fth encoding ladder candidate in the filtered encoding ladder candidate set 168 and a cost term associated with expected network bandwidth consumption in the adaptive streaming session defined by the streaming session trace 126(t).

[0136] The ladder evaluation and selection engine 480 can repeatedly calculate the value of each metric in the streaming evaluation metric set at any granularity (e.g., every minute) for any period of time (e.g., the entire simulation streaming session).

[0137] For ease of explanation, the metric value sets 470(f,1), ..., 470(f,t) may also be referred to herein as the fth "metric value set group." The fth metric value set group is associated with the composite streaming header 430(f) and, therefore, the fth encoding ladder candidate in the filtered encoding ladder candidate set 168.

[0138] The ladder evaluation and selection engine 480 may perform any number and / or types of evaluations and / or comparisons, including any other associated data, among any number of sets of index values ​​corresponding to any number of candidate coding ladders included in the filtered candidate coding ladder set 168. Based at least in part on the results of such evaluations and / or comparisons, the ladder evaluation and selection engine 480 may select any number of candidate coding ladders associated with the sets of index values ​​for further evaluation and deployment through the production coding pipeline 120.

[0139] In some embodiments, the ladder evaluation and selection engine 480 calculates an average streaming QoE for each encoding ladder candidate in the filtered encoding ladder candidate set 168 based at least in part on the corresponding set of metric values. The ladder evaluation and selection engine 480 may then select the encoding ladder candidate with the highest average streaming QoE vs. storage footprint trade-off for further evaluation and deployment via the production encoding pipeline 120 (not shown in FIG. 4 ).

[0140] In some other embodiments, the ladder evaluation and selection engine 480 performs a pair-by-pair comparison of the coding ladder candidates in the filtered coding ladder candidate set 168 based at least in part on the corresponding sets of index values. Then, based at least in part on the pair-by-pair comparison, the ladder evaluation and selection engine 480 may select any number of the coding ladder candidates in the filtered coding ladder candidate set 168 for further evaluation and deployment through the production coding pipeline 120.

[0141] For example, the ladder evaluation and selection engine 480 may compare two different values ​​for the same metric, specified in metric value set 470(1,1) and metric value set 470(2,1), respectively. Based on the results of this comparison, the ladder evaluation and selection engine 480 may determine that the first encoding ladder candidate associated with metric value set 470(1,1) should be used to stream the media title, rather than the second encoding ladder candidate associated with metric value set 470(2,1). Thus, the ladder evaluation and selection engine 480 may select the first encoding ladder candidate as the candidate to be deployed through the production encoding pipeline 120.

[0142] Based on the encoding ladder candidate selected through the comparison process described above, the ladder evaluation and selection engine 480 generates the encoding ladder 178. In some embodiments, the encoding ladder 178 is a copy of the selected encoding ladder candidate. In other embodiments, the ladder evaluation and selection engine 480 generates the encoding ladder 178 by performing one or more additional processes on the selected encoding ladder candidate. For example, in some embodiments, the ladder evaluation and selection engine 480 configures the shot-based encoding application 132 to regenerate coded video corresponding to each rung of the selected encoding ladder candidate using an encoding technique that is more computationally intensive than the encoding technique associated with the encoded chunk metadata 138. More specifically, coded video corresponding to each rung is typically regenerated with the same resolution, approximately the same bit rate, and a higher quality score than the information specified for each rung of the selected encoding ladder candidate.

[0143] The ladder evaluation and selection engine 480 may store the encoding ladder 178 in any type of memory accessible to the production encoding pipeline 120 and / or the encoding ladder workflow 140. In some embodiments, the ladder evaluation and selection engine 480 may transmit the encoding ladder 178 to the production encoding pipeline 120, the encoding ladder workflow 140, any number of other software applications, or any combination thereof. Also, in some embodiments, the encoding ladder 178 is the final encoding ladder used in streaming the media title to one or more client devices over a network.

[0144] In some embodiments, the ladder evaluation and selection engine 480 may perform any number and / or types of evaluations and / or comparisons, including any other associated data, among any number of sets of index values ​​corresponding to any number of coding ladder candidates included in the filtered coding ladder candidate set 168. Then, based at least in part on the results of such evaluations and / or comparisons, the ladder evaluation and selection engine 480 may select any number of coding ladder candidates associated with these sets of index values ​​for further evaluation and deployment through the production coding pipeline 120.

[0145] In some embodiments, the ladder evaluation and selection engine 480 calculates an average streaming QoE for each encoding ladder candidate in the filtered encoding ladder candidate set 168 based at least in part on the corresponding set of metric values. The ladder evaluation and selection engine 480 may then select the encoding ladder candidate with the highest average streaming QoE vs. storage footprint trade-off for further evaluation and deployment via the production encoding pipeline 120 (not shown in FIG. 4 ).

[0146] Again, it should be noted that the various techniques described herein are illustrative only and not limiting. That is, these techniques can be modified and implemented without departing from the broader spirit and scope of the present invention. For example, in some embodiments, the functionality of the simulation evaluation application 170, adaptive streaming simulator 106, and ABR algorithm 406 described herein can be integrated into a single software application or distributed across any number of software applications in any technically feasible manner. Also, in the same or other embodiments, the simulation evaluation application 170 can perform simulation evaluations of any number of candidate coding ladders and / or any number of coding ladders (e.g., the currently used production coding ladder) in any technically feasible manner using any number and / or type of ABR algorithms and any number and / or type of network throughput traces.

[0147] 5 is a flow diagram illustrating method steps for generating candidate encoding ladders for use in streaming a media title, according to various embodiments. Note that although the method steps are described with reference to the systems illustrated in FIGS. 1-3, one skilled in the art will understand that any system configured to perform the method steps in any order is within the scope of the present embodiments.

[0148] As shown, method 500 begins at step 502, in which the shot-based encoding application 132 divides a source video 102 corresponding to a media title into multiple shots. At step 504, the shot-based encoding application 132 generates multiple encoded shots based on the divided shots, any number of resolutions, and any number of sets of parameter values ​​for the encoding parameter sets. Then, at step 506, the shot-based encoding application 132 generates a convex hull of bitrate / quality assessment points for each resolution based on the encoded shots for each resolution.

[0149] In step 508, the coding ladder application 150 normalizes, if necessary, the bit rates and quality scores assigned to all bit rate / quality evaluation points in the convex hull so that they fall within the same range. Then, in step 510, the coding ladder application 150 defines a parameterized objective function 240, which represents a weighted trade-off between a weighted average of the quality scores (if necessary, normalized quality scores) of each rung in a candidate coding ladder and the sum of the bit rates (if necessary, normalized bit rates) of each rung. In step 512, the coding ladder application 150 defines parameterized constraints to be applied to the candidate coding ladder.

[0150] In step 514, the coded ladder application 150 generates one or more different ladder configurations, each with a different combination of values ​​for the total number of runs, rung quality weights, objective parameters, and constraint parameters. Then, in step 516, for each ladder configuration, the coded ladder application 150 generates an objective function based on the parameterized objective function 240 and generates constraints associated with the objective function based on the parameterized constraints.

[0151] In step 518, the encoding ladder application 150 uses a constrained optimization solver to solve the objective function corresponding to each ladder configuration under the constraints associated with the objective function to generate different candidate encoding ladders. In step 520, the encoding ladder application 150 then stores and / or transmits these candidate encoding ladders to any number and / or types of software applications for evaluation and selection of one or more candidate encoding ladders to be used as the encoding ladder for the media title. Thereafter, the method 500 ends.

[0152] 6 is a flow diagram illustrating method steps for determining an encoding ladder for a media title based on the results of an adaptive streaming simulation performed on candidate encoding ladders generated for the media title. Note that although the method steps are described with reference to the systems illustrated in Figures 1-4, one skilled in the art will understand that any system configured to perform the method steps in any order is within the scope of the present embodiments.

[0153] As shown, method 600 begins at step 602, where simulation evaluation application 170 selects a first candidate encoding ladder for a media title. At step 604, simulation evaluation application 170 generates a composite streaming header and subsequently selects a first streaming session trace based on the selected candidate encoding ladder and the bitrates and quality scores for the encoded chunks of the associated encoded video set. At step 606, simulation evaluation application 170 then runs adaptive streaming simulator 106 on the generated composite streaming header and the selected streaming session trace to determine a corresponding request sequence and set of index values.

[0154]

[0062] In step 608, the simulation evaluation application 170 determines whether the selected streaming session trace is the last streaming session trace. If in step 608 the simulation evaluation application 170 determines that the selected streaming session trace is not the last streaming session trace, the method 600 proceeds to step 610. Then, in step 610, the simulation evaluation application 170 selects the next streaming session trace. The method 600 then returns to step 606, where the simulation evaluation application 170 runs the adaptive streaming simulator 106 on the generated synthetic streaming header and the selected streaming session trace to determine a corresponding request sequence and a set of index values.

[0155] On the other hand, if the simulation evaluation application 170 determines at step 608 that the selected streaming session trace is the last streaming session trace, then the method 600 proceeds to step 612. Then, at step 612, the simulation evaluation application 170 determines whether the selected encoding ladder candidate is the last encoding ladder candidate for the media title. If, at step 612, the simulation evaluation application 170 determines that the selected encoding ladder candidate is not the last encoding ladder candidate for the media title, then the method 600 proceeds to step 614. Then, at step 614, the simulation evaluation application 170 selects the next encoding ladder candidate for the media title. Then, the method 600 returns to step 604, where the simulation evaluation application 170 generates a synthetic streaming header based on the selected encoding ladder candidate, and subsequently selects the first streaming session trace.

[0156] On the other hand, if, at step 612, the simulation evaluation application 170 determines that the selected candidate encoding ladder is the final candidate encoding ladder for the media title, then method 600 proceeds to step 616. Then, at step 616, the simulation evaluation application 170 selects one or more candidate encoding ladders for further evaluation and deployment through the production encoding pipeline 120 based on the corresponding sets of index values. At step 618, the simulation evaluation application 170 generates an encoding ladder for the media title based on the selected candidate encoding ladders. Then, at step 620, the simulation evaluation application 170 stores and / or transmits this encoding ladder for the media title to any number of software applications for processing, such as further evaluation and deployment through the production encoding pipeline 120. Thereafter, method 600 ends.

[0157] In other words, the techniques disclosed herein can be used to generate an encoding ladder for a media title with the inherent goal of improving the streaming QoE associated with the encoding ladder and reducing the storage footprint of the encoding ladder while simultaneously satisfying multiple constraints. In some embodiments, a shot-based encoding application divides source video corresponding to a media title into multiple different shots. The shot-based encoding application generates multiple encoded shots by encoding each shot using a set of multiple resolutions and a parameter set of multiple different encoding parameters. The shot-based encoding application then generates a convex hull of bitrate / quality evaluation points for each resolution based on the encoded shots corresponding to each resolution. Each convex hull optimizes the trade-off between bitrate and image quality level for the corresponding resolution. Each bitrate / quality evaluation point specifies a different encoded video and its corresponding resolution, bitrate, and quality score. Note that the resolution is constant for each encoded video, but the bitrate and quality score may differ.

[0158] The coding ladder application formulates the coding ladder generation problem as a parameterized constrained optimization problem for assigning bit rates / quality evaluation points to each rung of the coding ladder candidates. The coding ladder application also determines the constants of this parameterized constrained optimization problem based on the bit rates / quality evaluation points included in multiple convex hulls. More precisely, the coding ladder application normalizes the bit rates and image quality scores of the bit rates / quality evaluation points as needed to fall within the same range (e.g., a common range) to generate normalized bit rates and normalized quality scores. Furthermore, the coding ladder application organizes the resolutions, bit rates, and quality scores for the multiple bit rates / quality evaluation points, as well as the normalized bit rates and normalized quality scores as needed, into arrays, so that the bit rates / quality evaluation points corresponding to the elements in the arrays can be identified by indexes in each array.

[0159] The coding ladder application defines the objective and constraints of a parameterized constrained optimization problem using a parameterized objective function and parameterized constraints. The parameterized objective function represents a weighted trade-off between the weighted average of the normalized quality scores for each rung of a candidate coding ladder and the sum of the normalized bitrates for each rung. The parameters of the parameterized objective function are the number of runs, the rung quality weights, and the trade-off weights. The parameterized constraints include both intrinsic logical constraints and operational logical constraints. The intrinsic logical constraints ensure the validity of the candidate coding ladder itself. For example, a monotonically increasing bitrate constraint requires that the bitrate of coded video assigned to the rungs of a candidate coding ladder monotonically increase from the bottom rung to the top rung of the candidate coding ladder. The operational logical constraints reflect operational requirements and priorities related to client device capabilities, network capacity, CDNs, human perception of image quality, etc. For example, a parameterized bitrate spacing constraint is a constraint that the spacing between the bitrates of coded video assigned to successive stages of a candidate coding ladder must not exceed a relative bitrate spacing.

[0160] The coding ladder application generates multiple ladder configurations. Each configuration has a different combination of the number of steps, step quality weights, trade-off weights, and various parameters (e.g., relative bitrate intervals) in the parameterized constraints. For each ladder configuration, the coding ladder application generates an objective function based on the parameterized objective function and also generates constraints associated with the objective function based on the parameterized constraints. The coding ladder application then uses a constrained optimization algorithm to solve each objective function under the constraints associated with the objective function, generating a different allocation matrix for each ladder configuration. Each allocation matrix specifies the bitrate / quality score assigned to each step of a candidate coding ladder that can be used as an encoding ladder for the media title.

[0161] In some embodiments, a numerical evaluation application uses throughput distributions and bitrate demand distributions corresponding to historical streaming sessions to perform a numerical evaluation of the candidate coding ladders, and filters out any number (including zero) of candidate coding ladders that provide a poor trade-off between streaming QoE and storage footprint to generate a filtered set of candidate coding ladders.

[0162] In some embodiments, a streaming evaluation application performs a simulation-based evaluation of the coding ladder candidates included in the filtered coding ladder candidate set. For each coding ladder candidate included in the filtered coding ladder candidate set, the streaming evaluation application generates a different synthetic streaming header based on the coded chunk metadata associated with the coded video specified in the coding ladder candidate set or the coded chunk metadata estimated (e.g., based on a curve) for a hypothetical coded video associated with the coding ladder candidate. As used herein, the term "metadata estimated for a hypothetical coded video" refers to the metadata estimated for a coded video that can potentially be generated based on the source video 102. The synthetic streaming header of a given coding ladder candidate includes a different set of streaming metadata for each rung included in the coding ladder candidate. Each streaming metadata set specifies the coded chunk sequence of the corresponding coded video or the corresponding hypothetical coded video, the coded chunk bit rate, and the coded chunk quality score.

[0163] The streaming rating application emulates the operation of the ABR algorithm using an adaptive streaming simulator. The streaming rating application performs the emulation across multiple simulated streaming sessions defined by different streaming session traces, using each candidate encoding ladder in the filtered encoding ladder candidate set and its corresponding encoded chunk metadata. Each streaming session trace specifies the network throughput as a function of time for a different streaming session history. The result of each simulated emulation run is a request sequence for encoded chunks of the media title. For each request sequence, the streaming rating application calculates a different set of values ​​(value sets) for a set of metrics related to streaming QoE. The streaming rating application selects a candidate encoding ladder from the filtered encoding ladder candidate set that provides the best trade-off between streaming QoE and storage footprint across the multiple different simulated streaming sessions by performing any number and / or types of comparisons between the value sets of the metrics. In this specification, the expression "best streaming QoE to storage tradeoff" refers to a state in which the tradeoff between streaming QoE and storage footprint is closest to a target relationship between streaming QoE and storage footprint. Then, the streaming evaluation application generates an encoding ladder for the media title based on the selected encoding ladder candidates.

[0164] At least one technical advantage of the technology of the present disclosure over the prior art is that the technology of the present disclosure simultaneously considers multiple different ladder constraints from the beginning when generating an encoding ladder, thereby enabling the generation of an encoding ladder with the overall goal of increasing the quality level associated with the coded videos included in the encoding ladder while minimizing the storage footprint of the encoding ladder. This approach enables the generation of an encoding ladder to identify and exploit opportunities for using a single coded video that simultaneously satisfies multiple different ladder constraints, thereby improving the trade-off between the weighted average quality level associated with the coded videos included in the encoding ladder and the storage footprint of the encoding ladder. As a result, the trade-off between the streaming quality experience provided by the encoding ladder used for a given media title and the storage footprint of the encoding ladder can be significantly improved compared to the prior art. These technical advantages result in one or more technical improvements over prior art approaches.

[0165] Section 1 In some embodiments, a computer-implemented method for evaluating candidate encoding ladders for use in streaming a media title includes generating a first streaming header based on a first plurality of runs associated with a first candidate encoding ladder, wherein each rung in the first plurality of runs specifies a resolution and bit rate of a different encoded video included in a plurality of encoded videos; running an adaptive bitrate algorithm on the first streaming header based on a first network throughput trace to determine a first index value for a first index related to quality of experience; generating a second streaming header based on a second plurality of runs associated with a second candidate encoding ladder; running the adaptive bitrate algorithm on the second streaming header based on the first network throughput trace to determine a second index value for the first index; and determining that the first candidate encoding ladder should be used to stream the media title rather than the second candidate encoding ladder by comparing the first index value and the second index value.

[0166] Section 2 2. The computer-implemented method of claim 1, wherein generating the first streaming header includes determining an encoded chunk sequence based on a first encoded video specified in a first stage included in the first plurality of stages, and determining a plurality of bit rates associated with the encoded chunk sequence.

[0167] Section 3 3. The computer-implemented method of claim 1 or 2, wherein the step of performing the adaptive bitrate algorithm on the second streaming header includes: generating a first request for a first encoded chunk based on a first network throughput specified in the first network throughput trace; and calculating the first metric value based on a first quality score associated with the first encoded chunk.

[0168] Section 4 A computer-implemented method as described in any of claims 1 to 3, wherein the first indicator value represents at least one of a quality score, a total number of rebuffering events, or a total rebuffering time associated with streaming of the media title using the first streaming header.

[0169] Section 5 5. The computer-implemented method of any one of claims 1 to 4, wherein the quality score includes an average peak signal-to-noise ratio, an average multi-method fusion video assessment score, or a time-weighted video multimedia assessment fusion score.

[0170] Section 6 6. A computer-implemented method according to any one of claims 1 to 5, wherein the first metric value approximates a trade-off between streaming quality of experience and at least one of storage footprint or network bandwidth consumption associated with the first encoding ladder candidate.

[0171] Section 7 A computer-implemented method according to any one of claims 1 to 6, wherein the first coding ladder candidate and the second coding ladder candidate are included in a plurality of coding ladder candidates generated based on a parameterized objective function and a plurality of parameterized constraints.

[0172] Section 8 8. The computer-implemented method of any of claims 1 to 7, wherein the first network throughput trace comprises recorded measurements of one or more characteristics of the first network over a first period of time.

[0173] Section 9 9. The computer-implemented method of any of claims 1 to 8, wherein a first number of stages included in the first plurality of stages is different from a second number of stages included in the second plurality of stages.

[0174] Section 10 A computer-implemented method as described in any of claims 1 to 9, further comprising the step of performing one or more additional processes on the first encoding ladder candidate to generate a final encoding ladder for use in streaming the media title to one or more client devices over a network.

[0175] Section 11 In some embodiments, one or more non-transitory computer-readable media include instructions that, when executed by one or more processors, cause the one or more processors to evaluate candidate encoding ladders for use in streaming a media title. The instructions cause the one or more processors to perform the following steps to evaluate the encoding ladder candidates: generating a first streaming header based on a first plurality of stages associated with a first encoding ladder candidate, where each stage in the first plurality of stages specifies a resolution and bit rate of a different encoded video included in a plurality of encoded videos; running an adaptive bitrate algorithm on the first streaming header based on a first network throughput trace to determine a first indicator value for a first indicator related to quality of experience; generating a second streaming header based on a second plurality of stages associated with a second encoding ladder candidate; running the adaptive bitrate algorithm on the second streaming header based on the first network throughput trace to determine a second indicator value for the first indicator; and determining that the first encoding ladder candidate should be used for streaming the media title instead of the second encoding ladder candidate by comparing the first indicator value and the second indicator value.

[0176] Section 12 12. The one or more non-transitory computer-readable media of claim 11, wherein generating the first streaming header includes determining an encoded chunk sequence based on a first encoded video specified in a first stage included in the first plurality of stages, and determining a plurality of quality scores associated with the encoded chunk sequence.

[0177] Section 13 13. The one or more non-transitory computer-readable media of claim 11 or 12, wherein executing the adaptive bitrate algorithm on the first streaming header includes generating a sequence of requests for a sequence of encoded chunks based on the first streaming header and a network throughput sequence included in the first network throughput trace.

[0178] Section 14 One or more non-transitory computer-readable media described in any of paragraphs 11 to 13, wherein the first indicator value represents at least one of a quality score, a total number of rebuffering events, or a total rebuffering time associated with streaming of the media title using the first streaming header.

[0179] Section 15 One or more non-transitory computer-readable media described in any of paragraphs 11 to 14, wherein the second indicator value represents at least one of a quality score, a total number of stage switches, or a frequency of stage switches associated with streaming of the media title using the second streaming header.

[0180] Section 16 One or more non-transitory computer-readable media described in any of paragraphs 11 to 15, wherein the first metric value approximates a trade-off between streaming quality of experience and at least one of storage footprint or network bandwidth consumption associated with the first encoding ladder candidate.

[0181] Section 17 One or more non-transitory computer-readable media described in any of paragraphs 11 to 16, wherein the first coding ladder candidate and the second coding ladder candidate are included in a plurality of coding ladder candidates generated based on a parameterized objective function and a plurality of parameterized constraints.

[0182] Section 18 18. One or more non-transitory computer-readable media described in any of clauses 11-17, wherein the first network throughput trace includes recorded measurements of one or more characteristics of the first network over a first period of time.

[0183] Section 19 19. The one or more non-transitory computer-readable media of any of claims 11 to 18, wherein a first resolution specified in a first stage included in the first plurality of stages is different from a second resolution specified in a second stage included in the first plurality of stages.

[0184] Section 20 In some embodiments, a system comprises one or more memories storing instructions and one or more processors coupled to the one or more memories, wherein executing the instructions causes the one or more processors to perform the following steps: generating a first streaming header based on a first plurality of stages associated with a first encoding ladder candidate, wherein each stage in the first plurality of stages specifies a resolution and a bitrate of a different encoded video included in a plurality of encoded videos; running an adaptive bitrate algorithm on the first streaming header based on a first network throughput trace to determine a first index value for a first index related to quality of experience; generating a second streaming header based on a second plurality of stages associated with a second encoding ladder candidate; running the adaptive bitrate algorithm on the second streaming header based on the first network throughput trace to determine a second index value for the first index; and determining that the first encoding ladder candidate should be used for streaming a media title rather than the second encoding ladder candidate by comparing the first index value and the second index value.

[0185] Any combination of any element recited in any claim and / or any element described in this specification, regardless of the form of such combination, is intended to fall within the scope and protection contemplated by the present invention.

[0186] The various embodiments described above have been presented for illustrative purposes and are not intended to be exhaustive or limiting of the embodiments of the present disclosure. Those skilled in the art will recognize that many modifications and variations can be made without departing from the scope and spirit of the embodiments described herein.

[0187] Aspects of the present disclosure may be embodied as a system, method, or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be collectively referred to herein as a "module" or a "system." Furthermore, aspects of the present disclosure may take the form of a computer program product embodied as computer-readable program code recorded on one or more computer-readable medium(s).

[0188] One or more computer-readable media may be utilized in any combination. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific (but non-exhaustive) examples of computer-readable storage media include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. As used herein, a computer-readable storage medium may refer to any tangible medium capable of storing a program for use by or in connection with a system, apparatus, or device that executes instructions.

[0189] Aspects of the present disclosure have been described above with reference to flowchart diagrams and / or block diagrams illustrating methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart diagrams and / or block diagrams, and combinations of blocks in the flowchart diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a programmable data processing device, such as a general-purpose computer or a special-purpose computer, to produce a machine. The instructions, when executed by the processor of the computer or programmable data processing device, cause the implementation of the functions or acts specified in one or more blocks of the flowchart diagrams and / or block diagrams. Such a processor may include, but is not limited to, a general-purpose processor, a special-purpose processor, an application-specific processor, or a field-programmable gate array.

[0190] The flowcharts and block diagrams shown in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can be considered to represent a module, segment, or portion of code, including one or more executable instructions for implementing the logical function(s) defined in the block. It should also be noted that in some alternative implementations, the functions shown in the blocks can be executed in an order different from that shown in the figures. For example, two blocks shown as consecutive blocks may actually be executed substantially simultaneously, or in some cases, the blocks may be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by a dedicated hardware-based system that performs the functions or operations defined in the block, or a combination of dedicated hardware and computer instructions.

[0191] It should be noted that while the foregoing description is directed to embodiments of the present disclosure, other and further embodiments of the present disclosure may be devised without departing from the basic scope thereof, which scope is defined by the claims set forth below. [Explanation of symbols]

[0192] 100 systems 102 Source footage 104 Streaming Session History Database 106 Adaptive Streaming Simulator 110 compute instances 112 processors 116 memory 120 Production Encoding Pipeline 122 Throughput Distribution 124 Bitrate Demand Distribution 126 Streaming Session Traces 132 Shot-Based Coding Applications 136 Convex Hull Metadata 138 Encoded Chunk Metadata 140 Coding Ladder Workflow 150 Coding Ladder Applications 158 Coding Ladder Candidate Set 160 Numerical Evaluation Applications 168 Filtered Encoding Ladder Candidate Set 170 Simulation Evaluation Applications 178 Coding Ladder 180 steps 182, 212, 434 Encoded Video ID 184, 214, 442 resolution 186, 216, 444 bitrate 190 Ladder Deployment Application 210 bitrate / quality score 218 Quality Score 220 Normalization Engine 222 ID array 224 resolution array 226 bitrate array 228 Quality Score Array 230 Encoding Point Metadata 236 Normalized Bitrate Array 238 Normalized Quality Score Array 240 Parameterized Objective Function 250 parameterized constraint sets 260 Ladder Configuration 262 total steps 264-stage quality weight set 266 Objective Parameter Set 268 Constraint Parameter Set 270 Stage Allocation Engine 280 coding ladder candidates 322 Quality section 324 Footprint Section 340 Objective Function 350 Constraints 360 Constrained Optimization Solver 370 Allocation Matrix 372 Evaluation Point Axis 374 column axis 406 Adaptive Bitrate (ABR) Algorithm 410 Streaming Header Synthesis Engine 430 Synthetic Streaming Header 432 Streaming Metadata Set 436 Encoded Chunk IDs 446 encoded chunk bitrates 448 Encoded Chunk Quality Scores 450 Simulation Configuration 460 Request Sequence 470 Index Value Set 480 Ladder Evaluation and Selection Engine

Claims

1. 1. A computer-implemented method for evaluating candidate encoding ladders for use in streaming a media title, the method comprising: generating a first streaming header based on a first plurality of stages associated with a first candidate encoding ladder, each stage in the first plurality of stages specifying a resolution and bit rate of a different encoded video in the plurality of encoded videos; performing an adaptive bitrate algorithm on the first streaming header based on a first network throughput trace to determine a first metric value for a first metric related to quality of experience; generating a second streaming header based on a second plurality of runs associated with the second candidate encoding ladder; executing the adaptive bitrate algorithm on the second streaming header based on the first network throughput trace to determine a second metric value for the first metric; determining that the first candidate encoding ladder should be used for streaming the media title rather than the second candidate encoding ladder by comparing the first index value with the second index value; A method comprising:

2. generating the first streaming header comprises: determining a coded chunk sequence based on a first coded video designated in a first stage included in the first plurality of stages; determining a plurality of bit rates associated with the encoded chunk sequence; 10. The computer-implemented method of claim 1, comprising:

3. performing the adaptive bitrate algorithm on the second streaming header, generating a first request for a first encoded chunk based on a first network throughput specified in the first network throughput trace; calculating the first metric value based on a first quality score associated with the first encoded chunk; 10. The computer-implemented method of claim 1, comprising:

4. 2. The computer-implemented method of claim 1, wherein the first indicator value represents at least one of a quality score, a total number of rebuffering events, or a total rebuffering time associated with streaming of the media title using the first streaming header.

5. The computer-implemented method of claim 4 , wherein the quality score comprises an average peak signal-to-noise ratio, an average multi-method fusion video assessment score, or a time-weighted video multimedia assessment fusion score.

6. 2. The computer-implemented method of claim 1, wherein the first metric value approximates a trade-off between streaming quality of experience and at least one of storage footprint or network bandwidth consumption associated with the first encoding ladder candidate.

7. 2. The computer-implemented method of claim 1, wherein the first candidate coding ladder and the second candidate coding ladder are included in a plurality of candidate coding ladders generated based on a parameterized objective function and a plurality of parameterized constraints.

8. 10. The computer-implemented method of claim 1, wherein the first network throughput trace comprises recorded measurements of one or more characteristics of a first network over a first period of time.

9. 2. The computer-implemented method of claim 1, wherein a first number of stages in the first plurality of stages is different from a second number of stages in the second plurality of stages.

10. 10. The computer-implemented method of claim 1, further comprising performing one or more additional processes on the first candidate encoding ladder to generate a final encoding ladder for use in streaming the media title to one or more client devices over a network.

11. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to evaluate candidate encoding ladders for use in streaming a media title, the instructions causing the one or more processors to: generating a first streaming header based on a first plurality of stages associated with a first candidate encoding ladder, each stage in the first plurality of stages specifying a resolution and bit rate of a different encoded video in the plurality of encoded videos; performing an adaptive bitrate algorithm on the first streaming header based on a first network throughput trace to determine a first metric value for a first metric related to quality of experience; generating a second streaming header based on a second plurality of runs associated with the second candidate encoding ladder; executing the adaptive bitrate algorithm on the second streaming header based on the first network throughput trace to determine a second metric value for the first metric; determining that the first candidate encoding ladder should be used for streaming the media title rather than the second candidate encoding ladder by comparing the first index value with the second index value; and one or more non-transitory computer-readable media for performing the evaluation of the candidate coding ladders by causing the evaluation of the candidate coding ladders to execute.

12. generating the first streaming header comprises: determining a coded chunk sequence based on a first coded video designated in a first stage included in the first plurality of stages; determining a plurality of quality scores associated with the encoded chunk sequence; 12. The one or more non-transitory computer-readable media of claim 11, comprising:

13. 12. The one or more non-transitory computer-readable media of claim 11, wherein performing the adaptive bitrate algorithm on the first streaming header comprises generating a sequence of requests for a sequence of encoded chunks based on the first streaming header and a network throughput sequence included in the first network throughput trace.

14. 12. The one or more non-transitory computer-readable media of claim 11, wherein the first metric value represents at least one of a quality score, a total number of rebuffering events, or a total rebuffering time associated with streaming the media title using the first streaming header.

15. 12. The one or more non-transitory computer-readable media of claim 11, wherein the second metric value represents at least one of a quality score, a total number of stage switches, or a frequency of stage switches associated with streaming of the media title using the second streaming header.

16. 12. The one or more non-transitory computer-readable media of claim 11, wherein the first metric value approximates a trade-off between streaming quality of experience and at least one of storage footprint or network bandwidth consumption associated with the first encoding ladder candidate.

17. 12. The one or more non-transitory computer-readable media of claim 11, wherein the first candidate coding ladder and the second candidate coding ladder are included in a plurality of candidate coding ladders generated based on a parameterized objective function and a plurality of parameterized constraints.

18. 12. The one or more non-transitory computer-readable media of claim 11, wherein the first network throughput trace comprises recorded measurements of one or more characteristics of a first network over a first period of time.

19. 12. The one or more non-transitory computer-readable media of claim 11, wherein a first resolution assigned to a first stage in the first plurality of stages is different from a second resolution assigned to a second stage in the first plurality of stages.

20. one or more memories for storing instructions; one or more processors coupled to the one or more memories; A system comprising: Execution of the instructions causes the one or more processors to: generating a first streaming header based on a first plurality of stages associated with a first candidate encoding ladder, each stage in the first plurality of stages specifying a resolution and bit rate of a different encoded video in the plurality of encoded videos; performing an adaptive bitrate algorithm on the first streaming header based on a first network throughput trace to determine a first metric value for a first metric related to quality of experience; generating a second streaming header based on a second plurality of runs associated with the second candidate encoding ladder; executing the adaptive bitrate algorithm on the second streaming header based on the first network throughput trace to determine a second metric value for the first metric; determining that the first candidate encoding ladder should be used for streaming the media title rather than the second candidate encoding ladder by comparing the first index value with the second index value; To run the system.

Images