Optimal multi-codec ABR ladder design

Abstract

To provide a technique for the creation of multi-codec encoding profiles (or encoding ladders) that define the quality and bit rate for each of the streams made available to a client for streaming video.SOLUTION: In particular, optimization techniques can take into account the quality-rate function of each of the codecs when determining the encoding ladder. Further considerations may include network bandwidth distribution and / or client type distribution.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] Adaptive Bit Rate (ABR) streaming is a method of streaming video content in which the bit rate of the video stream delivered to a streaming client can be adjusted during playback to accommodate changes in available network bandwidth. To enable this functionality, an ABR streaming system may encode the source content into multiple streams at different bit rates. In this way, a streaming client can switch between different streams while streaming video, effectively receiving a composite stream that scales with the available network bandwidth.

[0002] The composition of streams into which source content is encoded can be determined by the ABR streaming system. In conventional ABR streaming systems, this decision is typically made independently for each codec. In other words, for each codec, an entirely new set of streams is generated to cover the range of bit rates required to accommodate the network. This results in significantly higher encoding and distribution costs. However, because many streaming clients can now switch between streams with different codecs, finding an optimal multi-codec composition of streams sufficient for ABR distribution can minimize this inefficiency. Summary of the Invention [Means for solving the problem]

[0003] The techniques described herein provide for the creation of a multi-codec encoding profile (or encoding ladder) that defines the quality and bitrate for each of the streams made available to a client for streaming video. In particular, optimization techniques can take into account the quality-rate function of each of the codecs when determining the encoding ladder. Further considerations can include network bandwidth distribution and / or client type distribution.

[0004] An exemplary method for creating a multi-codec encoding ladder according to this specification includes obtaining, by a computer system, source content including video, and generating an encoding ladder for the source content, wherein each video stream of a plurality of video streams defined by the encoding ladder includes a respective bit rate and a respective codec of a plurality of codecs for encoding the source content.

[0005]

number

[0006]

number

[0007]

number

[0008]

number

[0009]

number

[0010]

number

[0011]

number

[0012]

number

[0013] An exemplary computer system for creating a multi-codec encoding ladder according to the present disclosure includes a memory and one or more processing units communicatively coupled to the memory. The one or more processing units obtain source content including video, generate an encoding ladder for the source content, wherein each video stream of a plurality of video streams defined by the encoding ladder includes a respective bit rate and a respective codec of a plurality of codecs for encoding the source content, and the encoding ladder includes:

[0014]

number

[0015]

number

[0016]

number

[0017]

number

[0018]

number

[0019]

number

[0020]

number

[0021]

number

[0022] An exemplary non-transitory computer-readable medium according to the present disclosure has stored therein instructions for creating a multi-codec encoding ladder, the instructions, when executed by one or more processing units, causing the one or more processing units to obtain source content including video, generate an encoding ladder for the source content, wherein each video stream of a plurality of video streams defined by the encoding ladder includes a respective bitrate and a respective codec of a plurality of codecs for encoding the source content, and the encoding ladder

[0023]

number

[0024]

number

[0025]

number

[0026]

number

[0027]

number

[0028]

number

[0029]

number

[0030]

number

[0031] [Figure 1] 1 illustrates an ABR streaming system according to one embodiment. [Figure 2] 1 is a graph plotting available network bandwidth and client streaming rate. [Figure 3] 1 is a conceptual diagram of an ABR streaming system comprising a video source, two encoders, and three decoders with accompanying rate selectors, according to one embodiment. [Figure 4A-4B] 4 is a graph illustrating the video quality that different decoders (eg, the decoder of FIG. 3) can achieve at various bit rates, according to one embodiment. [Figure 5] FIG. 1 is a conceptual diagram of an ABR streaming system used in an exemplary embodiment. [Figure 6] FIG. 2 is a flow diagram illustrating the main steps of a method for determining an optimized multi-codec encoding ladder according to one embodiment. [Figure 7] 10 is a graph illustrating the shape of the obtained quality-rate function for a codec and exemplary source content; [Figure 8] 1 is a graph of network parameters for two network models used to obtain the experimental results described herein. [Figure 9]FIG. 1 is a conceptual diagram of a single-codec ABR streaming system used to obtain the experimental results described herein. [Figure 10] FIG. 1 is a conceptual diagram of a dual-codec ABR streaming system used to obtain the experimental results described herein. [Figure 11] 1 is a graph showing points on the encoding ladder and switching decisions made by an H.264 baseline client and an H.264 baseline / main switchable client. [Figure 12] FIG. 1 is a conceptual diagram of a dual-codec ABR streaming system with three client types used to obtain the experimental results described herein. [Figure 13] FIG. 1 is a conceptual diagram of a multi-codec ABR streaming system with four client types used to obtain the experimental results described herein. [Figure 14] 1 is a block diagram of a multi-codec ABR streaming system incorporating multi-codec ABR ladder generation using the methods described herein, according to one embodiment. [Figure 15] 1 is a flowchart of a method for determining a set of monotonically increasing points (streams in an encoding ladder) in terms of rate and quality. [Figure 16] FIG. 1 is a flow diagram illustrating a method for creating a multi-codec encoding ladder, according to one embodiment. [Figure 17] FIG. 1 is a block diagram of one embodiment of a computer system. DETAILED DESCRIPTION OF THE INVENTION

[0032] Like reference numerals in the various figures indicate like elements according to particular exemplary implementations. Additionally, multiple instances of an element may be designated using the first numeral of the element followed by a letter or hyphen and a second numeral. For example, multiple instances of element 110 may be designated as 110-1, 110-2, 110-3, etc., or as 110a, 110b, 110c, etc. When such an element is referred to by only the first numeral, it should be understood to refer to any instance of that element (e.g., element 110 in the previous example refers to elements 110-1, 110-2, and 110-3, or elements 110a, 110b, and 110c).

[0033] Detailed Description Certain exemplary embodiments will now be described with reference to the accompanying drawings, which form a part hereof. Specific embodiments in which one or more aspects of the present disclosure may be practiced are described below, but other embodiments may be used, and various changes may be made, without departing from the spirit of the scope of the disclosure or the appended claims.

[0034] Figure 1 illustrates an ABR streaming system 100 according to one embodiment. The ABR streaming system 100 includes a video source 110, an encoder 120, an origin server 130, a content delivery network (CDN) + network access 140, and a streaming client 150. As will be appreciated by those skilled in the art, different embodiments may include different numbers of each of the illustrated components. For example, a CDN + network access may serve many (e.g., tens, hundreds, thousands, or more) streaming clients 150.

[0035] To enable delivery of source content (e.g., one or more media files) stored in video source 110 to one or more streaming clients 150, encoder 120 may encode the source content into multiple streams with different bit rates. (For example, as shown in callout 160, the encoder may encode the source content to provide M bit rates and separate descriptions.) Each encoded stream may incorporate random access points (e.g., intra-frame (I) frames or instant decoding refresh (IDR) frames in the encoded video) to enable switching between streams. Such streams are then placed on origin server 130 and further pushed to CDN+Network Access 140 (which may include not only a CDN but also one or more data communication networks, such as the Internet) for scaled delivery to streaming clients 150.

[0036] During playback, each streaming client 150 may monitor the rate at which encoded content arrives. (For example, as shown in callout 170, the streaming client may estimate the bandwidth and then select an appropriate rate for the next segment, taking the bandwidth into account, before retrieving the next segment.) If such a rate becomes insufficient for continuous playback, the client may switch to a lower bitrate stream, thereby preventing buffering. On the other hand, if such a rate is greater than the bitrate of the current stream, the client may switch to a higher bitrate stream to provide better quality to the end user. Such switching mechanisms have since become widely adopted and are incorporated into all modern streaming protocols, such as HyperText Transfer Protocol (HTTP) Live Streaming (HLS) and MPEG Dynamic Adaptive Streaming over HTTP (DASH).

[0037] Thus, as a result, the streaming bitrate of video to streaming client 150 adapts to changes in available network bandwidth over time. For example, as shown in the graph of FIG. 2, streaming bitrate 210 may increase as available network bandwidth 220 increases and decrease as available network bandwidth 220 decreases. These changes in streaming bitrate 210 result from streaming client 150 switching from a stream having a first bitrate to a stream having a second bitrate. Thus, the more streams (at different bitrates) that encoder 120 creates for a given source content, the more finely tuned the changes in streaming bitrate 210 can be.

[0038] The configuration of video stream characteristics used for ABR streaming, such as bitrate, resolution, and codec constraints, is typically called an encoding profile or ladder. An exemplary encoding ladder for the High Efficiency Video Coding (HEVC) and H.264 / MPEG-4 AVC (or simply "H.264") codecs is set forth in Table 1 below, which is also set forth in the Apple® HLS deployment guidelines.

[0039] [Table 1] Recently, it has also been discovered that the performance of an ABR streaming system 100 can be improved by using a dynamic ladder generator that creates a custom encoding ladder that takes into account the rate-distortion characteristics of the content and / or the characteristics of the network used to deliver the stream. Such approaches have been known as "per-title," "content-aware encoding," and "context-aware encoding" techniques. Further information regarding ladder generators for ABR streaming is described in U.S. patent application Ser. No. 15 / 829,723, entitled "Optimization of Encoding Profiles for Media Streaming" (herein referred to as the "'723 application"), which is incorporated herein by reference in its entirety for all purposes.

[0040] Until recently, there was only one codec in the encoding ladder for ABR streaming. (In most cases, that codec was H.264, which is ubiquitous and has been supported by most existing devices for the past decade.) However, with the introduction of more codecs, such as VP9, ​​HEVC, and AV1, ABR streaming systems 100 generally must re-encode the entire source content for all additional supported codecs and create a new ABR ladder with all streams encoded using the new codec to help ensure that the content can be streamed reliably to a variety of devices that support different codecs. Current versions of streaming guidelines (e.g., the HLS guidelines and the DASH-IF implementation guidelines) define one ladder (a set of streams with a certain bitrate and resolution) for H.264 and another ladder for HEVC.

[0041] As previously mentioned, the deployment of multiple codecs for ABR streaming is currently based on the assumption that each codec has a separate ABR encoding ladder and a corresponding separate set of streams. Thus, the ABR streaming system 100 generates an encoding ladder for H.264 separately from the encoding ladder for HEVC.

[0042] Unfortunately, however, using such a single-codec encoding ladder is fundamentally suboptimal for at least three reasons: First, by generating the ladders separately, there may be no way to find the appropriate balance between the number of renditions allocated to the ABR encoding ladder associated with each codec. Thus, encoder 120 may generate as many renditions as it deems necessary for each codec without considering the fact that usage of such codecs across a set of viewers may vary. For example, the number of streaming clients 150 that support HEVC-encoded video may be much smaller than the number of streaming clients 150 that support H.264. Also, in an ABR streaming system 100 that has a fixed total budget for the number of renditions that can be generated, allocating more to H.264 may have a greater overall impact on the total quality delivered to end users.

[0043] Second, based on content characteristics, the coding gains of HEVC versus H.264 can vary significantly. This, in turn, can affect the balance of the number of renditions that should be used for each codec. For example, in one extreme scenario, all HEVC-enabled streaming clients 150 can also decode H.264, so an optimal ABR ladder design may not allocate any renditions to HEVC if HEVC does not offer any gain. Therefore, switching to H.264 does not reduce the system's reach.

[0044] Third, many new streaming clients 150 can switch between streams with both H.264 and HEVC codecs. Given this switching ability, such clients will likely be able to achieve better performance than clients that select either H.264 or HEVC streams, because they have more streams available. (Again, as noted with respect to FIG. 2, when more streams are available, the streaming bitrate 210 can adapt to changes in available network bandwidth 220 using finer steps. This greater granularity can make the ABR streaming system 100 more efficient.)

[0045] The embodiments described herein provide an optimized solution for ABR profile / ladder generation for multiple codec ladders. That is, the techniques provided describe an optimal multi-codec ABR streaming ladder generator implemented as software, hardware, or a combination thereof, and an ABR streaming system incorporating such a ladder generator. Among the benefits provided by the techniques herein are improved quality and / or reduced operational costs for ABR streaming systems.

[0046] FIG. 3 is a conceptual diagram of an ABR streaming system comprising a video source 110, two types of encoders (encoder 1 and encoder 2), and three types of clients each comprising decoders (decoder 1, decoder 2, and decoder 3) with accompanying rate selectors, according to one embodiment. Note that the encoders shown in FIG. 3 may correspond to a type of encoder 120 of FIG. 1 that may be implemented by one or more computers that ingest source content from video source 110 (which may also comprise one or more computers). Furthermore, each of clients 310 may correspond to a different type of streaming client 150 of FIG. 1 that may be implemented by an end-user device (e.g., a computer, a mobile phone, a television, etc.). (Note that the terms “client” and “decoder” are often used interchangeably herein when referring to the mathematical description of an ABR streaming system described below.) In this embodiment, and for purposes of establishing the mathematical description of one embodiment, decoders 1 and 2 are only capable of decoding the streams generated by encoders 1 and 2, respectively. Decoder 3 may select and decode a stream generated by either encoder 1 or encoder 2.

[0047] As used herein, the term "single-codec client" refers to a client (such as a client with decoder 1 or 2) that can decode video streams encoded with a single codec (e.g., from either encoder 1 or encoder 2). Similarly, the term "dual-codec client" refers to a client (such as a client with decoder 3) that can switch between video streams encoded with two different codecs (e.g., video streams encoded by either encoder 1 or encoder 2). Similarly, the term "switching client" may be used to refer to a decoder that can decode video encoded streams with more than one codec (from more than one encoder).

[0048] To provide a mathematical description of the ABR streaming system of Figure 3, the variable R is used to represent the bitrate and Q is used to represent the quality value achievable by the video codec. Here, the quality value Q is normalized so that a value Q = 0 represents the worst possible quality and Q = 1 represents an ideal reconstruction. A well-known example of a quality metric that satisfies such constraints is the Structural Similarity Index Measure (SSIM) metric, but in principle it could also be any other metric with a specific normalization applied, such as Peak Signal-to-Noise Ratio (PSNR), Multi-scale SSIM (MS-SSIM), or Video Multi-Method Assessment Fusion (VMAF).

[0049] For a given source content, encoders 1 and 2 each produce a set of encoded streams with the following (quality, rate) characteristics:

[0050]

number

[0051]

number

[0052] Here, the performance of the codec is modeled by specific quality-rate functions Q1(R) and Q2(R). The (quality, rate) points above (corresponding to different streams) can be understood as samples taken from these functions:

[0053]

number

[0054]

number

[0055] For notational convenience, such ladders always have a zero point that is the same for both codecs. (R 0 ,Q 0 )=(0,0) (5) can be extended by

[0056] In fact, one of the video parameters that may be similarly varied at different bit rates is resolution. In the embodiments described herein, resolution may be optimized and may be assumed to be captured by the quality-rate model of each codec. In other words, a set of allowed resolutions

[0057]

number

[0058]

number

[0059]

number

[0060] Modern streaming protocols, such as HLS or DASH, are based on using the Transmission Control Protocol (TCP) as the underlying transport protocol. TCP then performs retransmissions to eliminate packet loss and mask many natural statistics inherent in each type of physical network. However, at the TCP level, fluctuations in the transmission rate or bandwidth available at each point in time can still be observed.

[0061] Therefore, for the purposes of mathematical modeling, the network can be viewed as a continuous random variable R with a particular given probability density function p(R). In practice, such bandwidth density function p(R) may be different for different devices or their respective access networks. For example, considering a mobile client connected via a 4G / Long Term Evolution (LTE) network, known throughput measurements of TCP traffic over LTE may be used. More generally, such distributions can be measured experimentally given each specific streaming deployment and may, of course, be different for different devices, CDNs, distribution regions, etc.

[0062] The client model can be defined as follows: At any instant in time, given a particular available network bandwidth R, decoders 1 and 2 (single-codec clients) in Figure 3 receive from ladders L1 and L2, respectively,

[0063]

number

[0064]

number

[0065] In other words, decoders 1 and 2 are configured to operate at a maximum ladder rate R less than or equal to the available network bandwidth R. i Select. Therefore, the quality achieved by each decoder is

[0066]

number

[0067]

number

[0068] In practice, the rate selection algorithm in the streaming client may be more complex, but the above selection model may nevertheless be suitable for considering the average performance of a streaming system.

[0069] With reference to decoder 3 (of FIG. 3), for each bandwidth value R, decoder 3 can choose both the bit rate and codec that achieves the next best quality.

[0070]

number

[0071]

number

[0072] 4A and 4B are graphs showing the video quality that different clients / decoders can achieve at various bit rates. In FIG. 4A, the HEVC quality-rate function 310 (Q HEVC(R)) and H.264 quality rate function 320 (Q H.264 (R)) is plotted to show the achievable quality (SSIM) for a given source content over bit rates ranging from 0 to 3500 kbps. It can be seen that the plot of the HEVC quality-rate function 310 exceeds the plot of the H.264 quality-rate function 320, suggesting that HEVC is more efficient for a given source content.

[0073] As the bitrate increases, the quality selected by the HEVC decoder 330 and the quality selected by the H.264 decoder 340 each exhibit steps of increasing quality, indicating where the respective decoders switch from a stream with a lower bitrate / quality to a stream with a higher bitrate / quality. For example, the H.264 encoding ladder includes five bitrate points: 71, 268, 595, 1108, and 2149 kbps, respectively, resulting in a step-like function being exhibited by the quality selected by the H.264 decoder 340. The HEVC encoding ladder includes three bitrate points: 93, 459, and 1275 kbps, respectively, resulting in a step-like function being exhibited by the quality selected by the HEVC decoder 330. (The HEVC decoder can only select from among the three available HEVC rates.) FIG. 4B shows the quality selected by the dual-codec decoder 350 overlaid on the graph of FIG. 4A. As can be seen, the quality selected by the dual-codec decoder 350 partially coincides with the steps of both the H.264 and HEVC decoders, which select the highest quality available at each rate. Both codecs are used alternately, resulting in a total of seven steps. This allows the dual-codec decoder / client to more accurately adapt to changing network bandwidth, thereby achieving better network utilization than if the decoder were to work solely with either the H.264 or HEVC streams. Importantly, however, the dual-codec decoder can also skip points where switching is not worthwhile because there is no improvement in quality. For example, instead of using the 595 kbps H.264 point (which has lower quality than the 459 kbps HEVC), the dual-codec decoder stays at the 459 kbps HEVC.

[0074] Based on this, the conditions under which a dual-codec decoder / client achieves better performance can be formulated, where given ladders L1(1) and L2(2) of streams encoded using the first and second codecs respectively,

[0075]

number

[0076]

number

[0077]

number

[0078]

number

[0079]

number

[0080]

number

[0081]

number

[0082]

number

[0083]

number

[0084]

number

[0085] The average quality achievable by a dual-codec ABR streaming system can be determined as follows: Given the rate selection rule described above, and assuming that the network bandwidth is modeled as a continuous random variable R with probability density function p(R), an expression for the average quality achievable by three types of decoders in a streaming system can be written as follows:

[0086]

number

[0087]

number

[0088]

number

[0089]

number

[0090]

number

[0091]

number

[0092] Finally, by assuming that π={π1,π2,π3}, π1+π2+π3=1 is a distribution describing the presence of each type of client in the universe of clients, the overall average quality that the streaming system can achieve can be expressed as:

[0093]

number

[0094] Considering equations (1) to (16), and the average quality value

[0095]

number

[0096]

number

[0097]

number

[0098]

number

[0099]

number

[0100]

number

[0101]

number

[0102]

number

[0103] As can be easily noticed, the problem described in equation (17) is a nonlinear constrained optimization problem,

[0104]

number

[0105]

number

[0106] All the constraints introduced in (17) can be used in practical settings, e.g., the maximum rate limit R max prevents allocation of bit rates beyond what is physically possible. The minimum rate limit R min is usually related to the minimum quality level at which streaming as a service is still achievable.

[0107]

number

[0108] The problem formulated in equation (17) works with n total streams in a streaming system. However, if the number of streams is allowed to approach infinity, the resulting quality bound at the output of each decoder becomes

[0109]

number

[0110]

number

[0111]

number

[0112]

number

[0113] The relative distance between the ideal quality value and the best average quality achievable in an n-point system (called the "quality gap") can be defined as:

[0114]

number

[0115]

number

[0116]

number

[0117] In addition to the average quality, the average bandwidth consumed by each client can be expressed as:

[0118]

number

[0119]

number

[0120]

number

[0121]

number

[0122]

number

[0123] In principle, given all the definitions above, and considering that in practice bandwidth is usually a factor in the operational cost of a streaming system, the optimal ladder design problem can also be formulated as a problem of minimizing the average bandwidth.

[0124]

number

[0125] However, note that these problems are related and in some cases lead to exactly the same solution. Hence, for a given n and all other constraints, we find a quality bound that coincides with the solution of problem (17), i.e., the best quality achievable in such a system.

[0126]

number

[0127]

number

[0128] More generally, consider an ABR streaming system with k codecs and m clients, where the network bandwidth distribution p and client types π are known, and define a particular figure of merit function as the optimization criterion.

[0129]

number

[0130] Then, and under certain further conditions, the optimization problem is

[0131]

number

[0132]

number

[0133]

number

[0134]

number

[0135]

number

[0136]

number

[0137]

number

[0138] 6 is a flow diagram illustrating the main steps of a method for determining an optimized multi-codec encoding ladder according to one embodiment, in which the problem defined by equations (31) and (32) is solved. Below, we present specific examples of practical multi-codec systems and the respective solutions found by applying the proposed method. Some or all of the functions shown in the blocks of FIG. 6 may be performed by the encoder 120 (which, as mentioned above, may be performed by a computer server).

[0139] 6 may begin at block 605, which includes the process of defining a model of the quality-rate function for a given k codecs and content. This can be done, for example, by performing one or more probe encodings for each codec, and then fitting a model curve through the (quality, rate) points obtained after each probe, as described in the '723 application.

[0140] The combination of the functions of blocks 610, 640, and 650 describes a loop that finds a sufficient value n of the encoding ladder point (or total number of streams). The combination of the functions of blocks 615, 635, and 655 describes a loop that finds a sufficient value n of the number of streams assigned to each codec, n1,...,n k The number of such combinations is n1++n k = n, where n is given by the previous loop.

[0141] The function of block 620 is to calculate a figure of merit function (FMM) subject to some additional conditions, such as range or rate conditions.

[0142]

number

[0143]

number

[0144]

number

[0145]

number

[0146] Figure of merit function used in block 620

[0147]

number

[0148] The functions of blocks 620 and 625 are to find the best solution for a given total number of points n.

[0149]

number

[0150] Finally, as shown in block 640, the method of FIG. 6 includes determining whether the total number n of given streams is sufficient to generate a ladder, and if so, then outputting the parameters of such ladder for storage or use in the streaming system, as shown in block 645.

[0151] More generally, the function of the method for determining a multi-codec encoding ladder shown in FIG. 6 can be described as follows. 1) Select the total number of streams n to be used in the multi-codec ABR ladder.

[0152] 2) The number of streams assigned to each codec is n1,...,n k Select a) The number of such numbers is n1++n k = n, b) Some of these numbers may actually be set to 0, indicating that there is no advantage to using some codecs given the content, codec, client, network, and other constraints.

[0153] 3) Rate of each codec

[0154]

number

[0155] a) Quality rate functions Q1(R),...,Q k (R) acquired codec and content characteristics, b) The network characteristics obtained by the network bandwidth distribution p(R); c) the decoding and switching capabilities of the clients and the distribution of the clients π; and d) Further constraints defined by the operator, such as constraints on bitrate ranges.

[0156] The following description provides some example experimental results using the multi-codec ABR encoding ladder decision technique provided herein (e.g., as shown in FIG. 6), in which its advantages become apparent. These experiments used three video sequences created by selectively concatenating raw 720p50 video clips (YUV video sequences, available at https: / / media.xiph.org / video / derf / ). These sequences are referred to herein as "easy," "medium," and "complex" based on the difficulty they present to the encoder. The encoders used were the open-source x264 and x265 projects, which implement H.264 and the encoder, respectively. Typical codec constraints suitable for streaming (GOP, HRD, reference frames, and B-frames) were applied in both cases. The SSIM metric was used to measure quality. When operating the H.264 encoder, operation in the Baseline and Main profiles is considered separately, as their performance differs significantly.

[0157] To model the performance of all codecs, the following quality-rate model function was used:

[0158]

number

[0159] [Table 2] As is evident from Figure 7, the gains that the HEVC codec can achieve compared to H.264 are highly content-dependent, with little gain for "easy" content and significant gains for "normal" and "complex" content. The difference between the H.264 Baseline and Main profiles is also content-dependent, but to a lesser extent. Thus, for "easy" content, there is still a difference between the corresponding plots shown in Figure 7.

[0160] To obtain the network bandwidth model, we used throughput measurements of the LTE network and fitted the following analytical model: p(R)=αf(R,σ1)+(1-α)f(R,σ2) (34)

[0161]

number

[0162] Two models, referred to herein as Network 1 and Network 2, are obtained by scaling the throughput of the LTE network with the two possible numbers of users in the cell. The resulting model parameters and network model plots are shown in Table 3 and Figure 8, respectively.

[0163] [Table 3] Given the above quality rates and network model, optimal encoding ladders can be determined for several practically relevant configurations of streaming systems. In all example situations, the following constraints were used: Minimum bitrate limit: min=50[kbps], Maximum bitrate limit: max =10000[kbps], and Maximum bitrate limit for first stream:

[0164]

number

[0165]

number

[0166]

number

[0167] First, we consider the trivial case where a streaming system uses only one codec. In such a case, there is only one codec, one ladder, say L1, and one type of client that can decode the stream from this ladder. This leads to the following optimization problem:

[0168]

number

[0169]

number

[0170] The present system is a single-codec ABR streaming system, which is provided for comparison with the multi-codec ABR streaming system detailed below.

[0171] Tables 4 to 6 show examples of optimal ladders constructed by considering H.264 baseline, H.264 main, and HEVC codecs, respectively.

[0172] [Table 4]

[0173] [Table 5]

[0174] [Table 6] Several things can be seen from Tables 4 to 6.

[0175] First, the optimal encoding ladders designed for different networks look different: the encoding ladder designed for network model 1 has bit rates centered around 1 Mbps, which corresponds to the peak of the bandwidth distribution; the encoding ladder designed for network model 2 has bit rates centered around 2 Mbps, which corresponds to the peak of the bandwidth distribution.

[0176] Second, the optimal ladders designed for different content will look different. Complex content generally receives streams with higher bitrate allocations than normal and easy content. Complex content also requires more ladder points to achieve a small quality gap. For example, with Network 1 and the H.264 main codec, complex content requires eight streams to achieve a gap of less than 2%. In comparison, normal content requires only four streams, and easy content requires two streams.

[0177] Third, the optimal ladders designed for different codecs also look different. The more efficient the codec, the fewer encoding ladder streams are required. For example, for network 1, complex content, and a 2% quality gap limit, the encoding ladder has 9 streams for H.264 Baseline, 8 streams for H.264 Main, and 7 streams for HEVC.

[0178] Next, consider a two-codec ABR streaming system with H.264 Baseline and H.264 Main, as shown in Figure 10. Here, there are two types of clients: clients that can only decode H.264 Baseline streams (rate selector + decoder 1), and clients that can decode and switch between streams encoded using H.264 Baseline and H.264 Main codecs (rate selector + decoder 1). In this case,

[0179]

number

[0180] This system is a variation on the problem previously described with respect to Figure 5, except that there are no decoders that can only decode the second type of codec (H.264 Main). Such decoders have been eliminated because practically all H.264 Main Profile decoders should also be able to decode H.264 Baseline streams.

[0181] Examples of optimal encoding ladders constructed for such a system are shown in Tables 7 and 8. The design of these encoding ladders assumes that 10% of the total clients will have devices capable of decoding only H.264 Baseline, and 90% will have devices capable of decoding both H.264 Baseline and H.264 Main.

[0182] [Table 7]

[0183] [Table 8] The results presented in Tables 7 and 8 contradict the common belief that H.264 baseline encoding should always be performed at the lowest rate (and resolution) of the encoding ladder, and that H.264 Main Profile encoding should always be performed at the highest rate (and resolution) of the stream. Furthermore, according to Tables 7 and 8, the optimal ladder may not include an H.264 Main stream at all. This occurs, for example, for "easy" content and when the number of renditions is less than six. This also occurs for "medium" and "complex" content, but also when the number of allowed streams is low.

[0184] Additionally, according to Tables 7 and 8, at the point where the single-codec ladder switches between the dual-codec ladder (e.g., when n=4, which is typical for content), the single H.264 baseline stream is not assigned the lowest available bitrate. Instead, it is placed at an intermediate rate, maximizing the total average quality that can be provided to H.264 baseline clients. If n≧5 and two streams are assigned to the H.264 baseline, then again, that rate is not placed at the lowest bitrate. Instead, it is placed at an intermediate point between the rates assigned to the H.264 main, so that both types of clients can benefit.

[0185] Figure 11 is a graph showing the points on the encoding ladder and the switching decisions made by H.264 Baseline and H.264 Baseline / Main switchable clients. The ladder points in this case correspond to an eight-stream encoding ladder designed for "medium" content, Network 1 (Table 7). This ladder includes H.264 Baseline encoded streams at 179 and 874 kbps and H.264 Main encoded streams at 60, 217, 465, 821, 1362, and 2464 kbps.

[0186] 11, a client that can only use H.264 Baseline will use both the 179 and 874 kbps streams encoded with such codec. At the same time, a client that can decode both H.264 Baseline and H.264 Main will select six rates encoded using H.264 Main and one rate, 179 kbps, encoded by H.264 Baseline. This seven-rate configuration allows this client to achieve the best quality when streaming.

[0187] Again, selecting only a portion of the H.264 Baseline streams and not placing all such streams first in the ladder is new and non-trivial, demonstrating that the existing practice of allocating rates to the H.264 Baseline and H.264 Main profiles is suboptimal.

[0188] Next, consider a two-codec ABR streaming system with three types of clients, as shown in Figure 12. As shown, the three types of clients are: (i) clients (rate selector + decoder 1) that can only decode H.264 streams (e.g., a web layer on a PC), (ii) clients (rate selector + decoder 2) that can decode either H.264 or HEVC streams but cannot switch between them (e.g., a DASH player on an Android™ device, a smart TV), and (iii) clients (rate selector + decoder 3) that can decode and switch between H.264 and HEVC streams (e.g., the native HLS player found on recent Apple devices).

[0189] The optimization problem in this case is the same as the problem defined in equation (17), except that the achievable quality of decoder 2 is now:

[0190]

number

[0191] In this case,

[0192]

number

[0193] Examples of optimal ladders constructed for such a system are shown in Tables 9-14. For brevity, only the results for Network Model 1 are shown. However, several different distributions of various clients are possible. Tables 9-11 consider the case where switchable clients account for half of the number of HEVC-capable clients, while Tables 12-14 consider the case where there are no switchable clients.

[0194] [Table 9]

[0195] [Table 10]

[0196] [Table 11] Based on these tables, several things can be seen.

[0197] First, the table shows that for certain types of content, using HEVC may not improve quality, and for such content, it may be appropriate to generate ladders that include only H.264-encoded content.

[0198] Second, including HEVC streams may only make sense if there is a large percentage of HEVC-capable devices. Table 9 starts with the assumption that approximately 70% of all devices can decode HEVC, which seems to be the boundary where 12 or more streams are allowed, some of which may be HEVC-only. Note that this considers a situation where half of the HEVC-capable clients are also capable of switching. If they are unable to switch (as exemplified in Tables 12-14), then a greater deployment of HEVC-capable devices may be required to substantially improve overall system performance.

[0199] Third, including HEVC may require a sufficiently large total number of streams in the ladder. If HEVC is available on 70% of devices, we find that at least 10 streams are needed for moderate content and at least 12 streams are needed for complex content. A higher percentage of deployed HEVC clients may require fewer renditions. For example, if HEVC is available on 90% of devices, we find that the number of ladder points required to begin including it drops to about six renditions. However, given that H.264-only encoding is currently typically sufficient with about five streams, it becomes clear that HEVC deployment comes at the cost of extra renditions, even if the number of devices capable of decoding it is high.

[0200] Next, in the table below, a case where there is no H.264 / HEVC switchable client is considered.

[0201] [Table 12]

[0202] [Table 13]

[0203] [Table 14] Based on the results in Tables 12-14, we can see that not using the client's H264 / HEVC switching capability has a negative impact on system performance. First, the number of streams to support HEVC needs to be further increased. For example, for complex content, 12 streams are no longer sufficient for a 70% deployment of HEVC, and at least 7 streams are required for a 90% deployment. Furthermore, the overall quality and quality gap for the same number of streams are slightly lower.

[0204] The above differences explain why it can be useful to actually use clients that do the switching, and even more useful to generate a ladder of such clients, taking into account factors such as the % number of such clients in the overall client pool, content characteristics, network, etc.

[0205] The final exemplary ABR streaming system we consider is shown in Figure 13. In this example, H.264 Baseline Profile and H.264 Main Profile are treated as separate codecs, and HEVC is considered as another codec that the system must support. Furthermore, the exemplary ABR streaming system includes four types of clients: (i) clients (rate selector + decoder 1) that can only decode the H.264 baseline stream (e.g., a conventional portable device); (ii) clients (rate selector + decoder 2) that can only decode and switch between the H.264 baseline stream and the H.264 main stream (e.g., a web player on a PC); (iii) clients (rate selector + decoder 3) that can decode all H.264 and HEVC streams and can switch between H.264 baseline and H.264 main, but cannot switch between H.264 and HEVC (e.g., a DASH player on an Android device, a smart TV); and (iv) clients (rate selector + decoder 4) that can decode and switch between all streams (e.g., the native HLS player on recent Apple devices).

[0206] The optimization problem in this case is a generalization of the problem defined above in equation (17), and the final output at each client and the overall flow are illustrated in Figure 13. Furthermore, the method for solving the optimization problem described earlier is applied in this case.

[0207] Examples of optimal ladders constructed for the system shown in Figure 13 are shown in Tables 15 and 16. For compact presentation, only the results for Network Model 1 are shown.

[0208] [Table 15]

[0209] [Table 16] Tables 15 and 16 reveal the following:

[0210] First, similar to the previous results for H.264 / HEVC systems, the percentage of HEVC-capable devices must be high, and some content still does not use HEVC. In Table 14, the total percentage of HEVC-capable devices was 60%, and there were no streams allocated to HEVC. In Table 15, the total percentage of HEVC-capable devices was 70%, sufficient to include HEVC streams in the ladder for normal and complex content.

[0211] Additionally, if HEVC is included, it obviously comes at the cost of replacing the H.264 Main rendition and leaving the H.264 Baseline rendition. Therefore, with a limited number of renditions, it appears that the best ladder that can be created to support HEVC-enabled clients may include either H.264 Baseline and H.264 Main profile renditions, or H.264 Baseline and HEVC renditions, but not if all three codecs are used.

[0212] Optimization between mixing these two codecs appears to be content dependent and is also affected by the total number n of renditions that can be included. This again demonstrates the power of this optimization technique, showing that treating the H.264 Baseline Profile as a separate codec has a significant impact on the structure and shape of the final profile in a multi-codec use case.

[0213] Naturally, the proposed method can also be applied to different codecs such as VP9, ​​AV1, VVC, etc. Figure 14 is a block diagram of a multi-codec ABR streaming system 1400 incorporating multi-codec ABR ladder generation using the methods described herein, according to one embodiment. As can be seen, the components shown in Figure 14 may correspond to the ABR streaming system 100 of Figure 1. Figure 14 includes further details regarding the components used in determining and streaming the multi-codec encoding ladder. Furthermore, similar to Figure 1, an embodiment may have any number of individual components, which may be distributed across various geographic locations and / or executed by any number of computers (e.g., computer servers).

[0214] Video in the multi-codec ABR streaming system 1400 comes from a video source 1405. Depending on the desired functionality, this source may include an origin server that stores encoded video in some intermediate format, or it may be a live stream, delivered over the RTMP protocol, for example.

[0215] This video, along with some additional information, is then received by the multi-codec ABR profile generator 1410, which generates a description of the entire encoding ladder presented as a manifest 1415. Specific encoding instructions are distributed to the encoders 1420 (encoders 1-N) responsible for encoding each stream. The manifest 1415 and encoding instructions may include, for example, the codec type, target bit rate, resolution, frame rate, and other parameters for the streams to be generated. The encoded streams are then placed on the content origin server 1425 and pushed to the CDN+access network 1430 for distribution to clients 1435.

[0216] The manifest 1415 generated by the multi-codec ABR profile generator 1410 may be further processed by manifest filtering / generation logic 1440, which may retain only renditions relevant to each type of client in the streaming system based on the capabilities of such clients. For example, for a client that can only decode H.264 baseline content, only H.264 baseline encoded renditions may be retained. Alternatively, considering an H.264 / HEVC switchable codec, for example, each of the following rates R may be retained regardless of the codec being used: i+1 ≧R i About Q, better quality than the previous rate i+1 ≧Q i The ordered portions of the renditions can be left out so that the ladder is guaranteed to provide the best possible encoding. Naturally, this may result in some rate points being omitted, but otherwise produces the best possible ladder for such switchable codecs to use. Such filtering logic can be understood by considering FIG. 4B, which shows that in this case the 595 kbps H.264 stream should be omitted. Once filtered, the final encoding ladder can be stored as a DASH or HLS manifest 1445 on the manifest origin + CDN 1450.

[0217] During playback, as each type of client 1435 attempts to access a link to the content, device detection logic 1455 can parse the request to determine the type of client 1435 seeking the content. Such detection can be performed either by the receiving server or by Javascript logic embedded in the web page. Such detection can be based on several commonly available parameters of the client system, such as web browser type and version, OS type and version, device vendor and model, and chipset vendor and model.

[0218] Once the type of client 1435 has been identified, it can be directed to an appropriately filtered manifest containing the set of streams that the client can support. Once the manifest is received, each client 1435 can operate as normally expected in an ABR streaming system.

[0219] Some features of this multi-codec ABR streaming system 1400 that are not found in conventional ABR streaming systems include: a multi-codec ABR profile generator 1410 that generates an output manifest 1415 containing streams encoded with multiple codecs; Manifest filtering / generation logic 1440, which filters the output of the multi-codec ABR profile generator 1410 and customizes the ladder to suit the capabilities of each of the clients 1435 in the system (specifically, the filtering process may leave only streams encoded using a single codec, or a combination of streams encoded with multiple codecs, and also bitrate-sorted sequences of those to produce sequences of monotonically increasing quality levels); Device detection 1455, which identifies the type of client 1435 and selects the manifests filtered / generated by 1445 for the detected type of client 1435; and · A client 1435 then receives and plays the content described in the filtered manifest.

[0220] According to some embodiments, the manifest filtering / generation logic 1440 may rely on the quality annotations generated by the ABR profile generator 1410 or the "quality_rank" identifier defined in the DASH standard. If the "quality_rank" identifier is used, it must be properly assigned across all codec adaptation sets. Additional annotations that allow switching between adaptation sets in the case of DASH manifests must also be included in the case of switchable clients.

[0221] A specific example of a manifest filtering algorithm, leaving a set of points that are monotonically increasing in terms of rate and quality, is shown in Figure 15. Given the ladder points for two codecs, we begin by simply merging them and sorting them according to rate in block 1510. A selection loop 1520-1560 then follows, where at each step, only points from the rate-sorted list that also provide an increment in terms of quality 1530 are subsequently stored 1540. In step 1570, the final filtered ladder is obtained and sent to the output.

[0222] In the example of Figure 15, for example, codec 1 can be H.264 and codec 2 can be HEVC. A similar algorithm can also be applied recursively by considering multiple codecs. In this case, for example, it can be applied first to merge rates from the H.264 Baseline profile and the H.264 Main profile, and then to merge all H.264 ladder points into HEVC ladder points. In this way, a ladder for codecs that can switch between all three codecs (H.264 Baseline, H.264 Main, and HEVC) can be generated.

[0223] Depending on the circumstances, the proposed embodiments for ABR profile generation and profile filtering may be implemented in software and / or hardware (e.g., one or more hardware or software components of a computer such as that shown in FIG. 17 and described below), or may be compiled and stored on a computer medium as computer instruction code. Such code may then be executed on a local computer containing the media to be transcoded, or may be executed remotely (e.g., in a cloud instance). It may also be executed simultaneously on multiple such cloud instances, processing different media or different chunks of the same media. Execution of such operations may be orchestrated by creating and using web APIs.

[0224] FIG. 16 is a flow diagram illustrating a method 1600 for creating a multi-codec encoding ladder, according to one embodiment that may use one or more of the optimization techniques described above. It should be understood that the functions provided in the blocks illustrated in FIG. 16 are provided by way of example. Alternative embodiments may add, omit, combine, separate, and otherwise modify the illustrated functions. One or more functions of the blocks illustrated in FIG. 16 may be performed, for example, by the ABR profile generator 1410, the encoder 120, or other components of the multi-codec ABR streaming system described herein. As such, these functions may be implemented using software and / or hardware means of a computer system, such as the computer system illustrated in FIG. 17, described below.

[0225] At block 1610, method 1600 may begin by acquiring, by a computer system, source content including video. The source content may be provided in any of a variety of formats, including a digital master, a mezzanine file, an input stream (e.g., a live stream), or a separated video fragment stream. As mentioned above, the source content may be acquired from a video source 110, which may include an origin server.

[0226] In block 1620, the function includes generating an encoding ladder for the source content, wherein each video stream of the plurality of video streams defined by the encoding ladder includes a respective bit rate and a respective codec of the plurality of codecs for encoding the source content.

[0227]

number

[0228]

number

[0229]

number

[0230]

number

[0231]

number

[0232]

number

[0233]

number

[0234]

number

[0235]

number

[0236]

number

[0237]

number

[0238] As illustrated in the examples and embodiments described above, the process of creating an encoding ladder may be optimized to take into account any of a variety of factors. In some embodiments, for example, method 1600 may include obtaining, for each codec of a plurality of codecs, a quality-rate function of an individual codec of the source content that indicates a relationship between a bit rate and a quality value of the source content, and generating an encoding ladder for the source content based on the quality-rate function of the individual codec:

[0239]

number

[0240]

number

[0241]

number

[0242] Alternative embodiments may include additional or alternative considerations and / or optimization algorithms. For example, in some embodiments, the encoding ladder is further based on a network bandwidth distribution and a distribution of clients capable of streaming the source content once it is encoded using the encoding ladder, the distribution of clients including clients capable of switching between a first codec and a second codec. Additionally or alternatively, generating the encoding ladder may include determining a plurality of video streams using an iterative process in which an initial number is selected and the steps of (1) determining a figure of merit function for the selected number and (2) increasing the value of the selected number for the next iteration are repeated until the figure of merit function reaches a maximum value. In some embodiments, the figure of merit function is based on a quality-rate function for each codec of the plurality of codecs, a network bandwidth distribution, a client distribution, or any combination thereof. The network bandwidth distribution may include a probability density function determined based on bandwidth statistics collected taking into account device type, CDN, or distribution region, or any combination thereof.

[0243] Finally, some embodiments of method 1600 may further include encoding the content based on an encoding ladder. That is, some embodiments may further include, for each stream of the encoding ladder, creating individual pieces of encoded content by encoding the source content using the codec and bitrate of the respective stream, and storing the individual pieces of encoded content. As described in the above embodiments, the encoding and storage of the encoded content may be performed by one or more encoders and a content origin server, respectively (the content origin server may then transmit the encoded content to the CDN+access network).

[0244] FIG. 17 is a block diagram of one embodiment of a computer system 1700, which may be used, in whole or in part, to perform one or more of the functions of the methods described herein, including the methods illustrated in FIGS. 6, 15, and 16. Computer system 1700 may implement one or more of the components of an ABR streaming system (e.g., ABR streaming system 100 of FIG. 1 and / or multi-codec ABR streaming system 1400 of FIG. 14), including an ABR profile generator and / or encoder. It should be noted that FIG. 17 is merely intended to generally illustrate the various components, and any or all of these components may be utilized as appropriate. Thus, FIG. 17 broadly illustrates how individual system elements may be implemented in a relatively separate or relatively more integrated manner. Additionally, it should be noted that the components illustrated in FIG. 17 may be localized on a single device and / or distributed across various network devices that may be located in different geographic locations. As discussed above, the components of the ABR streaming system may be executed in the cloud. Thus, computer system 1700 may be one of many computer systems (eg, computer servers) configured to implement various components of an ABR streaming system.

[0245] Computer system 1700 is shown as comprising hardware elements that may be electrically coupled (or otherwise in communication as appropriate) via a bus 1705. The hardware elements may include a processing unit 1710, which may include, but is not limited to, one or more general-purpose processors, one or more special-purpose processors (such as digital signal processing chips and / or graphics acceleration processors), and / or other processing structures, and may be configured to perform one or more of the methods described herein. Computer system 1700 may also include one or more input devices 1715, which may include, but is not limited to, a mouse, keyboard, camera, and / or microphone, and one or more output devices 1720, which may include, but is not limited to, a display device and / or printer, and the like.

[0246] Computer system 1700 may further comprise (and / or communicate with) one or more non-transitory storage devices 1725, which may include, but are not limited to, local and / or network-accessible storage and / or may include, but are not limited to, disk drives, drive arrays, optical storage devices, solid-state storage devices, such as random access memory (RAM) and / or read-only memory (ROM), which may be programmable and / or flash-updateable, etc. Such storage devices may be configured to implement any suitable data store, including, but not limited to, various file systems and / or database structures, etc. Such data stores may include databases and / or other data structures used to store and manage messages and / or other information sent to one or more devices, as described herein.

[0247] Computer system 1700 may also include a communications subsystem 1730, which may include wireless and wired technologies (such as Ethernet, coaxial communication, and Universal Serial Bus (USB)) managed and controlled by a wireless communication interface. Thus, communications subsystem 1730 may include a modem, a network card (wireless or wired), an infrared communications device, a wireless communications device, and / or a chipset, etc., which may enable computer system 1700 to communicate over one or more communications networks with any device on the respective network (including operations and / or applications executing thereon), including other computer systems and / or any other electronic devices described herein. Thus, communications subsystem 1730 may be used to receive and transmit data as described in the embodiments herein.

[0248] In many embodiments, computer system 1700 further comprises working memory 1735, which may include RAM or ROM devices as described above. Software elements shown to be located in working memory 1735 may include an operating system 1740, device drivers, executable libraries, and / or other code, such as one or more applications 1745, which may comprise computer programs provided by various embodiments and / or may be designed to implement methods and / or configure systems provided by other embodiments as described herein. Merely by way of example, one or more procedures described with respect to the methods discussed above may be implemented as code and / or instructions executable by a computer (and / or a processing unit within a computer). In one aspect, such code and / or instructions may then be used to configure and / or adapt a general-purpose computer (or other device) to perform one or more operations in accordance with the described methods.

[0249] A set of these instructions and / or code may be stored on a non-transitory computer-readable storage medium, such as storage device(s) 1725 and / or working memory 1735 described above. In some cases, the storage medium may be incorporated within a computer system, such as computer system 1700. In other embodiments, the storage medium may be separate from the computer system (e.g., a removable medium such as an optical disk) and / or may be provided in an installation package, such that it may be used to program, configure, and / or adapt a general-purpose computer with the instructions / code stored therein. These instructions may take the form of executable code that can be executed by computer system 1700 and / or may take the form of source and / or installable code that, upon compilation and / or installation onto computer system 1700 (e.g., using any of various commonly available compilers, installation programs, compression / decompression utilities, etc.), takes the form of executable code.

[0250] It will be apparent to those skilled in the art that substantial variations can be made according to particular requirements. For example, customized hardware could also be used, and / or particular elements could be implemented in hardware, software (including portable software such as applets), or both. Furthermore, connection to other computing devices, such as network input / output devices, could be employed.

[0251] With reference to the accompanying drawings, components that may comprise memory may include non-transitory machine-readable media. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any storage medium that participates in providing data that causes a machine to operate in a specific manner. In the embodiments provided herein above, various machine-readable media may participate in providing instructions / code to a processing unit and / or other device for execution. Additionally or alternatively, machine-readable media may be used to store and / or transport such instructions / code. In many implementations, computer-readable media are physical and / or tangible storage media. Such media may take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Common forms of computer-readable media include, for example, magnetic and / or optical media, any other physical media with patterns of holes, RAM, programmable ROM (PROM), erasable PROM (EPROM), flash-EPROM, any other memory chip or cartridge, a carrier wave as described herein below, or any other medium from which a computer can read instructions and / or code.

[0252] The methods, systems, and devices discussed herein are examples. Various embodiments may omit, substitute, or add various procedures or components as appropriate. For example, features described with respect to one embodiment may be combined in various other embodiments. Different aspects and elements of the embodiments may be similarly combined. Various components of the figures provided herein may be embodied in hardware and / or software. Also, because technology evolves, many of the elements are examples that do not limit the scope of the disclosure to those specific examples.

[0253] Throughout this specification, references to "one example," "one example," "particular example," or "exemplary implementation" mean that a particular feature, structure, or characteristic described in connection with a feature and / or example may be included in at least one feature and / or example of the claimed subject matter. Thus, appearances of the phrases "in one example," "in one example," "particular example," or "in a particular implementation," or other similar phrases in various places throughout this specification do not necessarily all refer to the same features, examples, and / or limitations. Furthermore, particular features, structures, or characteristics may be combined in one or more examples and / or characteristics.

[0254] Some of the detailed descriptions contained herein are presented in terms of algorithms or symbolic representations of operations on binary digital signals stored within a memory of a specific apparatus or special-purpose computing device or platform. In the context of this particular specification, terms such as specific apparatus include a general-purpose computer that, when programmed, performs particular operations pursuant to instructions from program software. Algorithmic descriptions or symbolic representations are examples of techniques used by those skilled in the signal processing or related arts to convey the substance of their work to others skilled in the art. An algorithm is herein generally considered to be a self-consistent sequence of operations or similar signal processing leading to a desired result. In this context, operations or processing involve physical manipulations of physical quantities. Typically, though not necessarily, such quantities may take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, or otherwise manipulated. It has proven convenient at times, primarily for reasons of common usage, to refer to such signals as bits, data, values, elements, symbols, characters, terms, numbers, or numeric values, etc. It should be understood, however, that all of these or similar terms do not associate with appropriate physical quantities but are merely convenient labels. Unless specifically specified otherwise, as will be apparent from the discussion herein, it will be understood that throughout this specification, discussions utilizing terms such as "processing," "computing," "calculating," or "determining" refer to the actions or processing of a specific apparatus, such as a special purpose computer, a special purpose computing device, or a similar special purpose electronic computing device. In the context of this specification, therefore, a special purpose computer or similar special purpose electronic computing device can manipulate or transform signals that are typically represented as physical electronic or magnetic quantities in memory, registers, or other information storage devices, transmission devices, or display devices of the special purpose computer or similar special purpose electronic computing device.

[0255] As used herein, the terms "and," "or," and "and / or" can include a variety of meanings that are expected to also depend, at least in part, on the context in which such terms are used. Typically, when "or" is used in connection with a list such as A, B, or C, it is intended to mean A, B, and C when used in an inclusive sense, and A, B, or C when used in an exclusive sense. Additionally, as used herein, the term "one or more" can be used to describe any feature, structure, or characteristic in the singular, or to describe multiple features, structures, or characteristics or any other combination of features, structures, or characteristics. However, it should be noted that this is merely an illustrative example and claimed subject matter is not limited to this example.

[0256] While what are presently considered to be exemplary features have been shown and described, those skilled in the art will recognize that various other modifications can be made and equivalents can be substituted without departing from the claimed subject matter. Additionally, many modifications can be made to adapt a particular situation to the teachings of the claimed subject matter without departing from the central concept described herein. Therefore, it is intended that the claimed subject matter not be limited to the particular examples disclosed, but that such claimed subject matter can also include all aspects that come within the scope of the appended claims and equivalents thereof.

Claims

1. 1. A method for creating a multi-codec encoding ladder, comprising: obtaining, by a computer system, source content including video; determining a total number of streams of the video for an encoding ladder; generating the encoding ladder for the source content, Each video stream of the plurality of video streams defined by the encoding ladder includes a respective bit rate and a respective codec of a plurality of codecs for encoding the source content, and the video stream is a description of parameters, and the encoding ladder: [Equation 1] and [Equation 2] Individual bit rates and [Equation 3] and [Equation 4] a first video stream and a second video stream of a first codec, each having an individual quality value of [Equation 5] Bit rate and [Equation 6] a third video stream of a second codec having a quality value of Including, [Equation 7] and [Equation 8] and the encoding ladder is generated by outputting a description of parameters of all video streams generated for each codec from the plurality of codecs used to encode the source content; the encoding ladder defines quality values ​​and bit rates to alternate between the first codec and the second codec to stream the video.

2. The method of claim 1 , wherein the parameter description includes bit rate, resolution, codec type, and quality attributes.

3. obtaining, for each codec of the plurality of codecs, a quality-rate function of the respective codec for the source content, the quality-rate function indicating a relationship between a bit rate of the source content and a quality value; generating the encoding ladder for the source content based on the quality-rate functions of the individual codecs; [Equation 9] and [Equation 10] is determined using the quality rate function of the first codec; [0011] The method of claim 1 , wherein √{square root over ( ...

4. The method of claim 3 , wherein the quality-rate function for each codec of the plurality of codecs is determined from one or more probe encodings of the source content in each codec of the plurality of codecs.

5. The encoding ladder comprises: Network bandwidth distribution, a distribution of clients capable of streaming the source content once it is encoded using the encoding ladder, the distribution of clients including clients capable of switching between the first codec and the second codec; The method of claim 3 further based on

6. generating the encoding ladder includes determining the plurality of video streams using an iterative process, wherein an initial number is selected in the iterative process; (1) determining a figure of merit function for the selected numbers; (2) incrementing the value of the selected number for the next iteration; is repeated until the figure of merit function reaches a maximum value.

7. The figure of merit function is the quality-rate function for each codec of the plurality of codecs; the network bandwidth distribution; or said distribution of client types; or Any combination of these The method of claim 6, based on

8. the network bandwidth distribution, Device type, Content Delivery Network (CDN), or Distribution area, or Any combination of these The method of claim 5 , further comprising determining a probability density function based on collected bandwidth statistics for

9. The method of claim 1 , wherein the bit rate and corresponding quality value for each video stream of the plurality of video streams monotonically increases within the encoding ladder. [Request Item 10] [Number 12] 、 [0013] 、 or [0014] 、 or any combination thereof, Structural Similarity Index Index (SSIM) value, Peak signal-to-noise ratio (PSNR) value, Multi-scale SSIM (MS-SSIM) values, or Video Multi-Method Assessment Fusion (VMAF) Value The method of claim 1 , each comprising:

11. For each stream of the encoding ladder: creating respective encoded content by encoding the source content using the codec and the bit rate of the respective stream; storing each encoded content; The method of claim 1 further comprising:

12. 1. A computer system for creating a multi-codec encoding ladder, comprising: Memory and and one or more processing units communicatively coupled to the memory, the one or more processing units comprising: Get the source content, including the video, determining a total number of streams of said video for an encoding ladder; generating the encoding ladder for the source content; each video stream of the plurality of video streams defined by the encoding ladder includes a respective bit rate and a respective codec of a plurality of codecs for encoding the source content, and a video stream is a description of parameters; The encoding ladder comprises: [Equation 15] and [0016] Individual bit rates and [Equation 17] and [Equation 18] a first video stream and a second video stream of a first codec, each having an individual quality value of [Equation 19] Bit rate and [Equation 20] a third video stream of a second codec having a quality value of Including, [Equation 21] and [Equation 22] and the encoding ladder is generated by outputting a description of parameters of all video streams generated for each codec from the plurality of codecs used to encode the source content; The encoding ladder defines quality values ​​and bit rates to alternate between the first codec and the second codec to stream the video.

13. the one or more processing units: obtaining, for each codec of the plurality of codecs, a quality-rate function of the respective codec for the source content, the quality-rate function indicating a relationship between a bit rate of the source content and a quality value; generating the encoding ladder for the source content based on the quality-rate functions of the individual codecs; using the quality rate function of the first codec [Equation 23] and [0000] Determine using the quality rate function of the second codec [Equation 25] Determine 13. The computer system of claim 12, further configured to:

14. 14. The computer system of claim 13, wherein the one or more processing units are further configured to determine the quality-rate function for each codec of the plurality of codecs from one or more probe encodings of the source content in each codec of the plurality of codecs.

15. the one or more processing units: Network bandwidth distribution, a distribution of clients capable of streaming the source content once it is encoded using the encoding ladder, the distribution of clients including clients capable of switching between the first codec and the second codec; 14. The computer system of claim 13, further configured to generate the encoding ladder for the source content further based on:

16. To generate the encoding ladder, the one or more processing units are configured to determine the plurality of video streams using an iterative process, wherein an initial number is selected in the iterative process; (1) determining a figure of merit function for the selected numbers; (2) incrementing the value of the selected number for the next iteration; is repeated until the figure of merit function reaches a maximum value.

17. The figure of merit function is the quality-rate function for each codec of the plurality of codecs; the network bandwidth distribution; or said distribution of client types; or Any combination of these 17. The computer system of claim 16, wherein the one or more processing units are further configured to:

18. the one or more processing units: Device type, Content Delivery Network (CDN), or Distribution area, or Any combination of these 16. The computer system of claim 15, further configured to determine the network bandwidth distribution based on collected bandwidth statistics for:

19. 13. The computer system of claim 12, wherein the encoding ladder is generated to switch codecs between the first codec, the second codec, and a third codec.

20. 1. A non-transitory computer-readable medium having stored thereon instructions for creating a multi-codec encoding ladder, the instructions, when executed by one or more processing units, causing the one or more processing units to: Let them retrieve source content, including video, determining a total number of streams of said video for an encoding ladder; generating the encoding ladder for the source content; each video stream of the plurality of video streams defined by the encoding ladder includes a respective bit rate and a respective codec of a plurality of codecs for encoding the source content, and a video stream is a description of parameters; The encoding ladder comprises: [Equation 26] and [0000] Individual bit rates and [0000] and [0000] a first video stream and a second video stream of a first codec, each having an individual quality value of [Equation 30] Bit rate and [Equation 31] a third video stream of a second codec having a quality value of Including, [Equation 32] and [Equation 33] and the encoding ladder is generated by outputting a description of parameters of all video streams generated for each codec from the plurality of codecs used to encode the source content; A non-transitory computer-readable medium, wherein the encoding ladder defines quality values ​​and bit rates to alternate between the first codec and the second codec to stream the video.

Images