Optimal multi-codec abr ladder design
Optimizing ABR streaming systems with multi-codec encoding profiles that consider codec performance and network conditions addresses inefficiencies in traditional systems, enhancing quality and reducing costs by enabling more efficient codec switching and device support.
Patent Information
- Application Number
- JP2024148152
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2019-01-17
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-01-17
AI Technical Summary
Traditional ABR streaming systems generate separate encoding ladders for each codec, leading to inefficiencies in encoding and distribution costs, as they do not account for the varying usage of codecs among viewers and differing coding gains, and lack the ability to optimize the balance of renditions across multiple codecs.
The creation of multi-codec encoding profiles that consider the quality rate functions of codecs, network bandwidth distribution, and client type distribution to optimize the encoding ladder, allowing for more efficient switching between different codecs and reducing operational costs.
This approach enhances the quality and efficiency of ABR streaming by optimizing the encoding ladder to better adapt to changing network conditions and support a variety of devices, reducing operational costs and improving overall streaming performance.
Smart Images

Figure 2024159943000001_ABST
Abstract
Description
[Background technology]
[0001] Adaptive Bit Rate (ABR) streaming is a method of streaming video content in which the bit rate of the video stream delivered to a streaming client can be adjusted during playback to accommodate changes in available network bandwidth. To enable this functionality, an ABR streaming system can encode the source content into multiple streams of different bit rates. In this way, a streaming client can switch between different streams while streaming video, effectively receiving a composite stream that scales with the available network bandwidth.
[0002] The composition of streams into which the source content is encoded can be determined by the ABR streaming system. In conventional ABR streaming systems, this decision is typically made independently for each codec. In other words, for each codec, a whole new set of streams is generated that covers the range of bit rates required to accommodate the network. This results in significant encoding and distribution costs. However, now that many streaming clients can switch between streams of different codecs, such inefficiencies can be minimized by finding an optimal multi-codec composition of streams sufficient for ABR distribution. Summary of the Invention [Means for solving the problem]
[0003] The techniques described herein provide for the creation of a multi-codec encoding profile (or encoding ladder) that defines the quality and bitrate for each of the streams made available to the client for streaming video. In particular, optimization techniques can take into account the quality-rate function of each of the codecs when determining the encoding ladder. Further considerations can include network bandwidth distribution and / or client type distribution.
[0004] An exemplary method for creating a multi-codec encoding ladder according to the present disclosure includes obtaining, by a computer system, source content including video, and generating an encoding ladder for the source content, where each video stream of a plurality of video streams defined by the encoding ladder includes a respective bit rate and a respective codec of a plurality of codecs for encoding the source content.
[0005]
number
[0006]
number
[0007]
number
[0008]
number
[0009]
number
[0010]
number
[0011]
number
[0012]
number
[0013] An exemplary computer system for creating a multi-codec encoding ladder according to the present disclosure includes a memory and one or more processing units communicatively coupled to the memory. The one or more processing units obtain source content including video, generate an encoding ladder for the source content, wherein each video stream of a plurality of video streams defined by the encoding ladder includes a respective bit rate and a respective codec of a plurality of codecs for encoding the source content, and the encoding ladder includes:
[0014]
number
[0015]
number
[0016]
number
[0017]
number
[0018]
number
[0019]
number
[0020]
number
[0021]
number
[0022] An exemplary non-transitory computer-readable medium according to the present disclosure has stored therein instructions for creating a multi-codec encoding ladder, the instructions, when executed by one or more processing units, causing the one or more processing units to obtain source content including video, generate an encoding ladder for the source content, wherein each video stream of a plurality of video streams defined by the encoding ladder includes a respective bit rate and a respective codec of a plurality of codecs for encoding the source content, and the encoding ladder defines:
[0023]
number
[0024]
number
[0025]
number
[0026]
number
[0027]
number
[0028]
number
[0029]
number
[0030]
number
[0031] [Figure 1] 1 is an ABR streaming system according to one embodiment. [Diagram 2] 1 is a graph plotting available network bandwidth and client streaming rates. [Diagram 3] FIG. 1 is a conceptual diagram of an ABR streaming system with a video source, two encoders, and three decoders with associated rate selectors, according to one embodiment. [Figure 4A-4B] 4 is a graph illustrating the video quality that different decoders (eg, the decoder of FIG. 3) can achieve at various bitrates, according to one embodiment. [Diagram 5] FIG. 2 is a conceptual diagram of an ABR streaming system used in an exemplary embodiment. [Figure 6] FIG. 2 is a flow diagram illustrating the main steps of a method for determining an optimized multi-codec encoding ladder according to one embodiment. [Figure 7] 1 is a graph showing the shape of the obtained quality-rate function for a codec and exemplary source content; [Figure 8] 1 is a graph of network parameters for two network models used to obtain the experimental results described herein. [Figure 9]FIG. 1 is a conceptual diagram of a single-codec ABR streaming system used to obtain the experimental results described herein. [Figure 10] FIG. 1 is a conceptual diagram of a dual-codec ABR streaming system used to obtain the experimental results described herein. [Figure 11] 1 is a graph showing points on the encoding ladder and switching decisions made by an H.264 Baseline client and an H.264 Baseline / Main switchable client. [Figure 12] FIG. 1 is a conceptual diagram of a dual-codec ABR streaming system with three client types used to obtain the experimental results described herein. [Figure 13] FIG. 1 is a conceptual diagram of a multi-codec ABR streaming system with four client types used to obtain the experimental results described herein. [Figure 14] 1 is a block diagram of a multi-codec ABR streaming system incorporating multi-codec ABR ladder generation using the methods described herein, according to one embodiment. [Figure 15] 1 is a flowchart of a method for determining a set of monotonically increasing points (streams in an encoding ladder) in terms of rate and quality. [Figure 16] 1 is a flow diagram illustrating a method for creating a multi-codec encoding ladder according to one embodiment. [Figure 17] FIG. 1 is a block diagram of one embodiment of a computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0032] Like reference numbers in each drawing indicate like elements according to a particular exemplary implementation. Additionally, multiple instances of an element may be designated using the first numeral of the element followed by a letter or hyphen and a second numeral. For example, multiple instances of element 110 may be designated as 110-1, 110-2, 110-3, etc., or as 110a, 110b, 110c, etc. When such an element is referred to by only the first numeral, it should be understood as any instance of that element (e.g., element 110 in the previous example refers to elements 110-1, 110-2, and 110-3, or elements 110a, 110b, and 110c).
[0033] Detailed Description Several exemplary embodiments will now be described with reference to the accompanying drawings, which form a part of this specification. Specific embodiments in which one or more aspects of the present disclosure may be practiced are described below, but other embodiments may be used and various changes may be made without departing from the spirit of the scope of the present disclosure or the appended claims.
[0034] 1 illustrates an ABR streaming system 100 according to one embodiment. The ABR streaming system 100 includes a video source 110, an encoder 120, an origin server 130, a content delivery network (CDN)+network access 140, and a streaming client 150. As will be appreciated by those skilled in the art, different embodiments may have different numbers of each of the illustrated components. For example, the CDN+network access may serve many (e.g., tens, hundreds, thousands, or more) streaming clients 150.
[0035] To enable delivery of source content (e.g., one or more media files) stored in video source 110 to one or more streaming clients 150, encoder 120 may encode the source content into multiple streams with different bit rates. (For example, as shown in callout 160, the encoder may encode the source content to provide M bit rates and separate descriptions.) Each encoded stream may incorporate random access points (e.g., intraframe (I) frames or instant decoding refresh (IDR) frames in the encoded video) to allow switching between streams. Such streams are then placed on origin server 130 and further pushed to CDN+network access 140 (which may include not only CDNs but also one or more data communication networks such as the Internet) for scaled delivery to streaming clients 150.
[0036] During playback, each streaming client 150 may monitor the rate at which the encoded content arrives. (For example, as shown in callout 170, the streaming client may estimate the bandwidth and then select an appropriate rate for the next segment taking the bandwidth into account before acquiring the next segment.) If such rate becomes insufficient for continuous playback, the client may switch to a lower bitrate stream. This may prevent buffering. On the other hand, if such rate is greater than the bitrate of the current stream, the client may switch to a higher bitrate stream to provide better quality to the end user. Such switching mechanisms have since become widely adopted and are incorporated into all modern streaming protocols, such as HyperText Transfer Protocol (HTTP) Live Streaming (HLS), MPEG Dynamic Adaptive Streaming over HTTP (DASH), etc.
[0037] Thus, as a result, the streaming bitrate of the video to the streaming client 150 adapts to changes in available network bandwidth over time. For example, as shown in the graph of FIG. 2, the streaming bitrate 210 may increase as the available network bandwidth 220 increases and similarly decrease as the available network bandwidth 220 decreases. These changes in the streaming bitrate 210 are due to the streaming client 150 switching from a stream having a first bitrate to a stream having a second bitrate. Thus, the more streams (of different bitrates) the encoder 120 creates for a given source content, the more finely tuned the changes in the streaming bitrate 210 may be.
[0038] The configuration of video stream characteristics used for ABR streaming, such as bitrate, resolution, codec constraints, etc., is typically referred to as an encoding profile or ladder. An exemplary encoding ladder for the High Efficiency Video Coding (HEVC) and H.264 / MPEG-4 AVC (or simply "H.264") codecs is set forth below in Table 1, which is set forth in the Apple® HLS deployment guidelines.
[0039] [Table 1] Recently, it has also been found that the performance of the ABR streaming system 100 can be improved by using a dynamic ladder generator that creates a custom encoding ladder that takes into account the rate-distortion characteristics of the content and / or the characteristics of the network used to deliver the stream. Such approaches have been known as "per-title," "content-aware encoding," and "context-aware encoding" techniques. Further information regarding ladder generators for ABR streaming is described in U.S. patent application Ser. No. 15 / 829,723, entitled "Optimization of Encoding Profiles for Media Streaming" (herein referred to as the "'723 application"), which is incorporated herein by reference in its entirety for all purposes.
[0040] Until recently, there was only one codec in the encoding ladder for ABR streaming. (In most cases, the codec is H.264, which is ubiquitous and supported by most existing devices over the past decade.) However, with the introduction of more codecs, such as VP9, HEVC, and AV1, the ABR streaming system 100 must generally re-encode the entire source content for all additionally supported codecs and create a new ABR ladder with all streams encoded using the new codec to help ensure that the content can be streamed to a variety of devices that support different codecs. Current versions of streaming guidelines (e.g., HLS guidelines and DASH-IF implementation guidelines) define one ladder (a set of streams with a certain bitrate and resolution) for H.264 and another ladder for HEVC.
[0041] As previously mentioned, the deployment of multiple codecs for ABR streaming is currently based on the assumption that each codec has a separate ABR encoding ladder and a corresponding separate set of streams. Thus, the ABR streaming system 100 generates an encoding ladder for H.264 separately from the encoding ladder for HEVC.
[0042] Unfortunately, however, using such a single-codec encoding ladder is fundamentally suboptimal for at least three reasons: First, by generating the ladders separately, there may be no way to find the right balance between the number of renditions assigned to the ABR encoding ladder associated with each codec. Thus, the encoder 120 may generate as many renditions as it deems necessary for each codec without considering the fact that usage of such codecs across a set of viewers may vary. For example, the number of streaming clients 150 that support HEVC encoded video may be much smaller than the number of streaming clients 150 that support H.264. Also, in an ABR streaming system 100 that has a fixed total budget for the number of renditions that may be generated, allocating more to H.264 may have a larger overall impact on the total quality delivered to end users.
[0043] Second, based on the content characteristics, the coding gains of HEVC versus H.264 may differ significantly. This in turn may affect the balance of the number of renditions that should be used for each codec. For example, in one extreme scenario, an optimal ABR ladder design may not allocate a rendition to HEVC when HEVC does not provide any gain, since all HEVC-enabled streaming clients 150 can also decode H.264. Thus, switching to H.264 does not reduce the reach of the system.
[0044] Third, many new streaming clients 150 can switch between streams having both H.264 and HEVC codecs. Given this switching capability, such clients should likely be able to achieve better performance than clients that select H.264 or HEVC streams, since they have more streams available. (Again, as discussed with respect to FIG. 2, when more streams are available, the streaming bitrate 210 can use finer steps to adapt to changes in available network bandwidth 220. This greater granularity can make the ABR streaming system 100 more efficient.)
[0045] The embodiments described herein provide an optimized solution for ABR profile / ladder generation for multiple codec ladders. That is, the techniques provided describe an optimal multi-codec ABR streaming ladder generator implemented as software, or hardware, or a combination of both, and an ABR streaming system incorporating such a ladder generator. Among the advantages provided by the techniques herein are improved quality and / or reduced operational costs of the ABR streaming system.
[0046] FIG. 3 is a conceptual diagram of an ABR streaming system with a video source 110, two types of encoders (encoder 1 and encoder 2), and three types of clients with decoders (decoder 1, decoder 2, and decoder 3) with associated rate selectors, according to one embodiment. Note that the encoders shown in FIG. 3 may correspond to a type of encoder 120 of FIG. 1 that may be implemented by one or more computers that capture source content from the video source 110 (which may also comprise one or more computers). Furthermore, each of the clients 310 may correspond to a different type of streaming client 150 of FIG. 1 that may be implemented by an end-user device (computer, mobile phone, television, etc.). (Note that in this specification, the terms "client" and "decoder" are often used interchangeably when referring to the mathematical description of the ABR streaming system described below.) In this embodiment, and for the purposes of establishing the mathematical description of one embodiment, decoders 1 and 2 can only decode the streams generated by encoders 1 and 2, respectively. Decoder 3 may select and decode a stream generated by either encoder 1 or encoder 2.
[0047] As used herein, the term "single-codec client" refers to a client (such as a client with decoder 1 or 2) that can decode a video stream encoded with a single codec (e.g., from either encoder 1 or encoder 2). Similarly, the term "dual-codec client" refers to a client (such as a client with decoder 3) that can switch between video streams encoded with two different codecs (e.g., video streams encoded by either encoder 1 or encoder 2). Similarly, the term "switching client" may be used to refer to a decoder that can decode video encoded streams with more than one codec (from more than one encoder).
[0048] To provide a mathematical description of the ABR streaming system of Fig. 3, the variable R is used to represent the bit rate and Q is used to represent the quality value achievable by the video codec, where the quality value Q is normalized such that the value Q = 0 represents the worst possible quality and Q = 1 represents the ideal reconstruction. A well-known example of a quality metric that satisfies such constraints is the Structural Similarity Index Measure (SSIM) metric, but in principle it could be any other metric with a certain normalization applied, such as the Peak Signal-to-Noise Ratio (PSNR), Multi-scale SSIM (MS-SSIM), or Video Multi-method Assessment Fusion (VMAF).
[0049] For a given source content, encoders 1 and 2 each produce a set of encoded streams with the following (quality, rate) characteristics:
[0050]
number
[0051]
number
[0052] Here, the performance of the codec is modeled by specific quality-rate functions Q1(R) and Q2(R). The (quality, rate) points above (corresponding to the different streams) can be understood as samples taken from these functions:
[0053]
number
[0054]
number
[0055] For notational convenience, such ladders always have a zero point that is the same for both codecs. (R 0 ,Q 0 )=(0,0) (5) It can be expanded by:
[0056] In fact, one of the video parameters that may be similarly varied at different bit rates is the resolution. In the embodiments described herein, the resolution may be optimized and may be assumed to be captured by the quality-rate model of each codec. In other words, a set of allowed resolutions may be
[0057]
number
[0058]
number
[0059]
number
[0060] Modern streaming protocols such as HLS or DASH are based on using the Transmission Control Protocol (TCP) as the underlying transport protocol. TCP then implements retransmissions to eliminate packet loss and mask many natural statistics inherent to each type of physical network. However, at the TCP level, fluctuations in the transmission speed or bandwidth available at each moment can still be observed.
[0061] Thus, for the purposes of mathematical modeling, the network can be viewed as a continuous random variable R with a certain given probability density function p(R). In practice, such bandwidth density functions p(R) may be different for different devices or their respective access networks. For example, considering a mobile client connected over a 4G / Long Term Evolution (LTE) network, known throughput measurements of TCP traffic over LTE may be used. More generally, such distributions may be experimentally measured given each specific streaming deployment, and may of course be different for different devices, CDNs, distribution regions, etc.
[0062] The model of the client can be defined as follows: At any time, given a particular available network bandwidth R, decoders 1 and 2 (single-codec clients) in Figure 3 receive from ladders L1 and L2, respectively:
[0063]
number
[0064]
number
[0065] In other words, decoders 1 and 2 are connected to each other at a maximum ladder rate R i Select. Therefore, the quality achieved by each decoder is
[0066]
number
[0067]
number
[0068] In practice, the rate selection algorithm in the streaming client may be more complex, but the above selection model may nevertheless be suitable for considering the average performance of a streaming system.
[0069] With reference to decoder 3 (of FIG. 3), for each bandwidth value R, decoder 3 can choose both the bit rate and the codec that achieves the next best quality.
[0070]
number
[0071]
number
[0072] 4A and 4B are graphs showing the video quality that different clients / decoders can achieve at various bitrates. In FIG. 4A, the HEVC quality-rate function 310 (Q HEVC(R)) and H.264 quality rate function 320 (Q H.264 (R) is plotted to show the achievable quality (SSIM) for a given source content over bit rates ranging from 0 to 3500 kbps. It can be seen that the plot of the HEVC quality-rate function 310 outperforms the plot of the H.264 quality-rate function 320, suggesting that HEVC is more efficient for the given source content.
[0073] As the bitrate increases, the quality selected by the HEVC decoder 330 and the quality selected by the H.264 decoder 340 each exhibit steps of increasing quality showing where the respective decoders switch from a stream with a lower bitrate / quality to a stream with a higher bitrate / quality. For example, the H.264 encoding ladder includes five bitrate points, 71, 268, 595, 1108, and 2149 kbps, respectively, resulting in a step-like function exhibited by the quality selected by the H.264 decoder 340. The HEVC encoding ladder includes three bitrate points, 93, 459, and 1275 kbps, respectively, resulting in a step-like function exhibited by the quality selected by the HEVC decoder 330. (The HEVC decoder can only select from among the three available HEVC rates.) FIG. 4B shows the quality selected by the dual codec decoder 350 overlaid on the graph of FIG. 4A. As can be seen, the quality selected by the dual codec decoder 350 partially coincides with the steps of both the H.264 and HEVC decoders, which choose the best quality available at each rate. Both codecs are used alternately, resulting in a total of seven steps. This allows the dual codec decoder / client to adapt more accurately to the changing network bandwidth, and therefore achieve better network utilization than if the decoder only worked with H.264 or HEVC streams. However, importantly, the dual codec decoder can also skip points where it does not make sense to switch because there is no improvement in quality. For example, instead of using the 595 kbps H.264 point (which has a lower quality than the 459 kbps HEVC), the dual codec decoder stays at the 459 kbps HEVC.
[0074] Based on this, a condition can be formulated where a dual-codec decoder / client achieves better performance, where given ladders L1(1) and L2(2) of streams encoded with the first and second codecs, respectively,
[0075]
number
[0076]
number
[0077]
number
[0078]
number
[0079]
number
[0080]
number
[0081]
number
[0082]
number
[0083]
number
[0084]
number
[0085] The average quality achievable by a dual-codec ABR streaming system can be determined as follows: Given the rate selection rules above, and assuming that the network bandwidth is modeled as a continuous random variable R with probability density function p(R), an expression of the average quality achievable by three types of decoders in a streaming system can be written as follows:
[0086]
number
[0087]
number
[0088]
number
[0089]
number
[0090]
number
[0091]
number
[0092] Finally, by assuming that π={π1,π2,π3}, π1+π2+π3=1 is a distribution describing the presence of each type of client in the universe of clients, the overall average quality that the streaming system can achieve can be expressed as:
[0093]
number
[0094] Considering equations (1) to (16), and the average quality value
[0095]
number
[0096]
number
[0097]
number
[0098]
number
[0099]
number
[0100]
number
[0101]
number
[0102]
number
[0103] As can be easily noticed, the problem described in (17) is a nonlinear constrained optimization problem,
[0104]
number
[0105]
number
[0106] All the constraints introduced in (17) can be used in practical settings. For example, the maximum rate limit R max prevents the allocation of bit rates in excess of what is physically possible. The minimum rate limit R min is usually related to the minimum quality level at which streaming as a service is still achievable.
[0107]
number
[0108] The problem formulated in (17) works with n total streams in a streaming system. However, if the number of streams is allowed to approach infinity, the resulting quality bound at the output of each decoder becomes
[0109]
number
[0110]
number
[0111]
number
[0112]
number
[0113] The relative distance between the ideal quality value and the best average quality achievable in an n-point system (called the "quality gap") can be defined as:
[0114]
number
[0115]
number
[0116]
number
[0117] In addition to the average quality, the average bandwidth consumed by each client can be expressed as:
[0118]
number
[0119]
number
[0120]
number
[0121]
number
[0122]
number
[0123] In principle, given all the above definitions, and considering that in practice bandwidth is usually a factor in the operational cost of a streaming system, the optimal ladder design problem can also be formulated as a problem of minimizing the average bandwidth.
[0124]
number
[0125] However, note that these problems are related and in some cases lead to exactly the same solution. Thus, for a given n and all other constraints, we find a quality bound that coincides with the solution of problem (17), i.e., the best quality achievable in such a system.
[0126]
number
[0127]
number
[0128] More generally, we consider an ABR streaming system with k codecs and m clients, where the network bandwidth distribution p and client types π are known, and we define a particular figure of merit function as the optimization criterion.
[0129]
number
[0130] Then, and under certain further conditions, the optimization problem is
[0131]
number
[0132]
number
[0133]
number
[0134]
number
[0135]
number
[0136]
number
[0137]
number
[0138] Fig. 6 is a flow diagram illustrating the main steps of a method for determining an optimized multi-codec encoding ladder according to an embodiment in which the problem defined by equations (31) and (32) is solved. In the following, concrete examples of practical multi-codec systems and the respective solutions found by applying the proposed method are given. Some or all of the functions shown in the blocks of Fig. 6 may be performed by the encoder 120 (which, as mentioned above, may be performed by a computer server).
[0139] The method of Figure 6 may begin at block 605, which includes the process of defining a model of the quality-rate function for the given k codecs and content. This may be done, for example, by performing one or more probe encodings with each codec, and then fitting a model curve through the (quality, rate) points obtained after each probe, as described in the '723 application.
[0140] The combination of the functions of blocks 610, 640, and 650 describes a loop that finds a sufficient value n of the encoding ladder point (or total number of streams). The combination of the functions of blocks 615, 635, and 655 describes a loop that finds the number of streams n1,...,n assigned to each codec. k The number of such combinations is n1+···+n k =n, where n at this point is given by the previous loop.
[0141] The function of block 620 is to calculate a figure of merit function subject to some additional conditions, such as range or rate conditions.
[0142]
number
[0143]
number
[0144]
number
[0145]
number
[0146] Figure of merit function used in block 620
[0147]
number
[0148] The functions of blocks 620 and 625 are to find the best solution for a given total number of points n.
[0149]
number
[0150] Finally, as shown in block 640, the method of FIG. 6 includes checking whether the total number n of given streams is sufficient to generate a ladder, and if so, then outputting the parameters of such ladder for storage or use in the streaming system, as shown in block 645.
[0151] More generally, the function of the method for determining a multi-codec encoding ladder shown in FIG. 6 can be described as follows. 1) Select the total number of streams n to be used in the multi-codec ABR ladder.
[0152] 2) The number of streams assigned to each codec, n1,...,n k Select a) The number of such numbers is n1+...+n k = n, b) Some of these numbers may in fact be set to 0, indicating that there is no advantage to using some codecs given content, codecs, clients, networks, and other constraints.
[0153] 3) Rate of each codec
[0154]
number
[0155] a) Quality rate functions Q1(R),...,Q k (R) acquired codec and content characteristics, b) The network characteristics obtained by the network bandwidth distribution p(R); c) the decoding and switching capabilities of the clients and the distribution of the clients, π; and d) Further operator defined constraints, such as constraints on bit rate ranges.
[0156] In the following description, some examples of experimental results using the technique of multi-codec ABR encoding ladder decision provided herein (e.g., as shown in FIG. 6) are provided, in which the advantages become evident. In these experiments, three video sequences created by selective concatenation of raw 720p50 video clips were used (YUV video sequences, available at https: / / media.xiph.org / video / derf / ). These sequences are referred to herein as "easy", "medium", and "complex" based on the difficulty such sequences present to the encoder. The encoders used were the open source x264 and x265 projects that implement H.264 and encoders, respectively. Typical codec constraints suitable for streaming (GOP, HRD, reference frames, and B-frames) were applied in both cases. The SSIM metric was used to measure the quality. When operating the H.264 encoder, operation in the Baseline and Main profiles is considered separately, as their performance differs considerably.
[0157] To model the performance of all codecs, the following quality-rate model function was used:
[0158]
number
[0159] [Table 2] As is evident from Figure 7, the gains that the HEVC codec can achieve compared to H.264 are highly content dependent, with little gain for "easy" content and more noticeable gains for "medium" and "complex" content. The difference between the H.264 Baseline and Main profiles is also content dependent, but to a lesser extent. Thus, for "easy" content, there is still a difference between the corresponding plots in Figure 7.
[0160] To derive the network bandwidth model, we used throughput measurements of an LTE network and fitted the following analytical model: p(R)=αf(R,σ1)+(1-α)f(R,σ2) (34)
[0161]
number
[0162] Two models, referred to herein as Network 1 and Network 2, are obtained by scaling the throughput of the LTE network with the two possible numbers of users in a cell. The resulting model parameters and plots of the network models are shown in Table 3 and Figure 8, respectively.
[0163] [Table 3] Given the above quality rates and network models, optimal encoding ladders can be determined for several practically relevant configurations of streaming systems. In all example situations, the following constraints were used: Minimum bitrate limit: min=50[kbps], Maximum bitrate limit: max =10000[kbps], and · Maximum bitrate limit for first stream:
[0164]
number
[0165]
number
[0166]
number
[0167] First, we consider the trivial example where a streaming system uses only one codec. In such a case, there is only one codec, one ladder, say L1, and one type of client that can decode the stream from this ladder. This leads to the following optimization problem:
[0168]
number
[0169]
number
[0170] The present system is a single-codec ABR streaming system, which is provided for comparison with the multi-codec ABR streaming system detailed below.
[0171] Examples of optimal ladders constructed by considering H.264 baseline, H.264 main, and HEVC codecs, respectively, are shown in Tables 4 to 6.
[0172] [Table 4]
[0173] [Table 5]
[0174] [Table 6] Several things can be seen from Tables 4 to 6.
[0175] First, the optimal encoding ladders designed for different networks look different: the encoding ladder designed for network model 1 has bit rates centered around 1 Mbps, which corresponds to the peak of the bandwidth distribution; the encoding ladder designed for network model 2 has bit rates centered around 2 Mbps, which corresponds to the peak bandwidth distribution.
[0176] Secondly, the optimal ladders designed for different content look different. Complex content generally receives higher bitrate allocated streams compared to normal and easy content. Complex content also requires more ladder points to reach a small quality gap. For example, for network 1 and H.264 main codec, complex content requires 8 streams to reach a gap of less than 2%. In comparison, normal content requires only 4 streams and easy content requires 2 streams.
[0177] Third, the optimal ladders designed for different codecs also look different. With more efficient codecs, fewer encoding ladder streams are needed. For example, for network 1, complex content, and a limit of 2% quality gap, the encoding ladder has 9 streams for H.264 baseline, 8 streams for H.264 main, and 7 streams for HEVC.
[0178] Now consider a two-codec ABR streaming system with H.264 Baseline and H.264 Main, as shown in Figure 10. Here, there are two types of clients: clients that can only decode H.264 Baseline streams (rate selector+decoder1), and clients that can decode and switch between streams encoded with H.264 Baseline and H.264 Main codecs (rate selector+decoder1). In this case,
[0179]
number
[0180] This system is a variation of the problem previously described with respect to Figure 5, except that there are no decoders that can only decode the second type of codec (H.264 Main). Such decoders have been eliminated, since practically all H.264 Main Profile decoders should also be able to decode H.264 Baseline streams.
[0181] Examples of optimal encoding ladders constructed for such a system are shown in Tables 7 and 8. In designing these encoding ladders, it is assumed that 10% of the total number of clients have devices capable of decoding only H.264 Baseline, and 90% have devices capable of decoding both H.264 Baseline and H.264 Main.
[0182] [Table 7]
[0183] [Table 8] The results presented in Tables 7 and 8 contradict the common belief that H.264 baseline encoding should always be done at the lowest rate (and resolution) of the encoding ladder, and H.264 Main Profile encoding should always be done at the highest rate (and resolution) of the stream. Moreover, according to Tables 7 and 8, the optimal ladder may not contain any H.264 Main stream at all. This happens, for example, for "easy" content and when the number of renditions is less than six. This also happens for "medium" and "complex" content, but also when the number of allowed streams is low.
[0184] In addition, according to Tables 7 and 8, at the point where the single and dual codec ladders are switched over (e.g., n=4, for typical content), the single H.264 baseline stream is not assigned to the lowest available bitrate. Instead, it is placed at an intermediate rate to maximize the total average quality that can be provided to H.264 baseline clients. If n>=5 and two streams are assigned to H.264 baseline, then again, that rate is not placed at the lowest bitrate. Instead, it is placed at an intermediate point between the rates assigned to H.264 main so that it can be usefully used by both types of clients.
[0185] 11 is a graph showing the encoding ladder points and switching decisions made by H.264 Baseline and H.264 Baseline / Main switchable clients. The ladder points in this case correspond to an eight-stream encoding ladder designed for "medium" content, network 1 (Table 7). The ladder includes H.264 Baseline encoded streams at 179 and 874 kbps, and H.264 Main encoded streams at 60, 217, 465, 821, 1362, and 2464 kbps.
[0186] As can be further seen from Fig. 11, a client that can only use H.264 Baseline will use both the 179 and 874 kbps streams encoded with such codec. At the same time, a client that can decode both H.264 Baseline and H.264 Main will select six rates encoded with H.264 Main and one rate of 179 kbps encoded by H.264 Baseline. This seven rate configuration allows this client to achieve the best quality when streaming.
[0187] Again, selecting only a portion of the H.264 Baseline streams and not placing all such streams first in the ladder is new and non-trivial, demonstrating that the existing practice of assigning rates to the H.264 Baseline and H.264 Main profiles is suboptimal.
[0188] Now consider a two-codec ABR streaming system with three types of clients, as shown in Figure 12. As shown, the three types of clients are: (i) clients (rate selector + decoder 1) that can only decode H.264 streams (e.g., web layer on a PC), (ii) clients (rate selector + decoder 2) that can decode either H.264 or HEVC streams but cannot switch between them (e.g., DASH players on Android™ devices, smart TVs), and (iii) clients (rate selector + decoder 3) that can decode and switch between H.264 and HEVC streams (e.g., native HLS players on recent Apple devices).
[0189] The optimization problem in this case is the same as the problem defined in equation (17), except that the achievable quality of decoder 2 is now:
[0190]
number
[0191] In this case,
[0192]
number
[0193] Examples of optimal ladders constructed for such a system are shown in Tables 9-14. For the sake of brevity, only the results for Network Model 1 are shown. However, several different distributions of the various clients are possible. In Tables 9-11, we consider the case where switchable clients make up half of the number of HEVC-enabled clients, and in Tables 12-14, we consider the case where there are no switchable clients.
[0194] [Table 9]
[0195] [Table 10]
[0196] [Table 11] Based on these tables, several things can be noted.
[0197] First, the table illustrates that for certain types of content, the use of HEVC may not result in improved quality, and for such content it may be appropriate to generate ladders that include only H.264 encoded content.
[0198] Second, including HEVC streams may only make sense if there is a large percentage of HEVC-enabled devices. Table 9 starts with the assumption that about 70% of all devices can decode HEVC, which seems to be the borderline where some of them may be HEVC-only when 12 or more streams are allowed. Note that this considers a situation where half of the HEVC-enabled clients are also capable of switching. If they are not capable of switching (as exemplified in Tables 12-14), then a greater deployment of HEVC-enabled devices may be required to substantially improve overall system performance.
[0199] Third, including HEVC may require the total number of streams in the ladder to be sufficiently high. If HEVC is available on 70% of devices, we find that at least 10 streams are needed for moderate content and at least 12 streams for complex content. If a higher percentage of HEVC clients are deployed, the number of such renditions may be smaller. For example, if HEVC is available on 90% of devices, we find that the number of ladder points required to start including it drops to about 6 renditions. However, given that currently H.264-only encoding typically requires only about 5 streams, it becomes clear that HEVC deployment comes at a cost of extra renditions, even for a large number of devices capable of decoding.
[0200] Next, in the table below, a case where there is no H.264 / HEVC switchable client is considered.
[0201] [Table 12]
[0202] [Table 13]
[0203] [Table 14] Based on the results in Tables 12 to 14, it can be seen that not using the client's H264 / HEVC switching capability will have a negative impact on system performance. First, the number of streams to support HEVC needs to be further increased. For example, for complex content, 12 streams are no longer sufficient for 70% deployment of HEVC, and at least 7 streams are required for 90% deployment. Furthermore, the overall quality and quality gap with the same number of streams are also slightly lower.
[0204] The above differences explain why it may be useful to actually use clients that do switch, and even more useful to generate a ladder of such clients, taking into account factors such as the % of the number of such clients in the total client pool, the characteristics of the content, the network, etc.
[0205] The final exemplary ABR streaming system to consider is shown in Figure 13. In this example, H.264 Baseline Profile and H.264 Main Profile are treated as separate codecs, and HEVC is considered as just another codec that the system must support. Further, the exemplary ABR streaming system includes four types of clients: (i) clients (rate selector+decoder 1) that can only decode the H.264 baseline stream (e.g., traditional portable devices); (ii) clients (rate selector+decoder 2) that can only decode and switch between the H.264 baseline stream and the H.264 main stream (e.g., web players on PCs); (iii) clients (rate selector+decoder 3) that can decode all H.264 and HEVC streams and can switch between H.264 baseline and H.264 main, but cannot switch between H.264 and HEVC (e.g., DASH players on Android devices, smart TVs); and (iv) clients (rate selector+decoder 4) that can decode and switch all streams (e.g., native HLS players on recent Apple devices).
[0206] The optimization problem in this case is a generalization of the problem defined above in equation (17), and the final output at each client and the overall flow are illustrated in Figure 13. Furthermore, the method for solving the optimization problem described above is applied in this case.
[0207] Examples of optimal ladders constructed for the system shown in Figure 13 are shown in Tables 15 and 16. For compact presentation, only the results for Network Model 1 are shown.
[0208] [Table 15]
[0209] [Table 16] The following can be seen from Tables 15 and 16:
[0210] First, similar to the previous results for H.264 / HEVC systems, the percentage of HEVC-enabled devices must be high, and some content still does not use HEVC. In Table 14, the total percentage of HEVC-enabled devices was 60%, and there were no streams assigned to HEVC. In Table 15, the total percentage of HEVC-enabled devices was 70%, which was sufficient to include HEVC streams in the ladder for normal and complex content.
[0211] Additionally, if HEVC is included, it is obviously at the cost of replacing the H.264 Main rendition and leaving the H.264 Baseline rendition. Thus, with a limited number of renditions, it seems likely that the best ladder that can be generated to support HEVC-enabled clients might include either H.264 Baseline and H.264 Main profile renditions, or H.264 Baseline and HEVC renditions, but not if all three codecs are used.
[0212] Optimization between mixing these two codecs appears to be content dependent, and is also affected by the total number n of renditions that can be included. Again, this demonstrates the power of this optimization technique, showing that treating the H.264 Baseline Profile as a separate codec has a significant impact on the structure and shape of the final profile in a multi-codec use case.
[0213] Naturally, the proposed method can also be applied to different codecs such as VP9, AV1, VVC, etc. Figure 14 is a block diagram of a multi-codec ABR streaming system 1400 incorporating multi-codec ABR ladder generation using the methods described herein, according to one embodiment. As can be seen, the components shown in Figure 14 may correspond to the ABR streaming system 100 of Figure 1. Figure 14 includes further details regarding components used in determining and streaming the multi-codec encoding ladder. Additionally, similar to Figure 1, an embodiment may have any number of individual components, which may be distributed across various geographic locations and / or executed by any number of computers (e.g., computer servers).
[0214] Video in the multi-codec ABR streaming system 1400 comes from a video source 1405. Depending on the desired functionality, this source may include an origin server that stores encoded video in some intermediate format, or it may be a live stream, delivered, for example, over the RTMP protocol.
[0215] This video, along with some additional information, is then received by the Multi-Codec ABR Profile Generator 1410, which generates a description of the entire encoding ladder presented as a Manifest 1415, and specific encoding instructions are delivered to the Encoders 1420 (Encoders 1-N) that are responsible for encoding each stream. The Manifest 1415 and encoding instructions may include, for example, the codec type, target bit rate, resolution, frame rate, and other parameters for the streams to be generated. The encoded streams are then placed on the Content Origin Server 1425 and pushed to the CDN+Access Network 1430 for distribution to the Clients 1435.
[0216] The manifest 1415 generated by the multi-codec ABR profile generator 1410 may be further processed by the manifest filtering / generation logic 1440, which may retain only renditions relevant to each type of client in the streaming system based on the capabilities of such clients. For example, for a client that can only decode H.264 baseline content, only H.264 baseline encoded renditions may be retained. Or, considering for example an H.264 / HEVC switchable codec, each of the following rates R may be retained regardless of the codec being used: i+1 ≧R i About Q, better quality than previous rate i+1 ≧Q i 4B, which shows that in this case the 595 kbps H.264 stream should be omitted. Once filtered, the final encoding ladder may be stored as a DASH or HLS manifest 1445 on the manifest origin+CDN 1450.
[0217] During playback, as each type of client 1435 attempts to access a link to the content, device detection logic 1455 can parse the request to determine the type of client 1435 seeking the content. Such detection can be performed either by the receiving server or by Javascript logic embedded in the web page. Such detection can be based on several commonly available parameters of the client system, such as web browser type and version, OS type and version, device vendor and model, chipset vendor and model, etc.
[0218] Once the type of client 1435 has been identified, it can be directed to an appropriately filtered manifest that contains the set of streams that the client can support. Once the manifest is received, each client 1435 can operate as normally expected in an ABR streaming system.
[0219] Some features of this multi-codec ABR streaming system 1400 that are not present in conventional ABR streaming systems include the following: A multi-codec ABR profile generator 1410 that generates an output manifest 1415 that includes streams encoded with multiple codecs; Manifest filtering / generation logic 1440, which filters the output of the multi-codec ABR profile generator 1410 and customizes ladders to the capabilities of each of the clients 1435 in the system (specifically, the filtering process may leave only streams encoded using a single codec, or a combination of streams encoded with multiple codecs, and a bitrate-sorted sequence of those may also generate sequences of monotonically increasing quality levels); Device detection 1455, which identifies the type of client 1435 and selects the manifests filtered / generated by 1445 for the detected type of client 1435; and · A client 1435 then receives and plays the content described in the filtered manifest.
[0220] According to some embodiments, the manifest filtering / generation logic 1440 may rely on the quality annotations generated by the ABR profile generator 1410 or on the "quality_rank" identifier defined in the DASH standard. If the "quality_rank" identifier is used, it must be properly assigned across all adaptation sets for all codecs. Additional annotations that allow switching between adaptation sets in the case of DASH manifests must also be included in the case of switchable clients.
[0221] A specific example of a manifest filtering algorithm is shown in Figure 15, which leaves behind a set of points that are monotonically increasing in terms of rate and quality. Given the ladder points for the two codecs, we start by simply merging them and sorting them according to rate in block 1510. A selection loop 1520-1560 then follows, where at each step 1530 only those points of the rate-sorted list that also provide an increment in terms of quality are subsequently stored 1540. In step 1570 the final filtered ladder is obtained and sent to the output.
[0222] In the example of Figure 15, for example, codec 1 can be H.264 and codec 2 can be HEVC. A similar algorithm can also be applied recursively by considering multiple codecs. In this case, for example, it can be applied first to merge rates from H.264 Baseline Profile and H.264 Main Profile, and then to merge all H.264 ladder points into HEVC ladder points. In this way, a ladder for codecs that can switch between all three codecs (H.264 Baseline, H.264 Main, and HEVC) can be generated.
[0223] Depending on the circumstances, the proposed embodiments for ABR profile generation and profile filtering may be implemented in software and / or hardware (e.g., one or more hardware or software components of a computer such as that shown in FIG. 17 and described below), or may be compiled and stored on a computer medium as computer instruction code. Such code may then be executed on a local computer that contains the media to be transcoded, or may be executed remotely (e.g., in a cloud instance). It may also be executed simultaneously on multiple such cloud instances, processing different media or different chunks of the same media. Execution of such operations may be orchestrated by creating and using web APIs.
[0224] FIG. 16 is a flow diagram illustrating a method 1600 for creating a multi-codec encoding ladder according to an embodiment that may use one or more of the optimization techniques described above. It should be understood that the functions provided in the blocks illustrated in FIG. 16 are provided by way of example. Alternative embodiments may add, omit, combine, separate, and otherwise modify the illustrated functions. One or more functions of the blocks illustrated in FIG. 16 may be performed, for example, by the ABR profile generator 1410, the encoder 120, or other components of the multi-codec ABR streaming system described herein. As such, these functions may be implemented using software and / or hardware means of a computer system, such as the computer system illustrated in FIG. 17 described below.
[0225] At block 1610, the method 1600 may begin by obtaining, by a computer system, source content including video. The source content may be provided in any of a variety of formats, including a digital master, a mezzanine file, an input stream (e.g., a live stream), or a separated video fragment stream. As mentioned above, the source content may be obtained from a video source 110, which may include an origin server.
[0226] In block 1620, the function includes generating an encoding ladder for the source content, where each video stream of the plurality of video streams defined by the encoding ladder includes a respective bit rate and a respective codec of the plurality of codecs for encoding the source content.
[0227]
number
[0228]
number
[0229]
number
[0230]
number
[0231]
number
[0232]
number
[0233]
number
[0234]
number
[0235]
number
[0236]
number
[0237]
number
[0238] As illustrated in the examples and embodiments described above, the process of creating the encoding ladder may be optimized to take into account any of a variety of factors. In some embodiments, for example, the method 1600 may include obtaining, for each codec of a plurality of codecs, a quality-rate function of an individual codec of the source content indicating a relationship between a bit rate and a quality value of the source content, and generating an encoding ladder for the source content based on the quality-rate function of the individual codec:
[0239]
number
[0240]
number
[0241]
number
[0242] Alternative embodiments may include additional or alternative considerations and / or optimization algorithms. For example, in some embodiments, the encoding ladder is further based on a network bandwidth distribution and a distribution of clients that can stream the source content once it is encoded using the encoding ladder, the distribution of clients including clients that can switch between a first codec and a second codec. Additionally or alternatively, generating the encoding ladder may include determining the multiple video streams using an iterative process in which an initial number is selected and the steps of (1) determining a figure of merit function for the selected number and (2) increasing the value of the selected number for the next iteration are repeated until the figure of merit function reaches a maximum value. In some embodiments, the figure of merit function is based on a quality rate function for each codec of the multiple codecs, a network bandwidth distribution, or a distribution of clients, or any combination thereof. The network bandwidth distribution may include a probability density function determined based on bandwidth statistics collected taking into account device types, CDNs, or distribution regions, or any combination thereof.
[0243] Finally, some embodiments of the method 1600 may further include encoding the content based on the encoding ladder. That is, some embodiments may further include creating an individual encoded content for each stream of the encoding ladder by encoding the source content with a codec and bit rate of the respective stream, and storing the individual encoded content. As mentioned in the above embodiments, the encoding and storage of the encoded content may be performed by one or more encoders and a content origin server, respectively (the content origin server may then send the encoded content to the CDN+access network).
[0244] FIG. 17 is a block diagram of an embodiment of a computer system 1700 that may be used to perform, in whole or in part, one or more of the functions of the methods described herein, including the methods illustrated in FIG. 6, FIG. 15, and FIG. 16. The computer system 1700 may implement one or more of the components of an ABR streaming system (e.g., ABR streaming system 100 of FIG. 1 and / or multi-codec ABR streaming system 1400 of FIG. 14), including an ABR profile generator and / or encoder. It should be noted that FIG. 17 is merely intended to generally illustrate the various components, and any or all of these components may be utilized as appropriate. Thus, FIG. 17 broadly illustrates how individual system elements may be implemented in a relatively separated or relatively more integrated manner. In addition, it should be noted that the components illustrated in FIG. 17 may be localized on a single device and / or distributed across various network devices that may be located in different geographic locations. As discussed above, the components of the ABR streaming system may be executed in the cloud. Thus, computer system 1700 may be one of many computer systems (eg, computer servers) configured to implement various components of an ABR streaming system.
[0245] Computer system 1700 is shown as comprising hardware elements that may be electrically coupled (or otherwise in communication as appropriate) via a bus 1705. The hardware elements may include a processing unit 1710, which may include, but is not limited to, one or more general purpose processors, one or more special purpose processors (such as digital signal processing chips and / or graphic acceleration processors), and / or other processing structures, and may be configured to perform one or more of the methods described herein. Computer system 1700 may also comprise one or more input devices 1715, which may include, but is not limited to, a mouse, a keyboard, a camera, and / or a microphone, and one or more output devices 1720, which may include, but is not limited to, a display device, and / or a printer, and the like.
[0246] The computer system 1700 may further comprise (and / or be in communication with) one or more non-transitory storage devices 1725, which may include, but are not limited to, local and / or network accessible storage and / or may include, but are not limited to, disk drives, drive arrays, optical storage devices, solid-state storage devices, such as random access memory (RAM) and / or read-only memory (ROM), which may be programmable and / or flash updatable, etc. Such storage devices may be configured to implement any suitable data store, including, but not limited to, various file systems, and / or database structures, etc. Such data stores may include databases and / or other data structures used to store and manage messages and / or other information sent to one or more devices, as described herein.
[0247] The computer system 1700 may also include a communication subsystem 1730, which may include wireless communication technologies as well as wired technologies (such as Ethernet, coaxial communication, and Universal Serial Bus (USB)) managed and controlled by a wireless communication interface. As such, the communication subsystem 1730 may include a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and / or a chipset, etc., which may enable the computer system 1700 to communicate over one or more communication networks with any device on the respective network, including other computer systems and / or any other electronic devices described herein (including operations and / or applications executed thereon). As such, the communication subsystem 1730 may be used to receive and transmit data as described in the embodiments herein.
[0248] In many embodiments, the computer system 1700 further comprises a working memory 1735, which may include a RAM or ROM device as described above. Software elements shown to be located in the working memory 1735 may include an operating system 1740, device drivers, executable libraries, and / or other code, such as one or more applications 1745, which may comprise computer programs provided by various embodiments and / or may be designed to implement methods and / or configure systems provided by other embodiments as described herein. Merely by way of example, one or more procedures described with respect to the methods discussed above may be implemented as code and / or instructions executable by a computer (and / or a processing unit within a computer). In one aspect, such code and / or instructions may then be used to configure and / or adapt a general-purpose computer (or other device) to perform one or more operations in accordance with the methods described.
[0249] A set of these instructions and / or code may be stored on a non-transitory computer-readable storage medium, such as storage device 1725 and / or working memory 1735 described above. In some cases, the storage medium may be incorporated within a computer system, such as computer system 1700. In other embodiments, the storage medium may be separate from the computer system (e.g., a removable medium, such as an optical disk) and / or provided in an installation package, such that it may be used to program, configure, and / or adapt a general-purpose computer with the instructions / code stored therein. These instructions may take the form of executable code that can be executed by computer system 1700 and / or may take the form of source and / or installable code that, upon compilation and / or installation on computer system 1700 (e.g., using any of a variety of commonly available compilers, installation programs, compression / decompression utilities, etc.), takes the form of executable code.
[0250] It should be apparent to one skilled in the art that substantial variations can be made according to specific requirements. For example, customized hardware could also be used and / or particular elements could be implemented in hardware, software (including portable software such as applets), or both. Furthermore, connection to other computing devices, such as network input / output devices, could be employed.
[0251] With reference to the accompanying drawings, components that may comprise memory may include non-transitory machine-readable media. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any storage medium that participates in providing data that causes a machine to operate in a specific manner. In the embodiments provided herein above, various machine-readable media may participate in providing instructions / code to a processing unit and / or other devices for execution. Additionally or alternatively, machine-readable media may be used to store and / or transport such instructions / code. In many implementations, computer-readable media are physical and / or tangible storage media. Such media may take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Common forms of computer readable media include, for example, magnetic and / or optical media, any other physical media with a pattern of holes, RAM, Programmable ROM (PROM), Erasable PROM (EPROM), Flash-EPROM, any other memory chip or cartridge, a carrier wave as described herein below, or any other medium from which a computer can read instructions and / or code.
[0252] The methods, systems, and devices discussed herein are examples. Various embodiments may omit, substitute, or add various procedures or components, as appropriate. For example, features described with respect to one embodiment may be combined in various other embodiments. Different aspects and elements of the embodiments may be combined in a similar manner. Various components of the drawings provided herein may be embodied in hardware and / or software. Also, because technology evolves, many of the elements are examples that do not limit the scope of the disclosure to those specific examples.
[0253] Throughout this specification, references to "one example," "one example," "particular example," or "exemplary implementation" mean that a particular feature, structure, or characteristic described in connection with a feature and / or example may be included in at least one feature and / or example of the claimed subject matter. Thus, appearances of the phrases "in one example," "in one example," "particular example," or "in a particular implementation," or other similar phrases in various places throughout this specification do not necessarily all refer to the same features, examples, and / or limitations. Furthermore, particular features, structures, or characteristics may be combined in one or more examples and / or characteristics.
[0254] Some of the detailed descriptions contained herein are presented in terms of algorithms or symbolic representations of operations on binary digital signals stored in a memory of a specific apparatus or a special-purpose computing device or platform. In the context of this particular specification, terms such as specific apparatus include a general-purpose computer that, when programmed, performs specific operations according to instructions from the program software. Algorithmic descriptions or symbolic representations are examples of techniques used by those skilled in the art of signal processing or related arts to convey the substance of their work to others skilled in the art. An algorithm is generally considered here to be a self-consistent sequence of operations or similar signal processing that leads to a desired result. In this context, operations or processing involve physical manipulations of physical quantities. Typically, but not necessarily, such quantities may take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, or otherwise manipulated. It has proven convenient at times, primarily for reasons of common usage, to refer to such signals as bits, data, values, elements, symbols, characters, terms, numbers, or numeric values, or the like. It should be understood, however, that all of these or similar terms do not relate to appropriate physical quantities, but are merely convenient labels. Unless specifically specified otherwise, and as will be apparent from the discussion herein, it will be understood that throughout this specification discussions utilizing terms such as "processing," "computing," "calculating," or "determining" refer to the actions or processing of a specific apparatus, such as a special purpose computer, a special purpose computing device, or a similar special purpose electronic computing device. In the context of this specification, therefore, a special purpose computer or a similar special purpose electronic computing device can manipulate or transform signals that are typically represented as physical electronic or magnetic quantities in memories, registers, or other information storage devices, transmission devices, or display devices of the special purpose computer or similar special purpose electronic computing device.
[0255] As used herein, the terms "and," "or," and "and / or" may include a variety of meanings that are expected to also depend, at least in part, on the context in which such terms are used. Typically, when "or" is used in connection with a list such as A, B, or C, it is intended to mean A, B, and C when used in an inclusive sense, and A, B, or C when used in an exclusive sense. Additionally, as used herein, the term "one or more" may be used to describe any feature, structure, or characteristic in the singular, or may be used to describe multiple features, structures, or characteristics or any other combination of features, structures, or characteristics. However, it should be noted that this is merely an illustrative example and that claimed subject matter is not limited to this example.
[0256] While what are presently believed to be exemplary features have been shown and described, those skilled in the art will recognize that various other modifications can be made and equivalents can be substituted without departing from the claimed subject matter. Additionally, many modifications can be made to adapt a particular situation to the teachings of the claimed subject matter without departing from the central concept described herein. Therefore, it is intended that the claimed subject matter not be limited to the particular examples disclosed, but that such claimed subject matter also includes all aspects that come within the scope of the appended claims and equivalents thereof.
Claims
1. 1. A method for creating a multi-codec encoding ladder, comprising: obtaining, by a computer system, source content including a video; determining a total number of streams of the video for an encoding ladder; generating the encoding ladder for the source content, Each video stream of the plurality of video streams defined by the encoding ladder includes a respective bit rate and a respective codec of a plurality of codecs for encoding the source content, the video stream being a parameter description, and the encoding ladder: [0010] and [0025] The individual bit rates of [0030] and [0045] a first video stream and a second video stream of a first codec having respective quality values of [0050] Bit rate and [006] a third video stream of a second codec having a quality value of Including, [0070] and [0080] and the encoding ladder is generated by outputting a description of parameters of all video streams generated for each codec from the plurality of codecs used to encode the source content; the encoding ladder defines quality values and bit rates to alternate between the first codec and the second codec to stream the video.
2. The method of claim 1 , wherein the parameter description includes bit rate, resolution, codec type, and quality attributes.
3. obtaining, for each codec of the plurality of codecs, a quality-rate function of the respective codec for the source content indicating a relationship between a bit rate of the source content and a quality value; generating the encoding ladder for the source content based on the quality-rate functions of the individual codecs; [0090] and [0010] is determined using the quality rate function of the first codec; ##EQU00011## The method of claim 1 , wherein A is determined using the quality-rate function of the second codec.
4. The method of claim 3 , wherein the quality-rate function for each codec of the plurality of codecs is determined from one or more probe encodings of the source content in each codec of the plurality of codecs.
5. The encoding ladder comprises: Network bandwidth distribution; a distribution of clients capable of streaming the source content once the source content is encoded using the encoding ladder, the distribution of clients including clients capable of switching between the first codec and the second codec; The method of claim 3 further based on
6. Generating the encoding ladder includes determining the plurality of video streams using an iterative process, wherein an initial number is selected in the iterative process; (1) determining a figure of merit function for the selected numbers; (2) incrementing the value of the selected number for the next iteration; The method of claim 5 , wherein a is repeated until the figure of merit function reaches a maximum value.
7. The figure of merit function is the quality-rate function for each codec of the plurality of codecs; said network bandwidth distribution; or said distribution of client types; or Any combination of these The method according to claim 6, which is based on
8. said network bandwidth distribution being Device type, Content Delivery Network (CDN), or Distribution area, or Any combination of these The method of claim 5 , further comprising determining a probability density function based on collected bandwidth statistics for
9. The method of claim 1 , wherein the bitrate and corresponding quality value for each video stream of the plurality of video streams monotonically increases within the encoding ladder.
10. [Math. 12] , ##EQU00013## , or ##EQU14## , or any combination thereof, Structural Similarity Index Index (SSIM) value, Peak signal-to-noise ratio (PSNR) value, Multi-scale SSIM (MS-SSIM) values, or Video Multi-Method Assessment Fusion (VMAF) Value The method of claim 1 , each comprising:
11. For each stream of the encoding ladder: creating a respective encoded content by encoding the source content using the codec and the bit rate of the respective stream; storing each encoded content; The method of claim 1 further comprising:
12. 1. A computer system for creating a multi-codec encoding ladder, comprising: Memory, and one or more processing units communicatively coupled to the memory, the one or more processing units comprising: Get the source content, including the video, determining a total number of streams of said video for an encoding ladder; generating the encoding ladder for the source content; each video stream of the plurality of video streams defined by the encoding ladder includes a respective bit rate and a respective codec of a plurality of codecs for encoding the source content, a video stream being a parameter description; The encoding ladder comprises: ##EQU00015## and ##EQU00016## The individual bit rates of ##EQU00017## and [0018] a first video stream and a second video stream of a first codec having respective quality values of [0019] Bit rate and [0020] a third video stream of a second codec having a quality value of Including, ##EQU00021## and [0022] and the encoding ladder is generated by outputting a description of parameters of all video streams generated for each codec from the plurality of codecs used to encode the source content; The encoding ladder defines quality values and bit rates to alternate between the first codec and the second codec to stream the video.
13. the one or more processing units: obtaining, for each codec of the plurality of codecs, a quality-rate function of the respective codec for the source content, the quality-rate function indicating a relationship between a bit rate of the source content and a quality value; generating the encoding ladder for the source content based on the quality-rate functions of the individual codecs; Using the quality rate function of the first codec [0023] and ##EQU00024## Determine, Using the quality rate function of the second codec [0025] Determine, The computer system of claim 12 further configured to:
14. 14. The computer system of claim 13, wherein the one or more processing units are further configured to determine the quality-rate function for each codec of the plurality of codecs from one or more probe encodings of the source content in each codec of the plurality of codecs.
15. the one or more processing units: Network bandwidth distribution; a distribution of clients capable of streaming the source content once the source content is encoded using the encoding ladder, the distribution of clients including clients capable of switching between the first codec and the second codec; 14. The computer system of claim 13, further configured to generate the encoding ladder for the source content further based on:
16. To generate the encoding ladder, the one or more processing units are configured to determine the plurality of video streams using an iterative process, wherein an initial number is selected in the iterative process; (1) determining a figure of merit function for the selected numbers; (2) incrementing the value of the selected number for the next iteration; 16. The computer system of claim 15, wherein: ##EQU1## is repeated until the figure of merit function reaches a maximum.
17. The figure of merit function is the quality-rate function for each codec of the plurality of codecs; said network bandwidth distribution; or said distribution of client types; or Any combination of these 20. The computer system of claim 16, wherein the one or more processing units are further configured to:
18. the one or more processing units: Device type, Content Delivery Network (CDN), or Distribution area, or Any combination of these 16. The computer system of claim 15, further configured to determine the network bandwidth distribution based on collected bandwidth statistics for.
19. 13. The computer system of claim 12, wherein the encoding ladder is generated to switch codecs between the first codec, the second codec, and a third codec.
20. 1. A non-transitory computer readable medium having instructions stored therein for creating a multi-codec encoding ladder, the instructions, when executed by one or more processing units, causing the one or more processing units to: Get source content, including video, determining a total number of streams of said video for an encoding ladder; generating the encoding ladder for the source content; each video stream of the plurality of video streams defined by the encoding ladder includes a respective bit rate and a respective codec of a plurality of codecs for encoding the source content, a video stream being a parameter description; The encoding ladder comprises: [0026] and [0027] The individual bit rates of [0028] and [0029] a first video stream and a second video stream of a first codec having respective quality values of [0030] Bit rate and [0031] a third video stream of a second codec having a quality value of Including, [0032] and [Equation 33] and the encoding ladder is generated by outputting a description of parameters of all video streams generated for each codec from the plurality of codecs used to encode the source content; The encoding ladder defines quality values and bit rates to alternate between the first codec and the second codec to stream the video.
Citation Information
Patent Citations
Techniques to optimize bitrate and resolution during encoding
JP2018513604A
Variability in available levels of quality of encoded content
US20130268961A1
Optimization of encoding profiles for media streaming
WO2018102756A2
Iterative techniques for encoding video content
WO2018156997A1
