Fragment quality-oriented adaptive stream creation

Abstract

The present invention relates to segment quality-oriented adaptive streaming. Embodiments providing improved stream generation are provided. A video comprising a plurality of segments and an encoding ladder specifying a plurality of maximum average bitrates (MABs) are received. A plurality of intermediate bitrates interspersed between the plurality of MABs are selected. A target average bitrate (TAB) segment is generated by encoding a first segment using a first MAB, and a first intermediate average bitrate (IAB) segment is generated by encoding the first segment using a first intermediate bitrate. Quality scores are generated for the first TAB segment and the first IAB segment. A first output segment is selected for a first segment of the first MAB based on the quality score, wherein the first output segment is a first TAB segment or a first IAB segment. Upon receiving a request for a first segment of the first MAB, the first output segment is output.

Description

Background Art

[0001] Streaming services (e.g., video streaming, audio streaming, or multimedia streaming) have become increasingly common and desired by a wide variety of users. Streaming content enables users to receive the multimedia they want when they want / need it, rather than downloading it in advance. That is, rather than forcing users to wait for a long time for a large download (e.g., an entire movie, which also forces users to store a large amount of data), streaming enables users to obtain smaller segments of a larger video on demand (e.g., just before the segment begins).

[0002] To improve streaming services, adaptive bitrate streaming (ABR) has been developed. The premise of ABR is to provide multiple streams (often called variants) at different bitrates, allowing the client to dynamically select a variant based on network conditions and other factors. These variants are encoded using an encoding ladder with multiple rungs, each rung corresponding to a desired output bitrate. Often, multiple resolutions (e.g., 1920x1080, 1280x720, etc.) can coexist on one encoding ladder, with a given resolution on one or more consecutive rungs. ABR streams are segmented (e.g., divided into discrete sub-assets, each typically a few seconds long), and the client can adapt from one variant to another at segment boundaries as network conditions change. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] In order that the manner in which the above-described aspects are achieved may be understood in detail, a more particular description of the embodiments described herein, briefly summarized above, may be had by reference to the drawings.

[0004] It is to be noted, however, that the drawings illustrate typical embodiments and are therefore not to be considered limiting; other equally effective embodiments are contemplated.

[0005] Figure 1 Depicted is a system configured to perform segment quality-directed adaptive stream creation according to some embodiments disclosed herein.

[0006] Figure 2 A system configured to generate various encoded output streams to provide segment quality-oriented adaptive stream creation according to one embodiment disclosed herein is shown.

[0007] Figure 3 Depicted is a pool of segments for various different streams having different maximum average bit rates according to one embodiment disclosed herein.

[0008] Figure 4 Segment quality analysis for providing segment quality-oriented adaptive streaming according to one embodiment disclosed herein is shown.

[0009] Figure 5A Depicted is a collection of different bitrate streams generated using the segment quality-directed adaptation technique disclosed herein.

[0010] Figure 5B Depicted is an optimized set of different bitrate streams generated using the segment quality-directed adaptation technique disclosed herein.

[0011] Figure 6 An optimized reduction of the available output bitrate using the segment quality-directed adaptation technique disclosed herein is shown.

[0012] Figure 7 is a flow chart illustrating a method for segment quality-directed adaptive stream generation according to some embodiments disclosed herein.

[0013] Figure 8 is a flow chart illustrating a method for segment quality-directed adaptive stream generation according to some embodiments disclosed herein.

[0014] Figure 9 A streaming system configured to provide segment quality-directed adaptive streaming according to one embodiment disclosed herein is depicted. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure provide techniques for segment quality-oriented adaptive (SQA) stream generation. The SQA system achieves reduced resource usage, including lower storage requirements and reduced network bandwidth required to transmit the SQA stream. A major challenge in existing ABR applications is selecting an appropriate ABR encoding ladder. A coding stack or ladder refers to a set of variants / bitrates, where each bitrate corresponds to a step or rung on the ladder, with the highest bitrate at the top of the ladder and the lowest bitrate at the bottom. However, different content can involve significantly different encoding requirements and complexities.

[0016] A given encoding ladder targeting an output defined by its steps (e.g., resolution or bitrate) can easily produce more bits than required for a certain content. For example, an 8.5 megabits per second (Mb / s) stream is suitable for live-action video, but is too large for a simple animation. A more restricted ladder (e.g., with a lower bitrate, a lower resolution, or both) may be sufficient for such a simple animation, but is likely insufficient to produce sufficient quality for a certain content (e.g., live-action). When selecting an encoding ladder, existing systems are often forced to choose between existing ladders, some of which produce sufficient quality but waste bits, while others produce more conservative bits but at a lower quality than desired. Some attempts to alleviate these concerns involve title-based ABR encoding, where each title (e.g., each video source) can potentially have its own encoding stack.

[0017] However, not only is each content asset (title) often different from other content assets, but it can also vary within a single asset, which reduces the efficacy of selecting a single predefined ladder. Without adaptive encoding ladders, the system must again either select a ladder that is sufficient to produce the desired quality when needed within the asset, but produces wasted (unnecessary) bits when not needed, or select a second ladder that is optimal for most of the asset but insufficient for some parts. Many existing systems select a higher bitrate ladder, which results in excessive bits being generated to ensure high video quality. However, this choice results in wasted costs for the provider (including storage and delivery of unnecessary data), and increased costs for the customer (including increased data consumption, which is particularly harmful when the user is on a metered data plan). In addition, this approach can affect the quality of experience, such as by forcing more frequent rebuffering, longer download times, etc.

[0018] While some ABR format protocols allow for dynamic step changes within an asset, many do not. Embodiments of the present disclosure provide techniques for providing dynamic intra-title step changes that are protocol-agnostic. For example, in some embodiments, a client device can be presented with a set of bitrate options (e.g., an 8.5Mb / s stream, a 7Mb / s stream, a 5.5Mb / s stream, etc.). In various embodiments, the client device can automatically select between streams as conditions change, or the user can manually select a specific stream. Embodiments of the present disclosure enable provider systems to dynamically switch to lower bitrate streams when switching to a lower bitrate stream does not compromise the user experience (e.g., when segment quality is not compromised), thereby reducing network load and storage requirements. In particular, in some embodiments, the system dynamically selects different bitrate streams within each resolution. That is, the decision to select a lower bitrate segment can be performed on a per-resolution basis. In such embodiments, the system selects among lower bitrate segments with the same resolution as the original segment / stream (e.g., a 720p segment would not be selected for a 1080p stream).

[0019] Figure 1A system 100 is depicted that is configured to perform segment quality-oriented adaptive stream creation in accordance with some embodiments disclosed herein. In the illustrated embodiment, a media asset 105 can be provided. The asset 105 can include any media, such as audio, video, and multimedia that includes both video and audio. In the illustrated embodiment, the asset 105 is depicted as a sequence of segments 110A-N. Any number of techniques can be used to perform this segmentation. Generally speaking, each segment 110A-N is a portion or section of the asset 105, and the segments 110A-N are associated with an order or sequence. In this way, by receiving the segments 110A-N in sequence, the entire asset 105 can be received.

[0020] In the illustrated embodiment, an initial encoding ladder 115 is depicted. The encoding ladder 115 includes four rungs 120A-D, each rung associated with a corresponding target average bit rate (TAB). Although four rungs 120 are depicted, in various embodiments, the encoding ladder 115 may include any number of rungs / bit rates. In the illustrated embodiment, the first rung 120A corresponds to an 8.5 Mb / s TAB, rung 120B corresponds to a 7 Mb / s TAB, rung 120C corresponds to a 5.5 Mb / s TAB, and rung 120D corresponds to a 4.25 Mb / s TAB.

[0021] In existing systems, an encoding ladder 115 is used to encode each fragment 110 of an asset 105. That is, each fragment 110 is encoded at a bitrate specified by each rung 120 of the encoding ladder 115. Consequently, existing systems generate four separate output sequences, one for each TAB, each with a sequence of fragments encoded at the corresponding TAB. The client can then indicate its maximum average bitrate (MAB), which is used to select one of the four streams. For example, if the client requests an 8.5 Mbps MAB, the existing system provides a stream encoded at 8.5 Mbps (that is, each fragment 110 in that stream is encoded at 8.5 Mbps). Similarly, if the client requests a 7 Mbps MAB, the existing system provides a stream encoded at 7 Mbps. This results in some fragments 110 being transmitted with unnecessary / extra bits.

[0022] In the illustrated embodiment, the existing encoding ladder 115 is supplemented with a set of intermediate average bitrates (IABs), illustrated using rungs 125A-C. In the illustrated embodiment, each new rung 125A-C is inserted between two existing rungs 120A-D. In various embodiments, any number of IAB rungs 125 may exist. For example, two or more IAB rungs may exist between existing TAB rungs, or zero IAB rungs may exist between two given TAB rungs. Furthermore, in various embodiments, the system may utilize zero or more IAB rungs below the lowest bitrate TAB. In some embodiments, the IAB is selected to evenly divide the gap(s) between the existing TABs. As shown, combining the existing encoding rungs 115 with the additional IAB rungs 125 produces an enhanced encoding rung 130 having the additional rungs. Furthermore, in some embodiments, MAB / TAB / IAB are defined on a per-resolution basis. That is, each available resolution of the video may have an associated set of MABs, TAB segments, and IAB segments. For example, there may be one or more IABs above the highest TAB for 720p resolution, because these higher IABs are part of the existing 1080p resolution set.

[0023] As shown, the enhanced coding ladder 130 can be used to generate one or more output streams 135, each of which includes a sequence of coded segments 140A-N. In embodiments of the present disclosure, the system 100 allows each output stream 135 to include coded segments 140A-N encoded at different bit rates. That is, while existing systems force each segment in a given stream to be encoded at the same bit rate, the output stream 135 can vary. For example, coded segment 140A may be encoded using 8.5 TAB, while coded segment 140B may be encoded using 6.25 IAB, and coded segment 140C may be encoded using 7 TAB. Thus, while existing systems may allow clients to move up and down the ladder (e.g., select a higher or lower bit rate for the current segment or the next segment), embodiments of the present disclosure generate output streams 135 that include coded segments 140 that may have different bit rates. Therefore, using embodiments of the present disclosure, the system can output / transmit a lower bit rate when possible, even if the client continues to request a higher bit rate.

[0024] In an embodiment of the present disclosure, the system is quality-oriented and does not select a bitrate based solely on available bandwidth. Instead, the system adaptively selects the optimal or best possible bitrate for each rung of the ladder while maintaining a quality level for each segment. This allows the system to maintain a media quality similar to or matching the target bitrate while minimizing the number of bits required to deliver the content.

[0025] In some embodiments, a separate output stream 135 is generated for each MAB provided by the system 100. In some embodiments, the available MABs correspond to the TABs specified in the encoding ladder 115. When generating the output stream 135 for a given MAB, as discussed in more detail below, the system 100 can dynamically select appropriate encoded segments 140. In one embodiment, this includes dynamically selecting between segments encoded with the MAB and segments encoded at a lower bit rate if the lower bit rate does not degrade segment quality.

[0026] In one embodiment, to do this, the system 100 may iteratively encode each segment 110A-N using each of the available bit rates (e.g., each TAB and each IAB). In some embodiments, rather than iterative or serial encoding, the system 100 may encode each segment 100A-N in parallel using each available bit rate. The encoded segments may then be evaluated and combined to generate a set of MAB streams. In some embodiments, this evaluation is a post-processing step that occurs after all segments have been encoded at the available bit rates. Once the MAB streams are generated, they may be streamed or transmitted to any number of clients. Advantageously, by selecting lower bit rate segments when possible (e.g., when doing so does not degrade the quality of the segments received by the client), embodiments of the present disclosure reduce storage costs (because fewer resources can be used to store each MAB stream) and network costs (because fewer bits are transmitted for a given MAB).

[0027] Figure 2 A system 200 is shown according to one embodiment disclosed herein, configured to generate various encoded output streams to provide segment quality-oriented adaptive stream creation. In the illustrated embodiment, a source 205 and an encoding ladder 210 are provided to a transcoder 215. The source 205 is typically a content asset, such as a video asset, an audio asset, or a multimedia asset. As discussed above, the encoding ladder 210 typically specifies multiple bitrates. In some embodiments, the encoding ladder 210 is selected based on the content or type of the source 205. For example, the system 200 can utilize different ladders based on whether the source 205 includes two-dimensional animated content, three-dimensional computer-rendered content, or live-action video content. Other factors that increase complexity (and therefore recommend a more robust encoding ladder with a high bitrate, a larger number of ladders, or both) include significant motion in the frame, extensive color changes, etc. In some embodiments, one or more existing techniques are used to evaluate the source 205 to select an encoding ladder 210 appropriate for the type and content of the source 205.

[0028] In the illustrated embodiment, transcoder 215 utilizes a plurality of encoders 220A-N to encode source 205 into a set of encoded outputs 225A-N using encoding ladder 210. Specifically, encoder 220A generates encoded output 225A, encoder 220B generates encoded output 225B, and so on. In one embodiment, a corresponding encoder 220 is used for each rung / bitrate specified by encoding ladder 210. For example, if the highest bitrate in encoding ladder 210 is 8.5 Mbps, encoder 220A can be configured to encode source 205 at 8.5 Mbps. In some embodiments, encoding ladder 210 is enhanced with one or more IABs between traditional TABs. In other embodiments, transcoder 215 selects one or more IABs based on the TABs specified in encoding ladder 210 and configures encoders 220 accordingly for the TABs and IABs.

[0029] In one embodiment, encoding source 205 using a given encoder 220 includes encoding each individual segment of source 205 at the bit rate of encoder 220 so that the segments can be transmitted (e.g., over a network) at the corresponding bit rate. Thus, encoded output 225 may include a sequence of encoded segments of source 205. In some embodiments, as discussed in more detail below, when constructing an output stream for a given MAB, system 200 selects an encoded segment from encoded outputs 225A-N.

[0030] In one embodiment, each TAB specified in the encoding ladder 210 is generally equivalent to a corresponding MAB. That is, in a conventional system, when a client requests a stream for a given MAB, the system selects and outputs encoded video at a bitrate equal to that MAB. However, in embodiments of the present disclosure, the system 200 can intelligently select a lower bitrate (e.g., one or more IABs or lower TABs) to output, which reduces computational load. For example, such an embodiment can reduce storage requirements and delivery costs by providing segments at a lower bitrate than requested.

[0031] Although not included in the depicted embodiment, in some embodiments, the system 200 additionally performs quality assessment on each encoded output 225. That is, the system 200 can evaluate each individual segment of each encoded output 225 to generate a quantitative and objective quality score for each encoded segment. This allows the system 200 to know the visual quality of each segment at each available bitrate and resolution. In various embodiments, the quality assessment can utilize one or more objective quality algorithms, such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Video Multi-Method Assessment Fusion (VMAF), etc. This produces a quality assessment for each segment, which can be a single value (e.g., a weighted combination of one or more objective scores) or a set of values (e.g., a minimum score, a maximum score, an average score, a standard deviation, a rolling average score, etc.).

[0032] Figure 3 300 depicts segment pools for various streams with different maximum bit rates, according to one embodiment disclosed herein. In the illustrated embodiment, the enhanced coding ladder 130 is depicted to visualize the available bit rates, and a set of pools 315A-C is depicted for a number of potential MABs 305A-C. In some embodiments, after quality assessment is completed for each encoded segment, a segment is selected from the pool of encoded TAB / IAB segments for each MAB. That is, for each segment of the output, the system can select an encoded IAB or TAB segment for each MAB. For example, for the first segment of the original input source, the system would select the first segment of the first encoded version for the first MAB 305A (at 8.5 Mb / s), the first segment of the second encoded version for the MAB 305B (at 7 Mb / s), the first segment of the third encoded version for the MAB 305C (at 5.5 Mb / s), and so on. This process is repeated for each segment of the input source (and for each available MAB).

[0033] As discussed above, in conventional systems, the system simply selects TAB segments corresponding to the same bit rate as the requested MAB. Thus, for MAB 305A, existing systems simply select segments encoded at 8.5 Mb / s TAB. However, in the illustrated embodiment, the system defines a pool of potential segments for each MAB 305A-C. The pool of potential coded segments for MAB 305 includes the corresponding TAB for MAB 305, as well as a set of one or more TABs, IABs, or both IABs and TABs included below the corresponding TAB. As shown, for MAB 305A (corresponding to 8.5 Mb / s), pool 315A includes segments corresponding to 8.5 Mb / s TAB, segments corresponding to 7.75 Mb / s IAB, segments corresponding to 7 Mb / s TAB, segments corresponding to 6.25 Mb / s IAB, and segments corresponding to 5.5 Mb / s TAB.

[0034] Therefore, during segment selection for MAB 305A, the system can select any segment between 8.5 Mb / s and 6.25 Mb / s. In various embodiments, pools 315 can be static or dynamic. In one embodiment, pools 315 utilize a static configuration (e.g., a predefined fixed depth). In another embodiment, the depth of each pool 315 (e.g., the number of included tiers or bitrates) is predetermined. In yet another embodiment, the depth is set dynamically, for example, based on content characteristics, machine learning, etc. In various embodiments, the lowest depth (bitrate) of a given pool 315 represents the best-case (e.g., lowest) average bitrate output for that given MAB 315. For example, in the illustrated embodiment, pool 315b of MAB 305B with a 4.8 Mb / s IAB at its lowest end (7 Mb / s maximum bitrate output) indicates that the resulting average bitrate of the 7 Mb / s output will be greater than or equal to 4.8 Mb / s and less than or equal to 7 Mb / s.

[0035] In various embodiments, these pools 315 can be defined or configured within the encoder itself or in one or more downstream components. Although the illustrated enhancement coding ladder 130 utilizes seven rungs, in various embodiments, as discussed above, any number of layers / bitrates can exist. Additionally, while three MABs 305A-C are included in the illustrated embodiment, the system can provide any number and variety of MABs. Furthermore, while the depicted embodiment includes pools 315A-C having depths of five, four, and three, respectively, in various embodiments, the depth of each pool 315 can be any value, and each pool can include any number of potential bitrates. Additionally, as shown, each pool 315 can include not only the lower (one or more) IABs, but also the lower TABs.

[0036] Figure 4 A diagram 400 is depicted illustrating segment quality analysis for providing segment quality-oriented adaptive streaming, according to one embodiment disclosed herein. In some embodiments, the segment selection process for a given MAB (discussed above, from pool 315) includes evaluating the quality scores previously generated for each encoded version of a given (original) segment. In one embodiment, to select an encoded segment for a given MAB, the quality of each segment in the corresponding pool is evaluated. The system can select the encoded segment with the lowest bitrate whose quality score is within a predefined tolerance of the quality score of the MAB's corresponding TAB segment.

[0037] In some embodiments, the tolerance is configured to be less than a just noticeable difference (JND) so that a typical user cannot notice any difference in visual quality between the TAB segment and the selected lower bitrate segment. This ensures that the selected segment quality is no visually worse than the segment quality that would be produced by selecting the corresponding TAB segment of the MAB, while minimizing the final output bitrate. For example, for an 8.5 Mb / s MAB, the system can determine a quality score for the corresponding TAB segment (e.g., a segment encoded at 8.5 Mb / s). This TAB segment quality score can then be used to select a lower bitrate segment for inclusion in the MAB pool.

[0038] In the illustrated graph 400, segment quality scores are plotted on the vertical axis and the individual segments in the output are plotted on the horizontal axis. Points 405, 410, and 415 represent the quality scores of various encoded segments. For example, points 405A-D represent the quality scores of four different variants (e.g., different bit rates) of Seg1. Similarly, points 410A-D represent the quality scores of four different variants (e.g., different bit rates) of Seg2, and points 415A-D represent the quality scores of four different variants (e.g., different bit rates) of SegN. In the illustrated embodiment, the depth of the pool of MABs depicted is four. That is, when generating an output stream with a given MAB, there are four segment variants to choose from. Thus, for each segment in the stream, the system can choose between four potential encoded segments (e.g., four bit rates).

[0039] For example, assume that the depicted graph 400 is for 8.5 Mb / s MAB. In such an embodiment, the highest quality variant of each encoded segment (highest points 405, 410, and 415) would likely be a segment encoded at 8.5 Mb / s TAB (e.g., points 405A, 410A, and 415A). Continuing with this example, the next lower point (e.g., points 405B, 410B, and 415B) corresponds to the next lower bit rate (e.g., 7.75 Mb / s IAB), while the next lower point beyond the 7.75 Mb / s rate (points 405C, 410C, and 415C) corresponds to 7 Mb / s TAB. Additionally, the lowest quality segments in the pool (represented by points 405D, 410D, and 415D) correspond to segments encoded at 6.25 Mb / s IAB.

[0040] As shown in the figure, depending on the specific content of a given segment, the quality of each lower bit rate can vary significantly. For example, for the first segment (Seg1), segments encoded at 8.5Mb / s, 7.75Mb / s, and 7Mb / s have similar quality scores, while the segment encoded at 6.25Mb / s is significantly lower. For the second segment (Seg2), all four encoded segments have similar quality scores. Furthermore, for segment N (SegN), the segment encoded at 8.5Mb / s has relatively high quality, and the remaining segments (encoded at lower bit rates) do not have nearly as high visual quality.

[0041] In various embodiments, as discussed above, the system compares the quality score to one or more thresholds to determine which encoding variant should be selected for a given segment. In the illustrated embodiment, these thresholds are depicted by tolerances 420A-C. In at least one embodiment, tolerances 420A-C correspond to differences in visual quality that are imperceptible to a typical human user. As discussed above, the minimum acceptable quality score is defined based on the quality of the highest quality segment (e.g., a segment encoded in TAB corresponding to MAB). Thus, tolerances 420A-C are described as measuring the permissible variance in quality below the highest scoring encoding variant for each segment.

[0042] In some embodiments, other techniques may be used in addition or alternatively to define the allowable variance. For example, in one such embodiment, the system may optionally utilize a predefined lower limit, wherein segments below the lower limit are ignored even if they fall within the tolerance. Similarly, in some embodiments, the system may utilize a predefined upper limit, wherein if the quality score is above the predefined upper limit, the predefined tolerance may be ignored.

[0043] For the first segment (Segl), the encoding variant represented by point 405C is within tolerance 420A, while the next lower point 405D is not. Therefore, in the illustrated embodiment, the system will select the encoding segment represented by point 405C (e.g., 7 Mb / s TAB) to use as the first segment for output on the 8.5 Mb / s MAB stream. For the second segment (Seg2), the lowest bitrate segment that falls within tolerance 420B is the lowest segment in the pool represented by point 410D (e.g., 6.25 Mb / s segment). Therefore, for the second segment, the system will select that encoding variant. Additionally, for segment N (SegN), only the 8.5 Mb / s TAB segment is within tolerance 420C. Therefore, for this segment, the system will utilize the highest bitrate variant.

[0044] Advantageously, this enables the system to dynamically select segments encoded at a lower bit rate when doing so does not impact the visual quality of the stream. This can significantly reduce the required network bandwidth without impacting the user experience. Additionally, in some embodiments, the system can store only selected encoded segments of a stream and discard all other encoded segments. For example, the system can store the segments represented by points 405C, 410D, and 415A and discard the remaining encoded segments. This significantly reduces the storage cost required to store the MAB output stream. A similar evaluation can be performed for each available MAB. Additionally, as discussed in more detail below, in some embodiments, the system can intelligently identify segment(s) used in more than one MAB stream and merge these into a single shared copy, further reducing storage costs.

[0045] Figure 5A A collection 500A of different bitrate streams generated using the segment quality-guided adaptation techniques disclosed herein is depicted. The illustrated embodiment includes an 8.5 Mb / s MAB output 505A, a 7 Mb / s MAB output 505B, and a 5.5 Mb / s MAB output 505C. While three MABs are depicted, any number of MABs may be available in various embodiments. As discussed above, in various embodiments, each MAB typically corresponds to the highest bitrate to be used by the corresponding output. A client or provisioning system may typically select a MAB for a given client based on network conditions, computing conditions, and other factors. Each MAB output 505A-C includes a series of encoded segments 510.

[0046] As discussed above, in a traditional (non-SQA) transcoding system, the MAB outputs would only contain segments corresponding to the TAB of the MAB. For example, an 8.5 Mb / s MAB output would only contain segments from an 8.5 Mb / s TAB encoder. However, using embodiments of the present disclosure, each MAB output can include lower bitrates for various segments if visual quality is not compromised. However, it is worth noting that each MAB output 505A-C outputs an encoded variant of the same original segment for the corresponding portion of the output. For example, all three MAB outputs 505A-C begin with the encoded segments corresponding to "Seg1" (segments 510A, 510G, and 510M) and proceed in sequence to "Seg7." However, the encoding bitrate for each is different.

[0047] For example, in the illustrated embodiment, MAB output 505A includes segment 510A encoded at 5.5 Mb / s, segment 510B encoded at 8.5 Mb / s, segment 510C encoded at 8.5 Mb / s, segment 510D encoded at 6.25 Mb / s, segment 510E encoded at 6.25 Mb / s, and segment 510F encoded at 8.5 Mb / s. MAB output 505B includes segment 510G encoded at 5.5 Mb / s, segment 510H encoded at 7 Mb / s, segment 510I encoded at 6.25 Mb / s, segment 510J encoded at 5.5 Mb / s, segment 510K encoded at 6.25 Mb / s, and segment 510L encoded at 7 Mb / s. Additionally, MAB output 505C includes segment 510M encoded at 5.5 Mb / s, segment 510N encoded at 4.8 Mb / s, segment 510O encoded at 5.5 Mb / s, segment 510P encoded at 5.5 Mb / s, segment 510Q encoded at 4.8 Mb / s, and segment 510R encoded at 5.5 Mb / s.

[0048] Thus, as shown, by utilizing the embodiments disclosed herein, each MAB output 505A-C can selectively utilize segments with significantly reduced bandwidth. In one embodiment, this stream generation is performed during a preparation phase (e.g., when the asset is ingested), and the MAB outputs 505A-C are stored in one or more storage locations. This may include storing the sequential segments 510A-R themselves, storing a sequential list of segments (e.g., using pointers or links to the actual storage location of each segment), etc. Subsequently, when any client requests a maximum bitrate stream, the system can retrieve the matching MAB output 505 and begin streaming the corresponding segments 510 in sequence.

[0049] In particular, as discussed above, MAB output 505 requires fewer storage resources than conventional systems. For example, in the 8.5 MB / s MAB output 505A, only three segments, 510B, 510C, and 510F, are actually 8.5 MB / s segments. The remaining segments are encoded for a lower bit rate and therefore have a smaller file size and require fewer resources to store, process, and transmit. In some embodiments, additional optimizations can be applied to further reduce computing resources.

[0050] Figure 5B One such optimized set 500B of different bitrate streams generated using the segment quality-oriented adaptation techniques disclosed herein is depicted. As discussed above, with the SQA transcoding techniques disclosed herein, the segments in each output are selected from a pool of MABs, so a given MAB output may contain segments that were encoded for a lower bitrate than the MAB. Furthermore, because pools can overlap, the same encoded segment may exist in multiple outputs. For example, Figure 3 As shown, pools 315A and 315B overlap (eg, both include 7 MB / s TAB and 6.25 MB / s IAB), indicating that both 8.5 MB / s MAB output and 7 MB / s MAB output can use the same encoded segments (eg, segments encoded at 7 MB / s or 6.25 MB / s).

[0051] In some embodiments of the present disclosure, the encoding process includes attaching a unique identifier to each encoded segment (including both TAB and IAB segments). In one such embodiment, additional downstream processing can identify common / shared encoded segments between variants and store and deliver a common single copy of the shared segments. This improves efficiency and storage resources compared to storing copies of the same encoded segment. That is, the system only needs to store one copy of each encoded segment, and each MAB output 505 can include a pointer to that copy as needed.

[0052] Furthermore, optimizing these common segments can reduce the network resources required to deliver duplicate segments through a content delivery network (CDN). In some embodiments, many CDNs cache segments at relatively local locations. For example, received segments can be cached locally within an enterprise, at a location, or by an Internet Service Provider (ISP). These caches can significantly reduce network load. When multiple clients stream the same MAB output 505, the local cache can be used to provide the next segment (when it is available) rather than requesting it from the original stream provider.

[0053] By marking common segments with identifiers, a cache can provide locally cached segments that were initially downloaded for any MAB output. For example, assume an encoded segment is downloaded to a first client as part of a first stream at a first bandwidth (e.g., an 8.5Mb / s stream). If another client using the same MAB starts that stream, the cache can be used to provide the next segment. However, in existing systems, clients using another MAB (e.g., a 7Mb / s MAB) are forced to download the corresponding variant from the source provider. However, by marking overlapping segments (e.g., if both the 8.5Mb / s and 7Mb / s streams use the same encoded segment for one or more parts of the output), the cache can provide a cached copy of the common segment to a second client (streaming the 7Mb / s output). This significantly reduces the required bandwidth.

[0054] like Figure 5A As shown, all three MAB outputs 505A-C use a 5.5 Mb / s bit rate for Segl (segments 510A, 510G, and 510M). Figure 5B In the embodiment shown, the system has merged these common segments into a single shared segment 510A and replaced the other copies (in MAB outputs 505B and 505C) with pointers to the shared copy. Although depicted as being stored with MAB output 505A, in some embodiments, the system may store the shared copy in a separate location and include pointers to it in all MAB outputs 505A-C.

[0055] Similar optimizations are shown in the fourth and fifth segments of the stream. That is, MAB outputs 505B and 505C both use the 5.5 Mb / s coded segment 510 in Seg4, and the system has therefore eliminated one copy of segment 510P, replacing it with a pointer to the shared segment 510J. Additionally, because MAB outputs 505A and 505B share the 6.25 Mb / s coded segment 510 for Seg5, the system has similarly replaced one or both with a pointer to the shared copy of that coded segment.

[0056] Figure 6A graph 600 is depicted, which illustrates a further optimized reduction in available output bitrate using the segment quality-guided adaptation techniques disclosed herein. As discussed above, the average bitrate output by a given MAB can be anywhere between the highest bitrate of the MAB (the MAB itself) and the lowest bitrate included in the pool. Thus, the average output bitrate can often be very similar between different MABs (at a given screen resolution). Additionally, if two or more MABs in a pool share the same lowest bitrate, the average bitrates output by such MABs may be exactly the same (if each MAB uses the lowest bitrate in the pool) or very close to each other. This can provide various opportunities for further optimization.

[0057] Graph 600 depicts a visual representation of a possible distribution of SQA output bitrates. In the depicted embodiment, various assets are plotted along the horizontal axis, and the average bitrate per MAB for the assets is plotted on the vertical axis. For example, the top lines 605A, 610A, and 615A may represent the highest available MAB (e.g., 8.5 Mb / s), the next lines 605B, 610B, and 615B may represent the next lowest MAB (e.g., 7 Mb / s), and the bottom lines 605C, 610C, and 615C may represent the lowest MAB (e.g., 5.5 Mb / s). Of course, while three MABs are shown, in various embodiments, any number of MABs may be available.

[0058] As shown in the figure, for the first media source (labeled Asset1), the average bitrate varies significantly depending on the MAB selected. That is, there is a large gap in the actual average bitrate received by the client depending on the MAB they choose. This is visualized by the relatively wide spread between lines 605A-C. However, for Asset3, the gap between lines 615A-C is significantly reduced, indicating that the average bitrate changes very little regardless of the MAB selected. In the most extreme example, the lines will converge to a single point (for example, if the same segment bitrate is selected for two or more of the MAB outputs).

[0059] In many embodiments, the ABR ladder presented to the client is designed to have rungs whose rates are neither too close nor too far apart. If the rungs are too far apart, the system may try to provide the bandwidth required for the higher bitrate, but provide lower quality video at the next lowest bitrate. With the extra rungs in between, the system can find a balance. If the rungs are too close, the client may frequently move up and down the ladder to adjacent rates as conditions change slightly. This significantly reduces the efficiency that can be gained through local caching. However, as Figure 6As shown, some embodiments of the present disclosure can generate a ladder with steps that are relatively close to each other in some cases by optimizing the output bit rate.

[0060] Therefore, in some embodiments, the system can take further action. In at least one embodiment, the master list of ABR variants can still include all MAB, even though the actual average bitrate of each of these variants may be much less than their advertised rate. This can be beneficial if the peak bitrate of an asset differs significantly from the average bitrate. That is, a given MAB output can still have some segments encoded at the original MAB peak rate, even if the majority of segments are encoded at a much lower rate. By retaining all MAB outputs, content providers can ensure that each output stream maintains high quality, including any peaks.

[0061] In some embodiments, an analysis of the resulting MAB output average bitrate, segment peak rate, etc. can be performed. Based on this analysis, the system can determine to remove one or more of the MAB output variants. For example, suppose all MAB outputs are optimized to use the exact same set of encoded segments. In this case, all MAB outputs would be identical (having the same average bitrate and peak bitrate). Therefore, in such an embodiment, there is no benefit in retaining all MAB outputs or presenting them to the client.

[0062] Therefore, in some embodiments, the system can evaluate and identify MAB outputs that should be removed. In the illustrated embodiment, for Asset 1, the system has determined that the average bitrates are sufficiently dispersed, so all MAB options should be retained. However, for Asset 2, the system has determined that the variant represented by line 610B is within a predefined threshold distance of the variant represented by line 610A or 610C (or both). Therefore, as shown by the dashed line, the system has determined to remove the variant indicated by line 610B. In such an embodiment, when streaming Asset 2, the client can be presented with two variants: the one represented by line 610A (e.g., at 8.5 Mb / s) or the one represented by line 610C (e.g., at 5.5 Mb / s). In addition to removing the advertised intermediate variant (e.g., 7 Mb / s), in some embodiments, the system can also delete this MAB output from its storage to reduce storage usage.

[0063] Similarly, for Asset 3, based on determining that the highest variant (represented by dashed line 615A) and the lowest variant (represented by dashed line 615C) are within a predefined threshold difference from the variant represented by line 615B, the system has decided to eliminate the highest variant and the lowest variant. Therefore, in one embodiment, the system can determine to remove these variants (e.g., by avoiding advertising them, deleting them, etc.). This can significantly improve storage efficiency and further improve the overall operation of the system.

[0064] In some embodiments, eliminating MAB streams involves refraining from advertising MAB to clients and forcing them to choose from the remaining MAB(s). However, in at least one embodiment, the streaming system can continue to advertise all legacy MABs. Upon receiving a request for a MAB stream that has been eliminated, the system can instead provide an identified similar MAB stream. This allows the system to optimize its resources without any changes or notifications to the client devices.

[0065] Figure 7 7 is a flow chart illustrating a method 700 for segment quality-oriented adaptive stream generation according to some embodiments disclosed herein. Method 700 begins at block 705, where a streaming system receives a source asset. In various embodiments, as discussed above, the asset may include audio, video, a multimedia stream with both video and audio, and the like. In one embodiment, the source asset may include an uncompressed asset that may be compressed / encoded at any bit rate. Method 700 then proceeds to block 710.

[0066] At block 710, the streaming system receives an encoding ladder for use when encoding a source asset. In some embodiments, this is a predefined ladder, as discussed above. In at least one embodiment, the ladder is a type-specific or content-specific ladder that has been selected based on the type or content of the asset. In some embodiments, the ladder has IABs already integrated into it. In another embodiment, the streaming system may first select one or more IABs based on the TABs specified in the ladder.

[0067] Method 700 then proceeds to block 715, where the streaming system selects segments of the video. In some embodiments, the asset has been previously delineated into segments. In another embodiment, the streaming system uses one or more known techniques to evaluate the asset to define the segments. After the segments have been selected, method 700 proceeds to block 720, where the streaming system selects one of the encoding rungs specified in the (enhanced) encoding ladder. At block 725, the streaming system encodes the selected segments at the selected bitrate.

[0068] At block 730, the streaming system then uses one or more quality assessment techniques to assess the segments to generate a quality score for the encoded segments (that is, for the selected segments encoded at the selected bitrate). Method 700 then proceeds to block 735, where the streaming system determines whether there is at least one additional rung (i.e., bitrate) specified in the encoding ladder. If so, method 700 returns to block 720 to select the next rung. Otherwise, if the selected segment has been encoded at all possible bitrates, method 700 continues to block 740.

[0069] At block 740, the streaming system determines whether there is at least one additional segment in the asset that has not yet been encoded. If so, method 700 returns to block 715 to select the next segment. Otherwise, method 700 proceeds to block 745. In this way, the streaming system can generate encoding variants of each segment at every possible bitrate. These encoded segments are stored for evaluation and analysis to generate the MAB output set. In various embodiments, this storage can include any number of suitable alternatives, including on one or more hard drives or solid-state drives, in memory (e.g., random access memory), etc.

[0070] At block 745, the streaming system selects one of the available / potential MAB outputs. As discussed above, in various embodiments, each MAB represents a maximum average bitrate option. The client can select among the MABs, and the streaming system sequentially transmits segments from the selected MAB to the client. Method 700 then continues to block 750, where the streaming system selects one of the base segments. In one embodiment, a base segment refers to a segment encoded in the TAB of the selected MAB. For example, for an 8.5 Mb / s MAB, each base segment is one of the segments in the source asset that is encoded at 8.5 Mb / s. In another embodiment, a base segment refers to an original (unencoded) segment in the source asset.

[0071] Method 700 then proceeds to block 755, where the streaming system identifies a pool of potential segments for the base segment. In one embodiment, as discussed above, this includes determining a pool depth. The pool depth can be static or dynamic and can be manually selected based on machine learning, etc. In one embodiment, if the base segment is a TAB segment of an MAB, identifying the pool includes identifying potential alternative segments for the selected segment (e.g., variants of the same segment encoded at a lower bit rate). In embodiments where the base segment refers to a segment in the original source, identifying the pool includes identifying all encoded segments corresponding to the selected base segment (within the depth of the pool of the selected MAB).

[0072] At block 760, the streaming system selects and identifies the minimum bitrate segment in the pool. In one embodiment, as discussed above, this selection is based on the quality score of each segment. For example, the streaming system may select the potential segment with the lowest bitrate in the pool as long as the quality score is within a predefined criterion (e.g., a tolerance or threshold for the highest quality segment). If the quality score is too low, the streaming system may select a segment with the next highest bitrate. Once the minimum bitrate acceptable segment of the base segment has been identified, method 700 continues to block 765, where the streaming system determines whether there is at least one additional base segment to be evaluated for the selected MAB.

[0073] If yes, method 700 returns to block 750. Otherwise, method 700 continues to block 770. Thus, the streaming system can iteratively process each segment in the source asset to identify, for each corresponding source segment, the lowest bitrate encoded segment of sufficient quality for the selected MAB. At block 770, the streaming system stores the selected sequence of encoded segments as the generated MAB output. In this way, the streaming system can generate an optimized output for the selected MAB, reducing the bitrate of some segments without sacrificing quality. This improves computational efficiency.

[0074] Method 700 then proceeds to block 775, where the streaming system determines whether there is at least one additional MAB that has not yet been evaluated / generated. If so, method 700 returns to block 745. Otherwise, if all MAB outputs have been created, method 700 continues to block 780. In this manner, the streaming system can iteratively or in parallel generate optimized MAB outputs for all available MABs. At block 780, the streaming system can optionally continue optimizing segment storage, output, or both, as discussed above.

[0075] For example, in one such embodiment, as discussed above with respect to FIG5 , the streaming system may identify common coded segments between different MAB outputs and merge these common segments into a single shared copy. Each MAB output may then be modified to include a link or pointer to the single shared copy. In some embodiments, as discussed above with respect to FIG5 , the streaming system may identify common coded segments between different MAB outputs and merge these common segments into a single shared copy. Figure 6 As discussed, the streaming system can analyze the similarity between each overall MAB output to identify sufficiently similar outputs. MAB outputs that are sufficiently similar can be eliminated from the system.

[0076] Although method 700 is depicted as a sequential or serial process for conceptual clarity, in some embodiments, various aspects of method 700 can be effectively performed simultaneously (e.g., in parallel). For example, in some embodiments, the system can encode a segment at a given bit rate and determine the quality of the segment at that bit rate, and then iteratively repeat these steps for each bit rate and segment. However, in some embodiments, the system can encode segments at multiple bit rates in parallel, which can reduce latency in the process.

[0077] Figure 8 8 is a flow chart illustrating a method for segment quality-guided adaptive stream generation according to some embodiments disclosed herein. Method 800 begins at block 805, where a streaming system receives a video comprising a plurality of segments. At block 810, the streaming system receives an encoding ladder specifying a plurality of maximum average bitrates (MABs). Additionally, at block 815, the streaming system selects a plurality of intermediate bitrates interspersed between the plurality of MABs. Method 800 then continues to block 820, where the streaming system generates a first target average bitrate (TAB) segment by encoding a first segment of the plurality of segments using a first MAB from the plurality of MABs. At block 825, the streaming system generates a first intermediate average bitrate (IAB) segment by encoding the first segment using a first intermediate bitrate from the plurality of intermediate bitrates. Furthermore, at block 830, the streaming system generates corresponding quality scores for the first TAB segment and the first IAB segment. The method 800 then proceeds to block 835, where the streaming system selects a first output segment for the first segment of the first MAB based on the corresponding quality score, where the first output segment is (i) the first TAB segment or (ii) the first IAB segment. At block 840, upon receiving the request for the first segment of the first MAB, the streaming system outputs the first output segment.

[0078] Figure 9A streaming system 905 configured to provide segment quality-directed adaptive streaming according to one embodiment disclosed herein is depicted. Although depicted as a physical device, in various embodiments, the streaming system 905 can be implemented as a virtual device or service, or implemented between multiple devices (e.g., in a cloud environment). As shown, the streaming system 905 includes a processor 910, a memory 915, a storage device 920, a network interface 925, and one or more I / O interfaces 930. In the illustrated embodiment, the processor 910 retrieves and executes programming instructions stored in the memory 915, and stores and retrieves application data residing in the storage device 920. The processor 910 generally represents a single CPU, a GPU, a CPU and a GPU, multiple CPUs, multiple GPUs, a single CPU or GPU with multiple processing cores, etc. Memory 915 is generally included to represent random access memory. The storage device 920 can be any combination of disk drives, flash-based storage devices, etc., and can include fixed storage devices, removable storage devices, or a combination of both, such as fixed disk drives, removable memory cards, cache, optical storage, network attached storage (NAS), or storage area network (SAN).

[0079] In some embodiments, input and output devices (such as a mouse, keyboard, monitor, touch screen, etc.) are connected via (one or more) I / O interface(s) 930. Additionally, via network interface 925, streaming system 905 can be communicatively coupled to one or more other devices and components (e.g., directly or via network 980, which may include the Internet, (one or more) local area networks, etc.). Furthermore, network 980 may include wired connections, wireless connections, or a combination of wired and wireless connections. As shown, processor 910, memory 915, storage device 920, (one or more) network interface(s) 925, and (one or more) I / O interface(s) 930 are communicatively coupled via one or more buses 975. In various embodiments, streaming system 905 may be implemented as a standalone device, as part of a cloud deployment, as an application executed on a user's phone or computer, etc.

[0080] In the illustrated embodiment, storage device 920 includes one or more source assets 960 and one or more corresponding encoded streams 965. Although depicted as residing in storage device 920, source assets 960 and encoded streams 965 may be stored in any suitable location. In one embodiment, as discussed above, source asset 960 is video, audio, or multimedia content that can be encoded and streamed. Encoded streams 965 are typically encoded versions of source asset 960. In some embodiments, as discussed above, each encoded stream 965 is associated with a corresponding MAB. Additionally, in some embodiments, a single source asset 960 may be associated with any number of corresponding encoded streams 965 (e.g., in different MABs). As discussed above, in one embodiment, each encoded stream 965 is generated to include a sequence of segments encoded at or below the MAB of the encoded stream 965. In some embodiments, as discussed above, encoded streams 965 may utilize pointers to shared / overlapping common segments.

[0081] In the illustrated embodiment, memory 915 includes a streaming application 935. Streaming application 935 is generally configured to perform one or more of the embodiments discussed herein. Although depicted as software residing in memory 915, in various embodiments, the functionality of streaming application 935 may be implemented using software, hardware, or a combination of software and hardware. As shown, streaming application 935 includes an encoding component 940, a quality component 945, a selection component 950, and an optimizer 955. Although depicted as discrete components for conceptual clarity, in various embodiments, the operations of encoding component 940, quality component 945, selection component 950, and optimizer 955 may be combined or distributed across any number of components and devices.

[0082] In one embodiment, the encoding component 940 is configured to receive a source asset 960 and encode each segment at one or more bit rates to generate a set of encoded segments for the source asset 960. As discussed above, in some embodiments, the encoding component 940 generates a separate encoded variant of each source segment at each available bit rate (e.g., at each TAB and IAB). These variants can then be subsequently analyzed to construct an optimized encoded stream 965.

[0083] The quality component 945 is generally configured to evaluate the visual quality of each encoded segment generated by the encoding component 940. To do so, the quality component 945 can use any number and combination of visual quality algorithms. In one embodiment, the quality component 945 generates a quality score for each encoded segment based on the analysis. The quality score can be a single value or a composite set of values.

[0084] In the illustrated embodiment, the selection component 950 is configured to select the best coded segment for each segment of each coded stream 965. In various embodiments, the selection component 950 does so based in part on the generated quality scores. In some embodiments, as discussed above, the selection component 950 does so by identifying a pool of potential coded segments (of the MAB of the coded stream 965) for each segment in the source asset 960. The selection component 950 can then select the coded segment with the lowest bitrate, as long as the segment is associated with a sufficient quality score. This process can be repeated until the entire coded stream 965 is generated. The selection component 950 can then iteratively repeat this process for each MAB available for the source asset 960.

[0085] In some embodiments, the optimizer 955 is generally configured to perform other optimizations, such as reducing duplication between the coded streams 965, as discussed above. To do so, the optimizer 955 can identify shared coded segments in different coded streams 965 and replace one or more of them with pointers to a single shared copy of the coded segment. In some embodiments, the optimizer 955 can also evaluate the coded streams 965 available for a given source asset 960 to identify coded streams 965 that are sufficiently similar (e.g., above a predefined threshold). Similar streams can be condensed into a single coded stream 965 (e.g., by deleting one or more of the similar coded streams 965). In doing so, the optimizer 955 can reduce the storage and transmission requirements of the streaming system 905.

[0086] Although depicted as a component within the streaming application 935, in some embodiments, the optimizer 955 can be a standalone component in the system or can operate as a set of one or more functions distributed throughout the system. For example, in one such embodiment, after the streaming application 935 has generated the TAB and IAB segments and stored them (e.g., in storage 920 or in memory 915), a separate optimizer 955 (or a set of optimizers 955) can access and evaluate these stored segments (e.g., data in memory 915 or files in storage 920) and optimize them based on overlap and shared usage, as discussed above.

[0087] In the illustrated embodiment, the streaming system 905 can communicate with a client system 985 via a network 980. Although a single client system 985 is depicted, any number of client systems 985 may be present. The client system 985 generally represents any computing device configured to receive data (e.g., the encoded stream 965) from the streaming system 905, including but not limited to smartphones, laptops, desktop computers, Internet-connected gaming systems, tablet computers, televisions, etc. The client system 985 is depicted as including a streaming client 990. Although not included in the illustrated embodiment, the client system 985 may generally include any number and kind of computing elements, such as processors, storage, memory, interfaces, etc.

[0088] In various embodiments, the streaming client 990 can be implemented using hardware, software, or a combination of hardware and software. In one embodiment, the streaming client 990 can be used to select one or more assets provided by the streaming system 905. For example, a user can use the streaming client 990 to select a movie or show they wish to stream. In some embodiments, the streaming application 935 then provides a collection or list of available MABs for the assets, and the streaming client 990 can select among them. In one embodiment, the streaming client 990 automatically selects the MAB (e.g., based on available bandwidth, local computing resources, etc.). In one such embodiment, the streaming client 990 can automatically and dynamically change the selected MAB during streaming based on changing conditions (e.g., requesting a lower MAB when available bandwidth decreases). In at least one embodiment, the user can manually select the MAB or quality and lock the stream to that quality.

[0089] As discussed above, in some embodiments, the streaming application 935 dynamically selects output segments that may or may not align with the bitrate requested by the streaming client 990. For example, even if the streaming client 990 requests an 8.5 Mbps stream, the streaming application 935 may dynamically provide segments at a lower bitrate if dynamically providing the segments at the lower bitrate does not degrade the quality by more than a preconfigured threshold. This change may be format / protocol agnostic, and the streaming client 990 may be completely unaware of the changing bitrate being utilized. Advantageously, the streaming client 990 therefore does not need to be reprogrammed or reconfigured to apply the embodiments disclosed herein.

[0090] In the present disclosure, reference is made to various embodiments. However, it should be understood that the present disclosure is not limited to the embodiments specifically described. On the contrary, any combination of the following features and elements, whether or not related to different embodiments, is contemplated to implement and practice the teachings provided herein. In addition, when describing the elements of an embodiment in the form of "at least one of A and B", it will be understood that the embodiment that exclusively includes element A, the embodiment that exclusively includes element B, and the embodiment that includes element A and B are each contemplated. In addition, although some embodiments can achieve advantages that are superior to other possible solutions or superior to the prior art, whether a particular advantage is achieved by a given embodiment is not limiting the present disclosure. Therefore, the aspects, features, embodiments and advantages disclosed herein are merely exemplary and are not considered to be elements or limitations of the appended claims unless explicitly stated in (one or more) claims. Similarly, reference to "the present invention" should not be interpreted as a summary of any inventive subject matter disclosed herein and should not be considered to be elements or limitations of the appended claims unless explicitly stated in (one or more) claims.

[0091] As will be appreciated by those skilled in the art, the embodiments described herein may be embodied as systems, methods, or computer program products. Thus, the embodiments may take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware aspects, all of which may be collectively referred to herein as "circuits," "modules," or "systems." Additionally, the embodiments described herein may take the form of a computer program product embodied in (one or more) computer-readable media having computer-readable program code embodied thereon.

[0092] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0093] The computer program code for performing the operations of the embodiments of the present disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, and traditional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., over the Internet using an Internet service provider).

[0094] Various aspects of the present disclosure are described herein with reference to flowchart illustrations or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustration or block diagram and the combination of blocks in the flowchart illustration or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the function / action specified in the (one or more) blocks of the flowchart illustration or block diagram.

[0095] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other device to operate in a specific manner so that the instructions stored in the computer-readable medium produce an article of manufacture that includes instructions for implementing the functions / actions specified in (one or more) blocks of the flowchart illustration or block diagram.

[0096] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable data processing apparatus, or other device provide a process for implementing the functions / actions specified in the block(s) of the flowchart illustration or block diagram.

[0097] The flowchart illustrations and block diagrams in the accompanying drawings illustrate the architecture, functionality and operation of possible implementations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart illustration or block diagram may represent a module, fragment or code portion, which includes one or more executable instructions for implementing the specified (one or more) logical functions. It should also be noted that in some alternative embodiments, the functions marked in the blocks may not occur in the order marked in the figure. For example, depending on the functions involved, two blocks shown in succession may actually be executed substantially simultaneously, or sometimes the blocks may be executed in reverse order or out of order. It will also be noted that each block of the block diagram or flowchart illustration and the combination of blocks in the block diagram or flowchart illustration can be implemented by a dedicated hardware-based system that performs the specified function or action or a combination of dedicated hardware and computer instructions.

[0098] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, and the scope of the same is to be determined by the claims that follow.

Claims

1. A method comprising: receiving a video comprising a plurality of segments; receiving an encoding ladder specifying a plurality of rungs associated with a plurality of target average bitrates (TABs) for a plurality of output streams, each output stream having an associated maximum average bitrate (MAB) matching a TAB of the plurality of TABs; selecting a plurality of intermediate bit rates interspersed among the plurality of TABs; Generate a first TAB segment by encoding a first segment of the plurality of segments using a first TAB of the plurality of TABs; generating a first intermediate average bitrate (IAB) segment by encoding the first segment using a first intermediate bitrate of the plurality of intermediate bitrates, wherein the first intermediate bitrate is lower than the first TAB; generating a first quality score for the first TAB fragment and generating a second quality score for the first IAB fragment; selecting the first IAB segment as a first output segment of a first segment of a first output stream having a first associated MAB in the plurality of output streams based at least in part on determining that the second quality score is within a predefined tolerance of the first quality score; and Upon receiving a request for a first segment of the first associated MAB, a first output segment of the first output flow is output.

2. The method according to claim 1, further comprising: selecting a first output segment for a first segment of a second output stream having a second associated MAB among the plurality of output streams; and Upon determining that the first output segment is for both the first segment of the first output stream and the first segment of the second output stream: storing a shared copy of the first output fragment; and A first segment of the first output stream and a first segment of the second output stream are associated with a shared copy of the first output segment.

3. The method of claim 2, further comprising: selecting, for the first output stream, a first output segment sequence corresponding to the plurality of segments; selecting, for the second output stream, a second output segment sequence corresponding to the plurality of segments; and Upon determining that the first output segment sequence is within a predefined threshold difference from the second output segment sequence, the first output stream or the second output stream is eliminated.

4. The method of claim 1, further comprising: For a first segment of the first output stream, determining a pool of potential segments, the pool of potential segments comprising a first TAB segment, one or more additional TAB segments encoded at a lower bitrate than the first associated MAB, and one or more IAB segments encoded at a lower bitrate than the first associated MAB, wherein: selecting a first output segment for a first segment of the first output stream comprises selecting a segment from the pool of potential segments, and Selecting a first output segment for the first segment of the first output stream includes identifying a lowest bitrate segment in the pool of potential segments, wherein the lowest bitrate segment is associated with a quality score that is within the predefined tolerance of the first quality score.

5. The method of claim 1, further comprising: selecting, for the first output stream, an output segment sequence corresponding to the plurality of segments, wherein each output segment in the output segment sequence is encoded at a bit rate equal to or lower than the first associated MAB; and Upon receiving a request for video of the first associated MAB, a sequence of output segments of the first output stream is output.

6. A non-transitory computer readable medium containing computer program code, which, when executed by operation of one or more computer processors, performs operations comprising: receiving a video comprising a plurality of segments; receiving an encoding ladder specifying a plurality of rungs associated with a plurality of target average bitrates (TABs) for a plurality of output streams, each output stream having an associated maximum average bitrate (MAB) matching a TAB of the plurality of TABs; selecting a plurality of intermediate bit rates interspersed among the plurality of TABs; Generate a first TAB segment by encoding a first segment of the plurality of segments using a first TAB of the plurality of TABs; generating a first intermediate average bitrate (IAB) segment by encoding the first segment using a first intermediate bitrate of the plurality of intermediate bitrates, wherein the first intermediate bitrate is lower than the first TAB; generating a first quality score for the first TAB fragment and generating a second quality score for the first IAB fragment; selecting the first IAB segment as a first output segment of a first segment of a first output stream having a first associated MAB in the plurality of output streams based at least in part on determining that the second quality score is within a predefined tolerance of the first quality score; and Upon receiving a request for a first segment of a first associated MAB, a first output segment of the first output flow is output.

7. The computer-readable medium of claim 6, wherein the operations further comprise: selecting a first output segment for a first segment of a second output stream having a second associated MAB among the plurality of output streams; and Upon determining that the first output segment is for both the first segment of the first output stream and the first segment of the second output stream: storing a shared copy of the first output fragment; and A first segment of the first output stream and a first segment of the second output stream are associated with a shared copy of the first output segment.

8. The computer-readable medium of claim 7, the operations further comprising: selecting, for the first output stream, a first output segment sequence corresponding to the plurality of segments; selecting, for the second output stream, a second output segment sequence corresponding to the plurality of segments; and Upon determining that the first output segment sequence is within a predefined threshold difference from the second output segment sequence, the first output stream or the second output stream is eliminated.

9. The computer-readable medium of claim 6, wherein the operations further comprise: For a first segment of the first output stream, determining a pool of potential segments, the pool of potential segments comprising a first TAB segment, one or more additional TAB segments encoded at a lower bitrate than the first associated MAB, and one or more IAB segments encoded at a lower bitrate than the first associated MAB, wherein: selecting a first output segment for a first segment of the first output stream comprises selecting a segment from the pool of potential segments, and Selecting a first output segment for the first segment of the first output stream includes identifying a lowest bitrate segment in the pool of potential segments, wherein the lowest bitrate segment is associated with a quality score that is within the predefined tolerance of the first quality score.

10. The computer-readable medium of claim 6, the operations further comprising: selecting, for the pair of first output streams, a sequence of output segments corresponding to the plurality of segments, wherein each output segment in the sequence of output segments is encoded at a bit rate equal to or lower than the first associated MAB; and Upon receiving a request for video of the first associated MAB, a sequence of output segments of the first output stream is output.

11. A system comprising: one or more computer processors; as well as a memory containing a program that, when executed by the one or more computer processors, performs operations including: receiving a video comprising a plurality of segments; receiving an encoding ladder specifying a plurality of rungs associated with a plurality of target average bitrates (TABs) for a plurality of output streams, each output stream having an associated maximum average bitrate (MAB) matching a TAB of the plurality of TABs; selecting a plurality of intermediate bit rates interspersed among the plurality of TABs; Generate a first TAB segment by encoding a first segment of the plurality of segments using a first TAB of the plurality of TABs; generating a first intermediate average bitrate (IAB) segment by encoding the first segment using a first intermediate bitrate of the plurality of intermediate bitrates, wherein the first intermediate bitrate is lower than the first TAB; generating a first quality score for the first TAB fragment and generating a second quality score for the first IAB fragment; selecting the first IAB segment as a first output segment of a first segment of a first output stream having a first associated MAB in the plurality of output streams based at least in part on determining that the second quality score is within a predefined tolerance of the first quality score; and Upon receiving a request for a first segment of the first associated MAB, a first output segment of the first output flow is output.

12. The system of claim 11, wherein the operations further comprise: selecting a first output segment for a first segment of a second output stream having a second associated MAB among the plurality of output streams; and Upon determining that the first output segment is for both the first segment of the first output stream and the first segment of the second output stream: storing a shared copy of the first output fragment; and A first segment of the first output stream and a first segment of the second output stream are associated with a shared copy of the first output segment.

13. The system of claim 12, wherein the operations further comprise: selecting, for the first output stream, a first output segment sequence corresponding to the plurality of segments; selecting, for the second output stream, a second output segment sequence corresponding to the plurality of segments; and Upon determining that the first output segment sequence is within a predefined threshold difference from the second output segment sequence, the first output stream or the second output stream is eliminated.

14. The system of claim 11, wherein the operations further comprise: For a first segment of the first output stream, determining a pool of potential segments, the pool of potential segments comprising a first TAB segment, one or more additional TAB segments encoded at a lower bitrate than the first associated MAB, and one or more IAB segments encoded at a lower bitrate than the first associated MAB, wherein: selecting a first output segment for a first segment of the first output stream comprises selecting a segment from the pool of potential segments, and Selecting a first output segment for the first segment of the first output stream includes identifying a lowest bitrate segment in the pool of potential segments, wherein the lowest bitrate segment is associated with a quality score that is within the predefined tolerance of the first quality score.

15. The system of claim 11, the operations further comprising: selecting, for the first output stream, an output segment sequence corresponding to the plurality of segments, wherein each output segment in the output segment sequence is encoded at a bit rate equal to or lower than the first associated MAB; and Upon receiving a request for video of the first associated MAB, a sequence of output segments of the first output stream is output.

Images