Transcoder Adjustments for Segment Fluidity

Abstract

To provide a system capable of performing adaptive bit rate streaming (ABR) while keeping scene change detection enabled.SOLUTION: A transcoder 510 comprises a plurality of encoders 512 configured to generate a plurality of variant streams, and a cross-variant Instantaneous Decoder Refresh (IDR) identifier 518. The cross-variant Instantaneous Decoder Refresh (IDR) identifier inspects the plurality of variant streams, identifies IDR frames in each of the variant streams, determines which IDR frames correspond to cross-variant boundaries, and demarcates the IDR frames corresponding to the cross-variant boundaries.SELECTED DRAWING: Figure 5

Description

background

[0001] A video format is a container that contains a data stream and metadata. The data stream may include a video stream and an accompanying audio stream. The metadata contains information about the video stream, such as bit rate, resolution, codec, etc. The bit rate correlates with the quality of the video stream, while the resolution describes the image size of the video stream.

[0002] A codec is a system or program that encodes or decodes a data stream. An encoder is a system or program that executes a codec to encode a data stream, while a decoder is a system or program that executes a codec to decode a data stream. Codecs are often used to compress data streams to reduce the size of video files for transmission over computer networks. Codecs may also be used to decompress data streams for media playback or data stream file manipulation. [Brief explanation of the drawings]

[0003] So that the above aspects may be achieved and understood in detail, a more particular description of the embodiments described herein briefly summarized above may be had by reference to the accompanying drawings, in which:

[0004] It should be noted, however, that other equally effective embodiments are possible and that the accompanying drawings depict typical embodiments and are therefore not to be considered limiting. [Figure 1] 1 illustrates a system for generating and delivering variant streams aligned across IDR frames, according to one embodiment. [Figure 2] 1 illustrates a transcoder for conditioning a video data stream for downstream segmentation, according to one embodiment. [Figure 3A]1 illustrates a transcoder that conditions a video data stream for segment fluidity, according to one embodiment. [Figure 3B] 1 illustrates the configuration of segment resources of various time lengths according to one embodiment. [Figure 3C] 10 illustrates maximum GOP duration and target segment duration for reducing keyframe pulsation, according to one embodiment. [Figure 4] 10 illustrates a variant stream adjusted for segment fluidity, according to one embodiment. [Figure 5] 1 illustrates a transcoder that conditions a video data stream for segment fluidity, according to one embodiment. [Figure 6] 10 illustrates a flowchart of the operation of a cross-variant IDR identifier, according to one embodiment. [Figure 7A] 1 illustrates a transcoder that conditions a video data stream for segment fluidity, according to one embodiment. [Figure 7B] 1 illustrates segments and GOPs in a video data stream according to one embodiment. [Figure 8] 1 illustrates a flowchart of the operation of a variable boundary sizer, according to one embodiment. Detailed Description

[0005] To facilitate a detailed understanding of the present disclosure, embodiments of the present invention disclosed herein may refer to HTTP Live Streaming (HLS) as a computer network streaming protocol, although the disclosed embodiments should not be construed as limited to any particular network protocol.

[0006] HLS is a computer network streaming protocol that can deliver media content via a master playlist that references multiple variant streams, each containing a media playlist that references a collection of consecutive segments of media content to be played in sequence by a client device.

[0007] Segments are identified in a media playlist by one or more URIs (Uniform Resource Identifiers) and, optionally, byte ranges. A URI is a string that identifies a resource on the Internet.

[0008] Each segment in a media playlist begins at a segment boundary specified by an IDR frame and ends on the frame immediately preceding the next segment boundary specified by an IDR frame. Segment boundaries are typically synchronized across multiple variant streams. ABR is performed by switching variant streams, which adjusts streaming quality at these aligned segment boundaries. Switching variant streams involves changing from a group of segments in one variant stream to a contiguous group of segments in another variant stream.

[0009] When recording video, the audiovisual (AV) input produces an uncompressed data stream. An encoder compresses the video stream into groups of pictures (GOPs), which contain I-frames, P-frames, and B-frames. Compressing the video stream results in smaller video files that are easier to store and transfer across a computer network, as opposed to storing and transferring larger video files.

[0010] A GOP is a series of consecutive compressed video frames whose boundaries are specified by I-frames. Each GOP begins with an I-frame (inclusive) and ends with a P-frame or a B-frame. Rendering a GOP results in the display or media playback of the video content in the video stream.

[0011] A GOP can be either open or closed. An open GOP contains at least one frame referenced by a frame in the previous GOP. A closed GOP contains only frame references within the current GOP. A closed GOP begins with an Instantaneous Decoder Refresh (IDR) frame.

[0012] An I-frame (also called a key frame or intra-frame) is a video frame that contains all the video data necessary to display itself completely by itself, without reference to other frames. Because an I-frame contains a complete frame of video data, it can be compressed independently by an encoder; that is, it is compressed without reference to other frames. Additionally, an I-frame serves as a reference for predicting one or more frames in a video stream. I-frames are generally larger than P-frames, which are generally larger than B-frames.

[0013] A P frame (also called a predicted frame) is a video frame that contains video data that changes between the current P frame and a previous I frame or P frame. That is, a P frame excludes the same video data (redundancy) as in a previous reference frame, but contains video data that is not the same as in the previous reference frame. Thus, the current P frame is encoded with reference to a previous I frame or P frame, and the current P frame excludes the redundancy to reduce the storage size of the current P frame. P frames serve as references for future P frames or any B frames.

[0014] For example, if a video stream shows a person speaking in front of a static background, the current P frame may exclude information about the background, since this information about the background does not change from the previous reference frame to the current P frame. The current P frame may include information about the person's movement, since this information about the person's movement changes from the previous reference frame.

[0015] A B-frame (also called a bidirectional frame) is a video frame that contains a prediction of how one or more objects in the frame have changed between the previous or subsequent I-frame, P-frame, or B-frame. Thus, unlike P-frames, B-frames can look both backward and forward to more types of reference frames. The purpose is to identify redundancies to exclude from the B-frame. The encoder predicts the differences between the current B-frame and the reference frames to reduce the storage size of the current B-frame.

[0016] An IDR frame is an I-frame that specifies a frame reference barrier. IDR frames cannot be referenced by frames in a previous GOP. Furthermore, an IDR frame prevents frames in its own GOP from referencing frames that precede it. That is, when a decoder encounters an IDR frame, it designates all previous frames in the frame buffer as not allowing references from any frames after the IDR frame. Therefore, IDR frames are used to force the decoder to refresh the frame buffer, thereby ensuring that the first frame of a GOP is always an IDR frame and that subsequent P and B frames cannot reference any frames that precede the IDR frame. The frame reference barrier feature of IDR frames does not exist in non-IDR I-frames, which allow frames that reference the I-frame to also reference frames that precede it.

[0017] Starting a GOP at an IDR frame may, for example, improve media playback at seek points, which are specific locations within the media playback selected by an end user of a media player.

[0018] When an end user seeks to a point in media playback, that seek point may be located on a P-frame or a B-frame. Whenever media playback starts from this location, it may produce distorted output because the P-frames and B-frames do not contain enough video data to reconstruct the complete picture that corresponds to the GOP that contains them. Instead, the media player must look at the I-frames referenced by the P-frames and B-frames to render the complete picture and begin playback of the current GOP.

[0019] Furthermore, in adaptive bitrate streaming (ABR)-based protocols (e.g., HLS), the IDR frame is the only viable variant stream switching point. ABR is a computer network streaming technology that includes measuring the network bandwidth and data throughput of a client device in real time and accordingly switching between variant streams to adjust the streaming quality delivered to the client device.

[0020] Variant streams represent different streaming bitrates of the same media content represented by the data stream. Generally, a higher streaming bitrate correlates with higher streaming quality, while a lower streaming bitrate correlates with lower streaming quality. Furthermore, a higher streaming bitrate equates to a larger relative size of the data stream, while a lower streaming bitrate equates to a smaller relative size of the data stream. Thus, a higher-quality stream requires more network bandwidth and data throughput from the client device to ensure continuous, uninterrupted media playback on the client device. Similarly, a lower-quality stream requires less network bandwidth and data throughput from the client device to ensure continuous, uninterrupted media playback on the client device.

[0021] Because non-IDR I-frames allow subsequent frames to reference frames that precede them, if a seek results in a media player loading a different variant stream, frames in the current variant stream may reference frames that were present in the previous variant stream but are no longer present in the current variant stream. The absence of frames could be due to, for example, differences in encoding (such as differences due to the use of scene change functionality, as described below). These types of frame references can disrupt media playback in ABR-based protocols. The aforementioned problem is solved by using IDR frames as variant stream switching points, since IDR frames do not allow subsequent frames to reference frames that precede them.

[0022] In embodiments herein, a transcoder adjusts video frames for downstream processing to create segments that are compatible for use with segment fluidity, which is a technique for grouping segments into resources that allow different segment durations to be presented to different platforms or media players. A segment contains one or more GOPs.

[0023] 1 illustrates a system for generating and delivering an IDR frame-aligned variant stream according to one embodiment. In this embodiment, an AV input 110 generates a video stream, and an encoder (not shown) compresses the video stream before delivering it to a transcoder 130. The encoded video stream includes a resolution and a bit rate, which represent image size and video quality, respectively. For example, the video stream has a resolution of 1080p and a bit rate of 3 Mbps.

[0024] Transcoder 130 is a processing component that compresses and conditions a video stream for downstream processing. Transcoder 130 may include a decoder 132, one or more encoders, and a cross-variant IDR identifier 134. Decoder 132 may convert a video stream to its pre-encoded format so that an encoder within transcoder 130 can process the video stream. For example, camera 114 may generate a video stream in a RAW format. Camera 114 may have an integrated encoder (not shown) that converts the video stream to a first encoding format. A downstream transcoder 130 may not be compatible with the first encoding format. Thus, if transcoder 130 receives a video stream in a first encoding format, decoder 132 of transcoder 130 may convert the encoded video stream to a RAW format. An encoder within the transcoder may then convert the video stream to a second encoding format.

[0025] In one embodiment, the cross-variant IDR identifier 134 is a software module that runs on hardware (e.g., a processor and memory). The cross-variant IDR identifier 134 partitions frames, which the encoder may use to adjust the variant stream to perform segment fluidity. Alternatively, the cross-variant IDR identifier 134 may use metadata instead of directly partitioning video frames in the manner described above. In yet another implementation, the cross-variant IDR identifier 134 looks across the entire variant stream (post-construction) and partitions cross-variant aligned frames downstream, as described in more detail below.

[0026] Each encoder in the transcoder 130 converts the decoded video stream into a variant stream. Each variant stream contains the same video stream (e.g., the same media content) but at a different bitrate. The transcoder 130 may also change the resolution of the video stream for each variant stream. For example, encoder 1 136a generates variant stream 1, which has a resolution of 1080p and a bitrate of 3Mbps. Encoder 2 136b generates variant stream 2, which has a resolution of 720p and a bitrate of 2Mbps. Encoder N 136n generates variant stream N, which has a resolution of Xp and a bitrate of YMbps. The variant streams include partition boundaries from the cross-variant IDR identifier 134 with segment boundaries aligned throughout the variant stream. The variant streams are delivered to the packager 140.

[0027] The packager 140 generates a playlist for each variant stream and segments the variant stream at the selected partition frames. The packager 140 then distributes the playlists and variant streams to the distribution network 150, where, in one embodiment, the playlists and variant streams are sent directly to one or more servers 152. The playlists are then sent to the client devices 156. The one or more servers 152 handle any fetches / requests for segments of the media content from the client devices 156. That is, the client devices 156 can use the playlists to submit requests to the distribution network 150 for segments identified in those playlists in order to play the media content therein.

[0028] Alternatively, the playlist and variant streams are sent directly to one or more servers 152. The playlist and variant streams are then sent to a content delivery network (CDN) 154, which distributes the playlist to client devices 156 and handles client device 156 fetches / requests for media content segments. In yet another embodiment, the playlist and variant streams are first sent to the CDN 154, which distributes the playlist to client devices 156 and handles client device 156 fetches / requests for media content segments. Using a CDN to handle segment fetches / requests can improve media playback by localizing CDN edge servers to reduce delivery times for fetched / requested segments.

[0029] FIG. 2 illustrates a transcoder for preparing a video data stream for downstream segmentation, according to one embodiment. In this embodiment, a transcoder 204 receives a source video frame 202 and generates an encoded frame 206 by encoding the source video frame 202 using an indicator that marks selected IDR frames within the encoded frame 206 as segment boundaries. In this embodiment, the segment boundary indicator is in-band segment boundary metadata 208. A segmenter 210 downstream of the transcoder segments the encoded frame 206 only at the IDR frames marked as segment boundaries. In another embodiment, the transcoder 204 may insert IDR frames into the variant stream to enforce desired segment boundaries. A segmenter 201 downstream of the transcoder segments the encoded frame 206 at each IDR frame of the encoded frame 206.

[0030] The transcoder 204 can be configured with a maximum GOP duration, a target segment duration, and minimum / maximum segment durations. It is desirable to create segments with a segment duration equal to the target segment duration. However, the segment duration may be changed to accommodate other occurrences, such as advertisement breaks or chapter points.

[0031] A chapter point is a transition point in the media content. For example, a chapter may represent an advertisement break in the media content or the beginning or end of a natural break in a conversation between people. In one embodiment, a chapter acts as a segment boundary. A chapter may occur anywhere within the media playback, regardless of the chapter's location relative to non-chapter segment boundaries.

[0032] A segmenter 210, or packager, segments the encoded frame 206 with the partitioned IDR frames to generate segment 1, 214 and segment 2, 216. The segmenter 210 also creates a playlist 212, which references segment 1, 214 and segment 2, 216 as resources accessible by a client device via the playlist 212.

[0033] Although segment 1, 214, contains two IDR frames, either of which could have served as a segment boundary, segmenter 210 segmented only at the partitioned IDR frame. Therefore, segment 1, 214, begins with a partitioned IDR frame. Similarly, segment 2, 216, contains two IDR frames and one I-frame, either of which could have served as a segment boundary. However, because segmenter 210 segmented only at the partitioned IDR frame, segment 2, 216 begins with a partitioned IDR frame.

[0034] The unlabeled groups of frames within segment 1 214 and segment 2 216 may be P-frames, B-frames, or any combination of both. Both segment 1 214 and segment 2 216 have equal segment time lengths, and both segments begin with an IDR frame. Although both segments contain two IDR frames, the second IDR frame of each segment was not marked as a segment boundary and therefore was not segmented by segmenter 210 at that frame.

[0035] 3A illustrates a transcoder adjusting video data streams for segment fluidity, according to one embodiment, in which the transcoder adjusts the video data streams to ensure that all variant streams have identical segment boundaries with a minimum desired segment fluidity duration.

[0036] The transcoder adjusts the video stream for segment fluidity by using a target segment duration (SD) equal to the minimum segment fluidity duration (MinSFD). In one embodiment, MinSFD is a platform-specific or media player-specific minimum desired segment duration that allows segment boundaries to be aligned across all variant streams.

[0037] Although each segment has the same fixed segment time length, the segmenter may group multiple segments together to build playlists with various groups of segments. For example, if the MinSFD and matching SD are two seconds, a single playlist generated by the segmenter may first reference individual segments (segment A1 302, segment A2 304, segment A3 306, etc.), and then reference a group of segments including segment A4 308, segment A5 310, and segment A6 312. Thus, in this example, the playlist would begin with a reference to three individual two-second segments, followed by a reference to a group of segments spanning six seconds.

[0038] Variable grouping of segments enables a segmenter to generate playlists that reference segments with durations optimized for media playback on any platform. In this way, segment fluidity improves media playback performance on client devices implementing different operating systems and media players.

[0039] For example, in one embodiment, variable grouping of segments is used to provide shorter segment durations at the start and seek points of media playback, while providing longer segment durations during portions of the media playback where the end user is less likely to initiate a start or seek action. Variable grouping thus allows for improved media playback for media players that perform well with shorter durations at start or seek, and then switch to longer segments thereafter, reducing the load on the server.

[0040] 3B illustrates a configuration of segment resources of various time lengths according to one embodiment, in which each segment group includes multiple segments with SDs that match MinSFD.

[0041] Segment B1, 322, represents a group of two SDs, each matching a MinSFD. Segment C1, 332, represents a group of four SDs, each matching a MinSFD. Segment D1, 342, segment D2, 344, and segment D3, 346, each represent one SD, followed by segment D4, which represents a group of three SDs, each matching a MinSFD.

[0042] As shown in these SD grouping variations, each SD that matches the MinSFD begins with an IDR frame, so each SD grouping variation includes multiple IDR frames. For example, segment B1 322 includes two SDs that match the MinSFD, so segment B1 322 includes at least two IDR frames.

[0043] Because the MinSFD contains multiple matching SDs, in addition to the IDR frames present in the segment group, each segment may contain additional I-frames due to the encoder's scene change detection feature. Scene change detection occurs when the encoder detects a significant difference between adjacent video frames in the video stream. When a new scene is detected, the encoder inserts an I-frame that serves as a complete image containing all the video data for the new scene.

[0044] For example, segment C1, 332 begins with an IDR frame. Suppose this starting frame displays a red image, followed by a P-frame or B-frame (not labeled) that introduces small changes to the red hue in a small portion of the video stream. Suppose the next frame displays a detailed image of a rainforest. The encoder detects this significant change in the scene and encodes the rainforest video data into an I-frame (now an IDR frame). Thus, segment C1, 332 now has an additional IDR frame, but this IDR frame is not because its SD matches MinSFD.

[0045] 3A and 3B is suboptimal because video streams with many I-frames can cause keyframe pulsation, which is common when the SD is less than 4 seconds. Keyframe pulsation can manifest as a pulsating video image seen during media playback.

[0046] 3C shows the maximum GOP duration and target segment duration for reducing keyframe pulsation, according to one embodiment. In this embodiment, the solution to reducing keyframe pulsation is to configure the transcoder with SD equal to the target maximum segment fluidity duration (MaxSFD) and maximum GOP duration (GD) equal to MinSFD. In one embodiment, MaxSFD is the longest segment duration possible while still keeping segment boundaries aligned across all variant streams.

[0047] For example, the GD is equal to the set value, so the segment E1, 352 contains the same size GD. If the GD is large enough, the pulsation of the keyframes can be reduced.

[0048] One potential drawback of the configuration in Figure 3C is that detecting a scene change can disrupt GOP alignment across variant streams. Because scene change detection is dynamic, it can result in GOP misalignment, which may trigger I-frame insertion at different frames in each variant stream that correspond to the same media content. When the encoder inserts an I-frame, it resets the keyframe interval, which can result in GOP misalignment across variant streams.

[0049] Figure 4 shows variant streams adjusted for segment fluidity, according to one embodiment, in which the variant streams represented by bitrate 1, 410, and bitrate 2, 420, are aligned across GOPs because scene changes are disabled.

[0050] By disabling scene change detection, no additional keyframes are inserted into the variant stream. Therefore, the keyframe interval is not reset and the GD remains at its original size. If the GD is large enough, keyframe pulsation is minimized.

[0051] However, disabling scene change detection is undesirable because it may result in the IDR frame being placed in a non-ideal frame. For example, assume there is a scene change in the video data at the start of a car chase. Assume that the scene change occurs between the second and third IDR frames of segment F1, 412. If scene change detection is disabled, the next upcoming IDR frame will be the third frame, which is placed after the start of the car chase. Therefore, the end user will not be able to seek to the start of the car chase; instead, the end user will only be able to seek to the second IDR frame, which is before the start of the car chase, or the third IDR frame, which is after the start of the car chase.

[0052] Figure 5 illustrates a transcoder 510 for conditioning a video data stream for segment fluidity, according to one embodiment. Figure 6 illustrates a flowchart of the operation of a cross-variant IDR identifier, according to one embodiment. For clarity, Figure 5 will be described in conjunction with Figure 6.

[0053] In one embodiment, the transcoder uses large segment durations and small GOP durations, and enables scene change detection, which uses cross-variant IDR identifiers to minimize keyframe pulsation and eliminate segment drift.

[0054] The transcoder 510 includes multiple encoders 512 that convert a video stream into multiple variant streams that include coded frames 514. The coded frames 514 represent video frames within the variant streams.

[0055] IDR frames in the variant stream are shown as bold, open boxes. The target segment boundary represents the intended starting video frame at the start of a segment that is aligned with a segment containing the same media content across the variant stream. The target segment boundary across the variant stream is shown as a filled circle. The cross-stream GOP aligned segment boundary represents the video frame at the start of a closed GOP that is aligned with a segment containing the same media content across the variant stream. Each cross-stream GOP aligned segment boundary is shown as an unfilled circle.

[0056] When generating variant streams, each encoder 512 inserts IDR frames into its respective variant stream based on a dynamic evaluation of scene changes occurring within the video data of that variant stream. Dynamic evaluation of scene changes may result in an uneven distribution of IDR frames across the variant streams.

[0057] The cross-variant IDR identifier 518 receives the variant stream at block 602. The cross-variant IDR identifier 518 is a software module that runs on hardware (e.g., a processor and memory).

[0058] At block 604, the cross-variant IDR identifier 518 examines the variant streams and identifies the location of IDR frames within each variant stream, i.e., the cross-variant IDR identifier 518 may look within each segment of each variant stream to identify IDR frames.

[0059] In block 606, cross-variant IDR identifier 518 determines which IDR frames fall within cross-variant alignment target segment boundary 522 and cross-variant alignment target segment boundary 526. Each variant stream includes IDR frames at these target segment boundary locations so that the IDR frames at these locations are aligned across the variant streams. In contrast, if any variant stream has non-IDR frames at these locations, cross-variant IDR identifier 518 can determine that the IDR frames at these locations in the other variant stream do not fall within the cross-variant alignment target segment boundaries.

[0060] Additionally, based on the identification of an IDR frame at location 524, cross-variant IDR identifier 518 determines that the IDR frame also generates cross-variant GOP alignment segment boundary 524. In contrast, if neither frame at location 524 is an IDR frame, cross-variant IDR identifier 518 can determine that location 524 does not fall on a cross-variant GOP alignment segment boundary.

[0061] The cross-variant IDR identifier 518 can identify other target segment boundaries and GOP-aligned segment boundaries in a similar manner. The cross-variant IDR identifier 518 can also determine which IDR frames are inserted for scene change detection.

[0062] By determining an IDR frame that is a cross-variant target alignment boundary, as opposed to a cross-variant GOP alignment (non-target) boundary, the cross-variant IDR identifier 518 can segment the variant stream at the desired (target) cross-variant boundary. Therefore, any IDR frames inserted into the variant stream due to scene change detection do not prevent switching between variant streams when performing ABR. Therefore, ABR can be performed while scene change detection is enabled.

[0063] At block 608, the cross-variant IDR identifier 518 segments the target segment boundary or GOP-aligned segment boundary identified at block 606. The cross-variant IDR identifier 518 may segment other cross-variant aligned target segment boundaries and cross-variant GOP-aligned segment boundaries in a similar manner. In one embodiment, the cross-variant IDR identifier 518 uses metadata to segment boundaries for downstream segmentation.

[0064] The cross-variant IDR identifier 518 may be included in the transcoder 510, or may be a separate unit, or may be included in one or more segmenters. Regardless of its location, the cross-variant IDR identifier 518 operates to identify potential segment boundaries or GOP alignment boundaries throughout the variant stream.

[0065] Figure 7A illustrates a transcoder adjusting a video data stream for segment fluidity, according to one embodiment. Figure 8 illustrates a flowchart of the operation of a variable boundary sizer, according to one embodiment. For clarity, Figure 7A will be described in conjunction with Figure 8.

[0066] In one embodiment, the variable boundary sizer accommodates short segments or short GOP durations of the video stream at the start of media playback or at anticipated seek points. Because ABR allows a media player to switch between variant stream boundaries specified by IDR frames, shorter segments or GOPs may be desirable at these points. Because each segment or GOP begins with an IDR frame, shorter segments provide the media player with multiple opportunities for variant switching, allowing it to quickly adjust to optimal streaming quality immediately after starting / seeking. This embodiment may extend the functionality of all embodiments disclosed herein.

[0067] The transcoder 710 includes a variable boundary sizer 712 that provides partition boundaries to allow an encoder 722 of the transcoder 710 to push IDR frames into the partition boundaries. In block 802, the variable boundary sizer 712 receives the video stream 702.

[0068] In block 804, the variable boundary sizer 712 identifies the locations of segment or GOP boundaries. These segments or GOPs may have short and equal durations. For example, each segment may contain a single GOP.

[0069] At block 806, the variable boundary sizer 712 partitions segment boundaries for a first set of desired segment or GOP sizes within the video stream 702. In one embodiment, metadata is used to partition locations for IDR frames of the desired segment or GOP boundaries within the video stream. When performing downstream segmentation, the partitioned segment boundaries may be aligned across the variant stream. Thus, the partitioned segment boundaries may act as cross-variant boundaries for the segments or GOPs.

[0070] The variable boundary sizer delivers the metadata to the encoder 722 directly or via the video pre-processor pipeline 716. In one embodiment, the metadata partitions frames in a pre-processed video stream 714, which is forwarded to the video pre-processor pipeline 716. The video pre-processor pipeline 716 is an optional component that can help deliver one or more partitioned video streams 718 to the encoder 722 in a format that is optimal for the encoder 722.

[0071] Segment boundaries in a video stream can help create shorter segment or GOP sizes downstream. As noted above, shorter segments or GOPs may be desirable at startup because ABR allows a media player to switch between variant stream boundaries specified by an IDR frame. Therefore, a short segment or GOP that begins with an IDR frame provides the media player with multiple opportunities to switch variant streams. Therefore, regardless of the streaming quality at the start of media playback, the media player has multiple opportunities to quickly switch to the optimal variant stream that is adequately supported by the client device's available network bandwidth and data throughput.

[0072] Similarly, a media player on a client device that needs to download a certain amount of segments before switching to a different streaming quality can also reduce the time required to achieve optimal playback performance because the conditions for switching can be satisfied earlier during media playback.

[0073] A similar reason exists for seek points: if the client device does not have access to a cache containing the segment needed for immediate media playback, then the seek point acts like a starting point. Thus, the seek operation shares the aforementioned advantages of short segments or GOPs at the beginning of media playback.

[0074] At block 808, the variable boundary sizer 712 segments segment boundaries for a second set of desired segment or GOP sizes within the video stream 702. When performing downstream segmentation, the segmented segment boundaries may be aligned across the variant stream. Thus, the segmented segment boundaries may act as cross-variant boundaries for the segments or GOPs.

[0075] In one embodiment, variable boundary sizer 712 may partition longer segments or GOPs after partitioning a threshold amount of shorter segments or GOPs. This threshold may be based, for example, on the media player's variant switching rules, such as a media player's requirement to download a certain amount of segments or download segments for a certain amount of time before performing ABR.

[0076] Once a threshold amount of shorter segments or GOPs is exceeded, variable boundary sizer 712 may partition the video stream with larger segment boundaries and GOP sizes. Larger segments and GOPs reduce the number of resource requests made by the client device, thereby reducing the processing power required from the client device and easing the load on the server processing the requests.

[0077] Variable boundary sizer 712 may create these partition boundaries at the beginning of media playback and at common or frequent destination seek points, such as chapter points.

[0078] 7B illustrates segments and GOPs in a video data stream according to one embodiment, in which the segmenter output is displayed for segment boundaries with shorter segments and GOPs at the start and beginning of the seek point, and for segment boundaries with longer segments and GOPs.

[0079] At the start of the variant stream 750, shorter segments or GOPs are generated. These segments or GOPs are created by segmenting the variant stream at partition boundaries defined by an upstream variable boundary sizer. The segments and GOPs have the same time length (e.g., 752). In one embodiment, each segment contains a single GOP, and each segment or GOP begins with an IDR frame (e.g., 754).

[0080] In one embodiment, four segments or GOPs are segmented before the threshold is reached. If the threshold is exceeded, the segmenter generates a longer segment (e.g., 756).

[0081] A combination of shorter segments or GOPs followed by longer segments or GOPs occurs at the beginning 750 of the variant stream and at each anticipated seek point, such as chapter point 770. This combination of shorter and longer segments and GOPs may improve media playback.

[0082] Shorter segments or GOPs at startup are desirable to give the media player multiple opportunities to quickly switch to the optimal variant stream that is adequately supported by the client device's available network bandwidth and data throughput. Larger segments and GOPs reduce the number of resource requests made by the client device, thereby reducing the processing power required by the client device and the load on the server processing the requests.

[0083] In this disclosure, reference is made to various embodiments. However, it should be understood that the disclosure is not limited to the specific embodiments described. Instead, any combination of the following configurations and elements, whether related to various embodiments or not, is contemplated for implementing and practicing the teachings provided herein. Furthermore, when an element of an embodiment is described in the form of "at least one of A or B," it can be understood that an embodiment including only element A, including only element B, or including elements A and B is respectively contemplated. Furthermore, while some embodiments may achieve advantages over other possible solutions or the prior art, whether or not a particular advantage is achieved by a given embodiment does not limit the disclosure. Accordingly, the aspects, configurations, embodiments, and advantages disclosed herein are merely exemplary and should not be considered elements or limitations of the appended claims unless expressly recited in the claims. Similarly, references to the "present invention" should not be construed as generalizing the subject matter disclosed herein, nor should they be considered elements or limitations of the appended claims unless expressly recited in the claims.

[0084] As will be appreciated by those skilled in the art, the embodiments described herein may be embodied as a system, a method, or a computer program product. Accordingly, the embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which are collectively referred to herein as a "circuit," "module," or "system." Furthermore, the embodiments described herein may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code embodied therein.

[0085] The program code embodied on the computer readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, etc., or any suitable combination thereof.

[0086] Computer program code for carrying out operations of embodiments of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may run entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0087] Aspects of the present disclosure are described herein with reference to flowchart diagrams or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It can be understood that each block of the flowchart diagrams or block diagrams, and combinations of blocks in the flowchart diagrams or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus forming a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in the blocks of the flowchart diagrams or block diagrams.

[0088] These computer program instructions may also be stored on a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other device to function in a particular manner, whereby the instructions stored on the computer-readable medium form an article of manufacture including instructions that implement the functions / acts specified in the flowchart or block diagram blocks.

[0089] Computer program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device and caused to execute a series of operational steps on the computer, other programmable data processing apparatus, or other device to create a computer-implemented process, whereby the instructions executing on the computer, other programmable data processing apparatus, or other device provide a process for performing the functions / operations specified in the flowchart or block diagram blocks.

[0090] The flowchart diagrams and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart diagrams or block diagrams represents a module, segment, or portion of code, which may comprise one or more executable instructions for performing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or may be executed in the reverse order or out of order, depending on the functionality involved. It should also be noted that each block in the block diagrams or flowchart diagrams, and combinations of blocks in the block diagrams or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs particular functions or operations, or by a combination of special-purpose hardware and computer instructions.

[0091] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the present disclosure may be made without departing from the basic scope thereof, which scope is defined by the following claims.

Claims

1. a plurality of encoders configured to generate a plurality of variant streams; A cross-variant Instantaneous Decoder Refresh (IDR) identifier, inspecting a plurality of variant streams; identifying an IDR frame in each of a plurality of variant streams; determining a first set of IDR frames that fall within a first type of cross-variant boundary; determining a second set of IDR frames that fall into a second type of cross-variant boundary, the second type of cross-variant boundary being different from the first type of cross-variant boundary; and a cross-variant Instantaneous Decoder Refresh (IDR) identifier configured to include the steps of: The first set of IDR frames includes a cross-variant target segment boundary, which represents a starting video frame intended to be a segment start, which is aligned with a segment containing the same media content across the multiple variant streams. The system wherein the second set of IDR frames includes a cross-variant group of pictures (GOP) aligned segment boundary, the cross-variant GOP aligned segment boundary representing a starting video frame that is a closed GOP start, the closed GOP start being aligned with a starting video frame at the start of a set of closed GOPs that contain the same media content across the multiple variant streams.

2. The system of claim 1 , wherein the first set of IDR frames is partitioned at cross-variant target segment boundaries.

3. The system of claim 1 , wherein the second set of IDR frames is partitioned at cross-variant GOP aligned segment boundaries.

4. The cross-variant IDR identifier is determining a second set of IDR frames corresponding to the IDR frames inserted due to scene change detection; 2. The system of claim 1, further configured to include the step of: operatively not classifying IDR frames of the second set that correspond to IDR frames inserted due to scene change detection.

5. 2. The system of claim 1, wherein determining the second set of IDR frames includes identifying whether the IDR frames at the video frame positions are aligned across all of the multiple variant streams.

6. A non-transitory computer readable medium containing computer program code, the computer program code performing operations when executed by operation of one or more computer processors, comprising: inspecting the multiple variant streams output by each encoder; identifying an Instantaneous Decoder Refresh (IDR) frame within each variant stream of the plurality of variant streams; determining an IDR frame that falls on a cross-variant boundary; Segmenting an IDR frame corresponding to a cross-variant boundary; sending the plurality of variant streams to a segmenter together with the partitioned IDR frames, wherein the IDR frames are not added to the plurality of frames after being output from each encoder and before being sent to the segmenter; the cross-variant boundaries include cross-variant destination segment boundaries, the cross-variant destination segment boundaries representing starting video frames intended to be segment starts, the segment starts aligned with segments containing the same media content across the multiple variant streams; The cross-variant boundary includes a cross-variant group of pictures (GOP) aligned segment boundary, the cross-variant GOP aligned segment boundary representing a starting video frame that is a closed GOP start, the closed GOP start aligned with a starting video frame at the start of a set of closed GOP start times that include the same media content across the multiple variant streams.

7. The non-transitory computer-readable medium of claim 6 , wherein the determined IDR frame is partitioned at a cross-variant target segment boundary.

8. The non-transitory computer-readable medium of claim 6 , wherein the determined IDR frames are partitioned at cross-variant GOP alignment segment boundaries.

9. The operation is identifying the determined IDR frame corresponding to the IDR frame inserted due to scene change detection; and operating to not classify an IDR frame that corresponds to an IDR frame inserted due to scene change detection.

10. 7. The non-transitory computer-readable medium of claim 6, wherein determining an IDR frame that falls on a cross-variant boundary includes identifying whether the IDR frame at the video frame position is aligned across all of the multiple variant streams.

11. The non-transitory computer-readable medium of claim 6 , wherein the operations are performed by a cross-variant IDR identifier integrated into a packager.

Images