Transcoder adjustment for segment fluidity

Abstract

This invention provides a computer network streaming system that includes a process of switching between multiple variant streams to adjust the streaming quality delivered to client devices. [Solution] In the system, the transcoder 510 includes a plurality of encoders 512 that generate a plurality of variant streams, and a cross-variant instantaneous decoder refresh (IDR) identifier 518 that inspects the plurality of variant streams, identifies IDR frames in each of the plurality of variant streams, determines IDR frames corresponding to cross-variant boundaries, and divides the IDR frames corresponding to cross-variant boundaries.

Description

Background

[0001] A video format is a container that includes a data stream and metadata. The data stream may include a video stream and an accompanying audio stream. The metadata includes information about the video stream, such as bit rate, resolution, codec, etc. The bit rate correlates with the quality of the video stream, while the resolution represents the image size of the video stream.

[0002] A codec is a system or program that encodes or decodes a data stream. An encoder is a system or program that executes a codec to encode a data stream, while a decoder is a system or program that executes a codec to decode a data stream. Often, a codec is used to compress a data stream. This is to reduce the size of the video file for transfer through a computer network. Also, a codec may be used to decompress a data stream for media playback or data stream file operations.

Brief Description of the Drawings

[0003] To achieve the above aspects and for a more detailed understanding, a more specific description of the embodiments described herein, which were briefly outlined above, can be obtained by referring to the accompanying drawings.

[0004] However, it should be noted that other equally effective embodiments are also conceivable, and the accompanying drawings show typical embodiments and should not be regarded as limiting. [Figure 1] A system for generating and delivering a variant stream aligned throughout an IDR frame according to one embodiment is shown. [Figure 2] A transcoder for adjusting a video data stream for downstream segmentation according to one embodiment is shown. [Figure 3A]One embodiment of a transcoder that adjusts a video data stream for segment fluidity is shown. [Figure 3B] This shows the configuration of segment resources of various time lengths according to one embodiment. [Figure 3C] This shows the maximum GOP time length and target segment time length for reducing keyframe pulsation according to one embodiment. [Figure 4] This shows a variant stream adjusted for segmented fluidity according to one embodiment. [Figure 5] One embodiment of a transcoder that adjusts a video data stream for segment fluidity is shown. [Figure 6] A flowchart illustrating the operation of a cross-variant IDR identifier according to one embodiment is shown. [Figure 7A] One embodiment of a transcoder that adjusts a video data stream for segment fluidity is shown. [Figure 7B] This shows a segment and GOP within a video data stream according to one embodiment. [Figure 8] A flowchart illustrating the operation of a variable boundary sizer according to one embodiment is shown. Detailed explanation

[0005] To enable a detailed understanding of the structure of this disclosure, embodiments of the invention disclosed herein may refer to HTTP Live Streaming (HLS) as a computer network streaming protocol. However, the disclosed embodiments should not be construed as being limited to any particular network protocol.

[0006] HLS is a computer network streaming protocol that can deliver media content via a master playlist. The master playlist references multiple variant streams, each variant stream containing a media playlist that references a set of consecutive segments of media content that are played sequentially by client devices.

[0007] A segment is identified within a media playlist by one or more URIs (Uniform Resource Identifiers) and optionally by a byte range. A URI is a string that identifies a resource on the internet.

[0008] Each segment in a media playlist begins at a segment boundary specified by an IDR frame and ends at the frame immediately preceding the next segment boundary specified by an IDR frame. Segment boundaries are typically synchronized across multiple variant streams. ABR is performed by switching variant streams, which adjusts the streaming quality across these aligned segment boundaries. Switching variant streams involves changing from a group of segments in one variant stream to a contiguous group of segments in another variant stream.

[0009] When recording video, the audiovisual (AV) input generates an uncompressed data stream. The encoder compresses the video stream from the data stream into image groups (GOPs) containing I-frames, P-frames, and B-frames. Compressing the video stream makes the video file smaller. This smaller video file is easier to store and transfer over computer networks, in contrast to the storage and transfer of larger video files.

[0010] A GOP is a sequence of consecutive compressed video frames, bounded by I-frames. Each GOP begins with an I-frame (or contains one) and ends with a P-frame or B-frame. Drawing a GOP displays or plays the video content within the video stream.

[0011] A GOP can be either open or closed. An open GOP contains at least one frame referenced by a frame in the previous GOP. A closed GOP contains only frame references in the current GOP. A closed GOP begins with an instantaneous decoder refresh (IDR) frame.

[0012] An I-frame (also called a keyframe or intra-frame) is a video frame that contains all the video data necessary to display itself completely without referencing other frames. Because an I-frame contains a complete frame of video data, it is compressed independently by the encoder; that is, it is compressed without referencing other frames. Furthermore, an I-frame serves as a reference for predicting one or more frames in a video stream. I-frames are generally larger than P-frames, and P-frames are generally larger than B-frames.

[0013] A P-frame (also called a predictive frame) is a video frame that contains video data that will change between the current P-frame and the previous I-frame or P-frame. In other words, a P-frame contains video data that is not identical to the video data in the previous reference frame, but excludes the same video data (redundancy) that was in the previous reference frame. Therefore, the current P-frame is encoded by referencing the previous I-frame or P-frame, and the current P-frame excludes redundancy to reduce the memory size of the current P-frame. A P-frame serves as a reference for future P-frames or any B-frames.

[0014] For example, if a video stream shows a person speaking in front of an unchanging background, the current P-frame may exclude information about the background because this background information does not change from the preceding reference frame to the current P-frame. The current P-frame may include information about the person's movement because this information changes from the previous reference frame.

[0015] A B-frame (also called a bidirectional frame) is a video frame that contains predictions about how one or more objects within the frame have changed between previous or subsequent I-frames, P-frames, or B-frames. Therefore, unlike P-frames, B-frames can see more types of reference frames both backward and forward. Its purpose is to identify redundancy to exclude from B-frames. Encoders predict the differences between the current B-frame and reference frames to reduce the memory size of the current B-frame.

[0016] An IDR frame is an I-frame that specifies a frame reference barrier. An IDR frame cannot be referenced by a frame from a previous GOP. Furthermore, an IDR frame prevents frames within its own GOP from referencing frames preceding it. In other words, when a decoder encounters an IDR frame, it designates all frames in the frame buffer prior to it as not allowed to be referenced by any frames after the IDR frame. Therefore, using an IDR frame forces the decoder to refresh the frame buffer, thereby ensuring that the first frame of a GOP is always the IDR frame, and that subsequent P and B frames do not reference any frames preceding it. The frame reference barrier function of an IDR frame does not exist in a non-IDR I-frame. A non-IDR I-frame allows frames referencing it to reference frames preceding that I-frame.

[0017] Starting the GOP with an IDR frame can improve media playback, for example, at seek points. A seek point is a specific position within media playback selected by the end user of the media player.

[0018] When an end user seeks to a point in media playback, that seek point may be located in a P-frame or B-frame. Starting media playback from this position can always produce a distorted output because P-frames and B-frames do not contain enough video data to reconstruct the complete image corresponding to the GOP (Group of Pictures) that contains them. Instead, the media player must look at the I-frames referenced by the P-frames and B-frames to draw the complete image and begin playback of the current GOP.

[0019] Furthermore, in adaptive bitrate streaming (ABR) based protocols (e.g., HLS), only IDR frames are viable variant stream switching points. ABR is a computer network streaming technology that includes the steps of measuring the network bandwidth and data throughput of a client device in real time and, accordingly, switching variant streams to adjust the streaming quality delivered to the client device.

[0020] A variant stream represents various streaming bitrates of the same media content represented by a data stream. Generally, there is a correlation that the higher the streaming bitrate, the higher the streaming quality, while on the other hand, the lower the streaming bitrate, the lower the streaming quality. Furthermore, an increase in the streaming bitrate is equivalent to an increase in the relative size of the data stream, while a decrease in the streaming bitrate is equivalent to a decrease in the relative size of the data stream. Therefore, a higher-quality stream requires more network bandwidth and a larger data throughput on the client device. This is to ensure continuous and uninterrupted media playback on the client device. Similarly, for a lower-quality stream, less network bandwidth and data throughput of the client device are required to ensure continuous and uninterrupted media playback on the client device.

[0021] In a non-IDR I-frame, since subsequent frames can reference frames that precede the I-frame, when the media player loads different variant streams as a result of seeking, frames within the current variant stream may reference frames that existed in a previous variant stream but do not exist in the current variant stream. Note that the frames do not exist, for example, due to differences in encoding (such as differences due to the use of a scene change function, as described later). These types of frame references can potentially interrupt media playback in an ABR-based protocol. The aforementioned problem is solved by using an IDR frame as the switching point for variant streams. This is because an IDR frame does not allow subsequent frames to reference frames that precede the IDR frame.

[0022] In various embodiments of the present specification, the transcoder adjusts video frames for downstream processing to create segments that are compatible for use in segment fluidity. Segment fluidity is a technique that can present various segment durations to various platforms or media players by grouping segments into resources. A segment includes one or more GOPs.

[0023] FIG. 1 shows a system for generating and distributing a variant stream aligned across an entire IDR frame according to one embodiment. In this embodiment, an AV input 110 generates a video stream, and an encoder (not shown) compresses the video stream before distributing it to a transcoder 130. The encoded video stream includes a resolution and a bitrate, each of which represents an image size and video quality. For example, the resolution of the video stream is 1080p and the bitrate is 3 Mbps.

[0024] The transcoder 130 is a processing component that compresses and adjusts the video stream for downstream processing. The transcoder 130 may include a decoder 132, one or more encoders, and a cross-variant IDR identifier 134. By converting the video stream to its pre-encoded form, the decoder 132 may enable the encoder within the transcoder 130 to process the video stream. For example, a camera 114 may generate a video stream in RAW format. The camera 114 may have an integrated encoder (not shown) that converts the video stream to a first encoded format. The downstream transcoder 130 may not be compatible with the first encoded format. Thus, when the transcoder 130 receives a video stream in the first encoded format, the decoder 132 of the transcoder 130 may convert the encoded video stream to RAW format. Next, an encoder within the transcoder may convert the video stream to a second encoded format.

[0025] In one embodiment, the cross-variant IDR identifier 134 is a software module running on hardware (e.g., a processor and memory). The cross-variant IDR identifier 134 segments the frames, and the encoder may use these frames to adjust the variant stream to perform segment fluidity. Alternatively, the cross-variant IDR identifier 134 may use metadata instead of directly segmenting the video frames in the manner described above. In yet another embodiment, the cross-variant IDR identifier 134 oversees (post-hoc) the entire variant stream and segments the cross-variant aligned frames downstream, as will be described in more detail below.

[0026] Each encoder within the transcoder 130 converts the decoded video stream into a variant stream. Each variant stream contains the same video stream (e.g., the same media content) but has a different bitrate. The transcoder 130 can also change the resolution of the video stream for each variant stream. For example, encoder 1, 136a generates variant stream 1 with a resolution of 1080p and a bitrate of 3Mbps. Encoder 2, 136b generates variant stream 2 with a resolution of 720p and a bitrate of 2Mbps. Encoder N, 136n generates variant stream N with a resolution of Xp and a bitrate of YMbps. The variant streams include segment boundaries from the cross-variant IDR identifier 134, aligned throughout the variant stream. The variant streams are delivered to the packager 140.

[0027] The packager 140 generates a playlist for each variant stream and segments the variant streams in selected partition frames. The packager 140 then distributes the playlists and variant streams to the distribution network 150, where, in one embodiment, the playlists and variant streams are sent directly to one or more servers 152. The playlists are then sent to client devices 156. One or more servers 152 handle any fetch / requests from client devices 156 for segments of media content. In other words, client devices 156 can use the playlists to submit requests to the distribution network 150 for segments identified within those playlists in order to play the media content within them.

[0028] Alternatively, the playlist and variant stream are sent directly to one or more servers 152. The playlist and variant stream are then sent to a Content Delivery Network (CDN) 154, which delivers the playlist to client devices 156 and processes the client devices 156's fetch / requests for segments of media content. In yet another embodiment, the playlist and variant stream are first sent to a CDN 154, which delivers the playlist to client devices 156 and processes the client devices 156's fetch / requests for segments of media content. Using a CDN to process segment fetch / requests can improve media playback. To achieve this, localization of CDN edge servers reduces delivery time for fetched / requested segments.

[0029] Figure 2 shows a transcoder that adjusts a video data stream for downstream segmentation according to one embodiment. In this embodiment, transcoder 204 receives source video frame 202 and generates encoded frame 206 by encoding source video frame 202 with indicators that demarcate selected IDR frames within encoded frame 206 as segment boundaries. In this embodiment, the segment boundary indicator is in-band segment boundary metadata 208. A segmenter 210 downstream of the transcoder segments encoded frame 206 only at IDR frames demarcated as segment boundaries. In another embodiment, transcoder 204 may insert IDR frames into a variant stream to enforce desired segment boundaries. A segmenter 201 downstream of the transcoder segments encoded frame 206 at each IDR frame of encoded frame 206.

[0030] The transcoder 204 may consist of a maximum GOP time length, a target segment time length, and minimum / maximum segment time lengths. It is desirable to create segments with a segment time length equal to the target segment time length. However, the segment time length may be changed to accommodate other events such as ad interruptions or chapter points.

[0031] Chapter points are transition points in media content. For example, a chapter may represent an advertisement interruption within media content, or the beginning or end of a natural pause in a conversation between characters. In one embodiment, a chapter acts as a segment boundary. Regardless of the chapter's position relative to a non-chapter segment boundary, a chapter can occur at any point in media playback.

[0032] The segmenter 210, or packager, generates segments 1, 214 and segments 2, 216 by segmenting the encoded frame 206 with segmented IDR frames. The segmenter 210 also creates a playlist 212, which references segments 1, 214 and segments 2, 216 as resources accessible by client devices via the playlist 212.

[0033] Segment 1,214 contains two IDR frames, both of which could have functioned as segment boundaries. However, segmenter 210 only segmented using the partitioned IDR frame. Therefore, segment 1,214 begins with the partitioned IDR frame. Similarly, segment 2,216 contains two IDR frames and one I-frame, both of which could have functioned as segment boundaries. However, segmenter 210 only segmented using the partitioned IDR frame, so segment 2,216 also begins with the partitioned IDR frame.

[0034] The unlabeled groups of frames within segments 1,214 and 2,216 may be P-frames, B-frames, or any combination of both. Both segments 1,214 and 2,216 have equal segment time lengths, and both segments begin with IDR frames. Both segments contain two IDR frames, but the second IDR frame in each segment was not segmented by segmenter 210 because it was not demarcated as a segment boundary.

[0035] Figure 3A shows a transcoder according to one embodiment that adjusts a video data stream for segment fluidity. In this embodiment, the transcoder adjusts the video data stream to ensure that all variant streams have the same segment boundaries with the minimum desired segment fluidity time length.

[0036] The transcoder adjusts the video stream for segment fluidity by using a target segment length (SD) equal to the minimum segment fluidity time length (MinSFD). In one embodiment, the MinSFD is the shortest desired segment length specific to the platform or media player, so that segment boundaries can be aligned across all variant streams.

[0037] Each segment has the same fixed segment duration, but the segmenter may group multiple segments together to construct a playlist of various segments. For example, if the MinSFD and matching SD are 2 seconds, one playlist generated by the segmenter may first refer to individual segments (segments A1, 302, A2, 304, A3, 306, etc.), and then refer to a group of segments including segments A4, 308, A5, 310, and A6, 312. Thus, in this embodiment, the playlist begins by referring to three individual 2-second segments, followed by a reference to a group of segments spanning 6 seconds.

[0038] Variable segment grouping allows the segmenter to generate playlists that reference segments with optimized time lengths for media playback on any platform. Thus, segment fluidity improves media playback performance across client devices implementing various operating systems and media players.

[0039] For example, in one embodiment, variable segment grouping is used to provide shorter segment durations at media playback startup and seek points, while providing longer segment durations for media playback sections where the end user is less likely to initiate start or seek operations. Therefore, variable grouping makes it possible to improve media playback for media players that function well with shorter durations at startup or seek, and then reduce the load on the server by switching to longer segments.

[0040] Figure 3B shows the configuration of segment resources of various time lengths according to one embodiment. In this embodiment, each segment group includes multiple segments having SDs that match MinSFD.

[0041] Segments B1 and 322 represent groups of two SDs, each corresponding to MinSFD. Segments C1 and 332 represent groups of four SDs, each corresponding to MinSFD. Segments D1 and 342, D2 and 344, and D3 and 346 each represent one SD, followed by segment D4. Segment D4 represents groups of three SDs, each corresponding to MinSFD.

[0042] As shown in these SD grouping variations, each SD matching MinSFD starts with an IDR frame, so any SD grouping variation contains multiple IDR frames. For example, segment B1,322 contains two SDs matching MinSFD, so segment B1,322 contains at least two IDR frames.

[0043] Because MinSFD contains multiple SDs, each segment may contain an additional I-frame in addition to the IDR frames present in the segment group, due to the encoder's scene change detection feature. Scene change detection occurs when the encoder detects a significant difference between adjacent video frames in the video stream. When a new scene is detected, the encoder inserts an I-frame that acts as a complete image containing all the video data of the new scene.

[0044] For example, segment C1,332 begins with an IDR frame. Suppose this opening frame displays a red image, followed by a P-frame or B-frame (unlabeled) that makes a small change in the red hue over a small portion of the video stream. Suppose the next frame displays a detailed image of the rainforest. The encoder detects this significant change in the landscape and encodes the rainforest video data into an I-frame (in this case, an IDR frame). Thus, segment C1,332 now has an additional IDR frame, but this IDR frame is not because its SD matches MinSFD.

[0045] The embodiments shown in Figures 3A and 3B are suboptimal because video streams containing a large number of I-frames can generate keyframe pulsation, which often occurs when the standard deviation (SD) is less than 4 seconds. Keyframe pulsation can manifest as a pulsating video image during media playback.

[0046] Figure 3C shows the maximum GOP time length and target segment time length for reducing keyframe pulsation according to one embodiment. In this embodiment, the solution for reducing keyframe pulsation is to configure a transcoder having an SD equal to the target maximum segment fluidity time length (MaxSFD) and a maximum GOP time length (GD) equal to the MinSFD. In one embodiment, MaxSFD is the longest possible segment time length while maintaining segment boundaries aligned across all variant streams.

[0047] For example, since GD is equal to the set value, segment E1,352 contains a GD of the same size. If GD is large enough, keyframe pulsation can be reduced.

[0048] One potential drawback of the configuration in Figure 3C is that detecting a scene change can disrupt the GOP alignment across the entire variant stream. Because scene change detection is performed dynamically, GOP misalignment can occur, potentially triggering the insertion of I-frames at different frames in each variant stream corresponding to the same media content. When the encoder inserts I-frames, the keyframe interval is reset, which can result in GOP misalignment across the entire variant stream.

[0049] Figure 4 shows a variant stream adjusted for segment fluidity according to one embodiment. In this embodiment, the variant streams represented by bitrates 1,410 and 2,420 are aligned across the entire GOP because scene changes are disabled.

[0050] Disabling scene change detection prevents additional keyframes from being inserted into the variant stream. Therefore, the keyframe interval is not reset, and the GD remains at its original size. If the GD is large enough, keyframe pulsation is minimized.

[0051] However, disabling scene change detection is undesirable because it may result in IDR frames being placed in undesirable frames. For example, suppose there is a scene change at the start of a car chase in the video data. Suppose this scene change occurs between the second and third IDR frames of segment F1, 412. If scene change detection is disabled, the next IDR frame that comes along is the third frame, which is placed after the start of the car chase. Therefore, the end user cannot seek to the start of the car chase, and can only seek to the second IDR frame, which is before the start of the car chase, or the third IDR frame, which is after the start of the car chase.

[0052] Figure 5 shows a transcoder 510 that adjusts the video data stream for segment fluidity according to one embodiment. Figure 6 shows a flowchart of the operation of a cross-variant IDR identifier according to one embodiment. For clarity, Figure 5 is described together with Figure 6.

[0053] In one embodiment, the transcoder uses a large segment duration and a small GOP duration, and enables scene change detection. In this embodiment, cross-variant IDR identifiers are used to minimize keyframe pulsation and eliminate segment misalignment.

[0054] The transcoder 510 includes multiple encoders 512 that convert a video stream into multiple variant streams, each containing an encoded frame 514. The encoded frame 514 represents a video frame within the variant stream.

[0055] IDR frames within a variant stream are indicated by bold white rectangles. The target segment boundary represents the starting video frame intended to be the segment start, and at this segment start, it is aligned with segments containing the same media content throughout the variant stream. The target segment boundary for the entire variant stream is indicated by a filled circle. The cross-stream GOP alignment segment boundary represents the video frame at the start of a closed GOP, and at this closed GOP start, it is aligned with segments containing the same media content throughout the variant stream. Each cross-stream GOP alignment segment boundary is indicated by an unfilled circle.

[0056] When generating variant streams, each encoder 512 inserts IDR frames into its respective variant stream based on a dynamic evaluation of scene changes occurring within the video data of that variant stream. Dynamic evaluation of scene changes may result in an uneven distribution of IDR frames across the variant stream.

[0057] In block 602, the cross-variant IDR identifier 518 receives the variant stream. The cross-variant IDR identifier 518 is a software module running on hardware (e.g., processor and memory).

[0058] In block 604, the cross-variant IDR identifier 518 examines the variant streams and identifies the location of IDR frames within each variant stream. That is, the cross-variant IDR identifier 518 can examine each segment within each variant stream and identify IDR frames.

[0059] In block 606, the cross-variant IDR identifier 518 determines which IDR frames fall under the cross-variant alignment target segment boundaries 522 and 526. Each variant stream includes IDR frames at these target segment boundary locations so that the IDR frames at these locations are aligned throughout the variant stream. Conversely, if any variant stream has non-IDR frames at these locations, the cross-variant IDR identifier 518 can determine that the IDR frames at these locations in the other variant stream do not fall under the cross-variant alignment target segment boundaries.

[0060] In addition, based on the identification of an IDR frame at position 524, the cross-variant IDR identifier 518 determines that the IDR frame also generates a cross-variant GOP alignment segment boundary 524. Conversely, if none of the frames at position 524 are IDR frames, the cross-variant IDR identifier 518 can determine that position 524 does not correspond to a cross-variant GOP alignment segment boundary.

[0061] The cross-variant IDR identifier 518 can identify other target segment boundaries and GOP alignment segment boundaries in a similar manner. The cross-variant IDR identifier 518 can also determine which IDR frame was inserted for scene change detection.

[0062] By determining the IDR frame, which is the cross-variant target alignment boundary, the cross-variant IDR identifier 518 allows the variant stream to be segmented at the desired (target) cross-variant boundary, in contrast to the cross-variant GOP alignment (non-target) boundary. Therefore, any IDR frames inserted into the variant stream by scene change detection will not interfere with switching between variant streams when ABR is performed. Thus, ABR can be performed while scene change detection is enabled.

[0063] In block 608, the cross-variant IDR identifier 518 demarcates the target segment boundary or GOP alignment segment boundary identified in block 606. The cross-variant IDR identifier 518 may similarly demarcate other cross-variant alignment target segment boundaries and cross-variant GOP alignment segment boundaries. In one embodiment, the cross-variant IDR identifier 518 uses metadata to demarcate boundaries for downstream segmentation.

[0064] The cross-variant IDR identifier 518 may be included in the transcoder 510, or it may be an independent unit, or it may be included in one or more segmenters. Regardless of its location, the cross-variant IDR identifier 518 operates to identify potential segment boundaries or GOP alignment boundaries across the variant stream.

[0065] Figure 7A shows a transcoder that adjusts a video data stream for segment fluidity according to one embodiment. Figure 8 shows a flowchart of the operation of a variable boundary sizer according to one embodiment. For clarity, Figure 7A is described together with Figure 8.

[0066] In one embodiment, a variable boundary sizer adjusts a video stream to have a shorter segment or shorter GOP time length at the start of media playback or at an expected seek point. Because ABR allows the media player to switch between variant stream boundaries specified by IDR frames, shorter segments or GOPs may be desirable at these points. Since each segment or GOP begins with an IDR frame, shorter segments provide the media player with multiple opportunities to switch variants, thereby allowing for quick adjustment to the optimal streaming quality immediately after start / seek. This embodiment may extend the functionality of all embodiments disclosed herein.

[0067] The transcoder 710 includes a variable boundary sizer 712, which provides a partition boundary to cause the encoder 722 of the transcoder 710 to push the IDR frames into its partition boundary. In block 802, the variable boundary sizer 712 receives the video stream 702.

[0068] In block 804, the variable boundary sizer 712 identifies the location of the boundary of a segment or GOP. These segments or GOPs may have short, equal time lengths. For example, each segment may contain a single GOP.

[0069] In block 806, the variable boundary sizer 712 demarcates segment boundaries within the video stream 702 for a first set of desired segment or GOP sizes. In one embodiment, metadata is used to demarcate the positions of IDR frames for the boundaries of the desired segments or GOPs within the video stream. If segmentation is performed downstream, the demarcated segment boundaries can be aligned across the entire variant stream. Thus, the demarcated segment boundaries can act as cross-variant boundaries for segments or GOPs.

[0070] The variable boundary sizer delivers metadata directly to the encoder 722 or via the video preprocessor pipeline 716. In one embodiment, the metadata segments frames within a preprocessed video stream 714, which is then transferred to the video preprocessor pipeline 716. The video preprocessor pipeline 716 is an optional component that may help deliver one or more segmented video streams 718 to the encoder 722 in an optimal format.

[0071] Partition boundaries within a video stream can help create shorter segments or GOP sizes downstream. As mentioned above, shorter segments or GOPs may be desirable at startup. This is because ABR allows the media player to switch between variant stream boundaries specified by the IDR frame. Therefore, if the segment or GOP starts in the IDR frame, the media player is given multiple opportunities to switch variant streams. Thus, regardless of the streaming quality when media playback starts, the media player has multiple opportunities to quickly switch to the optimal variant stream, which is adequately supported by the available network bandwidth and data throughput of the client device.

[0072] Similarly, reducing the time required to achieve optimal playback performance is also beneficial for media players on client devices that need to download a certain amount of segments before switching to a different streaming quality, because the conditions for switching can be met earlier in media playback.

[0073] Similar reasons apply to seek points. If a client device cannot access a cache containing the segments necessary for immediate media playback, a seek point acts like a starting point. Therefore, the seek operation shares the aforementioned advantages of short segments or GOPs at the beginning of media playback.

[0074] In block 808, the variable boundary sizer 712 divides segment boundaries within the video stream 702 for a second set of desired segment or GOP sizes. If segmentation is performed downstream, the divided segment boundaries can be aligned across the entire variant stream. Thus, the divided segment boundaries can act as cross-variant boundaries for segments or GOPs.

[0075] In one embodiment, the variable boundary sizer 712 may, after classifying shorter segments or GOPs by a threshold amount, classify longer segments or GOPs. This threshold may be based, for example, on the variant switching rules of a media player, which include media player requirements such as downloading a predetermined amount of segments or downloading segments for a predetermined amount of time before performing ABR.

[0076] If the threshold for the amount of shorter segments or GOPs is exceeded, the variable boundary sizer 712 may divide the video stream into larger segment boundaries and GOP sizes. Increasing the segment and GOP size reduces the number of resource requests that the client device makes, thereby reducing the processing power required by the client device and easing the load on the server processing the requests.

[0077] The variable boundary sizer 712 can create these boundary lines at the start of media playback and at common or frequently used destination seek points such as chapter points.

[0078] Figure 7B shows segments and GOPs in a video data stream according to one embodiment. In this embodiment, the segmenter output is displayed for segment boundaries with shorter segments and GOPs at startup and the start of the seek point, and for segment boundaries with longer segments and GOPs.

[0079] At the start of the variant stream 750, shorter segments or GOPs are generated. These segments or GOPs are created by segmenting the variant stream at partition boundaries defined by an upstream variable boundary sizer. The time length (e.g., 752) of the segments and GOPs is the same. In one embodiment, each segment contains a single GOP, and each segment or GOP begins with an IDR frame (e.g., 754).

[0080] In one embodiment, four segments or GOPs are divided before a threshold is reached. If the threshold is exceeded, the segmenter generates longer segments (e.g., 756).

[0081] A combination of shorter segments or GOPs followed by longer segments or GOPs occurs at the start of the variant stream at 750 and at each expected seek point (e.g., chapter point 770). This combination of shorter and longer segments and GOPs can improve media playback.

[0082] Shorter segments or GOPs at startup are desirable to give the media player multiple opportunities to quickly switch to the optimal variant stream, which is adequately supported by the client device's available network bandwidth and data throughput. Larger segments and GOPs reduce the number of resource requests the client device makes, thereby reducing the processing power required from the client device and easing the load on the server processing the requests.

[0083] This disclosure references various embodiments. However, it should be understood that this disclosure is not limited to the specific embodiments described. Instead, any combination of the following configurations and elements, whether related to the various embodiments or not, is intended to perform and practice the teachings provided herein. Furthermore, where an element of an embodiment is described in the form of “at least one of A or B,” it should be understood that embodiments including only element A, only element B, and embodiments including both elements A and B are intended, respectively. Furthermore, while some embodiments may achieve effects that exceed other possible solutions or prior art, whether a particular effect is achieved by a given embodiment is not a limitation of this disclosure. Accordingly, the aspects, configurations, embodiments, and effects disclosed herein are merely illustrative and should not be considered elements or limitations of the appended claims unless expressly stated in the claims. Similarly, the reference to “the present invention” should not be interpreted as a generalization of the content of the invention disclosed herein, nor should it be considered elements or limitations of the appended claims unless expressly stated in the claims.

[0084] As those skilled in the art will understand, the embodiments described herein may be embodied as systems, methods, or computer program products. Accordingly, embodiments may take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware aspects, all of which are collectively referred to herein as “circuits,” “modules,” or “systems.” Furthermore, embodiments described herein may take the form of computer program products expressed in one or more computer-readable media in which computer-readable program code is embodied.

[0085] Program code, embodied on a computer-readable medium, may be transmitted using any suitable medium, including, but not limited to, wireless, wired, fiber optic cable, RF, or any suitable combination thereof.

[0086] Computer program code for performing the operations of embodiments of this disclosure may be written in any combination of one or more programming languages. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as traditional procedural programming languages ​​such as the C programming language or similar languages. The program code may be executed as a standalone software package, either entirely on the user's computer or partially on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN). Alternatively, it may be connected to an external computer (for example, via the Internet using an Internet service provider).

[0087] Aspects of the present disclosure are described herein with reference to flowcharts or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It can be understood that each block in a flowchart or block diagram, and combinations of blocks within a flowchart or block diagram, are executed by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device forming a machine, thereby generating means for performing the functions / operations specified in the blocks of the flowchart or block diagram when executed via the processor of the computer or other programmable data processing device.

[0088] These computer program instructions can also be stored in computer-readable media that can instruct a computer, other programmable data processing device, or other device to function in a specific way, thereby forming a product in which the instructions stored in the computer-readable media include instructions that perform functions / operations specified in the blocks of a flowchart or block diagram.

[0089] Furthermore, computer program instructions may be loaded into a computer, another programmable data processing device, or another device, and a series of operational steps may be executed on the computer, another programmable device, or another device to generate a computer execution process, thereby providing a process for instructions executed on the computer, another programmable data processing device, or another device to perform the functions / operations specified in the blocks of a flowchart or block diagram.

[0090] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible embodiments of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram represents a module, segment, or portion of code, which may contain one or more executable instructions for executing a specified logical function. It should also be noted that in some alternative embodiments, the functions described in a block may occur in a different order than that shown in the figure. For example, two blocks shown consecutively may actually be executed substantially simultaneously, in reverse order, or in any order depending on the related functions. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be executed by a special-purpose hardware-based system or a combination of special-purpose hardware and computer instructions to perform a particular function or operation.

[0091] While the above applies to embodiments of the present disclosure, other further embodiments of the present disclosure can be created without departing from its basic scope, the scope of which is defined by the following claims.

Claims

1. Multiple encoders configured to generate multiple variant streams, A cross-variant instantaneous decoder refresh (IDR) identifier, The process of inspecting multiple variant streams, A step of identifying IDR frames in each of multiple variant streams, A step to determine the IDR frame corresponding to the cross-variant boundary, A system comprising a cross-variant instantaneous decoder refresh (IDR) identifier configured to include a step of segmenting IDR frames corresponding to cross-variant boundaries.

2. The system according to claim 1, wherein the IDR frame corresponding to the cross-variant boundary includes the cross-variant target segment boundary, the cross-variant target segment boundary represents a starting video frame intended to be the segment start, and at the segment start, it is aligned with segments containing the same media content throughout the variant stream.

3. The system according to claim 2, wherein the IDR frame is divided by cross-variant target segment boundaries.

4. The system according to claim 1, wherein the IDR frame corresponding to the cross-variant boundary includes a cross-variant image group (GOP) aligned segment boundary, the cross-variant GOP aligned segment boundary represents the starting video frame which is the closed GOP start, and which is aligned with segments containing the same media content throughout the variant stream at the closed GOP start.

5. The system according to claim 4, wherein the IDR frame is divided by cross-variant GOP aligned arrangement segment boundaries.

6. The cross-variant IDR identifier is, A process to determine the IDR frame corresponding to the inserted IDR frame by scene change detection, The system according to claim 1, further configured to include a step of operating in such a way as not to classify IDR frames that correspond to IDR frames inserted by scene change detection.

7. The system according to claim 1, wherein the step of determining an IDR frame corresponding to a cross-variant boundary includes the step of identifying whether an IDR frame at a video frame position is aligned across all of a plurality of variant streams.

8. A non-temporary computer-readable medium containing computer program code, wherein the actions performed when the computer program code is executed by the operation of one or more computer processors are: The process of inspecting multiple variant streams, A step of identifying instantaneous decoder refresh (IDR) frames within each variant stream, A step to determine the IDR frame corresponding to the cross-variant boundary, A non-temporary computer-readable medium that includes a step of separating IDR frames corresponding to cross-variant boundaries.

9. The non-temporary computer-readable media according to claim 8, wherein the IDR frame corresponding to the cross-variant boundary includes the cross-variant target segment boundary, the cross-variant target segment boundary represents a starting video frame intended to be the segment start, and at the segment start, is aligned with segments containing the same media content throughout the variant stream.

10. The non-transient computer-readable medium according to claim 9, wherein the IDR frame is divided by cross-variant target segment boundaries.

11. The non-temporary computer-readable medium according to claim 8, wherein the IDR frame corresponding to the cross-variant boundary includes a cross-variant image group (GOP) aligned segment boundary, the cross-variant GOP aligned segment boundary represents a starting video frame which is the closed GOP start, and which is aligned with segments containing the same media content throughout the variant stream at the closed GOP start.

12. The non-transient computer-readable medium according to claim 11, wherein the IDR frame is separated by cross-variant GOP aligned segment boundaries.

13. The aforementioned operation is, A process to determine the IDR frame corresponding to the inserted IDR frame by scene change detection, The non-temporary computer-readable medium according to claim 8, further comprising the step of operating in a manner that does not classify IDR frames corresponding to IDR frames inserted by scene change detection.

14. The non-temporary computer-readable medium according to claim 8, wherein the step of determining an IDR frame corresponding to a cross-variant boundary includes the step of identifying whether an IDR frame at a video frame position is aligned across all of a plurality of variant streams.

15. The non-transient computer-readable medium according to claim 8, wherein the operation is performed by a cross-variant IDR identifier integrated into the packager.

16. The video stream receiving process, A process for identifying the location of the boundary, The process involves dividing the boundaries and creating the first set of segments, A method comprising the step of, after exceeding a threshold, dividing the boundary to create a second set of segments having a different time length from the first set of segments.

17. The method according to claim 16, wherein the first set of segments is divided at a position corresponding to at least one of the start of a video stream, a seek point, or a chapter point.

18. The method according to claim 16, wherein each segment includes at least one GOP.

19. The method according to claim 16, wherein all segments in the first set of segments have equal time lengths, and all segments in the second set of segments have equal time lengths.

20. The method according to claim 16, wherein the time length of each segment in the second set of segments is longer than the time length of each segment in the first set of segments.

Images