Adaptive Bitrate Streaming of Media Stored in Matroska Container Files Using the Hypertext Transfer Protocol

Abstract

To provide an adaptive bit rate streaming of a medium stored in a Matroska container file using a hypertext transfer protocol.SOLUTION: A system and a method for adaptive bit rate streaming of a medium stored in a Matroska container file, using a hypertext transfer protocol (HTTP) is disclosed. A processor requests a part of a file from a remote server, identifies the EBML container file, and reads the top level index data describing the maximum bit rate of an alternative stream to be contained in the EBML container file.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention generally relates to adaptive streaming, and more specifically, to adaptive bitrate streaming of encoded media contained within a Matroska container file using the Hypertext Transfer Protocol.

Background Art

[0002] The term media streaming describes the playback of media on a playback device, where the media is stored on a server and continuously transmitted to the playback device via a network during playback. Generally, the playback device completes the playback of all buffered media prior to receiving the next portion of the media. Thus, the playback device stores a sufficient amount of media in the buffer at any given time during playback to prevent interruption of playback. Adaptive bitrate streaming or adaptive streaming involves detecting this streaming state (e.g., the user's network bandwidth and CPU capacity) in real time and adjusting, as appropriate, the quality of the media being streamed. Generally, the source media is encoded at multiple bitrates, and the playback device or client switches between streaming different encodings depending on the available resources.

[0003] Adaptive streaming solutions generally utilize either the Hypertext Transfer Protocol (HTTP) published by the Internet Engineering Task Force and the World Wide Web Consortium as RFC2616, or the Real-Time Streaming Protocol (RTSP) published by the Internet Engineering Task Force as RFC2326 to stream media between a server and a playback device. HTTP is a stateless protocol that enables a playback device to request byte ranges within a file. HTTP is described as stateless because the server is not required to record information about the state of the requesting playback device or the byte ranges requested by the playback device in order to respond to requests received from the playback device. RTSP is a network control protocol used to control a streaming media server. A playback device issues control commands such as "play" and "pause" to a server streaming media to control the playback of the media file. When RTSP is utilized, the media server records the state of each client device and determines the media for streaming based on the commands received from the client device and the state of the client.

[0004] In an adaptive streaming system, the source media is typically stored on a media server as a top-level index file that points to several alternative streams containing actual video and audio data. Each stream is typically stored within one or more container files. Different adaptive streaming solutions generally utilize different indexes and media containers. The Synchronized Multimedia Integration Language (SMIL), developed by the World Wide Web Consortium, is used to create indexes in several adaptive streaming solutions, including IIS Smooth Streaming developed by Microsoft Corporation (Redmond, Washington) and Flash Dynamic Streaming developed by Adobe Systems Incorporated (San Jose, California). HTTP Adaptive Bitrate Streaming, developed by Apple Computer Incorporated (Cupertino, California), generally implements the index file using an extended M3U playlist file (.M3U8), which is a text file containing a list of URIs that identify the media container files. The most commonly used media container formats are the MP4 container format specified in MPEG-4 Part 14 (i.e., ISO / IEC 14496-14) and the MPEG Transport Stream (TS) container specified in MPEG-2 Part 1 (i.e., ISO / IEC standard 13818-1). The MP4 container format is utilized in IIS Smooth Streaming and Flash Dynamic Streaming. The TS container is used in HTTP Adaptive Bitrate Streaming.

[0005] The Matroska container is a media container developed as an open standard project by the Matroska non-profit organization (Aussonne, France). The Matroska container is based on the Extensible Binary Meta-Language (EBML), which is a binary derivative of the Extensible Markup Language (XML). Decoding of the Matroska container is supported by many consumer electronics (CE) devices. The DivX Plus file format, developed by DivX, LLC (San Diego, California), utilizes extensions to the Matroska container format (i.e., it is based on the Matroska container format but includes elements not specified within the Matroska format). SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0006] Systems and methods for adaptive bitrate streaming of media stored within a Matroska container file utilizing the Hypertext Transfer Protocol (HTTP) according to embodiments of the present invention are disclosed. In one embodiment, a processor is configured to request portions of a file from a remote server via a client application. Additionally, the client application further reads top-level index data that identifies a plurality of EBML container files and describes at least a maximum bitrate of alternative streams contained within the EBML container files, analyzes the top-level index data, obtains information identifying the plurality of EBML container files, requests at least one portion of at least one of the EBML container files that contains at least one element specifying an encoding parameter of a stream contained within the EBML container file, reads an index that references each element that contains a portion of an encoded video within at least one of the EBML container files, utilizes the index to request a portion of a first EBML container file that includes an element that contains a portion of an encoded video, receives and buffers the requested element, decodes the encoded video contained within the buffered element using the encoding parameter, measures a current streaming state, and configures the processor to select another one of the EBML container files from which to read an element that contains a portion of an encoded video for decoding based on the measured streaming state and the description of the bitrates of the alternative streams contained within the top-level data.

[0007] A further embodiment includes steps of identifying each of a plurality of EBML container files, reading top-level index data that describes at least a maximum bitrate of each of the alternative streams contained within the EBML container files, analyzing the top-level index data to obtain information identifying the plurality of EBML container files, requesting at least one portion of at least one of the EBML container files that contains at least one element specifying an encoding parameter of a stream contained within the EBML container file, receiving an index that references elements containing portions of each encoded video within at least one of the EBML container files, using the index to request a portion of a first EBML container file that includes elements containing portions of the encoded video, receiving and buffering the requested elements, decoding the encoded video contained within the buffered elements using the encoding parameter, measuring a current streaming state, and selecting another one of the EBML container files from which an element containing a portion of the encoded video should be received for decoding, the selection being based on the measured streaming state and the description of the bitrates of the alternative streams contained within the top-level index data.

[0008] Another embodiment of the present invention includes a processor configured to capture at least one multimedia file containing source video via a source encoding application. Additionally, the source encoding application further selects a portion of the source video, transcodes the selected portion of the source video into a plurality of alternative portions of the encoded video, each alternative portion being encoded using a different set of encoding parameters, starting with an intra-frame beginning a closed picture group (GOP), writes each of the alternative portions of the encoded video to an element of a different EBML container file, each element also containing another element indicating the encoding parameters used to encode the alternative portion of the encoded video, located within an EBML container file, and configures the processor to add an entry to at least one index identifying the location of an element containing one of the alternative portions of the encoded video within each of the EBML container files.

[0009] Yet another further embodiment includes the steps of repeatedly selecting a portion of source video using a source encoder, transcoding the selected portion of the source video into a plurality of alternative portions of the encoded video using a source encoder, each alternative portion being encoded using a different set of encoding parameters and starting with an intra-frame beginning a closed picture group (GOP), writing each of the alternative portions of the encoded video to an element of a different EBML container file using a source encoder, each element also containing another element containing a set of encoding parameters corresponding to the encoding parameters used to encode the portion of the video, located within the EBML container file, and adding an entry to at least one index identifying the location of an element containing one of the alternative portions of the encoded video within each of the EBML container files. The present invention provides, for example, the following. (Item 1) A playback device configured to perform adaptive bitrate streaming using a plurality of alternative streams of encoded video packaged within an Extensible Binary Markup Language (EBML) container file, wherein each of the plurality of alternative streams has each part of the encoded video packed into an element of the EBML container file starting with an intra-frame that begins a closed picture group (GOP), and each of the EBML container files is the same source video encoded using different encoding parameters such that each of the EBML container files includes at least one element specifying encoding parameters of the stream contained within the EBML container file, The playback device includes a processor configured to request portions of a file from a remote server via a client application, The client application identifies a plurality of EBML container files, reads top-level index data describing at least the maximum bitrate of the alternative streams contained within the EBML container files, analyzes the top-level index data to obtain information for identifying the plurality of EBML container files, requests at least one portion of the EBML container files that contains at least one element specifying the encoding parameters of the stream contained within the EBML container files, reads an index that references each element containing a portion of the encoded video within at least one of the EBML container files, uses the index to request a portion of a first EBML container file that includes an element containing a portion of the encoded video, receives and buffers the requested element, Decoding the encoded video contained within the buffered element using the encoding parameter; Measuring the current streaming state; Selecting another one of the EBML container files from which an element containing a portion of the encoded video should be read for decoding, the selection being based on the measured streaming state and the description of the bitrate of the alternative stream contained within the top-level data; A playback device that further configures the processor to perform the above. (Item 2) Each of the EBML container files includes a plurality of Cluster elements, and each Cluster element contains a portion of the encoded video. The portion of the encoded video within each of the Cluster elements starts with an intra-frame and includes at least one closed picture group. The playback device according to Item 1. (Item 3) The portion of the encoded video within each of the Cluster elements has the same duration. The playback device according to Item 2. (Item 4) The portion of the encoded video within each of the Cluster elements has a duration of 2 seconds. The playback device according to Item 2. (Item 5) Each Cluster element contains a time code, and each encoded frame of the portion of the encoded video contained within the Cluster element is contained within a separate BlockGroup element. The playback device according to Item 2. (Item 6) The first BlockGroup element in the Cluster element contains the intra-frame. The playback device according to Item 5. (Item 7) The playback device according to item 6, wherein the first BlockGroup element contains a Block element that specifies the time code attribute of the intra-frame with respect to the time code of the Cluster element. (Item 8) The playback device according to item 1, wherein each of the alternative streams of the encoded video is encoded at a different maximum bit rate. (Item 9) The playback device according to item 8, wherein at least two of the alternative streams of the encoded video are encoded at different resolutions that use different sample aspect ratios but have the same display aspect ratio. (Item 10) The playback device according to item 9, wherein at least one element in each of the EBML container files that specifies the encoding parameters of the stream contained in the EBML container file specifies the frame height, frame width, and sample aspect ratio of the video stream contained in the EBML container file. (Item 11) The playback device according to item 10, wherein the client application configures the processor to set the frame height, frame width, and sample aspect ratio prior to the start of decoding of the encoded video contained in an element buffered from one of the EBML container files. (Item 12) The playback device according to item 8, wherein at least two of the alternative streams of the encoded video are encoded at different frame rates. (Item 13) The playback device according to item 12, wherein at least one element in each of the EBML container files that specifies the encoding parameters of the stream contained in the EBML container file specifies the frame rate of the video stream contained in the EBML container file. (Item 14) The playback device according to item 13, wherein the client application configures the processor to set the frame rate prior to starting decoding of the encoded video contained within an element buffered from one of the EBML container files. (Item 15) The playback device according to item 11, wherein the client application configures the playback device to request a portion of a file from a remote server via a Hypertext Transfer Protocol (HTTP) byte range request. (Item 16) The playback device according to item 1, wherein the top-level index data is SMIL data, and the information identifying the plurality of EBML container files is a plurality of Universal Resource Indicators (URIs) identifying the location of each of the EBML container files. (Item 17) The playback device according to item 1, wherein the SMIL data specifies the maximum bitrate of the encoded stream contained within the EBML container file identified by each of the URIs. (Item 18) An index that references each of the elements containing portions of the encoded stream within the EBML container file is located within at least one element of the EBML container file. The SMIL file specifies a portion of the EBML container file that contains an element containing the encoding parameter. The client application parses the SMIL file to identify the portion of the EBML container file that contains the element containing the encoding parameter, and requests the portion of the EBML container file that contains the element containing the encoding parameter and configures the processor to do so. The playback device according to item 17. (Item 19) Each EBML container file includes at least one element containing an index that references each element containing a portion of the encoded video within the EBML container file. The client application configures the processor to read an index that references each element containing a portion of the encoded video within the EBML container file by requesting the portion of the EBML container file that includes at least one element containing the index, as described in item 1. (Item 20) The index is located within a Cue element in the EBML container file, as described in item 19 of the playback device. (Item 21) The Cue element includes a time attribute and includes a plurality of CuePoint elements that indicate the location of each element containing a portion of the encoded video within the EBML file, as described in item 20 of the playback device. (Item 22) The client application uses the time attribute of the CuePoint element in the index to search for the location of the element containing a specific portion of the encoded video within the EBML file, and requests the portion of the EBML file that includes at least the element identified by the CuePoint element to configure the processor to do so, as described in item 21 of the playback device. (Item 23) The encoding parameters used to decode the encoded video include at least one encoding parameter selected from the group consisting of frame rate, frame height, frame width, sample aspect ratio, maximum bit rate, and minimum buffer size, as described in item 1 of the playback device. (Item 24) The playback device according to item 1, wherein the client application configures the processor to measure the current streaming state by measuring the time taken to receive the requested element from the time the element was requested. (Item 25) The client application first configures the processor to request a portion of the first EBML container file that contains at least one element specifying the encoding parameters of the stream contained within the first EBML container file. When the client application selects to read an element containing a portion of the encoded video to be decoded from a second EBML container file based on the measured streaming state and the description of the bitrate of the alternative stream contained within the top-level index, the client application further configures the processor to request a portion of the second EBML container file that contains at least one element specifying the encoding parameters of the stream contained within the second EBML container file. The playback device according to item 1. (Item 26) The playback device according to item 1, further configured to perform adaptive bitrate streaming using an EBML container file containing a trick play track. (Item 27) The playback device according to item 26, wherein the client application configures the processor to select the EBML container file containing the trick play track as the EBML container file from which to read an element containing a portion of the encoded video in response to a user command. (Item 28) The playback device according to item 1, further configured to perform adaptive streaming using at least one EBML container file containing an encoded audio track. (Item 29) The client application reads, decodes the encoded audio from one of the at least one EBML container file containing the audio track, and configures the processor to synchronize the decoded audio with the decoded video, the playback device according to item 28. (Item 30) Each of the EBML container files includes an element containing at least one encoded audio track multiplexed with the encoded video, the playback device according to item 1. (Item 31) The client application reads a portion of the EBML file containing an element including a portion of one of the encoded audio tracks, decodes the read and encoded audio, and synchronizes the decoded audio with the decoded video and is configured to perform, the playback device according to item 28. (Item 32) The playback device is further configured to perform adaptive streaming using at least one EBML container file containing a subtitle track, the playback device according to item 1. (Item 33) The client application reads subtitles from one of the at least one EBML container file containing the subtitle track, synchronizes the subtitles with the decoded video, and overlays the subtitles on the decoded video and configures the processor to perform, the playback device according to item 32. (Item 34) Each of the EBML container files contains an element that contains at least one subtitle track multiplexed with the encoded video, the playback device according to item 1. (Item 35) The client application reads a portion of the EBML file that contains an element that contains a portion of one of the subtitle tracks, synchronizes the subtitle with the decoded video, and overlays the subtitle on the decoded video and is configured to perform the method of the playback device according to item 34. (Item 36) A method for performing adaptive bitrate streaming of media on a playback device using a plurality of alternative streams of an encoded video packaged within an Extensible Binary Markup Language (EBML) container file, wherein each of the alternative streams of the video is such that each portion of the encoded video packed into an element of the EBML container file starts with an intra frame that begins a closed picture group (GOP), and each of the EBML container files is the same source video encoded using different encoding parameters, including at least one element that specifies the encoding parameters of the stream contained within the EBML container file. The method identifies each of the plurality of EBML container files and reads top-level index data that describes at least the maximum bitrate of each of the alternative streams contained within the EBML container file, analyzes the top-level index data to obtain information for identifying the plurality of EBML container files, requests at least a portion of the EBML container file that contains at least one element that specifies the encoding parameters of the stream contained within the EBML container file, Reading an index that references each element containing a portion of the encoded video within at least one of the EBML container files, Using the index to request a portion of a first EBML container file that includes elements containing portions of the encoded video, Receiving and buffering the requested elements, Using the encoding parameters to decode the encoded video contained within the buffered elements, Measuring the current streaming state, For decoding, selecting another one of the EBML container files from which to read elements containing portions of the encoded video, the selection being based on the measured streaming state and the description of the bitrate of the alternative stream contained within the top-level index data, Including, a method. (Item 37) A source encoder configured to encode source video as a plurality of alternative video streams packed within a container file, the container file being an Extensible Binary Markup Language (EBML) file, The source encoder, Comprises a processor configured to take in at least one multimedia file containing the source video via a source encoding application, The source encoding application, Selecting a portion of the source video, Transcoding the selected portion of the source video into a plurality of alternative portions of encoded video, each alternative portion being encoded using a different set of encoding parameters and starting with an intra-frame beginning a closed picture group (GOP), Writing each alternative portion of the encoded video to an element of a different EBML container file, where each element is located within an EBML container file that also contains another element indicating the encoding parameters used to encode the alternative portion of the encoded video, and adding an entry to at least one index that identifies the location of an element containing one of the alternative portions of the encoded video in each of the EBML container files, and configuring the processor to perform the above, a source encoder. (Item 38) Transcoding the selected portion of the source video further includes transcoding the selected portion into at least one closed picture group, the source encoder according to item 37. (Item 39) The portion of the source video is selected based on the duration of the selected portion of the source video, the source encoder according to item 37. (Item 40) The source encoding application configures the processor to select a portion of the source video having a duration of 2 seconds, the source encoder according to item 39. (Item 41) Each of the alternative portions of the encoded video is encoded at a different maximum bitrate, the source encoder according to item 37. (Item 42) At least two of the alternative portions of the encoded video are encoded at different resolutions, the source encoder according to item 41. (Item 43) At least two of the alternative portions of the encoded video are encoded at different frame rates, the source encoder according to item 41. (Item 44) The element of the EBML container file into which each alternative part of the encoded video is written is a Cluster element that contains a time code, and the part of the encoded video is contained within a BlockGroup element within the Cluster element, the source encoder described in item 37. (Item 45) Each encoded frame of the alternative part of the encoded video contained within the Cluster element is contained within a separate BlockGroup element, the source encoder described in item 44. (Item 46) The first BlockGroup element within the Cluster element contains an IDR frame, the source encoder described in item 45. (Item 47) The first BlockGroup element contains a Block element that specifies the time code attribute of the IDR frame with respect to the time code of the Cluster element, the source encoder described in item 46. (Item 48) Each element into which each alternative part of the encoded video is written is assigned the same time code, the source encoder described in item 37. (Item 49) The source encoding application further configures the processor to create an index for each of the EBML container files, the source encoder described in item 37. (Item 50) The source encoding application further configures the processor to add the location of an element that contains one of the alternative parts of the encoded video within each of the EBML container files to the index for the EBML container file, the source encoder described in item 49. (Item 51) The source encoding application further configures the processor to pack the index for each EBML container file into the EBML container file, the source encoder according to item 49. (Item 52) Each index is the source encoder according to item 51, which includes a Cues element. (Item 53) Each Cue element is the source encoder according to item 52, and includes a CuePoint element that indicates the location of an element containing one of the alternative portions of the encoded video within the EBML file. (Item 54) The source encoding application further configures the processor to create a top-level index file that identifies each of the EBML container files, the source encoder according to item 37. (Item 55) The captured multimedia file also includes source audio, the source encoder according to item 37. (Item 56) The source encoding application configures the processor to multiplex the audio into each of the EBML container files, the source encoder according to item 55. (Item 57) The source encoding application configures the processor to write the audio to a separate EBML container file, the source encoder according to item 55. (Item 58) The source encoding application further configures the processor to transcode at least one of the at least one audio track, the source encoder according to item 55. (Item 59) The captured multimedia file further includes subtitles, the source encoder according to item 55. (Item 60) The source encoding application is the source encoder according to item 59, which configures the processor to multiplex the subtitles into each of the EBML container files. (Item 61) The source encoding application is the source encoder according to item 59, which configures the processor to write the subtitles to a separate EBML container file. (Item 62) The source encoding application is the source encoder according to item 37, which further configures the processor to transcode the source video to create a lower frame rate trick play track and write the trick play track to a separate EBML container file. (Item 63) The trick play track is also of a lower resolution than the source video, for the source encoder according to item 62. (Item 64) The source encoding application is the source encoder according to item 37, which further configures the processor to write an element containing a set of encoding parameters into each of the EBML container files. (Item 65) The set of encoding parameters includes at least one parameter selected from the group consisting of frame rate, frame height, frame width, sample aspect ratio, maximum bit rate, and minimum buffer size, for the source encoder according to item 64. (Item 66) A method for encoding a source video as a plurality of alternative streams packed in an Extensible Binary Markup Language (EBML) file using a source encoder, The method includes repeatedly selecting portions of the source video using the source encoder, and Transcoding a selected portion of the source video into multiple alternative portions of an encoded video using the source encoder, each alternative portion being encoded using a different set of encoding parameters and starting with an intra-frame that begins a closed picture group (GOP). Writing each of the alternative portions of the encoded video into elements of different EBML container files using the source encoder, each element being located within an EBML container file that also includes another element containing a set of encoding parameters corresponding to the encoding parameters used to encode the portion of the video. Adding an entry to at least one index that identifies the location of an element containing one of the alternative portions of the encoded video in each of the EBML container files A method comprising.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4a

Figure 4b

Figure 4c

Figure 4d

Figure 4e

Figure 5

Figure 5a

Figure 6

Figure 7

Figure 8

Figure 9a

Figure 9b

[0011] Next, referring to the drawings, a system and method for adaptive bitrate streaming of media stored within a Matroska container file that utilizes the Hypertext Transfer Protocol (HTTP), according to an embodiment of the present invention, are illustrated. In some embodiments, the source media is encoded as a number of alternative streams. Each stream is stored within a Matroska (MKV) container file. In many embodiments, the Matroska container file is a special Matroska container file in that the manner in which the media within each stream is encoded and stored within the container is constrained to improve streaming performance. In some embodiments, the Matroska container file is further special in that additional index elements (i.e., elements not specified as part of the Matroska container format) can be included within the file to facilitate the reading of the desired media during adaptive bitrate streaming. In some embodiments, each stream (i.e., audio, video, or subtitles) is stored within a separate Matroska container file. In other embodiments, the encoded video stream is multiplexed with one or more encoded audio and / or subtitle streams within each Matroska container file. A top-level index file containing an index to the streams contained within each of the container files is also generated to enable adaptive bitrate streaming of the encoded media. In many embodiments, the top-level index file is a Synchronized Multimedia Integration Language (SMIL) file containing URIs for each of the Matroska container files. In other embodiments, any of a variety of file formats can be utilized in the generation of the top-level index file.

[0012] According to embodiments of the present invention, the performance of an adaptive bitrate streaming system can be significantly improved by encoding each portion of a source video at each bitrate such that portions of the video are encoded within each stream as a single (or at least one) closed picture group (GOP) that begins with an instantaneous decode refresh (IDR) frame. The GOP for each stream can then be stored as a Cluster element within a Matroska container file for the stream. In this way, a playback device can switch between streams upon completion of playback, and regardless of the stream from which a cluster is obtained, the first frame within the cluster is an IDR frame and can be decoded without reference to any encoded media other than the encoded media contained within the Cluster element. In many embodiments, all sections of the source video encoded as a GOP have the same duration. In some embodiments, each two-second sequence of the source video is encoded as a GOP.

[0013] During adaptive streaming, the reading of media using HTTP can be improved by adding additional index information to the Matroska container file that contains each of the encoded streams. In some embodiments, the index is a simplified index in that the index points only to the IDR at the start of each cluster. In many embodiments, the index of the Matroska container file includes additional non-standard attributes (i.e., attributes that do not form part of the Matroska container file format specification) that specify the size of each cluster such that a playback device can read Cluster elements from the Matroska container file via HTTP using byte range requests.

[0014] Adaptive streaming of the source media encoded as described above can be adjusted by a playback device according to an embodiment of the present invention. The playback device obtains information regarding each of the available streams from a top-level index file, selects one or more streams, and uses them in media playback. The playback device can then obtain header information from a Matroska container file containing one or more bitstreams or streams, where the header provides information regarding decoding of the stream. The playback device can also request index information that indexes the encoded media stored within the associated Matroska container file. The index information can be stored within the Matroska container file, or separately from the Matroska container file, within a top-level index or a separate index file. The index information enables the playback device to request, from a server via HTTP, the byte range corresponding to a Cluster element within a Matroska container file containing a particular portion of the encoded media. As the playback device receives Cluster elements from the HTTP server, the playback device can evaluate the current streaming state and determine whether the bitrate of the streamed media should be increased or decreased. If the playback device determines that a change within the bitrate is required, the playback device can obtain the header information and index information for the container file containing the desired stream (assuming the playback device has not already obtained this information). The index information can then be used to identify the byte range of a Cluster element containing the next portion of the source media encoded at the desired bitrate, and the identified Cluster element can be read from the server via HTTP. The next portion of the source media requested is generally identified based on the Cluster elements already requested by the playback device and the Cluster elements buffered by the playback device.The next portion of the source media requested from the substitute stream is required to minimize the likelihood that the buffer of the playback device will underflow (i.e., run out of media to play) prior to the playback device receiving a Cluster element containing the next portion of the source media. In this way, the playback device can achieve adaptive bitrate streaming by reading successive Cluster elements from the various streams in accordance with their suitability for the streaming state, using the top-level index and index information that describe the Cluster elements within each Matroska container file.

[0015] In some embodiments, the variation in bitrate between different streams can be achieved by modifying the encoding parameters for each stream, including but not limited to bitrate, frame rate, and resolution. When different streams include different resolutions, the display aspect ratio of each stream is the same and the sample aspect ratio is modified to ensure a smooth transition from one resolution to another. The encoding of source video for use in adaptive bitrate streaming and the playback of encoded source video using HTTP requests to achieve adaptive bitrate streaming according to embodiments of the present invention are discussed further below.

[0016] (Adaptive Streaming System Architecture) An adaptive streaming system according to an embodiment of the present invention is illustrated in FIG. 1. The adaptive streaming system 10 includes a source encoder 12 configured to encode source media as a number of alternative streams. In the illustrated embodiment, the source encoder is a server. In other embodiments, the source encoder can be any processing device that includes a processor and sufficient resources for transcoding source media (including but not limited to video, audio, and / or subtitles). As further discussed below, the source encoding server 12 generates a top-level index to a plurality of container files containing the streams, at least a plurality of which are alternative streams. Alternative streams are streams that encode the same media content in different ways. In many cases, alternative streams encode media content (such as, but not limited to, video) at different bitrates. In some embodiments, alternative streams are encoded at different resolutions and / or different frame rates. The top-level index file and the container files are uploaded to an HTTP server 14. Various playback devices can then request portions of the top-level index file and the container files via a network 16, such as the Internet, using HTTP or another suitable stateless protocol.

[0017] In many embodiments, the top-level index file is an SMIL file, and the media is stored within a Matroska container file. As further discussed below, the media can be stored within the Matroska container file to facilitate adaptive bitrate streaming of the media. In many embodiments, the Matroska container file is a special Matroska container file that includes enhancements (i.e., elements that do not form part of the Matroska file format specification) that facilitate reading of specific portions of the media via HTTP during adaptive bitrate streaming of the media.

[0018] In the illustrated embodiment, the playback devices include personal computer 18 and mobile phone 20. In other embodiments, the playback devices can include consumer electronic devices such as DVD players, Blu-ray™ players, televisions, set-top boxes, video game consoles, tablets, and other devices that can connect to a server via HTTP and play the encoded media. The specific architecture is shown in FIG. 1, but any of a variety of architectures can be utilized such that the playback device can request portions of the top-level index file and the container file in accordance with embodiments of the present invention.

[0019] (File Structure) For streaming to a playback device according to an embodiment of the present invention, files generated by a source encoder and / or stored on an HTTP server are illustrated in FIG. 2. Files utilized within adaptive bitrate streaming of source media include a top-level index 30 and a plurality of container files 32, each containing at least one stream. The top-level index file describes the respective content of the container files. As further discussed below, the top-level index file can take various forms, including an SMIL file, and the container files can take various forms, including special Matroska container files.

[0020] In many embodiments, each Matroska container file contains a single stream. For example, the stream can be one of several alternative video streams, an audio stream, one of several alternative audio streams, a subtitle stream, one of several alternative subtitle streams, a trick play stream, or one of several alternative trick play streams. In some embodiments, the Matroska container file includes a plurality of multiplexed streams. For example, the Matroska container can include a video stream, and one or more audio streams, one or more subtitle streams, and / or one or more trick play streams. As further discussed below, in many embodiments, the Matroska container file is a special file. The manner in which media is encoded and stored within the Cluster elements within the Matroska container file can be conditioned on constraints designed to improve the performance of an adaptive bitrate streaming system. Additionally, the Matroska container file can include an index element that facilitates the identification and download of Cluster elements from various Matroska container files during adaptive streaming of media. The top-level index file and Matroska container file that can be used in an adaptive bitrate streaming system according to embodiments of the present invention are discussed below.

[0021] (Top-level index file) According to many embodiments of the present invention, a playback device uses a top-level index file to identify a container file that contains streams available to the playback device for use in adaptive bitrate streaming. In many embodiments, the top-level index file can include references to container files each containing an alternative stream of encoded media. The playback device can use the information in the top-level index file to read the encoded media from each of the container files according to the streaming state experienced by the playback device.

[0022] In some embodiments, the top-level index file provides information that enables the playback device to read information regarding the encoding of the media within each of the container files and an index to the encoded media within each of the container files. In some embodiments, each container file includes information regarding the encoded media contained within the container file and an index to the media encoded within the container file, and the top-level index file indicates the portions of each container file that contain this information. Thus, the playback device can read the top-level index file and use the top-level index file to request one or more portions of the container files that include information regarding the encoded media contained within the container files and an index to the media encoded within the container files. Various top-level index files that can be used in an adaptive bitrate streaming system according to embodiments of the present invention are further discussed below.

[0023] (Top-Level Index SMIL File) In some embodiments, the top-level index file utilized in the adaptive bitrate streaming of media is an SMIL file, which is an XML file that includes a list of URIs that describe each of the streams and the container files that contain the streams. The URI can include information, such as the "system-bitrate" of the stream contained within the stream and the location of specific pieces of data within the container file.

[0024] The basic structure of an SMIL file involves providing an XML declaration and an SMIL element. The SMIL element defines the streams available for use in adaptive bitrate streaming and generally includes a HEAD element, which is left empty, and a BODY element, which generally contains only a PAR (parallel) element. The PAR element describes streams (i.e., including media that can be presented simultaneously) that can be played simultaneously.

[0025] The SMIL specification defines several child elements for the PAR element, which can be used to specify the streams available for use in adaptive bitrate streaming. The VIDEO, AUDIO, and TEXTSTREAM elements can be used to define specific video, audio, or subtitle streams. The VIDEO, AUDIO, and TEXTSTREAM elements can collectively be referred to as media objects. The basic attributes of a media object are the SRC attribute that specifies the full path or URI to the container file containing the associated stream and the XML:LANG attribute that includes a three-letter language code. Additional information regarding the media object can be specified using the PARAM element. The PARAM element is the standard way within the SMIL format to provide a pair of a common name and a value. In some embodiments of the present invention, specific PARAM elements are defined to be utilized during adaptive bitrate streaming.

[0026] In many embodiments, the "header-request" PARAM element is defined to specify the size of the header section of a container file that contains a stream. The value of the "header-request" PARAM element generally specifies the number of bytes between the start of the file and the start of the encoded media within the file. In many embodiments, the header contains information regarding the manner in which the media is encoded, and the playback device reads the header prior to playing the encoded media in order to be able to configure a decoder for playback of the encoded media. Examples of the "header-request" PARAM element are

[0027] [Number] as follows.

[0028] In some embodiments, the "mime" PARAM element is defined to specify the MIME type of a stream. A "mime" PARAM element that identifies a stream as an H.264 stream (i.e., a stream encoded according to the MPEG-4 Advanced Video Coding standard) is

[0029] [Number] as follows.

[0030] The MIME type of a stream can be specified using the "mime" PARAM element depending on the appropriateness for the encoding of a particular stream (e.g., an AAC audio or UTF-8 text stream).

[0031] When the media object is a VIDEO element, additional attributes are defined within the SMIL file format specification that includes a systemBitrate attribute that specifies the bitrate of the stream within the container file identified by the VIDEO element, and width and height attributes that specify the dimensions of the encoded video in pixels. Additional attributes can also be defined using the PARAM element. In some embodiments, the "vbv" PARAM element is defined to specify the VBV buffer size of the video stream in bytes. A video buffering verifier (VBV) is a theoretical MPEG video buffer model used to ensure that an encoded video stream can be accurately buffered and played back on a decoder device. An example of a "vbv" PARAM element specifying a VBV size of 1000 bytes is

[0032]

Number

[0033] Examples of VIDEO elements that include the aforementioned attributes are

[0034]

Number

[0035] An adaptive bitrate streaming system according to an embodiment of the present invention can support trick play streams, which can be used to provide smooth visual seeking of source content encoded for adaptive bitrate streaming. A trick play stream, during playback, can be encoded through the source media as if visual seeking were accelerated. In fact, a trick play stream is simply a separate track that encodes the source media at a lower frame rate. In many embodiments of the system, the reference to the trick play track is indicated by the systemProfile attribute of the VIDEO element. In other embodiments, any of a variety of techniques can be utilized to indicate within the top-level index file that a particular stream is a trick play stream. An example of a trick play stream VIDEO element according to an embodiment of the present invention is

[0036] [Number] as follows.

[0037] In some embodiments of the present invention, a "reservedBandwidth" PARAM element can be defined for the AUDIO element. The "reservedBandwidth" PARAM element specifies the bitrate of the audio stream in Kbps. An example of an AUDIO element specified according to an embodiment of the present invention is

[0038] [Number] as follows.

[0039] In some embodiments, the "reservedBandwidth" PARAM element is also defined for the TEXTSTREAM element. An example of a TEXTSTREAM element that includes the "reservedBandwidth" PARAM element, according to an embodiment of the present invention, is

[0040]

Number

[0041] In other embodiments, any of a variety of mechanisms can be utilized to specify information regarding VIDEO, AUDIO, and SUBTITLE elements, depending on their appropriateness for a particular application.

[0042] The SWITCH element is a mechanism defined within the SMIL file format specification that can be utilized to define an adaptive or alternative stream. An example of a style in which the SWITCH element can be utilized to specify alternative video streams at different bitrates is

[0043]

Number

[0044] The SWTICH element specifies the URLs of three alternative video streams. The file names indicate the respective different bitrates of the streams. As further discussed below, the SMIL file format specification provides a mechanism that can be utilized in accordance with embodiments of the present invention to specify additional information regarding the stream and the container file in which it is contained, within the top-level index SMIL file.

[0045] In many embodiments of the present invention, the EXCL (exclude) element is used to define non-adaptive alternative tracks during playback with a streaming state. For example, the EXCL element can be used to define alternative audio tracks or alternative subtitle tracks. An example of a manner in which the EXCL element can be utilized to specify alternative English and French audio streams is

[0046]

Number

[0047] An example of a top-level index SMIL file that defines the attributes and parameters of two alternative video levels, audio streams, and subtitle streams according to an embodiment of the present invention is

[0048]

Number

[0049]

Number

[0050] A top-level index SMIL file can be generated when source media is encoded for playback via adaptive bitrate streaming. Alternatively, a top-level index SMIL file can be generated when a playback device requests the start of playback of the encoded media. When the playback device receives the top-level index SMIL file, the playback device can parse the SMIL file and identify the available streams. The playback device can then select a stream and use it to play the content, use the SMIL file to identify portions of the container file, obtain information regarding the encoding of a particular stream, and / or download an index to the encoded media within the container file.

[0051] Although the top-level index SMIL file has been described above, any of the various top-level index file formats can be utilized to create a top-level index file, depending on the suitability for a particular application according to an embodiment of the present invention. The use of a top-level index file to enable playback of encoded media using adaptive bitrate streaming according to an embodiment of the present invention is further discussed below.

[0052] (Store media in a MATROSKA file for adaptive bitrate streaming) The Matroska container file used to store the encoded video according to an embodiment of the present invention is illustrated in FIG. 3. The container file 32 is an Extensible Binary Markup Language (EBML) file, which is an extension of the Matroska container file format. The special Matroska container file 32 includes standard EBML elements 34 and a standard Segment element 36 that includes a standard Seek Head element 40, a standard Segment Information element 42, and a standard Tracks element 44. These standard elements describe the media contained within the Matroska container file. The Segment element 36 also includes a standard Clusters element 46. As will be described below, the manner in which the encoded media is inserted within the individual Cluster elements 48 within the Clusters element 46 is constrained to improve the playback of the media within an adaptive streaming system. In many embodiments, the constraints imposed on the encoded video are consistent with the specifications of the Matroska container file format and involve encoding the video such that each cluster includes at least one closed GOP that starts with an IDR frame. In addition to the aforementioned standard elements, the Segment element 36 also includes a modified version of the standard Cues element 52. As will be further discussed below, the Cues element includes special CuePoint elements (i.e., non-standard CuePoint elements) that facilitate the retrieval of the media contained within a particular Cluster element via HTTP.

[0053] The constraints imposed on the encoding of media for adaptive bitrate streaming according to embodiments of the present invention, the format of the encoded media within the Clusters element of the Matroska container file, and the additional index information inserted into the container file are further discussed below.

[0054] (Encoding of Media for Insertion within Cluster Elements) An adaptive bitrate streaming system provides the playback device with options to select between different streams of encoded media during playback, according to the streaming state experienced by the playback device. In many embodiments, switching between streams is facilitated by separately pre-encoding discrete portions of the source media according to the encoding parameters of each stream, and then including each separately encoded portion within its own Cluster element in the stream's container file. Further, the media contained within each cluster is encoded such that the media is playable without referring to media contained within any other cluster within the stream. Thus, each stream contains Cluster elements corresponding to the same discrete portions of the source media, and at any time, the playback device can select a Cluster element from the stream that is most appropriate for the streaming state experienced by the playback device and begin playback of the media contained within the Cluster element. Thus, the playback device can select clusters from different streams as the streaming state experienced by the playback device changes over time. In some embodiments, the Cluster elements are further constrained such that each Cluster element contains a portion of media encoded from source media having the same duration. In some embodiments, each Cluster element includes 2 seconds of encoded media. Depending on the type of media (i.e., video, audio, or subtitles), the specific constraints applied to the media encoded within each Cluster element are discussed below.

[0055] The Clusters element of a Matroska container file containing a video stream according to an embodiment of the present invention is illustrated in FIG. 4a. The Clusters element 46 includes a plurality of Cluster elements 48, each containing a discrete portion of the encoded video. In the illustrated embodiment, each Cluster element 48 includes 2 seconds of encoded video. In other embodiments, the Cluster element includes encoded video having more or less than 2 seconds. The smaller the Cluster element (i.e., the shorter the duration of the media encoded within each Cluster element), the higher the overhead associated with requesting each Cluster element. Thus, there is a trade-off between the responsiveness of the playback device to changes in the streaming state and the effective data rate of an adaptive streaming system for a given set of streaming states (i.e., the portion of the available bandwidth actually utilized to transmit the encoded media). In some embodiments, the encoded video sequences within the Cluster element have different durations. Each Cluster element 48 includes a Timecode element 60 indicating the start time of the encoded video within the Cluster element and the plurality of BlockGroup elements. As described above, the encoded video stored within a cluster is constrained so that it can be played back without referencing the encoded video contained within any of the other Cluster elements within the container file. In many embodiments, the encoding of the video contained within the Cluster element, as a GOP where the first frame is an IDR frame, enforces the constraint. In the illustrated embodiment, the first BlockGroup element 62 contains an IDR frame. Thus, the first BlockGroup element 62 does not include a ReferenceBlock element. The first BlockGroup element 62 includes a Block element 64 that specifies the Timecode attribute of the frame encoded within the Block element 64 relative to the Timecode of the Cluster element 48.Subsequent BlockGroup elements 66 are not restricted as to the types of frames they can contain (except that they cannot reference frames not contained within a Cluster element). Thus, subsequent BlockGroup elements 66 can include ReferenceBlock elements 68 that reference other BlockGroup elements utilized in the decoding of frames contained within the BlockGroup, or can contain IDR frames, and are similar to the first BlockGroup element 62. As described above, the manner in which the encoded video is inserted within the Cluster element of the Matroska file conforms to the specifications of the Matroska file format.

[0056] The insertion of encoded audio and subtitle information into the Cluster element 46 of a Matroska container file according to an embodiment of the present invention is illustrated in FIGS. 4b and 4c. In the illustrated embodiment, the encoded media is inserted into the Cluster element 48 subject to the same constraints applied to the encoded video described above in relation to FIG. 4a. Additionally, the duration of the encoded audio and subtitle information within each Cluster element corresponds to the duration of the encoded video within the corresponding Cluster element of the Matroska container file containing the encoded video. In other embodiments, the Cluster elements within a container file containing an audio and / or subtitle stream need not correspond to the start time and duration of the Cluster elements within a container file containing an alternative video stream.

[0057] (Multiplexing of Streams within a Single MKV Container File) Assume that the cluster elements shown in FIGS. 4a - 4c contain a single stream within each Matroska container file. In some embodiments, media from multiple streams are multiplexed within a single Matroska container file. Thus, a single container file can contain a video stream multiplexed with one or more corresponding audio streams and / or one or more corresponding subtitle streams. Storing the streams in this way can result in duplication of audio and subtitle streams across multiple alternative video streams. However, the search time for reading the encoded media from the video stream and the associated audio and / or subtitle streams can be reduced for adjacent storage of the data on the server. The cluster element 46 of a Matroska container file containing multiplexed video, audio, and subtitle data according to an embodiment of the present invention is illustrated in FIG. 4d. In the illustrated embodiment, each Cluster element 48 includes an additional BlockGroup element for each of the multiplexed streams. The first Cluster element contains an encoded video frame and includes a first BlockGroup element 62v for the encoded video, which includes a Block element 64v indicating the Timecode attribute of the frame (i.e., Timecode attribute 60) relative to the start time of the Cluster element. The second BlockGroup element 62a includes an encoded audio sequence and includes a Block element 64a indicating the Timecode of the encoded audio relative to the start time of the Cluster element, and the third BlockGroup element 62s contains encoded subtitles and includes a Block element 64s indicating the Timecode of the encoded subtitles relative to the start time of the Cluster element. Although not shown, in the illustrated embodiment, each Cluster element 48 is likely to include additional BlockGroup elements containing additional encoded video, audio, or subtitles.The same restrictions regarding the encoded media apply regardless of the multiplexing of the encoded video, audio, and / or subtitle streams.

[0058] (Incorporation of a trick play track into an MKV container file for use in an adaptive bitrate streaming system) The incorporation of a trick play track into a Matroska container file was proposed by DivX, LLC in U.S. Patent Application No. 12 / 260,404, "Application Enhancement Tracks," filed on October 29, 2008, the disclosure of which is hereby incorporated by reference in its entirety. A trick play track similar to the trick play track described in U.S. Patent Application No. 12 / 260,404 can be used to provide a trick play stream to an adaptive bitrate streaming system according to an embodiment of the present invention, and to provide smooth visual seeking through encoded source content for adaptive bitrate streaming. A separate trick play track can be encoded to accelerate visual seeking through the source media during playback, but in practice, a trick play track is simply a separate track that encodes the source media at a lower frame rate. In some embodiments, a trick play stream is created by generating a trick play track and inserting the trick play track into a Matroska container file, subject to the aforementioned constraints regarding the insertion of a video stream into the Matroska container file, as outlined in U.S. Patent Application No. 12 / 260,404. In many embodiments, the trick play track is also subject to the further constraint that each frame within the GOP of each Cluster element within the trick play track is encoded as an IDR frame. Similar to other video streams, each Cluster element contains a GOP corresponding to the same two seconds of source media as the corresponding Cluster element within another stream. Within the GOP of the trick play track, there are simply fewer frames, and each frame has a longer duration. Thus, transitions to and from the trick play stream can be processed in the same manner as transitions between any of the other encoded streams are processed within an adaptive bitrate streaming system according to an embodiment of the present invention.To achieve acceleration of visual search, the playback of frames contained within a trick play track generally involves the playback device manipulating the timecode assigned to frames of the encoded video prior to providing the frames to the decoder of the playback device in order to achieve a desired increase in the rate of acceleration of the search (e.g., x2, x4, x6, etc.).

[0059] A Cluster element containing encoded media from a trick play track is shown in FIG. 4e. In the illustrated embodiment, the encoded trick play is inserted into Cluster element 48, subject to the same constraints applied to the encoded video described above in connection with FIG. 4a. However, each Block element contains an IDR. In other embodiments, the Cluster elements within the container file containing the trick play track need not correspond to the start time and duration of the Cluster elements within the container file containing the alternative video stream.

[0060] In many embodiments, the source content can be encoded to provide a single trick play track or multiple trick play tracks for use by an adaptive bitrate streaming system. When a single trick play track is provided, the trick play track is generally encoded at a low bitrate. When multiple alternative trick play tracks are provided, adaptive rate streaming can also be performed with respect to the trick play tracks. In some embodiments, the multiple trick play tracks are provided to accommodate acceleration of visual search at different rates throughout the encoded media.

[0061] (Incorporation of Indexing Information into MKV Container File) The specification for the Matroska container file format provides an optional Cue element that is used to index Block elements within the container file. A modified Cue element 52 incorporated in a Matroska container file according to an embodiment of the present invention and capable of facilitating the request for clusters by a playback device using HTTP is illustrated in FIG. 5. The modified Cue element 52 includes a plurality of CuePoint elements 70 each including a CueTime attribute 72. Each CuePoint element includes a CueTrackPosition element 74 containing CueTrack76 and CueClusterPosition78 attributes. In many embodiments, the CuePoint elements are configured to identify a particular Cluster element, as opposed to a particular Block element within a Cluster element. However, in some applications, the ability to search for a particular BlockGroup element within a Cluster element is required and additional index information is included within the Cue element.

[0062] The use of a modified Cues element for indexing encoded media within the Clusters element of a Matroska file according to an embodiment of the present invention is illustrated in FIG. 6. CuePoint elements are generated to correspond to each Cluster element within the Matroska container file. The CueTime attribute 72 of the CuePoint element 70 corresponds to the Timecode attribute 60 of the corresponding Cluster element 48. In addition, the CuePoint element contains a CueTrackPosition element 74 having a CueClusterPosition attribute 78 that points to the start of the corresponding Cluster element 48. The CueTrackPosition element 74 can also include a CueBlockNumber attribute, which is generally used to indicate the Block element containing the first IDR frame within the Cluster element 48.

[0063] As can be easily understood, the modified Cue element 52 forms an index to each of the Cluster elements 48 within the Matroska container file. Further, the CueTrackPosition element provides information that can be used by a playback device to request a byte range of a particular Cluster element 48 from a remote server via HTTP or another suitable protocol. The Cue element of a conventional Matroska file does not provide the playback device with information regarding the number of bytes to request from the start of the Cluster element in order to obtain all of the encoded video contained directly within the Cluster element. The size of the Cluster element can be inferred within the modified Cue element by using the CueClusterPosition attribute of the CueTrackPosition element that indexes the first byte of the next Cluster element. Alternatively, an additional CueTrackPosition element that indexes the last byte of the Cluster element (in addition to the CueTrackPosition element that indexes the first byte of the Cluster element), and / or a non-standard CueClusterSize attribute that specifies the size of the Cluster element pointed to by the CueClusterPosition attribute can be added to the modified Cue element according to embodiments of the present invention and included within each CueTrackPosition element to assist in the reading of a particular Cluster element within the Matroska container file via an HTTP byte range request or a similar protocol.

[0064] The modification of the Cue elements as described above significantly simplifies the reading of Cluster elements from a Matroska container file via the HTTP or a similar protocol during adaptive bitrate streaming. Additionally, the size of the index is significantly reduced by only indexing the first frame within each cluster. Assuming that the index is generally downloaded prior to playback, the reduction in the size of the Cue element (i.e., the index) means that playback can start more quickly. Using the CueClusterPosition element, the playback device can request a specific Cluster element from the stream most suitable for the streaming state experienced by the playback device by simply referring to the index of the associated Matroska container file using the Timecode attribute for the desired Cluster element.

[0065] In some embodiments, some attributes within the Cue element are not utilized during adaptive bitrate streaming. Thus, the Cue element can be further modified by removing the unused attributes and reducing the overall size of the index for each Matroska container file. A modified Cue element that can be utilized within a Matroska container file, according to an embodiment of the present invention, including a single encoded stream, is illustrated in FIG. 5a. The Cue element 52’ shown in FIG. 5a is similar to the Cue element 52 shown in FIG. 5, except that the CuePoint element 70’ does not include the CueTime attribute (see 72 in FIG. 5) and / or the CueTrackPosition element 74’ does not include the CueTrack attribute (76 in FIG. 5). When the portions of the encoded media within each Cluster element within the Motroska container file have the same duration, the CueTime attribute is not necessary. When Matroska contains a file including a single encoded stream, the CueTrack attribute is not necessary. In other embodiments, the Cue element and / or other elements of the Matroska container file can be modified to remove elements and / or attributes that are not necessary for adaptive bitrate streaming of the encoded stream contained within the Matroska container file, assuming the manner in which the stream is encoded and inserted into the Matroska container file.

[0066] Contains information about the size of each Cluster element within a Matroska container file, and various modifications to the Cue elements to eliminate unnecessary attributes are described above. However, many embodiments of the present invention utilize a conventional Matroska container. In some embodiments, the playback device simply uses the information obtained from the conventional Cues element to determine, on the fly, the size of the Cluster element and / or relies on a separate index file containing information about the size and / or location of the Cluster elements within the MKV container file. In some embodiments, additional index information is stored within the top-level index file. In some embodiments, the additional index information is stored within a separate file identified within the top-level index file. When the index information used to read the Cluster elements from the Matroska container file is stored separately from the container file, the Matroska container file is still generally constrained to encode the media for inclusion within the Cluster elements as described above. Additionally, regardless of where the index information is located, the index information will generally index each Cluster element and include information about at least the start location of each Cluster element and, in many cases, its size (but not limited thereto).

[0067] (Encoding of Source Media for Adaptive Bitrate Streaming) A process for encoding source media as a top-level index file according to an embodiment of the present invention, and a plurality of Matroska container files for use in an adaptive bitrate streaming system, are illustrated in FIG. 7. Encoding process 100 begins with a step (102) of selecting a first portion of the source media and a step (104) of encoding the source media using encoding parameters for each stream. When the portion of the media is video, the portion of the source video is encoded as a single GOP starting with an IDR frame. In many embodiments, the encoding parameters used to create alternative GOPs vary based on bitrate, frame rate, encoding parameters, and resolution. Thus, the portion of the media is encoded as a set of interchangeable alternatives, and the playback device can select the alternative most appropriate for the streaming state experienced by the playback device. When accommodating different resolutions, the encoding of the stream is constrained such that each stream has the same display aspect ratio. A constant display aspect ratio can be achieved across different resolution streams by varying the sample aspect ratio with the resolution of the stream. In many cases, a lower resolution can result in a higher quality video compared to a higher resolution video encoded at the same bitrate. In many embodiments, the source media itself is encoded, and encoding process (104) includes transcoding or translating the encoded source media according to the respective encoding parameters of the alternative streams accommodated by the adaptive bitrate streaming system.

[0068] When the source media is encoded as a set of alternative portions of the encoded media, each of the alternative portions of the encoded media is inserted into a Cluster element within a Matroska container file corresponding to the stream to which the portion of the encoded media belongs (106). In many embodiments, the encoding process also constructs an index for each Matroska container file as the media is inserted into the Cluster elements within the container. Thus, process 100 can also include the step of creating a CuePoint element that points to the Cluster element inserted into the Matroska container file. The CuePoint element can be held in a buffer until the source media is fully encoded. The foregoing process describes encoding each of the alternative portions of the encoded media in succession through the source media in a single pass, although many embodiments of the present invention involve making separate passes through the source media and encoding each of the alternative streams.

[0069] Referring back to FIG. 7, the process continues with the selection (102) and encoding (104) of portions of the source media until the entire source media is encoded (108) for adaptive bitrate streaming, and then inserts (106) the encoded portions of the media into a Matroska container file corresponding to the appropriate stream. At that point, the process can insert (110) an index into the Matroska container for each stream and create (112) a top-level index file that indexes each of the encoded streams contained within the Matroska container file. As described above, the index is created as encoded media and the CuePoint elements can be inserted into the Matroska container file such that each Cluster element within the Mastroska container file is indexed. In response to completion of the encoding, each CuePoint element can be contained within a Cue element, and the Cue element can be inserted into the Matroska container file following the Cluster element.

[0070] Following the encoding of the source media, after generating a Matroska container file containing each of the streams generated during an encoding process that can include the generation of trick play streams, and a top-level index file that indexes each of the streams within the Matroska container file, the top-level index file and the Matroska container file can be uploaded to an HTTP server for adaptive bitrate streaming to a playback device. Adaptive bitrate streaming of media encoded in accordance with embodiments of the present invention using HTTP requests is further discussed below.

[0071] (Adaptive Bitrate Streaming from MKV Container Files Using HTTP) When the source media is encoded such that there is an alternative stream contained within a separate Matroska container file for at least one of video, audio, and subtitle content, adaptive streaming of the media contained within the Matroska container file can be achieved using an HTTP request or a similar stateless data transfer protocol. In many embodiments, the playback device requests a top-level index file resident on the server and uses the index information to identify the streams available to the playback device. The playback device can then read the index for one or more of the Matroska files and use the index to request media from one or more of the streams contained within the Matroska container file using an HTTP request or a similar stateless protocol. As described above, many embodiments of the present invention implement an index for each of the Matroska container files using a modified Cue element. However, in some embodiments, the encoded media for each stream is contained within a standard Matroska container file and a separate index file can also be provided for each of the container files. Based on the streaming state experienced by the playback device, the playback device can select media from alternative streams encoded at different bitrates. When the media from each of the streams is inserted into the Matroska container file as described above, transitions between streams can occur in response to the completion of the playback of the media within a Cluster element. Thus, the size of the Cluster element (i.e., the duration of the encoded media within the Cluster element) is generally selected such that the playback device can respond sufficiently quickly to changes in the streaming state and commands from the user involving the use of trick play tracks.The smaller the Cluster elements (i.e., the shorter the duration of the encoded media within each Cluster element), the higher the overhead associated with the requirements of each Cluster element. Thus, there is a trade-off between the responsiveness of the playback device to changes in the streaming state and the effective data rate of the adaptive streaming system for a given set of streaming states (i.e., the portion of the available bandwidth that is actually utilized to transmit the encoded media). In many embodiments, the size of the Cluster elements is selected such that each Cluster element contains 2 seconds of encoded media. In other embodiments, the duration of the encoded media can be greater than or less than 2 seconds and / or the duration of the encoded media can vary from Cluster element to Cluster element.

[0072] Communication between a playback device or client and an HTTP server during playback of media encoded in a separate stream contained within a Matroska container file indexed by a top-level index file, according to an embodiment of the present invention, is illustrated in FIG. 8. In the illustrated embodiment, the playback device 200 begins playback by requesting the top-level index file from the server 202 using an HTTP request or similar protocol to receive data. The server 202 provides the bytes corresponding to the request. The playback device 200 then parses the top-level index file and identifies the respective URIs of the Matroska container files containing the streams of encoded media derived from specific fragments of the source media. The playback device can then request the byte ranges corresponding to one or more headers of the Matroska container files via HTTP or a similar protocol, the byte ranges being determined using the information contained within the URI for the associated Matroska container file (see discussion above). The server responds to the request for the byte range containing the header of the Matroska container file,

[0073]

Number

[0074] EBML elements are generally processed by the playback device to ensure compatibility with the file version. The SeekHead element is parsed to find the location of the Matroska index element, and the SegmentInfo element contains two important elements used in playback, namely, TimecodeScale and Duration. TimecodeScale specifies the timecode scale for all timecodes within a Segment of the Matroska container file, and Duration specifies the duration of the Segment based on the TimecodeScale. The Tracks element contains information used by the playback device to decode the encoded media contained within the Clusters element of the Matroska file. As described above, the adaptive bitrate streaming system according to an embodiment of the present invention can handle different encoded streams encoded using different encoding parameters including, but not limited to, frame rate and resolution. Therefore, the playback device can configure the decoder each time a transition is made between the encoded streams using the information contained within the header of the Matroska container file.

[0075] In many embodiments, the playback device does not receive headers for all of the Matroska container files indexed within the top-level index file. Instead, the playback device first determines the streams that will be utilized to begin playback and requests headers from the corresponding Matroska container files. Depending on the structure of the URIs contained within the top-level index file, the playback device can request byte ranges from a server containing at least portions of the index from the relevant Matroska container files, using either information from the URIs or information from the headers of the Matroska container files. The byte range can correspond to the entire index. The server provides the relevant byte range containing the index information to the playback device, and the playback device can use the index information to request the byte ranges of Cluster elements containing the media encoded using this information Once the Cluster elements are received, the playback device can extract the encoded media from the Block elements within the Cluster elements and decode and playback the media within the Block elements according to their associated Timecode attributes.

[0076] In the illustrated embodiment, the playback device 200 requests sufficient index information from the HTTP server prior to the start of playback such that the playback device can stream the entirety of each of the selected streams using the index information. In other embodiments, the playback device continuously reads index information as the media is being played back. In some embodiments, all of the index information for the lowest bitrate stream is requested prior to playback so that the index information for the lowest bitrate stream is available to the playback device in case the streaming state suddenly degrades during playback.

[0077] (Switching between streams) The communication illustrated in FIG. 8 assumes that the playback device continues to request media from the same stream (i.e., a Matroska container file) throughout the playback of the media. In fact, the streaming state experienced by the playback device is likely to change during the playback of the streaming media, and the playback device can request media from an alternative stream (i.e., a different Matroska container file) to provide the best picture quality for the streaming state experienced by the playback device. In addition, the playback device may switch the stream to perform a trick play function that utilizes trick play track streams.

[0078] When a playback device switches to a new stream according to an embodiment of the present invention, the communication between the playback device and the server is illustrated in FIG. 9a. The communication illustrated in FIG. 9a assumes that the download of Cluster elements from the old stream proceeds while index information for the new stream has not been previously requested by the playback device and the information is being acquired regarding the Matroska container file containing the new stream. When the playback device 200 detects a change in the streaming state and determines that a higher bitrate stream can be utilized in this streaming state or receives a trick play command from the user, the playback device can use the top-level index file to identify a URI for an alternative stream that is more suitable for at least one of the video, audio, or subtitle streams that the playback device is currently requesting encoded media for. The playback device can save information regarding the current stream and use the parameters of the corresponding URI to request the byte range of the header for the Matroska container file containing the new stream. Caching the information in this way can be beneficial when the playback device attempts to adapt the bitrate of the stream downward. When the playback device experiences a reduction in available bandwidth, the playback device will ideally quickly switch to a lower bitrate stream. Due to the bandwidth reduction experienced by the playback device, the playback device is unlikely to have additional bandwidth for requesting header and index information. Ideally, the playback device utilizes all available bandwidth to download the previously requested higher rate Cluster elements and uses the locally cached index information to initiate a request for Cluster elements from the Matroska container file containing the lower bitrate stream.

[0079] The byte ranges for index information for a Matroska container file containing a new stream can be requested from the HTTP server 202, as described above in connection with FIG. 8. At that point, the playback device can stop downloading Cluster elements from the previous stream and use the index information from the Matroska container file to initiate a request for the byte ranges of the appropriate Cluster elements from the Matroska container file containing the new stream from the HTTP server, and can identify the Cluster elements containing the encoded media that follows the encoded media within the last Cluster element read by the playback device. As described above, a smooth transition from one stream to another is facilitated by encoding each of the alternative streams such that the corresponding Cluster elements start with the same Timecode element and IDR frame.

[0080] When a playback device caches the entire headers and indexes for each stream utilized in the playback of media, the process of switching back to a previously used stream can be simplified. The playback device already has the header and index information for Matroska files containing previously utilized streams, and the playback device can simply use this information to initiate a request for Cluster elements from the Matroska container file of the previously utilized stream via HTTP. The communication between the playback device and the HTTP server when the playback device switches back to a stream for which it has cached the header and index information, according to an embodiment of the present invention, is illustrated in FIG. 9b. The process illustrated in FIG. 9b is preferably done when adapting the bitrate downward because the reduction in available resources can be exacerbated by the need to download index information in addition to the media. The likelihood of playback interruption is reduced by increasing the speed at which the playback device switches between streams and reducing the amount of overhead data downloaded to achieve the switch.

[0081] The invention has been described in a certain specific aspect, but many additional modifications and variations will be apparent to those skilled in the art. Accordingly, the invention may be practiced otherwise than as specifically described, including various changes in implementation that utilize encoders and decoders corresponding to features other than those specified within the particular standards to which they conform, without departing from the scope and spirit of the invention, for example. Therefore, the embodiments of the invention should be regarded as illustrative rather than restrictive in every aspect.

Claims

1. A method for performing adaptive bitrate streaming using a playback device, the playback device selecting among a plurality of different streams of encoded media during playback according to the streaming state experienced by the playback device, the method comprising: Using a playback device to obtain a top-level index file for a fragment of media content including an audio sequence and a video sequence via a network, the top-level index file comprising: Identifying alternative streams of encoded video available to the playback device, each of the identified alternative streams of encoded video being an encoding of the video sequence, each of the identified alternative streams of encoded video being encoded as a plurality of parts of the encoded video, the identified alternative streams of encoded video being encoded at a plurality of different bitrates; Identifying at least one stream of encoded audio, each of the identified at least one stream of encoded audio being an encoding of the audio sequence, each of the identified at least one stream of encoded audio being encoded as a plurality of parts of the encoded video; Identifying a separate index file containing information about the location of a container file, the container file containing the identified alternative streams of encoded video and the identified at least one stream of encoded audio; and performing; Using the playback device to select a first stream of encoded video to be utilized to start playback from the identified alternative streams of encoded video; Using the playback device, obtaining, via the network, a first index file from the separate index files identified in the top-level index file, wherein the first index file is an index file for a first stream of the encoded video, and the first index file contains information regarding the location of at least one container file containing the first stream of the encoded video, Using the playback device, obtaining, via the network, a header for a first stream of the encoded video by utilizing information from the first index file, Configuring a video decoder within the playback device by using information contained within the header for a first stream of the encoded video, Using the playback device to select a first stream of encoded audio to be utilized to start playback from at least one identified stream of the encoded audio, Using the playback device, obtaining, via the network, an audio index file from the separate index files identified in the top-level index file, wherein the audio index file is an index file for a first stream of the encoded audio, and the audio index file contains information regarding the location of at least one container file containing the first stream of the encoded audio, Using the playback device, obtaining, via the network, a header for a first stream of the encoded audio by utilizing information from the audio index file, Configuring an audio decoder within the playback device by using information contained within the header for a first stream of the encoded audio, Using the playback device, via the network, obtaining at least a first portion of the encoded video from a first stream of the encoded video using information from the first index file; Using the playback device, via the network, obtaining at least a first portion of the encoded audio from a first stream of the encoded audio using information from the audio index file; Starting playback of the media content, the starting comprising: Using the video decoder in the playback device to decode the encoded video within at least the first portion of the encoded video from the first stream of the encoded video; Using the audio decoder in the playback device to decode the encoded audio within at least the first portion of the encoded audio from the first stream of the encoded audio; being performed by; As the playback device obtains additional portions of the encoded media, evaluating, in the playback device, the current streaming state experienced by the playback device; Using the playback device to select a second stream of encoded video from an identified alternative stream of the encoded video based on the current streaming state experienced by the playback device; Storing in the memory of the playback device the first index file and the information contained in the header for the first stream of the encoded video used to configure the video decoder; Using the playback device, via the network, obtaining a second index file from the separate index files identified in the top-level index file, wherein the second index file is an index file for a second stream of the encoded video, and the second index file contains locations of at least one container file containing a plurality of portions of the encoded video of the second stream of the encoded video, Using the playback device, via the network, obtaining a header for a second stream of the encoded video by utilizing information from the second index file, Configuring a video decoder in the playback device using information contained in the header for the second stream of the encoded video, Using the playback device, via the network, obtaining at least one portion of the encoded video from the second stream of the encoded video by utilizing information from the second index file, Using the playback device, via the network, obtaining at least one additional portion of the encoded audio from the first stream of the encoded audio by utilizing information from the audio index file, Continuing the playback of the media content, wherein the continuing is Decoding the encoded video in at least one portion of the encoded video from the second stream of the encoded video using the video decoder in the playback device, Decoding the encoded audio in at least one additional portion of the encoded audio from the first stream of the encoded audio using the audio decoder in the playback device and being performed by A method comprising

2. The method further comprises evaluating a current streaming state experienced by the playback device as the playback device obtains further additional portions of the encoded media; selecting, using the playback device, a first stream of the encoded video from an identified alternative stream of the encoded video based on the current streaming state experienced by the playback device; reading, from the memory of the playback device, the first index file and the information contained in the header for the first stream of the encoded video used to configure the video decoder; configuring a video decoder within the playback device using the information contained in the header for the first stream of the encoded video read from the memory of the playback device; using the playback device to obtain at least one portion of the encoded video from the first stream of the encoded video using information from the first index file read from the memory of the playback device via the network; using the playback device to obtain at least one further portion of the encoded audio from the first stream of the encoded audio using information from the audio index file via the network; continuing playback of the media content, the continuing comprising decoding, using the video decoder within the playback device, the encoded video within at least one portion of the encoded video from the first stream of the encoded video; Decoding the encoded audio in at least one further portion of the encoded audio from the first stream of the encoded audio using the audio decoder within the playback device as performed by The method according to claim 1, further comprising

3. Storing the information contained in the header for the first stream of the encoded video used to configure the video decoder includes storing the header from the first stream of the encoded video. The method according to claim 1.

4. Evaluating the current streaming state includes evaluating at least one streaming state selected from the group consisting of buffer occupancy, network bandwidth, and playback device processor capacity. The method according to claim 1.

5. Prior to selecting the second stream based on the current streaming state, the playback device obtains the second index file and the header for the second stream of the encoded video. The method according to claim 1.

6. Prior to starting playback, the playback device reads the second index file and the header for the second stream of the encoded video. The method according to claim 5.

7. The first index file contains information regarding the location of the container file containing the first stream of the encoded video, and information regarding the location of the portion of the video within the container file and contains At least a first portion of the encoded video from the first stream of the encoded video is obtained by the playback device, via the network, using a byte range request generated using the information regarding the location of the portion of the video within the container file, according to the method of claim 1.

8. The information regarding the location of the portion of the video within the container file includes information for each portion of the encoded video selected from the group consisting of a start location and a size, according to the method of claim 7.

9. The first index file contains information regarding the locations of a plurality of container files, each of the plurality of container files containing a portion of the encoded video from the first stream of the encoded video, At least a first portion of the encoded video from the first stream of the encoded video is obtained by the playback device, via the network, by requesting one of the plurality of container files using the information regarding the locations of the plurality of container files, according to the method of claim 1.

10. Each portion of the encoded video from at least one portion of the encoded video from the second stream of the encoded video comprises a closed picture group (GOP) starting with an instantaneous decoding refresh (IDR) frame, according to the method of claim 1.

11. The top-level index file contains an extensible markup language (XML) including a list of uniform resource identifiers (URIs) describing media streams including the identified alternative streams of the encoded video, The method further includes parsing the top-level index file using the playback device to identify the URIs describing each of the identified alternative streams of the encoded video, according to the method of claim 1.

12. The method according to claim 11, wherein the list of URIs includes at least one URI describing an audio stream.

13. The method according to claim 12, wherein the list of URIs includes at least one URI describing a subtitle stream.

14. The method according to claim 11, wherein the list of URIs includes at least one URI describing a trick play stream.

15. The method according to claim 11, wherein the top-level index file is obtained by the playback device via the network using an HTTP request.

16. The second stream of the encoded video is a portion of the encoded video already requested from a server system by the playback device, and a portion of the encoded video buffered by the playback device and is selected based on at least one factor selected from the group consisting of. The method according to claim 1.

17. Each of the at least one container file containing the first stream of the encoded video is an Extensible Binary Markup Language (EBML) file. The method according to claim 1.

18. Each of the identified alternative streams of the encoded video has a different maximum bitrate. The method according to claim 1.

19. A plurality of the identified alternative streams of the encoded video have different resolutions. The method according to claim 1.

20. The method according to claim 1, wherein a plurality of the identified alternative streams of the encoded video have pixels with different sample aspect ratios.

21. A method of performing adaptive bitrate streaming using a playback device, the playback device selecting among a plurality of different streams of encoded media during playback according to the streaming state experienced by the playback device, the method comprising: Obtaining, using the playback device, via a network, a top-level index file for a fragment of media content including an audio sequence and a video sequence, the top-level index file comprising: Identifying alternative streams of the encoded video available to the playback device, each of the identified alternative streams of the encoded video being an encoding of the video sequence, each of the identified alternative streams of the encoded video being encoded as a plurality of parts of the encoded video, the identified alternative streams of the encoded video being encoded at a plurality of different bitrates; Identifying at least one stream of the encoded audio, each of the identified at least one stream of the encoded audio being an encoding of the audio sequence, each of the identified at least one stream of the encoded audio being encoded as a plurality of parts of the encoded video; Identifying a separate index file containing information about the location of the container file, the container file containing the identified alternative streams of the encoded video and the identified at least one stream of the encoded audio; and performing; To start playback from the identified alternative stream of the encoded video, use the playback device to select a first stream of the encoded video to be utilized; Use the playback device to obtain, via the network, a first index file from the separate index files identified in the top-level index file, where the first index file is an index file for the first stream of the encoded video, and the first index file contains information regarding the location of at least one container file containing the first stream of the encoded video; Use the playback device to obtain, via the network, a header for the first stream of the encoded video using the information from the first index file; Use the information contained in the header for the first stream of the encoded video to configure a video decoder within the playback device; To start playback from the identified at least one stream of the encoded audio, use the playback device to select a first stream of the encoded audio to be utilized; Use the playback device to obtain, via the network, an audio index file from the separate index files identified in the top-level index file, where the audio index file is an index file for the first stream of the encoded audio, and the audio index file contains information regarding the location of at least one container file containing the first stream of the encoded audio; Use the playback device to obtain, via the network, a header for the first stream of the encoded audio using the information from the audio index file; Configuring an audio decoder in the playback device using the information contained in the header for the first stream of the encoded audio; Using the playback device to obtain at least a first portion of the encoded video from the first stream of the encoded video via the network, utilizing information from the first index file; Using the playback device to obtain at least a first portion of the encoded audio from the first stream of the encoded audio via the network, utilizing information from the audio index file; Starting playback of the media content, wherein starting comprises: Decoding the encoded video within at least the first portion of the encoded video from the first stream of the encoded video using the video decoder in the playback device; Decoding the encoded audio within at least the first portion of the encoded audio from the first stream of the encoded audio using the audio decoder in the playback device; which is performed by: As the playback device obtains additional portions of the encoded media, evaluating, in the playback device, the current streaming state experienced by the playback device; Using the playback device to select a second stream of encoded video from an identified alternative stream of the encoded video based on the current streaming state experienced by the playback device; Storing in the memory of the playback device the first index file and the information contained in the header for the first stream of the encoded video used to configure the video decoder; Using the playback device, via the network, to obtain a second index file from the separate index files identified in the top-level index file, wherein the second index file is an index file for a second stream of the encoded video, and the second index file contains locations of at least one container file containing a plurality of portions of the encoded video of the second stream of the encoded video, and Using the playback device, via the network, to obtain a header for a second stream of the encoded video by utilizing information from the second index file, and Using the information contained in the header for the second stream of the encoded video to configure a video decoder within the playback device, and Using the playback device, via the network, to obtain at least one portion of the encoded video from the second stream of the encoded video by utilizing information from the second index file, and Using the playback device, via the network, to obtain at least one additional portion of the encoded audio from the first stream of the encoded audio by utilizing information from the audio index file, and Continuing the playback of the media content, and the continuing is Using the video decoder within the playback device to decode the encoded video within at least one portion of the encoded video from the second stream of the encoded video, and Using the audio decoder within the playback device to decode the encoded audio within at least one additional portion of the encoded audio from the first stream of the encoded audio is performed by As the playback device obtains additional portions of the encoded media, evaluating the current streaming state experienced by the playback device; Using the playback device to select a first stream of the encoded video from the identified alternative streams of the encoded video based on the current streaming state experienced by the playback device; Reading from the memory of the playback device the first index file and the information contained in the header for the first stream of the encoded video used to configure the video decoder; Using the information contained in the header for the first stream of the encoded video read from the memory of the playback device to configure a video decoder within the playback device; Using the playback device to obtain at least one portion of the encoded video from the first stream of the encoded video using information from the first index file read from the memory of the playback device via the network; Using the playback device to obtain at least one further portion of the encoded audio from the first stream of the encoded audio using information from the audio index file via the network; Continuing the playback of the media content, the continuing comprising: Using the video decoder within the playback device to decode the encoded video within at least one portion of the encoded video from the first stream of the encoded video; Using the audio decoder within the playback device, decoding the encoded audio within at least one further portion of the encoded audio from the first stream of the encoded audio as performed by A method comprising.

Images