System and method for providing optimized time scales and accurate presentation timestamps

Abstract

The computer-implemented method of the present disclosure includes identifying, for each of a plurality of different media items, at least two of the plurality of media items encoded at different frame rates, a current timescale used in encoding the media item for distribution. The method also includes identifying, for the plurality of media items, a unified timescale that provides a constant frame spacing for each of the plurality of media items. The method also includes modifying at least one of the plurality of media items from the current timescale to the identified unified timescale, thereby providing a constant frame spacing for the modified media item. Various other methods, systems, and computer-readable media are disclosed in the present disclosure.

Description

cross reference

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 17 / 359,468, filed June 25, 2021, entitled "SYSTEMS AND METHODS FOR PROVIDING OPTIMIZED TIME SCALES AND ACCURATE PRESENTATION TIME STAMPS," the entire disclosure of which is incorporated herein by reference. [Background technology]

[0002] Media items, including audio and video media items, are typically produced with a presentation time stamp (PTS). A presentation time stamp (PTS) is a metadata field applied to a media item to keep the various elementary streams (e.g., audio, video, subtitles, etc.) of the media item synchronized. This metadata indicates specific points in time defined by a reference clock that is also transmitted as part of the media item. These points in time define the points to which the underlying elementary streams are synchronized. The presentation time stamp itself is an N-bit number representing the value of a counter driven by a clock with a time scale of X Hertz (Hz). For example, when encoding in the Moving Pictures Expert Group (MPEG) 1 or MPEG 2 formats, the N value of the presentation time stamp is 33 bits, and the X value of the time scale is 90,000 Hz. Summary of the Invention [Problem to be solved by the invention]

[0003] However, even when presentation timestamps are assigned, seamless synchronization can sometimes be hindered due to the approximate duration of presentation of various audio and video frames during playback and the fluctuations in such duration. [Means for solving the problem]

[0004] As explained in more detail below, this disclosure describes methods and systems for providing optimized time scales and accurate presentation timestamps for media items.

[0005] In one example, a computer-implemented method for providing optimized timescales and accurate presentation timestamps includes, for each of a plurality of different media items, at least two of the plurality of media items being encoded at different frame rates, identifying a current timescale used when encoding the media item for distribution. The method also includes, for the plurality of media items, identifying a unified timescale that provides a constant frame spacing for each of the plurality of media items. The method also includes changing at least one of the plurality of media items from the current timescale to the identified unified timescale, thereby providing a constant frame spacing for the changed media item.

[0006] In some embodiments, the media items are video media items. In some cases, the video media items are encoded at a frame rate of 23.97 frames / second, 24 frames / second, 25 frames / second, 29.97 frames / second, 30 frames / second, 59.94 frames / second, 60 frames / second, 120 frames / second, 240 frames / second, or 300 frames / second.

[0007] In other embodiments, the media items are audio media items. In some examples, audio media items have a frame size of 1024 samples, 1536 samples, or 2048 samples per frame.

[0008] In some cases, each media item in a specified group of media items among the plurality of media items has a specified video frame rate and audio frame rate, and in such cases, the unified time scale is calculated to optimize the specified video frame rate and specified audio frame rate of the media items in the group of media items.

[0009] In some embodiments, a unified time scale is implemented to generate one or more presentation timestamps for the multiple media items. In some cases, the presentation timestamps are monotonically increasing and the units used for the unified time scale are selected to maximize the wraparound time of the multiple media items.

[0010] In some cases, the unit selected to maximize the wraparound time of the multiple media items is selected based on the video frame rate. In some examples, the identified unitary time scale includes a presentation timestamp interval. In such cases, the presentation timestamp interval may be configured as a minimum frame interval or a multiple of the minimum frame interval. In some embodiments, the method further includes restoring the presentation timestamp interval to a specified resolution. In some cases, the identified unitary time scale enables streaming of the multiple media items at variable frame rates while maintaining constant frame spacing for each frame rate. In some embodiments, media items with different frame rates are streamed at variable frame rates while maintaining constant frame spacing for each frame rate using the unitary time scale. In some examples, media items having video content captured using variable refresh rates are streamed at variable frame rates while maintaining constant frame spacing using the unitary time scale.

[0011] In some embodiments, identifying the unitary time scale includes converting one or more input presentation timestamps from a plurality of different media items having different time scales into presentation timestamps based on the unitary time scale. In some cases, implementing converted presentation timestamps derived from the input presentation timestamps avoids wraparound of the presentation timestamp counter. In some examples, converting at least one of the plurality of media items from a current time scale to the identified unitary time scale enables implementing a single fixed vertical synchronization interrupt during playback of the plurality of media items. In some cases, the method further includes optimizing the presentation timestamps of the plurality of media items such that the scaled presentation timestamps match native presentation timestamps without rounding errors.

[0012] Additionally, a corresponding system includes at least one physical processor and a physical memory containing computer-executable instructions that, when executed by the physical processor, cause the physical processor to perform the following steps: for each of a plurality of different media items, at least two of the plurality of media items being encoded at different frame rates, determine a current timescale that was used when encoding the media item for distribution; for the plurality of media items, identify a unified timescale that maintains a constant frame spacing for each of the plurality of media items; and change at least one of the plurality of media items from the current timescale to the identified unified timescale, thereby maintaining a constant frame spacing for the at least one media item after the change.

[0013] In some examples, the above-described methods may also be encoded as computer-readable instructions on a computer-readable medium. For example, the computer-readable medium may include one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to perform the following steps: for each of a plurality of different media items, at least two of the plurality of media items being encoded at different frame rates, determine a current timescale used when encoding the media item for distribution; for the plurality of media items, identify a unified timescale that maintains a constant frame spacing for each of the plurality of media items; and scale at least one of the plurality of media items from the current timescale to the identified unified timescale, thereby maintaining a constant frame spacing for the at least one media item after the scale change.

[0014] The features of the embodiments described herein may be used in combination with one another, in accordance with the general principles described herein, regardless of the features of any embodiment. These and other embodiments, features, and advantages will be better understood from the following detailed description, taken in conjunction with the accompanying drawings and claims.

[0015] The accompanying drawings show, by way of example, numerous exemplary embodiments and are a part of this specification, and together with the following description, demonstrate and explain various principles of the present disclosure. [Brief explanation of the drawings]

[0016] [Figure 1] Diagram showing a computing environment with optimized time scales and accurate presentation timestamps [Figure 2]Flow diagram illustrating an exemplary method for providing optimized time scales and accurate presentation timestamps for media items. [Figure 3] Flow diagram showing time scale information for media items in various layers [Figure 4] A table showing different frame intervals for media items encoded at various frame rates. [Figure 5] A table clearly showing how rounding errors in the presentation timestamps can change the length of frame intervals. [Figure 6] Table showing how rounded and unrounded presentation timestamps relate to vertical sync [Figure 7] A table showing video frame spacing for media items encoded at different frame rates [Figure 8] A table showing audio frame spacing for media items encoded at different frame rates [Figure 9] Another table showing audio frame spacing for media items encoded at different frame rates [Figure 10] Table showing audio / video frame intervals for sampling rates up to 96kHz [Figure 11] A table showing the scaled frame rates for video coded at 23.976fps. [Figure 12] Table showing example time scales for 48kHz audio [Figure 13] Table showing example time scales for 44.1kHz audio [Figure 14] Table showing example time scales and wraparound times for 48kHz audio [Figure 15] Table showing example scaling of presentation timestamps [Figure 16] FIG. 1 illustrates an embodiment in which minimum spacing is calculated and implemented. [Figure 17]A table showing optimized submission timestamps with an initial value of zero. [Figure 18] FIG. 1 illustrates an embodiment in which the frame rate varies between different video segments. [Figure 19] Block diagram illustrating an exemplary content distribution ecosystem [Figure 20] A block diagram illustrating an exemplary distribution infrastructure within the content distribution ecosystem shown in FIG. [Figure 21] A block diagram illustrating an exemplary content player within the content distribution ecosystem shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0017] Throughout the drawings, like reference symbols and descriptions indicate similar, but not necessarily identical, elements. While the exemplary embodiments described herein are susceptible to various modifications and variations, specific embodiments thereof have been shown by way of example in the drawings and are herein described in detail. However, the exemplary embodiments described herein are not intended to be limited to the particular forms disclosed. Rather, the present disclosure covers all modifications, equivalents, and alternatives falling within the scope of the appended claims.

[0018] The present disclosure is generally directed to providing optimized timescales and accurate presentation timestamps for media items. As described in more detail below, embodiments of the present disclosure generate a unified timescale that maintains constant frame spacing during playback of media items, even if the media items are encoded at different frame rates.

[0019] As mentioned above, media items are typically encoded with presentation timestamps (PTS), which indicate where different elementary streams can be synchronized. Such elementary streams include audio streams, video streams, and subtitles. If these elementary streams lead or lag each other during playback of a media item, the out-of-sync state is often noticeable to the user, potentially resulting in a poor playback experience.

[0020] The presentation timestamp itself is an N-bit number representing the value of a counter driven by a clock with a particular time scale (X Hz) measured in hertz (Hz). For MPEG1 and MPEG2 formats, N = 33 bits and X = 90,000 Hz. Different time scales are used for media items encoded at different frame rates. For example, video can be encoded at frame rates of 24 frames per second (fps), 29.97 fps, 30 fps, 50 fps, 59.94 fps, 60 fps, etc. The amount of division indicating the display time of each frame varies depending on the time scale. For example, if the frame rate is 24 fps and the time scale is 1000 divisions per second, the display time of each frame (i.e., the "frame interval") is 1000 / 24 ​​= 41.66667 milliseconds. Therefore, in this example, the frame interval for video encoded at 24 fps would be 41 or 42 milliseconds. If the time scale is changed to, say, 90,000 Hz, the frame interval becomes (90,000 / 24) divisions, or 3,750 divisions, which is more accurate. If the time scale is increased to, say, 1,000,000 divisions / second, the frame interval becomes 41,666 or 41,667 microseconds (1,000,000 / 24).

[0021] Because presentation timestamp values ​​are presented as integers, fractional values ​​(e.g., 41.66667 milliseconds, 41666.6667 microseconds, etc.) must be rounded up or down to the nearest integer. This rounding to the nearest integer results in rounding errors in the presentation timestamps. These rounding errors then cause the frame interval (the display time of each frame) to fluctuate up or down. For example, a video encoded at 24 fps with a time scale of 1000 units would have a frame interval of 41.66667 milliseconds, but because 41.66667 is rounded up or down to maintain synchronization, the actual frame interval will frequently switch between 41 milliseconds and 42 milliseconds. However, frequent switching between different frame intervals can have adverse effects on the playback of a media item. For example, switching between frame intervals can cause eye strain to the user. Furthermore, in at least some cases, switching between different frame intervals in this manner can result in other problems, such as dropped frames, frozen frames, duplicate frames, blank frames, as well as other playback or synchronization issues that are noticeable to the user and that detract from the user's viewing experience.

[0022] In contrast, the embodiments described herein are designed to provide a universal time scale that applies to a variety of different frame rates. The universal time scale ensures that each frame of a media item is played at a consistent frame interval. The system described herein identifies the current time scale used to encode a set of media items (e.g., audio or video media items). The system also identifies the frame rates used to encode these media items. The system then determines the least common multiple of the various frame rates used by these media items. For example, by identifying the least common multiple of the scaled frame rates of a group of media items, the system can divide each media item into an equal number of divisions. This uniform number of divisions ensures a consistent frame interval for the media items, regardless of the frame rate used to encode the media items. By identifying the least common multiple of the scaled frame rates used by the group of media items, each frame interval can be expressed as an integer value. By forcing the frame interval for each media item to be an integer value, the fluctuations that would occur between different values ​​of frame interval (e.g., the fluctuations between 41 ms and 42 ms in the example above) are eliminated, and problems associated with frame interval fluctuations are also reduced or eliminated entirely because the frame interval is constant due to the unified time scale.

[0023] In some cases, embodiments described herein can also optimize a unified time scale for a particular audio or video format combination. For example, rather than finding the least common multiple for a wide range of different media items encoded at different frame rates, the system herein finds the least common multiple for a selected audio / video format combination and then performs a time scale change specifically for that combination. This reduces the likelihood of overflowing a memory buffer configured to store presentation timestamp values. Indeed, as described above, presentation timestamps are monotonically increasing integer values ​​and therefore will reach very large values ​​at some point. If the bit field for storing the presentation timestamps were too small, the monotonically increasing presentation timestamps could overflow the buffer very early in the playback of the media items. Therefore, optimizing the time scale for a particular audio / video format combination allows for the use of a smaller time scale, which allows for a longer frame interval and therefore fewer increments of the presentation timestamp counter. By optimizing the time scale for each media item or for a specific group of media items, embodiments described herein can provide a smoother and more reliable playback experience for the viewing user that is less susceptible to memory overrun errors.

[0024] Various tables, figures, and flow diagrams are described in detail below with reference to Figures 1-21. Figure 1 illustrates, by way of example, a computing environment 100 in which a unified time scale is identified and applied to media items. Figure 1 includes various electronic components and elements, including a computer system 101. The computer system 101 is used alone or in combination with other computer systems to perform tasks associated with storing digital content. The computer system 101 can be virtually any type of computer system, including a local computer system or a distributed (e.g., cloud) computer system. The computer system 101 includes at least one processor 102 and at least some system memory 103. The computer system 101 includes program modules for performing various different functions. The program modules can be hardware-based, software-based, or can include a combination of hardware and software. Each program module utilizes computing hardware, computing software, or both to perform specific functions, including those described below.

[0025] In some cases, the communications module 104 is configured to communicate with other computer systems. The communications module 104 comprises substantially any wired or wireless communications means capable of receiving, transmitting, or transmitting and receiving data to and from other computer systems. These communications means include, for example, hardware radios, such as a hardware-based receiver 105, a hardware-based transmitter 106, or a combined hardware-based transceiver capable of both receiving and transmitting data. The radios can be types of radios, such as Wi-Fi radios, cellular radios, Bluetooth® radios, or Global Positioning System (GPS) radios. The communications module 104 is configured to interact with types of computing systems, such as databases, mobile computing devices (such as mobile phones or tablets), or embedded computing systems.

[0026] The computer system 101 also includes a verification module 107 configured to determine the current frame rate 123 of various media items 122. In some cases, these media items 122 are selected from various stored media items 121 maintained in a data store 120. The data store 120 may be a local, remote, or distributed (e.g., cloud-based) data store. The media items 122 may include any type of media item, such as audio or video items, that is encoded for playback on a user device (e.g., the device 116 of the user 115). The verification module 107 is configured to access these media items 122 and determine the frame rate 123 with which the media items 122 were encoded. The frame rate defines the number of frames per second that are designed to be presented on the playback device. In some cases, for example, video media items are encoded at frame rates of 23.97 frames per second (fps), 24 fps, 25 fps, 29.97 fps, 30 fps, 59.94 fps, 60 fps, 120 fps, 240 fps, or 300 fps. Audio frame rates may include encoded frame rates of 23.4375 fps, 31.25 fps, etc.

[0027] The identification module 107 also determines the current time scale 108 to use in conjunction with the media items 122. The time scale is measured in divisions per second (units per second). For example, if the time scale is 1000, then dividing this time scale by the frame rate (e.g., 24 fps) (1000 / 24 ​​= 41.6667 milliseconds) determines the frame interval, or the length of time each frame is displayed on a playback device. Many different time scales may be used in conjunction with different types of media items. Therefore, the identification module 107 determines both the encoded frame rate 123 of a media item and the current time scale 108 used by that media item to determine the frame interval ([time scale] / [frame rate]) for each media item.

[0028] Next, the identification module 109 of the computer system 101 identifies or calculates a unified time scale 110 for the media items 122. The unified time scale 110 is applied to media items with different encoding frame rates 123 to ensure consistent frame spacing. As discussed above, continued rounding of non-integer values ​​(e.g., 41.6667 milliseconds) to integers will eventually result in playback errors such as extra frames, dropped frames, and blank frames. These errors are noticeable to the user, cause eye fatigue and strain, and disrupt a smooth, seamless playback experience. The time scale modification module 111 receives the identified unified time scale 110 and modifies one or more of the media items 122, resulting in modified media items 112 that are ready to be played using the unified time scale 110. Thus, when a user (e.g., user 115) selects a media item (e.g., from data store 120) for streaming or downloading via input 117, the selected media item is scaled to a uniform time scale with constant frame spacing. Constant frame spacing allows for smooth and reliable playback, eliminating noticeable artifacts such as blank frames, dropped frames, and extra frames. These embodiments are described in more detail below in the description of method 200 of FIG. 2 and in the description of the embodiments shown in FIGS. 3-21.

[0029] Figure 2 is a flow diagram illustrating an exemplary computer-implemented method 200 for providing optimized time scales and accurate presentation timestamps for media items. Each step illustrated in Figure 2 may be performed by any suitable computer-executable code, a computing system, including the system illustrated in Figure 1, or both. In one example, each step illustrated in Figure 2 represents an algorithm that includes and / or has a structure represented by multiple sub-steps, examples of which are described in more detail below.

[0030] As shown in FIG. 2, at step 210, one or more of the systems described herein may identify, for each of a plurality of different media items (e.g., media items 122 of FIG. 1), the current timescale 108 used when encoding the media item for distribution. At least two of the media items 122 are encoded at different frame rates. Then, at step 220, the systems described herein identify, for the media items 122, a unified timescale 110 that maintains a constant frame spacing for each of the media items 122. Then, at step 230, the system changes at least one of the media items from the current timescale 108 to the identified unified timescale 110, thereby maintaining a constant frame spacing for the changed media item.

[0031] As shown in FIG. 3 and described above, timestamps are added to each video frame and audio frame at regular time intervals to synchronize audio and video playback (e.g., synchronize audio and lip movements). As described above, a timestamp is an N-bit number representing the value of a counter driven by a clock with a time scale of X Hertz (Hz). Note that a decoder (e.g., decoder 303) does not necessarily need to obtain time scale information or presentation timestamps directly from the encoding / streaming container 301. Instead, in at least some cases, such timing information is provided by or specified in the media playback / demuxing device. The playback / demuxer 302 can perform a scaling process to determine a unified time scale, which simplifies implementation and control. The video decoder 303, the audio decoder 304, or both, use time information from the playback / demuxer 302 if available. If this time information is not available, the audio or video decoder uses the time information from the elementary stream. If neither time information is available, the hardware-based decoder attempts to play the media item without the time information. Thus, in at least some cases, the time information of the Playback / Demultiplexer 302 takes highest priority, and the time information of the elementary streams takes a relatively lower priority.

[0032] A presentation timestamp indicates where in time a particular access unit (AU) is located. When a decoder (e.g., decoder 303 or 304) receives a video access unit or an audio access unit, it decodes it and stores it in a memory buffer. When a reference clock reaches the timestamp value, the decoded access unit is rendered on a playback device (e.g., playback device 116 in FIG. 1). The presentation timestamp controls how the access unit is rendered. Video frame rates vary, including 23.976 fps, 24 fps, 29.97 fps, 30 fps, 50 fps, 59.94 fps, and 60 fps. A V-Sync interrupt handler uses the presentation timestamp to determine whether the corresponding audio / video frame should be rendered. Here, the rule for rendering an access unit is to select the closest access unit with a presentation timestamp (PTS) value less than Ti, where Ti is the system time of the vertical synchronization interrupt or the audio serial output interrupt of the presentation device, PTS' = PTS + Δ, and Δ is the tolerance (Δ≧0).<Ti)。PTS’> For Ti, the currently rendering access unit is repeated. Note that for simplicity of explanation, at least some embodiments herein assume that Δ=0.

[0033] The presentation timestamps play a similar role for audio signals that use a serial output interrupt handler. Audio frequencies, in at least some instances, are 32,000 Hz, 48,000 Hz, or 96,000 Hz. Applications primarily focused on high-quality music distribution use the 44.1 kHz range. However, in at least some cases, the video vertical sync interrupt and the audio serial output interrupt may not be aligned during rendering. Therefore, to properly synchronize audio and video, both the video presentation timestamps and the audio presentation timestamps reference a common clock (e.g., system time) with the same time scale. In some implementations, the audio clock may also be used as the system time. This method is sometimes referred to as the audio master A / V sync method.

[0034] The precision of the presentation timestamp is determined by the time scale (e.g., the current time scale 108). The precision of the presentation timestamp can be expressed in the format (number_of_ticks, time_scale) ([number of ticks], [time scale]). The time scale is the total number of ticks per second. The frame interval (i.e., the duration (length of time) each frame is presented on the playback device) is calculated by calculating time_scale / frame_rate ([time scale] / [frame rate]). Table 400 in Figure 4 shows several different frame intervals when media items encoded at different frame rates (402) are presented at different time scales (401). As can be seen, the frame intervals for time scales of 1000, 90000, and 1000000 are decimal values, especially for frame rates such as 23.97 fps and 59.94 fps. However, because the presentation timestamp is expressed as an integer, the actual presentation timestamp will contain rounding errors due to the decimal part. Table 500 in Figure 5 shows the presentation timestamp values ​​for the first 10 frames of a 23.97 fps video, with a time scale value of 1000 (corresponding to 1000 milliseconds). Each frame number 0-9 (column 501) corresponds to a source timestamp (column 502), starting at 0. The source timestamps advance every 41 or 42 milliseconds (as shown in column 503), resulting in a non-uniform frame spacing (column 504) that varies between 42 and 41 milliseconds.

[0035] A problem that can be seen from table 500 is that the presence of rounding errors of up to 0.5 milliseconds can affect the accuracy of frame rate conversion (e.g., frame rendering conditions: Ti > PTS). Table 600 in FIG. 6 shows that, according to the rendering conditions (Ti > PTS), frame #35 (column 602) should be displayed in three vertical synchronization (V-Sync) slots (column 601) between 1460 milliseconds and 1500 milliseconds. However, due to rounding errors, frame #34 is actually displayed three times between 1420 milliseconds and 1460 milliseconds (column 605). Similar irregularities occur when rounding presentation timestamp values ​​to the nearest integer, and these irregularities will occur throughout the playback of the entire media item (see the difference between the unrounded presentation timestamp values ​​(column 603) and the rounded presentation timestamp values ​​(column 604)).

[0036] Table 500 shows how the frame interval varies from 41 milliseconds to 42 milliseconds. This irregular frame interval disrupts the regularity exhibited by the media item's frame rate. It also makes it difficult to implement wake-up hardware or software using a continuous timer to render the media item's frame rate accurately enough to provide smooth and reliable playback. This imprecise time scale for presentation timestamps results in fractional frame intervals, which creates many problems (as discussed above), but using a unified time scale overcomes at least some of these problems.

[0037] The embodiments described herein provide a solution that provides a consistent frame spacing for different video frame rates (e.g., 23.97 fps, 24 fps, 25 fps, 29.97 fps, 30 fps, 50 fps, 59.94 fps, 60 fps, 120 fps, 240 fps, or 300 fps), where [frame spacing] = [time scale] / [frame rate]. In at least some embodiments, this may amount to calculating the least common multiplier (LCM) of all frame rates in the media item set (e.g., LCM(24, 25, 30, 60, 120, 240, 24000, 30000, 60000)). This example calculation yields LCM(24000, 60000) = 120000. In some cases, a lookup table containing pre-calculated LCM values ​​for various frame rate combinations can be prepared and the LCM can be obtained from this lookup table. Thus, as shown in table 700 of FIG. 7 , when the time scale is changed from 1,000 ticks / second, 90,000 ticks / second, or 1,000,000 ticks / second to 120,000 ticks / second (shown in the "Time Scale" row 701), the frame intervals for each frame rate (shown in the "Frame Rate" column) are all integers. For example, a video encoded at 24 fps has a frame interval of 5,000 milliseconds, a video encoded at 59.94 fps has a frame interval of 2,002 milliseconds, and a video encoded at 120 fps has a frame interval of 1,000 milliseconds. Since these frame intervals are all integer values, the frame duration remains constant. By keeping the frame duration constant in this way, anomalies that occur when the frame duration changes during playback, such as blank frames, dropped frames, double frames, playback errors and abnormal playback, are avoided.

[0038] For audio media items, the frame rate of the HE-AAC encoding method (HEAAC) is 2048 samples or 1024 samples per frame, which corresponds to 23.4375 fps and 46.875 fps, respectively, when the audio sampling rate is 48 kHz. Furthermore, Dolby Digital Atmos (DDP / Atomos) encodes at 1536 samples per frame, resulting in encoding at 31.25 fps. Table 800 in Figure 8 shows a table similar to Figure 7, except for audio items. In table 800, when the frame rate (column 802) is 23.4273, the frame intervals are decimal values ​​on the 1000 and 1000000 time scales (row 801). However, using a uniform time scale of 120,000 ticks / second, the frame spacing for a frame rate of 23.4273 fps is 5,120 ticks, and for a frame rate of 31.25 fps the frame spacing is 3,840 ticks. As shown in table 900 of Figure 9, for an audio sampling rate of 32 kHz, the frame spacing is constant at time scales of 1,000, 90,000, and 120,000 for the frame rate (column 902) and time scale (column 901) shown.

[0039] In some embodiments, 44100 Hz audio is used in conjunction with the video access units. In this case, a very large time scale (17640000) is used to accommodate all possible situations. Table 1000 of FIG. 10 shows various sampling rates (column 1001) along with frame sizes (column 1002), frame intervals (column 1003), and various time scales (columns 1004-1008). As can be seen from table 1000, a time scale of 120000 results in a constant frame interval for all of the illustrated video / audio formats except for the 44100 Hz audio. However, if audio is provided at 44100 Hz, then a universal time scale value of 17640000 would produce a constant frame interval for any of the access unit formats shown in table 1000.

[0040] In some embodiments, the system herein focuses on a set of video frequencies and audio sampling rates. In such cases, the minimum time scale capable of generating consistent frame intervals for audio and video can be determined by implementing an algorithm including the following steps (1) to (4): (1) Determine the greatest common divisor (GCD) (D) of ([audio sampling rate], [frame size]). For example, if the sampling rate is 48000 Hz, the frame sizes are 1536 (DDP) and 2048 (HE-AAC), so D = GCD(48000, 1536, 2048) = 128. (2) Determine A = [audio sampling rate] / D. In the above example, the sampling rate is 48000 Hz, so A = 48000 / 128 = 375. (3) Obtain the scaled video frame rate V from table 1100 in FIG. 11. Table 1100 lists various frame rates of media items in column 1101 and scaled frame rates in column 1102. (4) Calculate [Time Scale] = LCM(V, A). For example, if the audio frequency is 48000 Hz, the time scale can be calculated using the frame rate (column 1201) and the scaled frame rate (column 1202) as shown in table 1200 of Figure 12. In this embodiment, one unified time scale is used for each available media item format.

[0041] Because 33 bits are allocated for a 33-bit presentation timestamp (e.g., in MPEG1 or MPEG2 formats), in at least some embodiments, larger time scale values ​​may be more susceptible to wraparound (overflow) into a 33-bit unsigned integer. For example, with a time scale of 90,000 ticks / second, it would take approximately 26 hours for the presentation timestamp to wrap around (i.e., overflow the memory buffer). With a time scale of 120,000 ticks / second, it would take approximately 19 hours for the presentation timestamp to wrap around. However, with a time scale of 17,640,000 ticks / second, it would only take approximately 8 minutes. Thus, in at least some embodiments, it may be preferable to calculate the time scale for each particular video frame rate to be combined with 44.1 kHz audio. For 44.1 kHz audio, A=11,025, as shown in column 1303 of table 1300 of FIG. 13. Table 1300 also shows other frame rates (column 1301), scaled frame rates (column 1302), and time scale values ​​(column 1303).

[0042] The presentation timestamp is represented by a fixed number of integer bits. Therefore, as described above, if the presentation timestamp increases monotonically, wraparound will eventually occur, where the presentation timestamp value overflows from the (e.g., 33-bit) memory buffer for the presentation timestamp. In one embodiment, an optional timing information container is provided by AV1, defined as a 32-bit field. For 48 kHz audio, the above algorithm is implemented to determine the time scale value (column 1203) shown in table 1200 and the corresponding time until wraparound occurs for a 32-bit unsigned value (hereinafter referred to as the wraparound time). For example, when the time scale is 90,000 divisions per second and 32 bits are used, the wraparound time is approximately 13 hours. In an embodiment aimed at optimizing the wraparound time, an optimized time scale is selected based on the video frame rate. Therefore, by using the time scale (column 1403) shown in table 1400 in FIG. 14, the wraparound time 1404 for each selected frame rate 1401 and its scaled frame rate 1402 is indicated.

[0043] In some embodiments, higher resolution presentation timestamps may be used to avoid dropped frames or freezes during playback. When the resolution of presentation timestamps input to a decoder or system is low, scaling techniques using the formula PTS' = PTS * TS' / TS can be used to convert the presentation timestamps from one time scale (TS) to another (TS'). However, in at least some cases, it may be difficult to recover the precision of the presentation timestamps. For example, as shown in table 1500 of FIG. 15, let TS = 1000 and TS' = 1000000. As can be seen from FIG. 15, for each frame # (column 1501), the PTS (column 1502) is shown along with the scaled presentation timestamp (column 1503) and the native presentation timestamp (column 1504). Each of these frames contains a resulting rounding error (column 1505). Because the system knows that the minimum frame spacing based on the frame rate will be constant, the frame spacing of the presentation timestamps will be the minimum frame spacing (D) or a multiple of the minimum frame spacing (N x D). For example, if the time scale is 120000, the minimum frame spacing D for a frame rate of 23.97 fps is 5005 tick marks. The actual frame spacing should be 5005 tick marks or a multiple of 5005 tick marks, as shown in embodiment 1600 of Figure 16. Figure 16 shows the minimum spacing (D) as the spacing between frames 1601 and 1602, as well as N times the spacing of D (the spacing of "10010" shown between frames 1602 and 1603).

[0044] In some embodiments, an algorithm can be implemented to optimize (remaster) the presentation timestamps. In this example, the presentation timestamp of the first frame is set to PTS in0 And the first PTS out0 Scaling is performed for PTS out0 =PTS in0 *TS out / TS in is.

[0045] This algorithm includes the following steps (1) to (3): (1) Determine the frame rate (F) and the input time scale (TS in ) and output time scale (TS out ) and the frame interval (D in / D out ) is calculated. in =TS in / F, D out =TS out / F. (2) Current PTS in Based on this, the rounding function I=round((PTS in -PTS in0 ) / D in (3) The presentation timestamp after optimization (remastering) is expressed as PTS out =I*D out +PTS out0 Let's say.

[0046] PTS in If PTS starts from 0, the above algorithm can recover (perfectly, at least in some cases) the presented timestamp to virtually any resolution. in If does not start at 0, there is a constant initial offset value, which is out0 All PTS from out 17 shows each frame # (column 1701), presentation timestamp value (column 1702), remastered presentation timestamp value (column 1703), and native presentation timestamp value (column 1704), and as shown, the rounding error (column 1705) of the remastered presentation timestamp of each frame # is zero. The fact that the rounding error of the remastered presentation timestamp value is zero means that the frame spacing obtained by each remastered presentation timestamp value will be constant for a given appropriate output time scale, which can mitigate issues with media items where frame spacing fluctuates.

[0047] In another embodiment, variable frame rate streaming is based on a unified time scale. Variable frame rate streaming can be used in several situations. For example, different frame rates may be used between video segments. Using a unified time scale allows for seamless frame rate transitions to avoid blank screens, such as black screens. In this case, different time scales are used between video segments, such as to optimize wraparound times. Another embodiment is for capturing and streaming variable refresh rate (VRR) content. This is a source-side encoding use case, where a consistent time scale is required across various frame rates. In this case, the 120,000 ticks / second time scale in Table 700 is implemented as the time scale for capturing and streaming VRR content. This is because the 120,000 ticks / second time scale ensures that the frame intervals for each frame rate in Table 700 are integer values.

[0048] In the first case, the presentation timestamps are given as pairs ([tick value], [time scale]). Therefore, a unified time scale (TS) is used for rendering on the client playback device under conditions where the frame rate changes from F0 to F1. out In this embodiment, the following algorithm is defined as an algorithm for providing presentation timestamp information based on the first frame. in0 , T.S. in =TS in0 , F=F0, the first tick out0 Scaling is performed for tick out0 =tick in0 *TS out / TS inNext, the system converts ([scale value], [time scale]) into a new scale value of the unified time scale using the following routines (1) to (4). (1) The frame rate (F) and the input time scale (TS in ) and output time scale (TS out ) and the frame interval (D in / D out ) is calculated. in =TS in / F, D out =TS out / F. (2) Current tick in Based on this, the rounding function I=round((tick in -tick in0 ) / D in ) to get the frame index (I). (3) The scale value after remastering is set as tick out =I*D out +tick out0 (4) The frame rate changes (F≠F1) and a new time scale (TS in1 ) updated (TS in ≠TS in ) and TS in =TS in1 , tick in0 =tick out *TS in / TS out , tick out0 =tick out , set F=F1.

[0049] Thus, the above algorithm converts input presentation timestamps into more accurate unified timescale presentation timestamps, even when timescales differ between video segments. Because the above process, in at least some embodiments, is performed after encoding on the client playback device, the process can use a more precise timescale (e.g., 64-bit precision) for the unified timescale without having to monitor wraparound. For this reason, a 120,000 ticks / second timescale is an optimal choice in at least some embodiments.

[0050] 1 and 2, methods and systems can be provided that are configured to identify, for each of a variety of different media items (e.g., media items 122), a current timescale 108 that was used when encoding the media item for distribution, identify or calculate, for the different media items, a unified timescale 110 that maintains a constant frame spacing for each of the media items, and scale at least one of the different media items from the current timescale 108 to the identified unified timescale 110, thereby maintaining a constant frame spacing for the resulting media item 112. In some cases, the different media items are video media items encoded at different frame rates (e.g., 23.97 frames / second, 24 frames / second, 25 frames / second, 29.97 frames / second, 30 frames / second, 59.94 frames / second, 60 frames / second, 120 frames / second, 240 frames / second, or 300 frames / second), and in other cases, the different media items are audio media items with different frame rates (e.g., frame rates of 1024 samples per frame, 1536 samples per frame, or 2048 samples per frame). In some examples, the different media items are interactive media items that allow a user to make decisions that affect the order of scenes during playback.

[0051] In some embodiments, a particular set of media items is selected. This set of media items may include video items, audio items, or a combination thereof. At least some of the media items in the set of media items, at least in this example, have various designated video and / or audio frame rates. In such cases, the maximum range of frame rates for the set of media items is known (e.g., a range of 24 fps to 60 fps). Thus, when the identification module 109 shown in FIG. 1 identifies a unitary timescale 110, such as by computation, it does not need to consider many possible frame rates (in this case, frame rates above 60 fps and frame rates below 24 fps) but can focus only on frame rates between 24 and 60 fps. Thus, the identification module 109 can identify, such as by computation, a unitary timescale 110 that is optimized specifically for the set of media items, which may be audio and / or video items. In this case, the least common multiple can be significantly smaller than for other media items consisting of audio and / or video items, so that even a unitary time scale 110 with a small number of divisions per second can provide consistent frame intervals for various frame rates (frame rates of 24 to 60 fps in this example). Reducing the number of divisions per second shortens the wraparound time of monotonically increasing presentation timestamp values, reducing the likelihood of presentation timestamps overflowing the memory buffer. Once the unitary time scale 110 for the selected media items has been identified, the time scale modification module 111 modifies the media items in the selected media items to conform to the unitary time scale 110 by generating new presentation timestamps for those media items.

[0052] In some embodiments, when the presentation timestamps are monotonically increasing, the number of units (i.e., the number of divisions per second) used in the unitary time scale is selected to maximize the wraparound time of the media items. In at least some cases, maximizing the wraparound time requires selecting the smallest number of units in the unitary time scale 110 that allows for a constant frame spacing between each media item. The smaller the number of units, the slower the rate at which the monotonically increasing presentation timestamp values ​​increase. This slower rate of monotony maximizes the time it takes to reach the wraparound time. In some cases, the number of units selected to maximize the wraparound time of media items in a group of media items is selected based on the video frame rate of the video items in the group, while in other cases, the number of units selected to maximize the wraparound time of media items in a group of media items is selected based on the audio frame rate of the audio items in the group.

[0053] In some cases, the identified unified time scale 110 includes a presentation timestamp interval. The presentation timestamp interval may differ from a constant minimum frame interval. The presentation timestamp interval is configured as a minimum frame interval or a multiple of the minimum frame interval. As shown in FIG. 16, for example, the interval between frames 1601 and 1602 is 5005 milliseconds, which represents the minimum frame interval between frames 1601 and 1602. On the other hand, the frame interval between frames 1602 and 1603 is a multiple of 5005 (in this case, 5005 x 2 milliseconds, or 10010 milliseconds). In some cases, the presentation timestamp interval can also be restored (remastered) to a specified resolution. In such cases, a constant minimum frame interval or a multiple of the minimum frame interval is determined as the presentation timestamp interval and applied to the media item. Thus, the remastering process includes identifying a constant frame interval and using the identified constant frame interval to output another temporal resolution by modifying the input temporal resolution (e.g., milliseconds, microseconds, nanoseconds, etc.) so that the scaled presentation timestamp interval matches the native presentation timestamp interval without rounding errors. The remastered presentation timestamp interval is then applied to each frame of the media item. Thus, as shown in Figure 17, the remastered presentation timestamps (column 1703) have a presentation timestamp interval that is equal to the native presentation timestamps (column 1704), thereby eliminating rounding errors (column 1705) that could lead to playback errors such as skipped frames, extra frames, and blank frames.

[0054] In some embodiments, a unified timescale 110 may be identified that allows multiple media items to be streamed at variable frame rates while maintaining constant frame spacing. In some cases, frame rates may be switched within the same media item. For example, if the media item is an interactive video, the server (distribution) may encode portions of the video at different frame rates. In such cases, the server provides the interactive media item for playback on a user's playback device (e.g., playback device 116) at these different frame rates. In this case, playback device 116 calculates and / or applies a unified timescale 110 that allows media items to be streamed at variable frame rates. This unified timescale ensures that the frame spacing of the provided media items is constant, including the frame spacing of media items encoded at different frame rates, such as interactive media items. Thus, in such embodiments, media items with different frame rates may be streamed at variable frame rates while maintaining constant frame spacing using the unified timescale. Similarly, in at least some embodiments, media items having video content captured using a variable refresh rate can be streamed (from the server) at a variable frame rate, but played back at constant frame intervals using the unified timescale 110.

[0055] In some cases, identifying or calculating the unitary time scale 110 involves converting input presentation timestamps from various media items with different time scales into presentation timestamps based on the unitary time scale. This is generally illustrated in FIG. 17, where the remastered presentation timestamp values ​​are shown to be equal to the native presentation timestamp values ​​because any rounding errors inherent in the original presentation timestamps have been eliminated. In some instances, using these converted presentation timestamps derived from the input presentation timestamps can prevent wraparound of the presentation timestamp counter, or at least extend the time until wraparound occurs. Reducing wraparounds can also reduce the number of steps that must be taken to handle wraparounds smoothly without introducing additional playback errors. Thus, providing a unitary time scale in this manner ensures that the interframe spacing between frames is consistent during playback of the media item. This can be done both when the source encoding is performed at the same frame rate within the media item and when the source encoding frame rate varies within the media item. A consistent time interval avoids many common playback problems, providing users with a smoother and more reliable (and therefore more enjoyable) media playback experience.

[0056] In some embodiments, a corresponding system is provided, comprising at least one physical processor and a physical memory containing computer-executable instructions that, when executed by the physical processor, cause the physical processor to perform the following steps: for each of a plurality of different media items, at least two of the plurality of media items being encoded at different frame rates, identify a current timescale used when encoding the media item for distribution, for the plurality of media items, a unified timescale that provides a constant frame spacing for each of the plurality of media items, and modify at least one of the plurality of media items from the current timescale to the identified unified timescale, thereby providing a constant frame spacing for the modified at least one media item.

[0057] In another embodiment, a non-transitory computer-readable medium is provided that includes one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to perform the following steps: for each of a plurality of different media items, at least two of the plurality of media items being encoded at different frame rates, identify a current timescale used when encoding the media item for distribution, for the plurality of media items, a unified timescale that provides a constant frame spacing for each of the plurality of media items, and modify at least one of the plurality of media items from the current timescale to the identified unified timescale, thereby providing a constant frame spacing for the modified at least one media item.

[0058] An exemplary ecosystem in which content is provided to end nodes and content request requests are directed to specific end nodes is described in detail below with reference to Figure 19. The description of Figures 20 and 21 provides an overview of an exemplary distribution infrastructure and an exemplary content player used during a playback session, respectively. These exemplary ecosystems and distribution infrastructures may be implemented in any of the embodiments described above with reference to Figures 1-18.

[0059] 19 is a block diagram of a content distribution ecosystem 1900 that includes a distribution infrastructure 1910 in communication with a content player 1920. In some embodiments, the distribution infrastructure 1910 is configured to encode data at a particular data rate and transmit the encoded data to the content player 1920. The content player 1920 is configured to receive the encoded data via the distribution infrastructure 1910, decode the data, and play it for a user. The data provided by the distribution infrastructure 1910 may include streaming types of data, such as audio, video, text, images, animation, interactive content, haptic data, virtual (or augmented) reality data, location data, and gaming data.

[0060] The distribution infrastructure 1910 broadly represents any services, hardware, software, or other infrastructure components configured to distribute content to end users. For example, the distribution infrastructure 1910 may include one or more of various types of hardware and software, such as content aggregation systems, media transcoding and packaging services, and network components. In some cases, the distribution infrastructure 1910 may be implemented as a highly complex distribution system, a media server or media device alone, or anything in between. In some examples, regardless of size or complexity, the distribution infrastructure 1910 may include at least one physical processor 1912 and at least one memory device 1914. The memory 1914 may store or load one or more modules 1916 that enable adaptive streaming as described herein.

[0061] The content player 1920 broadly represents any type or form of device or system capable of playing audio and / or video content provided via the distribution infrastructure 1910. Examples of the content player 1920 include, but are not limited to, any type or form of device capable of rendering digital content, such as a mobile phone, a tablet, a laptop, a desktop, a television, a set-top box, a digital media player, a virtual reality headset, or an augmented reality pair of glasses. Like the distribution infrastructure 1910, the content player 1920 also includes a physical processor 1922, memory 1924, and one or more modules 1926. Module 1926 performs or enables some or all of the adaptive streaming processes described herein, and in some examples, module 1916 of the distribution infrastructure 1910 cooperates with module 1926 of the content player 1920 to adaptively stream digital content.

[0062] In particular embodiments, one or more of the modules 1916, 1926 illustrated in Figure 19 represent one or more software applications or programs that, when executed by a computing device, cause the computing device to perform one or more tasks. For example, as described in further detail below, one or more of the modules 1916, 1926 represent modules stored and configured to execute on one or more general-purpose computing devices. One or more of the modules 1916, 1926 illustrated in Figure 19 also represent, in whole or in part, one or more special-purpose computers configured to perform one or more tasks.

[0063] Additionally, one or more of the modules, processes, algorithms, or steps described herein may transform the form of data, a physical device, a representation of a physical device, or any combination thereof. For example, one or more of the modules described herein may receive audio data to be encoded, transform the audio data by encoding it, output the encoded result for an adaptive audio bitrate system, send the transformed result to a content player, and render the transformed data for viewing by an end user. Additionally or alternatively, one or more of the modules described herein may transform the form of any part of a physical computing device, such as a processor, volatile memory, or non-volatile memory, by executing on or interacting with a physical computing device, such as by storing data on the physical computing device.

[0064] Physical processors 1912, 1922 broadly represent any type or form of hardware-implemented processing unit capable of interpreting and / or executing computer-readable instructions. In one example, physical processor 1912 accesses, modifies, or both, one or more of modules 1916, and physical processor 1922 accesses, modifies, or both, one or more of modules 1926. Additionally or alternatively, physical processors 1912, 1922 execute one or more of modules 1916, 1926 to facilitate adaptive streaming of digital content. Examples of physical processors 1912, 1922 include, but are not limited to, any suitable physical processor, such as a microprocessor, a microcontroller, a central processing unit (CPU), a field programmable gate array (FPGA) implementing a soft-core processor, an application-specific integrated circuit (ASIC), portions of one or more of these, variations or combinations of one or more of these, etc.

[0065] Memory 1914, 1924 broadly represents any type or form of volatile or non-volatile storage device or medium capable of storing data, computer-readable instructions, or both. In one example, memory 1914, 1924 stores, loads, retains, or any combination thereof, one or more of modules 1916, 1926. Examples of memory 1914, 1924 include, but are not limited to, any suitable memory device or memory system, such as random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), optical disk drive, cache, or any variation or combination of one or more of these.

[0066] FIG. 20 is a block diagram illustrating exemplary components of a content distribution infrastructure 1910 according to certain embodiments. The distribution infrastructure 1910 includes a storage device 2010, a service 2020, and a network 2030. The storage device 2010 broadly represents any device, set of devices, system, or any combination thereof capable of storing content for distribution to end users. The storage device 2010 may comprise a central repository with devices capable of storing terabytes or petabytes of data. Alternatively or additionally, the storage device 2010 may comprise a distributed storage system (e.g., devices that mirror (cache) content at locations interconnected by the Internet to provide high-speed access to the mirrored replicated content within a particular geographic area). The storage device 2010 may also be configured in any other suitable manner.

[0067] As shown, storage device 2010 can store a variety of different items, including content 2012, user data 2014, log data 2016, or any combination thereof. Content 2012 includes any suitable type or form of content, such as television programs, movies, video games, user-generated content, etc. User data 2014 includes information associated with a particular user or content player, such as personally identifiable information (PII), payment information, preference settings, language and accessibility settings, or any combination thereof. Log data 2016 includes metrics associated with a user's connection to or interaction with distribution infrastructure 1910, such as browsing history information, network throughput information, etc., or any combination thereof.

[0068] Services 2020 may include a personalization service 2022, a transcoding service 2024, a packaging service 2026, or any combination thereof. The personalization service 2022 personalizes user experience elements, such as recommendations and content streams, for users of the distribution infrastructure 1910. The transcoding service 2024 compresses media at multiple different bit rates, allowing for real-time switching between different encoding schemes, as described in more detail below. The packaging service 2026 packages the encoded video. Once packaged, the video is deployed for streaming over a distribution network, such as network 2030.

[0069] Network 2030 broadly represents any medium or architecture capable of supporting communications or data transfer. Network 2030 may utilize wireless connections, wired connections, or both to support communications or data transfer. Examples of network 2030 include, but are not limited to, any suitable network, such as an intranet, a wide area network (WAN), a local area network (LAN), a personal area network (PAN), the Internet, power line communications (PLC), a cellular network (e.g., a Global System for Mobile Communications (GSM) network), portions of one or more of these, variations, or combinations of one or more of these. For example, as shown in FIG. 20 , network 2030 may include an Internet backbone 2032, an Internet service provider network 2034, a local network 2036, or any combination thereof. As will be described in more detail below, bandwidth limitations or bottlenecks in one or more of these network segments may trigger video and / or audio bitrate throttling.

[0070] Figure 21 is a block diagram of an example implementation of the content player 1920 shown in Figure 19. The content player 1920 broadly represents any type or form of computing device capable of reading computer-executable instructions. The content player 1920 may include any suitable computing device, such as, but not limited to, a laptop computer, a tablet computer, a desktop computer, a server, a mobile phone, a multimedia player, an embedded system, a wearable device (e.g., a smart watch, smart glasses, etc.), a smart vehicle, a gaming console, an Internet of Things (IoT) device such as a smart home appliance, or any variation or combination of one or more of the foregoing.

[0071] 21 , the content player 1920 further includes a processor 1922 and a memory 1924, as well as a communication interface 2122 coupled to the communication infrastructure 2102 and a network connection 2124. The content player 1920 also includes a graphics interface 2126 coupled to a graphics device 2128, an input interface 2134 coupled to an input device 2136, and a storage interface 2138 coupled to a storage device 2140.

[0072] Communications infrastructure 2102 broadly represents any type or form of infrastructure capable of facilitating communication between one or more components of a computing device. Examples of communications infrastructure 2102 include, but are not limited to, any type or form of communications bus (e.g., peripheral component interconnect (PCI) bus, PCI Express (PCIe) bus, memory bus, front-side bus, integrated drive electronics (IDE) bus, control or register bus, host bus, etc.).

[0073] As mentioned above, memory 1924 broadly represents any type or form of volatile or non-volatile storage device or medium capable of storing data, other computer-readable instructions, or both. In some examples, memory 1924 stores and / or loads operating system 2108, which is executed by processor 1922. In one example, operating system 2108 includes and / or represents software that manages computer hardware and software resources, provides common services to computer programs and / or applications on content player 1920, or both.

[0074] The operating system 2108 performs various system management functions, such as managing hardware components (e.g., graphics interface 2126, audio interface 2130, input interface 2134, and / or storage interface 2138). The operating system 2108 also provides a process and memory management model for the playback application 2110. Modules of the playback application 2110 include, for example, a content buffer 2112, an audio decoder 2118, and a video decoder 2120.

[0075] The playback application 2110 is configured to retrieve digital content via the communications interface 2122 and play the digital content via the graphics interface 2126. The graphics interface 2126 is configured to send rendered video signals to the graphics device 2128. In typical operation, the playback application 2110 first receives a request from a user requesting playback of a particular title or particular content. The playback application 2110 then identifies one or more encoded video and audio streams associated with the title requested in the request (hereinafter, the request title). After the playback application 2110 locates the encoded streams associated with the request title, the playback application 2110 then downloads, from the distribution infrastructure 1910, sequence header indexes associated with each of the encoded streams associated with the request title. The sequence header indexes associated with the encoded content contain information related to the encoded sequence of the data included in the encoded content.

[0076] In one embodiment, the playback application 2110 begins downloading the content associated with the requested title by downloading sequence data for the content. Note that the sequence data is encoded at the lowest audio and / or video playback bitrate to minimize playback startup time. The digital content file requested in the request is then downloaded to the content buffer 2112. The content buffer 2112 is configured to function as a first-in, first-out queue. In one embodiment, data is downloaded in units of data, including one unit of video data or one unit of audio data. As several video data units are downloaded to the content player 1920 for the video data associated with the digital content file requested in the request, these video data units are pushed to the content buffer 2112. Similarly, as several audio data units are downloaded to the content player 1920 for the audio data associated with the digital content file requested in the request, these audio data units are pushed to the content buffer 2112. In one embodiment, video data units are stored in a video buffer 2116 within the content buffer 2112 and audio data units are stored in an audio buffer 2114 of the content buffer 2112 .

[0077] The video decoder 2120 reads video data units from the video buffer 2116 and outputs the read video data units as a sequence of video frames having a duration corresponding to a fixed playback time. Reading a video data unit from the video buffer 2116 effectively dequeues that video data unit from the video buffer 2116. The sequence of video frames is then rendered by the graphics interface 2126 and sent to the graphics device 2128 for display to the user.

[0078] The audio decoder 2118 reads audio data units from the audio buffer 2114 and outputs the read audio data units as a sequence of audio samples that are generally synchronized in time with the sequence of decoded video frames. In one embodiment, the sequence of audio samples is sent to an audio interface 2130, which converts the sequence of audio samples into an electrical audio signal. The electrical audio signal is then sent to a speaker of an audio device 2132, which generates an acoustic output responsive to the electrical audio signal.

[0079] In situations where the delivery infrastructure 1910 is experiencing limited or changing bandwidth, the playback application 2110 may download and buffer successive portions of video and audio data, switching between different bitrate video encodings based on various factors (e.g., scene complexity, audio complexity, network bandwidth, device capabilities, etc.). In some embodiments, video playback quality is prioritized over audio playback quality. Audio playback quality and video playback quality may also be balanced, and in some embodiments, audio playback quality may be prioritized over video playback quality.

[0080] Graphics interface 2126 is configured to generate frames of video data and send the frames of video data to graphics device 2128. In one embodiment, graphics interface 2126 is included as part of an integrated circuit with processor 1922. Alternatively, graphics interface 2126 may be configured as a hardware accelerator separate from (i.e., not integrated into) a chipset that includes processor 1922.

[0081] Graphics interface 2126 broadly represents any type or form of device configured to transfer images for display on graphics device 2128. For example, graphics device 2128 may be fabricated using liquid crystal display (LCD), cathode-ray, or light-emitting diode (LED) display technology (whether organic or inorganic). In some embodiments, graphics device 2128 may further comprise a virtual reality display, an augmented reality display, or both. Graphics device 2128 may comprise any technically feasible means for generating images for display. In other words, graphics device 2128 broadly represents any type or form of device capable of visually displaying information transferred by graphics interface 2126.

[0082] 21 , the content player 1920 further includes at least one input device 2136 coupled to the communications infrastructure 2102 via an input interface 2134. The input device 2136 broadly represents any type or form of computing device capable of providing computer- or human-generated input to the content player 1920. Examples of the input device 2136 include, but are not limited to, any type or form of electronic input mechanism, such as a keyboard, a pointing device, a voice recognition device, a touch panel, a wearable device (e.g., a glove, a watch, etc.), a controller, or any variation or combination of one or more of these.

[0083] The content player 1920 further includes a storage device 2140 coupled to the communications infrastructure 2102 via a storage interface 2138. The storage device 2140 broadly represents any type or form of storage device or storage medium capable of storing data, other computer-readable instructions, or both. For example, the storage device 2140 may be a magnetic disk drive, a solid-state drive, an optical disk drive, a flash drive, etc. The storage interface 2138 broadly represents any type or form of interface or device for transmitting data between the storage device 2140 and other components of the content player 1920.

[0084] Illustrative Embodiments 1. A computer-implemented method comprising the steps of: for each of a plurality of different media items, at least two of the plurality of media items being encoded at different frame rates, identifying a current time scale used when encoding the media item for distribution; identifying a unified time scale for the plurality of media items that will result in a constant frame spacing for each of the plurality of media items; and converting at least one of the plurality of media items from the current time scale to the identified unified time scale, thereby resulting in a constant frame spacing for the at least one media item after the conversion.

[0085] 2. The computer-implemented method of embodiment 1, wherein the media items include video media items.

[0086] 3. The computer-implemented method of embodiment 2, wherein the video media items are encoded at a frame rate of 23.97 frames / second, 24 frames / second, 25 frames / second, 29.97 frames / second, 30 frames / second, 59.94 frames / second, 60 frames / second, 120 frames / second, 240 frames / second, or 300 frames / second.

[0087] 4. The computer-implemented method of embodiment 1, wherein the media items include audio media items.

[0088] 5. The computer-implemented method of embodiment 4, wherein the audio media item has a frame rate of 1024 samples, 1536 samples, or 2048 samples per frame.

[0089] 6. A computer-implemented method as described in embodiment 1, wherein each media item in a specified group of media items among the plurality of media items has a specified video frame rate and audio frame rate, and the unified time scale is calculated to optimize the specified video frame rate and specified audio frame rate of the media items in the group of media items.

[0090] 7. The computer-implemented method of embodiment 1, wherein the unified time scale is implemented to generate one or more presentation timestamps for multiple media items.

[0091] 8. The computer-implemented method of embodiment 6, wherein one or more presentation timestamps are monotonically increasing and the units used for the unitary time scale are selected to maximize the wraparound time of multiple media items.

[0092] 9. The computer-implemented method of embodiment 8, wherein the unit selected to maximize the wraparound time of the multiple media items is selected based on the video frame rate.

[0093] 10. A computer-implemented method as described in embodiment 1, wherein the identified unitary time scale has a presentation timestamp (PTS) interval, and the presentation timestamp interval is configured as a minimum frame interval or a multiple of the minimum frame interval.

[0094] 11. The computer-implemented method of embodiment 10, further comprising restoring the presented timestamp interval to a specified resolution.

[0095] 12. A system comprising at least one physical processor and a physical memory containing computer-executable instructions that, when executed by the physical processor, cause the physical processor to perform the following steps: for each of a plurality of different media items, at least two of the plurality of media items being encoded at different frame rates, identify a current timescale used when encoding the media item for distribution; for the plurality of media items, identify a unified timescale that maintains a constant frame spacing for each of the plurality of media items; and change at least one of the plurality of media items from the current timescale to the identified unified timescale, thereby maintaining a constant frame spacing for the at least one media item after the change.

[0096] 13. The system of embodiment 12, wherein the identified unified time scale enables streaming of multiple media items at variable frame rates while maintaining constant frame spacing.

[0097] 14. The system of embodiment 13, wherein media items with different frame rates are streamed at variable frame rates while maintaining constant frame spacing for each frame rate using a unified time scale.

[0098] 15. The system of embodiment 12, wherein a media item having video content captured using variable refresh rates is streamed at variable frame rates while maintaining a constant frame spacing for each frame rate using a unified time scale.

[0099] 16. The system of embodiment 12, wherein the step of identifying a unified time scale includes converting one or more input presentation timestamps (PTS) obtained from a plurality of different media items with different time scales into presentation timestamps based on the unified time scale.

[0100] 17. The system of embodiment 16, wherein wraparound of the presentation timestamp counter is avoided by implementing a transformed presentation timestamp derived from the input presentation timestamp.

[0101] 18. The system of embodiment 12, wherein the step of changing at least one of the plurality of media items from a current time scale to an identified unified time scale enables the implementation of a single fixed vertical synchronization interrupt during playback of the plurality of media items.

[0102] 19. The system of embodiment 12, further comprising a step of optimizing the presentation timestamps of multiple media items at an appropriate time scale so that the scaled presentation timestamps match the native presentation timestamps without rounding errors.

[0103] 20. A non-transitory computer-readable medium containing one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to perform the following steps: for each of a plurality of different media items, at least two of the plurality of media items being encoded at different frame rates, identify a current timescale used when encoding the media item for distribution; for the plurality of media items, identify a unified timescale that maintains a constant frame spacing for each of the plurality of media items; and change at least one of the plurality of media items from the current timescale to the identified unified timescale, thereby maintaining a constant frame spacing for the at least one media item after the change.

[0104] As detailed above, the computing devices and systems shown and described herein broadly represent any type or form of computing device or system capable of executing computer-readable instructions, such as those contained in the modules described herein. In their most basic configurations, each of these computing devices may include at least one memory device and at least one physical processor.

[0105] In some examples, the term "memory device" refers generally to any type or form of volatile or non-volatile storage device or storage medium capable of storing data, computer-readable instructions, or both. In one example, a memory device may store, load, retain, or any combination thereof, one or more of the modules described herein. Examples of memory devices include, but are not limited to, any suitable storage memory, such as random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), optical disk drive, cache, or any variation or combination of one or more of these.

[0106] In some examples, the term "physical processor" refers generally to any type or form of hardware-implemented processing unit capable of interpreting, executing, or both, computer-readable instructions. In one example, a physical processor may access, modify, or both, one or more modules stored in the memory devices described above. Examples of physical processors include, but are not limited to, any suitable physical processor, such as a microprocessor, a microcontroller, a central processing unit (CPU), a field programmable gate array (FPGA) implementing a soft-core processor, an application specific integrated circuit (ASIC), a portion of one or more of these, a variation or combination of one or more of these, etc.

[0107] The modules shown and described herein, while illustrated as separate elements, may represent portions of a single module or application. Moreover, in particular embodiments, one or more of these modules may represent one or more software applications or programs that, when executed by a computing device, cause the computing device to perform one or more tasks. For example, one or more of the modules shown and described herein may represent modules stored and configured to execute on one or more of the computing devices or systems shown and described herein. One or more of these modules may represent, in whole or in part, one or more special-purpose computers configured to perform one or more tasks.

[0108] Additionally, one or more of the modules described herein may transform the form of data, physical devices, representations of physical devices, or any combination thereof. For example, one or more of the modules described herein may receive data to be transformed, transform the data, output the transformation results to identify an optimized time scale, use the transformation results to generate accurate presentation timestamps, and store the transformation results. Additionally or alternatively, one or more of the modules described herein may transform the form of any portion of a physical computing device, such as a processor, volatile memory, non-volatile memory, etc., by interacting with the physical computing device, such as by executing on the physical computing device or storing data on the physical computing device.

[0109] In some embodiments, the term "computer-readable medium" refers generally to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media include, but are not limited to, transmission-type media such as carrier waves, non-transitory media such as magnetic storage media (e.g., hard disk drives, tape drives, floppy disks), optical storage media (e.g., compact discs (CDs), digital video discs (DVDs), Blu-ray discs), electronic storage media (e.g., solid-state drives, flash media), and other distribution systems.

[0110] The process parameters and order of steps shown and described herein are for illustrative purposes only and may be modified as desired. For example, although the steps shown and described herein may be shown and described in a particular order, the steps do not necessarily have to be performed in the order shown and described. Also, although various exemplary methods are shown and described herein, such methods may omit one or more of the steps shown and described herein or may include additional steps in addition to those disclosed.

[0111] The foregoing description has been provided to enable those skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or to be limited to the precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the present disclosure. The presently disclosed embodiments are therefore to be considered in all respects as illustrative and not restrictive. Reference should be made to the appended claims and their equivalents in determining the scope of the present disclosure.

[0112] Unless otherwise noted, the terms "connected to" and "coupled to" (and their derivatives) used in this specification and claims should be interpreted to allow both direct and indirect (i.e., via other elements or components) connections. Furthermore, the terms "a" or "an" used in this specification and claims should be interpreted to mean "at least one of." Finally, although the terms "including" and "having" (and their derivatives) are used in this specification and claims for convenience, they may be replaced with "comprising" and have the same meaning. Preferred embodiments of the present invention will be described below in detail. Embodiment 1 for each of a plurality of different media items, at least two of the plurality of media items being encoded at different frame rates, identifying a current time scale used in encoding the media item for distribution; identifying a unified time scale for the plurality of media items that provides a uniform frame spacing for each of the plurality of media items; scaling at least one of the plurality of media items from the current time scale to the identified unified time scale, thereby maintaining a constant frame spacing for the at least one media item after the scaling; A computer-implemented method comprising: Embodiment 2 2. The computer-implemented method of embodiment 1, wherein the media items include video media items. Embodiment 3 3. The computer-implemented method of embodiment 2, wherein the video media items are encoded at a frame rate of 23.97 frames / second, 24 frames / second, 25 frames / second, 29.97 frames / second, 30 frames / second, 59.94 frames / second, 60 frames / second, 120 frames / second, 240 frames / second, or 300 frames / second. Embodiment 4 2. The computer-implemented method of embodiment 1, wherein the media items include audio media items. Embodiment 5 5. The computer-implemented method of embodiment 4, wherein the audio media item has a frame rate of 1024 samples, 1536 samples, or 2048 samples per frame. Embodiment 6 each media item in a designated group of media items among the plurality of media items has a designated video frame rate and audio frame rate; 2. The computer-implemented method of claim 1, wherein the unified time scale is calculated to optimize a specified video frame rate and a specified audio frame rate of the media items in the group of media items. Embodiment 7 2. The computer-implemented method of embodiment 1, wherein the unified time scale is implemented to generate one or more presentation timestamps for the plurality of media items. Embodiment 8 the one or more presentation timestamps are monotonically increasing; 7. The computer-implemented method of embodiment 6, wherein the units used for the unitary time scale are selected to maximize wraparound time for the plurality of media items. Embodiment 9 9. The computer-implemented method of embodiment 8, wherein the unit selected to maximize wraparound time for the plurality of media items is selected based on a video frame rate. Embodiment 10 the identified unitary time scale comprises a presentation timestamp interval; 2. The computer-implemented method of embodiment 1, wherein the presentation timestamp interval is configured as a minimum frame interval or a multiple of the minimum frame interval. Embodiment 11 11. The computer-implemented method of embodiment 10, further comprising restoring the presented timestamp interval to a specified resolution. Embodiment 12 at least one physical processor; a physical memory containing computer-executable instructions; A system comprising: The computer-executable instructions, when executed by the physical processor, for each of a plurality of different media items, at least two of the plurality of media items being encoded at different frame rates, identifying a current time scale used in encoding the media item for distribution; identifying a unified time scale for the plurality of media items that provides a uniform frame spacing for each of the plurality of media items; scaling at least one of the plurality of media items from the current time scale to the identified unified time scale, thereby maintaining a constant frame spacing for the at least one media item after the scaling; on the physical processor. Embodiment 13 13. The system of claim 12, wherein the identified unified time scale enables streaming of the plurality of media items at variable frame rates while maintaining the constant frame spacing. Embodiment 14 14. The system of embodiment 13, wherein media items with different frame rates are streamed at variable frame rates while maintaining the constant frame spacing for each frame rate using the unified time scale. Embodiment 15 A system as described in embodiment 12, wherein a media item having video content captured using a variable refresh rate is streamed at a variable frame rate while maintaining the constant frame spacing using the unified time scale. Embodiment 16 The system of embodiment 12, wherein the step of identifying the unitary time scale includes converting one or more input presentation timestamps obtained from the plurality of different media items having different time scales into presentation timestamps based on the unitary time scale. Embodiment 17 17. The system of embodiment 16, wherein wraparound of a presentation timestamp counter is avoided by implementing a converted presentation timestamp derived from the input presentation timestamp. Embodiment 18 The system of embodiment 12, wherein the step of changing at least one of the plurality of media items from the current time scale to the identified unified time scale enables the implementation of a single fixed vertical synchronization interrupt during playback of the plurality of media items. Embodiment 19 13. The system of embodiment 12, further comprising optimizing the presentation timestamps of the plurality of media items so that the scaled presentation timestamps match the native presentation timestamps without rounding errors. Embodiment 20 A non-transitory computer-readable medium containing one or more computer-executable instructions that, when executed by at least one processor of a computing device, for each of a plurality of different media items, at least two of the plurality of media items being encoded at different frame rates, identifying a current time scale used in encoding the media item for distribution; identifying a unified time scale for the plurality of media items that provides a uniform frame spacing for each of the plurality of media items; scaling at least one of the plurality of media items from the current time scale to the identified unified time scale, thereby maintaining a constant frame spacing for the at least one media item after the scaling; a non-transitory computer-readable medium for causing the computing device to execute the [Explanation of symbols]

[0113] 100 Computing Environments 101 Computer Systems 102 processors 103 System Memory 104 Communication Module 105 Receiver 106 Transmitter 107 Verification Module 108 Current Time Scale 109 Identification Module 110 Unitary Time Scale 111 Time Scale Change Module 112 Media items after change 115 users 116 Playback Devices 117 Input 120 data stores 121 Stored Media Items 122 media items 123 encoding frame rate 301 Encoded / Streaming Container 302 Regeneration / Demultiplexer 303, 2120 Video decoder 304, 2118 Audio decoder 1900 Content Delivery Ecosystem 1910 Content Delivery Infrastructure 1912, 1922 physical processors 1914, 1924 memory 1916, 1926 modules 1920 Content Player 2010 storage device 2012 Content 2014 User Data 2016 log data 2020 Services 2022 Personalized Services 2024 encoding and conversion service 2026 Packaging Services 2030 Network 2032 Internet Backbone Network 2034 Internet Service Provider Network 2036 Local Network 2102 Communication Infrastructure 2108 Operating System 2110 Playback Application 2112 Content Buffer 2114 Audio Buffer 2116 Video Buffer 2122 Communication Interface 2124 Network Connection 2126 Graphics Interface 2128 Graphics Device 2130 Voice Interface 2132 Audio Device 2134 Input Interface 2136 Input Devices 2138 Storage Interface 2140 storage device

Claims

1. for each of a plurality of different media items, at least two of the plurality of media items being encoded at different frame rates, identifying a current time scale used in encoding the media item for distribution; identifying a unified time scale for the plurality of media items that provides a uniform frame spacing for each of the plurality of media items; scaling at least one of the plurality of media items from the current time scale to the identified unified time scale, thereby maintaining a constant frame spacing for the at least one media item after the scaling; Including, A computer-implemented method for selecting the number of units of the unitary time scale to be the smallest number of units that results in a constant frame spacing for each of the plurality of media items, thereby maximizing wraparound time for the plurality of media items.

2. The computer-implemented method of claim 1 , wherein the media items include video media items.

3. 3. The computer-implemented method of claim 2, wherein the video media items are encoded at a frame rate of 23.97 frames / second, 24 frames / second, 25 frames / second, 29.97 frames / second, 30 frames / second, 59.94 frames / second, 60 frames / second, 120 frames / second, 240 frames / second, or 300 frames / second.

4. The computer-implemented method of claim 1 , wherein the media items include audio media items.

5. 5. The computer-implemented method of claim 4, wherein the audio media item has a frame rate of 1024 samples, 1536 samples, or 2048 samples per frame.

6. each media item in a designated group of media items among the plurality of media items has a designated video frame rate and audio frame rate; 2. The computer-implemented method of claim 1, wherein the unified time scale is calculated to optimize a specified video frame rate and a specified audio frame rate of the media items in the group of media items.

7. The computer-implemented method of claim 1 , wherein the unified time scale is implemented to generate one or more presentation timestamps for the plurality of media items.

8. the one or more presentation timestamps are monotonically increasing; The computer-implemented method of claim 7 , wherein the units used for the unified time scale are selected to maximize wraparound time for the multiple media items.

9. 9. The computer-implemented method of claim 8, wherein the unit selected to maximize wraparound time for the plurality of media items is selected based on a video frame rate.

10. the identified unitary time scale comprises a presentation timestamp interval; The computer-implemented method of claim 1 , wherein the presentation timestamp interval comprises a minimum frame interval or a multiple of a minimum frame interval.

11. 11. The computer-implemented method of claim 10, further comprising restoring the presented timestamp interval to a specified resolution.

12. at least one physical processor; a physical memory containing computer-executable instructions; A system comprising: The computer-executable instructions, when executed by the physical processor, for each of a plurality of different media items, at least two of the plurality of media items being encoded at different frame rates, identifying a current time scale used in encoding the media item for distribution; identifying a unified time scale for the plurality of media items that provides a uniform frame spacing for each of the plurality of media items; scaling at least one of the plurality of media items from the current time scale to the identified unified time scale, thereby maintaining a constant frame spacing for the at least one media item after the scaling; on the physical processor; The system selects the number of units of the unitary time scale as the smallest number that keeps frame spacing constant for each of the plurality of media items, thereby maximizing wraparound time for the plurality of media items.

13. The system of claim 12 , wherein the identified unified time scale enables streaming of the plurality of media items at variable frame rates while maintaining the constant frame spacing.

14. 14. The system of claim 13, wherein media items with different frame rates are streamed at variable frame rates while maintaining the constant frame spacing for each frame rate using the unified time scale.

15. 13. The system of claim 12, wherein a media item having video content captured using a variable refresh rate is streamed at a variable frame rate while maintaining the constant frame spacing using the unified time scale.

16. 13. The system of claim 12, wherein identifying the unitary time scale comprises converting one or more input presentation timestamps obtained from the plurality of different media items having different time scales into presentation timestamps based on the unitary time scale.

17. 17. The system of claim 16, wherein wraparound of a presentation timestamp counter is avoided by implementing a transformed presentation timestamp derived from the input presentation timestamp.

18. 13. The system of claim 12, wherein changing at least one of the plurality of media items from the current time scale to the identified unified time scale enables implementing a single fixed vertical synchronization interrupt during playback of the plurality of media items.

19. 13. The system of claim 12, further comprising optimizing the presentation timestamps of the plurality of media items such that the scaled presentation timestamps match the native presentation timestamps without rounding errors.

20. A non-transitory computer-readable medium containing one or more computer-executable instructions that, when executed by at least one processor of a computing device, for each of a plurality of different media items, at least two of the plurality of media items being encoded at different frame rates, identifying a current time scale used in encoding the media item for distribution; identifying a unified time scale for the plurality of media items that provides a uniform frame spacing for each of the plurality of media items; scaling at least one of the plurality of media items from the current time scale to the identified unified time scale, thereby maintaining a constant frame spacing for the at least one media item after the scaling; on the computing device, The non-transitory computer-readable medium selects the minimum number of units of the unified time scale that results in a constant frame spacing for each of the plurality of media items, thereby maximizing a wraparound time for the plurality of media items.

Images