Systems and methods for providing optimized time scales and accurate presentation time stamps

An integrated time scale addresses synchronization issues in media playback by providing constant frame intervals, enhancing playback quality and reducing errors through optimized presentation time stamps.

KR103046902B1Active Publication Date: 2026-11-11NETFLIX INC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
KR1020247002062
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-06-25
Filing Date
2022-06-24
Publication Date
2026-11-11
Estimated Expiration
2042-06-24

AI Technical Summary

Technical Problem

Existing media playback technologies experience synchronization issues due to rounding errors in presentation time stamps, leading to irregular frame intervals and playback anomalies such as frame drops, freezes, and eye strain.

Method used

Implementing an integrated time scale that provides constant frame intervals by identifying a common multiplier for media items encoded at different frame rates, converting presentation time stamps to maintain synchronization and avoid rounding errors.

Benefits of technology

Ensures smooth and reliable playback by maintaining consistent frame intervals, reducing noticeable playback defects, and optimizing memory buffer usage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure R1020247002062_ABST
    Figure R1020247002062_ABST
Patent Text Reader

Abstract

A method implemented by a disclosed computer comprises, for a plurality of different media items, determining a current time scale in which the media items are encoded for distribution, wherein at least two of the media items are encoded at different frame rates. Then, the method comprises, for the media items, identifying a combined time scale that provides a constant frame interval for each of the media items. The method also comprises, for at least one of the media items, changing it from the current time scale to the identified combined time scale to provide a constant frame interval for the changed media item(s). Various other methods, systems, and computer-readable media are also disclosed.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] Cross-reference

[0002] This application claims priority to U.S. Regular Application No. 17 / 359,468, filed on June 25, 2021, under the title “SYSTEMS AND METHODS FOR PROVIDING OPTIMIZED TIME SCALES AND ACCURATE PRESENTATION TIME STAMPS”, the entire contents of which are incorporated herein by reference. Background Technology

[0003] Media items containing audio and video media items are typically generated with a presentation time stamp. A presentation time stamp (PTS) is a metadata field applied to a media item to keep its various underlying streams (e.g., audio, video, subtitles, etc.) synchronized. The metadata identifies specific points in time defined by the clock reference that is transmitted as part of the media item. These points in time are the references that define the locations where the underlying underlying streams are synchronized. The presentation time stamp itself is an N-bit number representing the value of a counter driven by the clock on a time scale of X-Hz. In MPEG (Moving Pictures Expert Group) 1 and 2 encodings, for example, the N value of the PTS is 33 bits, and the X value for the time scale is 90,000 Hz.

[0004] However, despite having these presentation time stamps, seamless synchronization may be disrupted by approximations and variations in the amount of time during which various audio and video frames are presented during playback.

[0005] As described in more detail below, the present disclosure describes methods and systems for providing optimized time scales and accurate presentation time stamps in media items.

[0006] In one example, a method implemented by a computer for providing optimized time scales and accurate presentation time stamps comprises, for a plurality of different media items, determining a current time scale in which the media items are encoded for distribution, wherein at least two of the media items are encoded at different frame rates. The method also comprises, for the media items, identifying a combined time scale that provides a constant frame interval for each of the media items. The method further comprises the step of changing at least one of the media items from the current time scale to the identified combined time scale to provide a constant frame interval for the changed media item(s).

[0007] In some embodiments, the media items are video media items. In some cases, the video media items are encoded at 23.97, 24, 25, 29.97, 30, 59.94, 60, 120, 240, or 300 frames per second.

[0008] In other embodiments, the media items are audio media items. In some examples, the audio media items have a frame size of 1024, 1536, or 2048 samples per frame.

[0009] In some cases, each media item within a specified media item group has a specified video frame rate and audio frame rate. In these cases, the combined time scale is calculated to optimize the specified video frame rate and specified audio frame rate of the media items within the group.

[0010] In some embodiments, the integrated time scale is implemented to generate one or more presentation time stamps (PTS) for media items. In some cases, the PTSs are monotonically increasing, and the units used in the integrated time scale are selected to maximize the wrap-around time for the media items.

[0011] In some cases, the unit selected to maximize the wrap-around time for media items is selected based on the video frame rate. In some examples, the identified integrated time scale includes a presentation time stamp interval. In these cases, the PTS interval may include a minimum frame interval or a multiple of the minimum frame interval. In some embodiments, the method also includes the step of restoring the PTS interval to a specified resolution. In some cases, the identified integrated time scale allows media items to be streamed at a variable frame rate while maintaining a constant frame interval for each frame rate. In some embodiments, media items having different frame rates are streamed at a variable frame rate while maintaining a constant frame interval for each frame rate using the integrated time scale. In some examples, media items having video content captured using a variable refresh rate are streamed at a variable frame rate while maintaining a constant frame interval using the integrated time scale.

[0012] In some embodiments, the step of identifying an integrated time scale includes converting input presentation time stamps from different media items having different time scales into PTSs based on the integrated time scale. In some cases, implementing the converted input PTSs avoids PTS counter wrap-around. In some examples, changing at least one of the media items from the current time scale to the identified integrated time scale allows a single fixed V-Synch interrupt to be implemented during the playback of multiple media items. In some cases, the method further includes the step of optimizing the PTSs for the media items so that the scaled presentation time stamps match the unique PTSs without resulting rounding error.

[0013] Additionally, the corresponding system includes at least one physical processor and physical memory containing computer-executable instructions, wherein when the computer-executable instructions are executed by the physical processor, the physical processor determines, for each of a plurality of different media items, a current time scale in which the media items are encoded for distribution—at least two of the plurality of media items are encoded at different frame rates—identifies, for the plurality of media items, an integrated time scale that provides a constant frame interval for each of the plurality of media items, and changes at least one of the plurality of media items from the current time scale to the identified integrated time scale to provide a constant frame interval for at least one changed media item.

[0014] In some examples, the method described above may be encoded as computer-readable instructions on a computer-readable medium. For example, the computer-readable medium may include one or more computer-executable instructions, and when the one or more computer-executable instructions are executed by at least one processor of a computing device, the computing device may determine, for each of a plurality of different media items, a current time scale at which the media items are encoded for distribution—at least two of the plurality of media items are encoded at different frame rates—identify, for the plurality of media items, an integrated time scale that provides a constant frame interval for each of the plurality of media items, and change at least one of the plurality of media items from the current time scale to the identified integrated time scale to provide a constant frame interval for at least one changed media item.

[0015] Features from any of the embodiments described herein may be used in combination with one another in accordance with the general principles described herein. These and other embodiments, features, and advantages will be more fully understood when reading the following detailed description together with the accompanying drawings and claims. Brief explanation of the drawing

[0016] The attached drawings illustrate a number of exemplary embodiments and are part of the specification. Together with the following description, these drawings illustrate and explain the various principles of the present disclosure. Figure 1 illustrates a computing environment in which accurate presentation time stamps and optimized time scales are provided. Figure 2 is a flowchart of an exemplary method for providing optimized time scales and accurate presentation time stamps in media items. Figure 3 is a flowchart illustrating time-scale information in media items at different levels. Figure 4 illustrates a table showing different time frame intervals for media items encoded at different frame rates. Figure 5 illustrates a table highlighting the occurrence of presentation time stamp rounding errors that cause different frame interval lengths. Figure 6 illustrates a table showing how V-synchrony is related to rounded PTS and non-rounded PTS. Figure 7 illustrates a table showing video frame intervals for media items encoded at different frame rates. Figure 8 illustrates a table showing audio frame intervals for media items encoded at different frame rates. Figure 9 illustrates an alternative table showing audio frame intervals for media items encoded at different frame rates. Figure 10 illustrates a table showing audio / video frame intervals for sample rates up to 96 kHz. Figure 11 illustrates a table showing scaled frame rates for a video encoded at 23.976fps. Figure 12 illustrates a table showing an exemplary time scale for 48kHz audio. Figure 13 illustrates a table showing an exemplary time scale for audio at 44.1 kHz. Figure 14 illustrates a table showing an exemplary time scale and wrap-around time for 48kHz audio. Figure 15 illustrates a table showing an example of presentation time stamp scaling. FIG. 16 illustrates an embodiment in which the minimum interval is calculated and implemented. Figure 17 illustrates a table showing optimized presentation time stamps with an initial PTS starting from 0. FIG. 18 illustrates an embodiment in which the frame rate is changed for video segments with different frame rates. Figure 19 is a block diagram of an exemplary content distribution ecosystem. FIG. 20 is a block diagram of an exemplary distribution infrastructure within the content distribution ecosystem illustrated in FIG. 19. FIG. 21 is a block diagram of an exemplary content player within the content distribution ecosystem illustrated in FIG. 19. Throughout the drawings, the same reference characters and descriptions indicate elements that are similar but not necessarily identical. While various modifications and alternative forms are possible for the exemplary embodiments described herein, specific embodiments are illustrated by way of example in the drawings and will be described in detail herein. However, the exemplary embodiments described herein are not intended to be limited to the specific forms disclosed. Rather, the present disclosure covers all modifications, equivalents, and alternatives that fall within the scope of the appended claims. Specific details for implementing the invention

[0017] The present disclosure generally relates to providing optimized time scales and accurate presentation time stamps in media items. As described in more detail below, embodiments of the present disclosure generate an integrated time scale that provides constant frame intervals during media item playback, even if the media items are encoded at different frame rates.

[0018] As previously mentioned, media items are typically encoded with presentation time stamps (PTS). Presentation time stamps provide points where different underlying streams can be synchronized. These underlying streams include audio streams, video streams, and subtitles. If any of these underlying streams are ahead or behind each other during the playback of a media item, users will typically perceive a lack of synchronization and may be hindered by a poor playback experience.

[0019] The presentation time stamp itself is an N-bit number representing the value of a counter driven by a clock at a specified time scale measured in Hertz (Hz). For MPEG 1 and 2, N=33 bits and X=90,000 Hz. Different time scales are used for media items encoded at different frame rates. For example, video can be encoded at 24 frames / second (fps), 29.97fps, 30fps, 50fps, 59.94fps, 60fps, or other frame rates. The number of ticks for the duration each frame is displayed varies across different time scales. For example, at a frame rate of 24fps and a time scale of 1,000 ticks / second, the duration each frame is displayed (i.e., the "frame interval") would be 1,000 / 24 ​​= 41.66667ms. Therefore, in this example, a video encoded at 24fps will have a frame interval of 41ms or 42ms. If the time scale is changed, for example, to 90,000Hz, the frame interval will be much more accurate at (90,000 / 24) or 3750 ticks. For example, if the time scale is increased to 1,000,000 ticks / second, for example, the frame interval will be 41,666 or 41,667 microseconds (1,000,000 / 24).

[0020] Since the values ​​of presentation time stamps are presented as integers, these fractions (e.g., 41.66667ms, 41,666.6667us, etc.) need to be rounded up or down to the nearest integer. This rounding process to the nearest integer introduces rounding errors into the presentation time stamp. These rounding errors cause the frame interval (i.e., the duration for which each frame is displayed) to shift up or down. For example, in the case of video encoded at 24fps with a time scale of 1,000 units, a frame interval of 41.66667ms consequently continuously switches between 41ms and 42ms as 41.66667 is rounded up or down to maintain synchronization. This continuous shifting between different frame intervals can have an adverse effect on the playback of media items. For instance, the shifting between frame intervals can tire the user's eyes. Furthermore, in at least some cases, such shifting between different frame intervals may cause other problems including frame drops, frame freezes, repeated frames, and blank frames, or lead to other playback or synchronization problems that are noticeable to the user and impair the user's viewing experience.

[0021] Meanwhile, the embodiments described herein are designed to provide a universal time scale applicable to various and different frame rates. The universal time scale ensures that each frame of a media item is played at a constant frame interval. The systems described herein identify the current time scale at which a set of media items (e.g., audio or video media items) is encoded. These systems also determine the frame rate at which the media items are encoded. Subsequently, the system determines the minimum common multiplier of the various frame rates used in the media items. For example, in the integrated time scale determined by identifying the minimum common multiplier of the scaled frame rates for a group of media items, each media item will be divided into the same number of ticks per second. This integrated number of ticks per second yields constant frame intervals for each media item, regardless of the frame rate at which the media item is encoded. By identifying the minimum common multiplier applied to each scaled frame rate used in a group of media items, each frame interval can be presented as a whole integer value. By providing the frame interval as a whole integer value for each media item, the frame intervals no longer vary between different values ​​(e.g., the variation between 41ms and 42ms in the example above). And, because the frame interval is kept constant by the integrated time scale, problems associated with varying frame intervals are suppressed or completely eliminated.

[0022] In some cases, the embodiments described herein may also optimize the integrated time scale for specific audio or video format combinations. For example, instead of finding the minimum common multiplier for a wide range of different media items encoded at different frame rates, the systems described herein find the minimum common multiplier for a selected audio / video format combination and specifically modify the time scale for that combination. This reduces the likelihood of memory buffers configured to store presentation time stamp values ​​overflowing. Indeed, as previously discussed, the presentation time stamp is a monotonically increasing integer value that, at some point, grows to a very large number. If the bit field for storing the PTS is too low, the monotonically increasing PTS value will potentially overfill the buffer very early in the playback of the media item. By optimizing the time scale for a specific audio / video format combination, a smaller time scale may be utilized where the frame intervals are longer and the number of increments in the presentation time stamp counter is correspondingly lower. By optimizing the time scale for each media item or for each specific group of media items, the embodiments of this specification provide the viewer with a smoother and more reliable playback experience that is less susceptible to memory overrun errors.

[0023] The following provides detailed descriptions of various tables, examples, and flowcharts with reference to FIGS. 1 through 21. FIG. 1 illustrates, for example, a computing environment (100) in which an integrated time scale is identified and applied to a media item. FIG. 1 includes various electronic components and elements, including a computer system (101) used alone or in combination with other computer systems to perform tasks associated with storing digital content. The computer system (101) may be substantially any type of computer system, including a local computer system or a distributed (e.g., cloud) computer system. The computer system (101) includes at least one processor (102) and at least a predetermined system memory (103). The computer system (101) includes program modules that perform various and different functions. The program modules may be hardware-based, software-based, or include a combination of hardware and software. Each program module performs specified functions, including those described below, using computing hardware and / or software.

[0024] In some cases, the communication module (104) is configured to communicate with other computer systems. The communication module (104) includes substantially any wired or wireless communication means capable of receiving and / or transmitting data to / from other computer systems. Such communication means include, for example, hardware radios such as a hardware-based receiver (105), a hardware-based transmitter (106), or a combined hardware-based transceiver capable of receiving and transmitting data. The radios may be Wi-Fi radios, cellular radios, Bluetooth radios, Global Positioning System (GPS) radios, or other types of radios. The communication module (104) is configured to interact with databases, mobile computing devices (e.g., mobile phones or tablets), embedded computing systems, or other types of computing systems.

[0025] The computer system (101) further includes a determination module (107). The determination module (107) is configured to determine the current frame rate (123) of various media items (122). In some cases, these media items (122) are selected from various stored media items (121) held in a data store (120). The data store (120) may be local or remote, and may be a distributed (e.g., cloud-based) data store. The media items (122) include audio items, video items, or any type of media item encoded for playback on a user device (e.g., the device (116) of the user (115)). The determination module (107) is configured to access these media items (122) and determine the frame rate (123) at which they are encoded. The frame rate defines how many frames are designed to be presented on the playback device per second. In some cases, for example, video media items are encoded at 23.97, 24, 25, 29.97, 30, 59.94, 60, 120, 240, or 300 frames per second. Audio frame rates may include 23.4375 frames per second, 31.25 frames per second, or any other encoded frame rate.

[0026] The determination module (107) also determines the current time scale (108) used with the media items (122). The time scale is measured in ticks or per second. In cases where the time scale is 1,000, for example, the time scale divided by the frame rate (e.g., 24 fps) (1000 / 24 ​​= 41.6667 ms) yields the frame interval, or the amount of time each frame is displayed on the playback device. Many different time scales may be used with different types of media items. Accordingly, the determination module (107) determines both the encoded frame rate (123) of the media item and the current time scale (108) used with that media item to determine the frame interval (time scale / frame rate) of each media item.

[0027] Subsequently, the identification module (109) of the computer system (101) identifies or calculates an integrated time scale (110) for media items (122). The integrated time scale (110), when applied to media items having different encoded frame rates (123), produces constant frame intervals. As previously mentioned, continuous rounding of non-integer values ​​(e.g., 41.6667ms) to integers results in added frames, dropped frames, blank frames, and other playback errors. These errors are noticeable to users, cause eye strain and overuse, and impair a smooth and uninterrupted playback experience. The time scale change module (111) takes the identified integrated time scale (110) and changes one or more of the media items (122) to produce modified media items (112) that are ready to be played using the integrated time scale (110). Then, when a user (e.g., 115) selects a media item to stream or download (e.g., from a data store (120)) via input (117), the media item will be changed to an integrated time scale that provides a constant frame interval. The constant frame interval, in turn, provides a smooth and reliable playback experience, without blank frames, dropped frames, added frames, or other noticeable defects. These embodiments will be described in more detail below in relation to the method (200) of FIG. 2 and the embodiments illustrated in FIG. 3 through 21.

[0028] FIG. 2 is a flowchart of an exemplary computer-implemented method (200) for providing optimized time scales and accurate presentation time stamps in media items. The steps illustrated in FIG. 2 may be performed by any suitable computer-executable code and / or computing system, including the system illustrated in FIG. 1. In one example, each of the steps illustrated in FIG. 2 represents an algorithm whose structure includes a plurality of sub-steps and / or is represented by a plurality of sub-steps, examples thereof will be provided in more detail below.

[0029] As illustrated in FIG. 2, in step (210), one or more of the systems described herein may determine, for each of a plurality of different media items (e.g., 122 in FIG. 1), a current time scale (108) in which the media items are encoded for distribution. At least two of these media items (122) are encoded at different frame rates. Next, in step (220), the systems described herein identify a combined time scale (110) for the media items that provides a constant frame interval for each of the media items (122). Then, in step (230), the systems change at least one of the media items from the current time scale (108) to the identified combined time scale (110) to provide a constant frame interval for the changed media item(s).

[0030] As illustrated in FIG. 3 and as described above, time stamps are periodically applied to each video and audio frame to keep audio and video playback in a synchronized state (e.g., lip-sync). As described above, a time stamp is an N-bit number representing the value of a counter driven by a clock at a time scale of X-Hz. Decoders (e.g., 303) do not necessarily need to obtain time scale information and presentation time stamps directly from the encoding / streaming container (301). Rather, in at least some cases, this timing information is provided by or determined by a media player / demux device. The player / demux (302) may perform a scaling process to provide an integrated time scale for ease of implementation and control. The video decoder (303) and / or audio decoder (304) will use time information from the player / demux (302) if available. Otherwise, the audio or video decoders will use the basic stream time information. If neither is available, the hardware-based decoder will attempt to play the media item without time information. As such, in at least some cases, the player / demux (302) time information has the highest priority, while the basic stream time information has a lower priority.

[0031] Presentation time stamps indicate where a specific access unit (AU) belongs in time. When a decoder (e.g., 303 or 304) receives video and audio AUs, the decoder decodes each AU and stores it in a memory buffer. When the reference clock reaches the value of the time stamp, the decoded AU is rendered on a playback device (e.g., 116 in FIG. 1). The PTS controls how the AU is rendered. Video frame rates vary from 23.976fps, 24fps, 29.97fps, 30fps, 50fps, 59.94fps to 60fps or other frame rates. The presentation time stamp is used by the V-synchronous interrupt handler to determine whether the corresponding audio / video frame should be rendered. One rule for rendering AUs is to select the nearest AU with a PTS value smaller than It (PTS' < Ti), where Ti is the system time of the presentation device's V-synchronous interrupt or audio serial output interrupt, and PTS' = PTS + is, is the tolerance. ≥0). If PTS'>Ti, the currently rendered AU is repeated. For the sake of simplicity, at least some of the embodiments of this specification are Assume =0.

[0032] Presentation time stamps perform a similar role in audio signals using serial output interrupt handlers. Audio frequencies are, in at least some examples, 32,000 Hz, 48,000 Hz, or 96,000 Hz. The 44.1 kHz domain is primarily used for delivering high-quality music. In at least some cases, video V-sync interrupts and audio serial output interrupts are not aligned during rendering. As such, to achieve proper A / V synchronization, both video and audio presentation time stamps are referenced to a common clock (e.g., system time) having the same time scale. Some implementations may use the audio clock as system time, which may be referred to as an audio master A / V synchronization scheme.

[0033] Presentation time stamp precision is defined by the time scale (e.g., the current time scale (108)). PTS precision may be presented in the format (number_of_ticks, time_scale). The time scale is the total number of ticks per second. The frame interval (i.e., the duration or amount of time each frame is presented on the playback device) is obtained by calculating time_scale / frame_rate. Table (400) in FIG. 4 illustrates multiple different frame intervals for media items encoded at different frame rates (402) presented on different time scales (401). As can be seen, time scales of 1,000, 90,000, or 1,000,000 yield fractional frame intervals, particularly for frame rates such as 23.97fps and 59.94fps. Since PTS is presented as an integer, these fractions cause rounding errors in the actual presentation time stamps. Table (500) of FIG. 5 illustrates presentation time stamp values ​​for the first 10 frames of a 23.97fps video with a time scale value of 1,000 (equivalent to milliseconds). Each of the frame numbers 0-9 (501) has a corresponding source time stamp (502) starting from 0, which proceeds every 41ms or 42ms (as shown in 503), which results in an uneven and continuously varying frame interval (504) that varies between 42ms and 41ms.

[0034] From the table (500), the following issues can be identified: with a rounding error of up to 0.5 ms, the rounding error will affect the accuracy of the frame rate conversion (e.g., frame rendering condition: Ti > PTS). The table (600) in FIG. 6 shows that under the rendering condition (Ti > PTS), frame #35 (column 602) should be represented by three V-sync slots (column 601) from 1460 ms to 1500 ms. However, due to the rounding error, frame (34) is instead represented three times (column 605) from 1420 ms to 1460 ms. Similar irregularities will occur throughout the playback of the entire media item due to irregularities that occur when rounding PTS values ​​to the nearest integer (see the differences between PTS values ​​without rounding (column 603) and PTS values ​​with rounding (column 604)).

[0035] Table (500) shows how the frame interval varies between 41ms and 42ms. These irregular frame intervals disrupt the regularity of the frame rate of the media item. The irregular frame intervals also prevent hardware or software implementations using continuous timers from waking up the rendering to a frame rate of the media item with sufficient accuracy to provide smooth and reliable playback. This lack of accuracy in the PTS time scale, which causes fractional frame intervals, leads to many problems (as mentioned above), at least some of which can be overcome by using a unified time scale.

[0036] The embodiments described herein provide a solution for achieving constant frame intervals for different video frame rates (e.g., 23.97, 24, 25, 29.97, 30, 50, 59.94, 60, 120, 240, or 300 fps), where (frame interval = time scale / frame rate). In at least some embodiments, this may be equivalent to calculating the least common multiplier (LCM) of all frame rates within a set of media items (e.g., LCM(24, 25, 30, 60, 120, 240, 24000, 30000, 60000)). This example is reduced to LCM(24,000, 60,000) = 120,000. In some cases, the LCM can be obtained from a lookup table that pre-calculates LCM values ​​for different frame rate combinations. Accordingly, as shown in the table (700) of FIG. 7 and as shown in the “Time” row (701), when the time scale changes from 1,000, 90,000, or 1,000,000 ticks / second to 120,000 ticks / second, the frame interval for each frame rate (shown in the “Scale Frame Rate” column) will be an integer. For example, the frame interval for a 24fps encoded video is 5,000ms, the frame interval for 59.94fps is 2,002ms, and the frame interval for a 120fps encoded video is 1,000ms. As can be seen, each of these is an integer value that provides a constant frame duration. This constant frame duration avoids blank frames, dropped frames, duplicate frames, playback errors or irregularities, and other anomalies that occur when different frame durations exist during playback.

[0037] For audio media items, HE-AAC encoding has a frame rate of 2048 or 1024 samples / frame, and at an audio sample rate of 48KHz, the frame rate is 23.4375 or 46.875fps, respectively. For Dolby Digital Atmos, encoding has 1536 samples / frame and is encoded at 31.25fps. Table (800) in FIG. 8 illustrates a chart similar to that of FIG. 7, except for audio items. In Table (800), a frame rate of 23.4273 (802) yields fractions for time scales (801) of 1,000 and 1,000,000. However, when an integrated time scale of 120,000 ticks / second is used, frame rates of 23.4273fps and 31.25 yield frame intervals of 5120 ticks and 3840 ticks, respectively. As shown in Table (900) of FIG. 9, at an audio sample rate of 32KHz, time scales of 1000, 90,000, and 120,000 provide constant frame intervals for the illustrated frame rates (902) and time scales (901).

[0038] In some embodiments, 44,100Hz audio is used with video access units. In these cases, the time scale is much larger (17,640,000) to cover all scenarios. Accordingly, as shown in Table (1000) of FIG. 10, various sample rates are shown (1001), along with frame size (1002), frame interval (1003), and different time scales (1004-1008). As can be seen from Table (1000), a time scale with a value of 120,000 provides a fixed frame interval for the video / audio formats in the list, except for 44,100Hz audio. When 44,100Hz audio is provided, a time scale with a value of 17,640,000 would be universal to produce fixed frame intervals for each of the access unit formats shown in Table (1000).

[0039] In some embodiments, the systems of this specification focus on a single video frequency and audio sample rate. In these cases, the following algorithm may be implemented to determine a minimum time scale capable of generating a fixed frame interval for audio and video. The algorithm comprises 1) calculating the greatest common divisor (D) of (audio sample rates, frame sizes). For example, at a sample rate of 48,000 Hz, the frame sizes are 1536 (DDP) and 2048 (HE-AAC), and D = GCD(48000, 1536, 2048) = 128. 2) A = audio sample rate / D. From the above example, for a sample rate of 48,000 Hz, A = 48,000 / 128 = 375. 3) Find the scaled video frame rate V from the table (11) of FIG. 11, which shows different media item frame rates in column (1101) and scaled frame rates in column (1102). 4) Time scale = LCM (V, A). For example, if the audio frequency is 48,000 Hz, the time scale can be calculated using the frame rate (1201) and the scaled frame rate (1202) as shown in the table (1200) of FIG. 12. In this embodiment, a single integrated time scale is used for each of the available media item formats.

[0040] (For example, in MPEG 1 / 2) because 33 bits are allocated for the PTS 33 bits, in at least some embodiments, it may be easier to obtain an unsigned 33-bit integer wrap-around (overflow) when the time scale value is larger. At a time scale of 90,000 ticks / second, it will take about 26 hours to wrap the PTS (i.e., overflow the memory buffer). Using a time scale of 120,000 ticks / second, it will take about 19 hours to wrap the PTS. However, at a time scale of 17,640,000 ticks / second, it takes only about 8 minutes. In at least some embodiments, it may be desirable to calculate the time scale for each specific video frame rate for 44.1KHz audio. For 44.1KHz audio, A = 11,025, as shown in column (1303) of the table (1300) of FIG. 13. Other frame rates (1301), scaled frame rates (1302), and time scale values ​​(1303) are shown in the table (1300).

[0041] PTS is presented as an integer value with a fixed number of bits. Thus, as previously described, when PTS increases monotonically, a wrap-around occurs, where the PTS value will overrun its (e.g., 33 bits) memory buffer. In one embodiment, AV1 provides an optional timing information container and is defined in a 32-bit field. For 48KHz audio, the algorithm described above can be implemented to obtain the corresponding wrap-around duration for the time scale values ​​(1203) in the table (1200) and the unsigned 32-bit values. For comparison, when the time scale is 90,000 ticks / second, using 32 bits, the wrap-around time is approximately 13 hours. In embodiments where wrap-around time optimization is the goal, these embodiments select an optimized time scale based on the video frame rate. Next, using the time scale (1403) of the table (1400) of FIG. 14, the resulting wrap-around time (1404) for each selected frame rate (1401) and scaled frame rate (1402) is shown.

[0042] In some embodiments, a higher resolution PTS may be provided to avoid frame drops or freezing during playback. When the input PTS to the decoder or system is at a lower resolution, a scaling approach may be used to convert the PTS from one time scale (TS) to another time scale (TS') using the following formula: PTS' = PTS * TS' / TS. However, in at least some cases, the PTS precision may be difficult to recover. For example, as shown in the table (1500) of FIG. 15, TS = 1,000 and TS' = 1,000,000. As can be seen, for each frame # (1501), a PTS (1502) is provided along with a scaled PTS (1503) and an intrinsic PTS (1504). Each of these frames has a resulting rounding error (1505). Since the system knows that the minimum frame interval is constant based on the frame rate, the PTS interval is the minimum frame interval (D) or a multiple of the minimum interval (N x D). For example, given a time scale of 120,000, the minimum frame interval is D = 5005 ticks for a frame rate of 23.97 fps. The actual frame interval must be 5005 ticks or a multiple of 5005 ticks, as illustrated in the embodiment (1600) of FIG. 16 showing the minimum interval (D) between frame (1601) and frame (1602) and (N) times (D) (10,010 between frame (1602) and frame (1603).

[0043] In some embodiments, an algorithm for optimizing or remastering presentation timestamps may be implemented. In this example, PTS in0 Present the PTS for this first frame, and the first PTS out0 It is said to be scaling, and here PTS out0 = PTS in0 * TS out / TS in am.

[0044] The algorithm may include the following steps: 1) frame rate (F) and input / output time scale (TS in , TS out Based on ), frame interval (D in / D out Calculate ), where D in = TS in / F and, D out = TS out / F. 2) Rounding function: I = round ((PTS in - PTS in0 Current PTS as ) / Din) in Retrieves the frame index (I) based on . 3) Optimized / remastered PTS is PTS out = I * D out + PTS out0 am.

[0045] PTS in If starting from this 0, the above algorithm can reconstruct the PTS at practically arbitrary resolution (at least in some cases, without defects). Otherwise, all PTS out PTS transmitted through out0 A constant initial offset may exist from. As shown in the table (1700) of FIG. 17, the remastered PTS represents a rounding error (1705) of 0 for each of the frame numbers (1701), PTS values ​​(1702), remastered PTS values ​​(1703), and unique PTS values ​​(1704). Having a rounding error of 0 indicates that each remastered PTS value will cause a constant frame interval at a given appropriate output time scale, thus mitigating the problems identified in media items with fluctuating frame intervals.

[0046] In other embodiments, an integrated time scale provides the basis for variable frame rate streaming. There are multiple scenarios in which variable frame rate streaming can be utilized. For example, video segments having different frame rates are taken. An integrated time scale is used for seamless frame rate transitions to avoid black or blank displays. In this case, the different video segments have different time scales, for example, for wrap-around time optimization. Another embodiment is variable refresh rate (VRR) content capture and streaming. This use case occurs at the source encoding end, and the time scale must be consistent across different frame rates. In this case, from Table (700), a time scale of 120,000 ticks / second is implemented to time scale VRR content capture and streaming, because this provides frame intervals of full integer values ​​for each frame rate in Table (700).

[0047] In the case of Example 1, since PTS is given as a pair (tick, time scale), the embodiments of this specification provide an integrated time scale (TS) for rendering on client playback devices where the frame rate F0 is changed to F1. out The following algorithm is defined to provide PTS information based on ). Initially, tick in0 This first frame TS in = TS in0 , present PTS for F = F0, and the first tick out0 It is said to scale, and here, tick out0 = tick in0 * TS out / TS in...is. Subsequently, the system uses the following routine to convert (tick, time scale) into a new tick with an integrated time scale: 1) Frame rate (F) and input / output time scale (TS in , TS out Based on ), frame interval (D in / D out Calculate ), where D in = TS in / F and, D out = TS out / F. 2) Rounding function: I = round ((tick in - tick in0 Current tick as ) / Din) in Retrieves the frame index (I) based on . 3) The remastered tick is tick out = I * D out + tick out0 is. 4) New time scale (TS in1 When ) is updated with a frame rate change (F ≠ F1) (TS in ≠ TS in ), TS in = TS in1 , tick in0 = tick out * TS in / TS out , tick out0 = tick out , set F = F1.

[0048] Thus, even if different video segments have different time scales, the algorithm described above converts the input PTS into a more accurate PTS at an integrated time scale. Since the above process is executed after encoding on a client playback device in at least some embodiments, the process can use a higher precision time scale as the integrated time scale (e.g., having 64-bit precision) without needing to monitor wrap-around. For this reason, in at least some embodiments, a time scale of 120,000 ticks / second represents an optimized choice.

[0049] As mentioned above with reference to FIGS. 1 and 2, methods and systems configured to determine a current time scale (108) in which each media item is encoded for distribution for different media items (e.g., 122). These methods and systems are also configured to identify or calculate a combined time scale (110) that provides a constant frame interval for each of the media items, and then change at least one of the media items from the current time scale (108) to the combined time scale (110) to provide a constant frame interval for the changed media item(s) (112). In some cases, these media items are video media items encoded at different frame rates (e.g., 23.97, 24, 25, 29.97, 30, 59.94, 60, 120, 240, or 300 frames per second), whereas in other cases, the media items are audio media items with different frame rates (e.g., 1024, 1536, or 2048 samples per frame). In some examples, the media items are interactive media items that allow users to make decisions that affect the order of scenes during playback.

[0050] In some embodiments, a specific group of media items is selected. This group of media items may include video items, audio items, or a combination thereof. At least some of the media items within the group have different specified video frame rates and / or audio frame rates, at least in this example. In these cases, a maximum range of frame rates (e.g., from 24fps to 60fps) is known for the group. Subsequently, when the identification module (109) of FIG. 1 calculates or otherwise identifies an integrated time scale (110), it is not necessary to examine a plurality of possible frame rates (in this case, greater than 60fps and less than 24fps), and can focus on frame rates between 24 and 60fps. Thus, the identification module (109) can calculate or otherwise identify an integrated time scale (110) that is specific to and optimized for this group of audio and / or video items. In these cases, the least common multiple may be much smaller than for other groups of audio and / or video items, and thus, the integrated time scale (110) will have fewer ticks / second while still providing a constant frame interval for various frame rates (in this example, between 24 and 60 fps). This smaller number of ticks / second will reduce the chance of overrunning PTS memory buffers by reducing the wrap-around time of monotonically increasing PTS values. Once the integrated time scale (110) is determined for a selected group of media items, the time scale change module (111) will change the media items to the integrated time scale (110) by generating new presentation time stamps for those media items.

[0051] In some embodiments, when presentation time stamps are monotonically increasing, the unit used in the integrated time scale (i.e., number of ticks / second) is selected to maximize the wrap-around time for the media items. Maximizing the wrap-around time involves selecting the minimum number of units in the integrated time scale (110) that, at least in some cases, still allow each media item to have a constant frame interval. Having a smaller number of units allows the monotonically increasing PTS value to increase more slowly. This slower monotonically increasing makes it possible to maximize the amount of time required before reaching the wrap-around time. In some cases, the unit selected to maximize the wrap-around time for the media items is selected based on the video frame rate of the video items, whereas in other cases, the unit selected to maximize the wrap-around time for the media items is selected based on the audio frame rate of the audio items.

[0052] In some cases, the identified integrated time scale (110) includes a presentation time stamp interval. The PTS interval may differ from the minimum constant frame interval. The PTS interval includes the minimum frame interval or a multiple of the minimum frame interval. As illustrated in FIG. 16, for example, the interval between frames (1601 and 1602) is 5005 ms, which represents the minimum frame interval between frames (1601 and 1602). On the other hand, the frame interval between frames (1602 and 1603) is a multiple of 5005 (in this case, 5005 x 2 or 10010 ms). In some cases, the PTS interval may be restored to a specified resolution or remastered. In these cases, the PTS interval is calculated as the minimum constant frame intervals or as multiples of the minimum frame intervals and applied to the media item. Accordingly, the remastering process includes determining a constant frame interval, using the determined constant frame interval to change the input time resolution (e.g., milliseconds, microseconds, nanoseconds, etc.), and outputting a different time resolution to ensure that the scaled PTS interval matches the original PTS interval without rounding error. Subsequently, the remastered PTS interval is applied to each frame within the media item. Thus, as illustrated in FIG. 17, the remastered PTS (1703) will have the same PTS interval as the original PTS (1704), thereby eliminating rounding error (1705) that would otherwise lead to skipped frames, added frames, blank frames, or other similar playback errors.

[0053] In some embodiments, the identified integrated time scale (110) allows multiple media items to be streamed at variable frame rates while maintaining a constant frame interval. In some cases, the same media item may even be switched between different frame rates. For example, if the media item is an interactive video, parts of the video may be encoded at different frame rates on the server (distribution) side. In these cases, the server will provide the interactive media item at different frame rates for playback on the user's playback device (e.g., 116). Then, the playback device (116) calculates and / or applies an integrated time scale (110) that enables the media items to be streamed at variable frame rates. Then, this integrated time scale results in a constant frame interval for the media items, including for the interactive or other media items encoded at different frame rates. Thus, in these embodiments, media items having different frame rates can be streamed at variable frame rates while maintaining a constant frame interval using the integrated time scale. Similarly, in at least some embodiments, media items having video content captured using a variable refresh rate are streamed (from a server) at a variable frame rate and played back at constant frame intervals using an integrated time scale (110).

[0054] In some cases, identifying or calculating the integrated time scale (110) involves converting input presentation time stamps from different media items having different time scales into PTS based on the integrated time scale. As previously mentioned, this is generally illustrated in FIG. 17, where the remastered PTS values ​​are designed to eliminate original rounding errors and are identical to the original PTS values. In some examples, using these converted input PTSs avoids PTS counter wrap-arounds or at least increases the amount of time before a wrap-around occurs. Reducing the occurrence of wrap-arounds also reduces the various measures that need to be taken to ensure that wrap-arounds are processed smoothly without introducing additional playback errors. Thus, in this way, providing the integrated time scale ensures that each frame of the media item will have the same frame interval during playback. This is true regardless of whether the source encoding is encoded at the same frame rate overall or whether the source encoding is changed overall. Providing regular time intervals avoids many common playback issues and offers users a smoother and more reliable (and therefore more enjoyable) media playback experience.

[0055] In some embodiments, a corresponding system is provided comprising at least one physical processor and physical memory including computer-executable instructions, wherein when the computer-executable instructions are executed by the physical processor, the physical processor determines, for each of a plurality of different media items, a current time scale in which the media items are encoded for distribution—at least two of the plurality of media items are encoded at different frame rates—identifies, for the plurality of media items, an integrated time scale that provides a constant frame interval for each of the plurality of media items, and changes at least one of the plurality of media items from the current time scale to the identified integrated time scale to provide a constant frame interval for at least one changed media item.

[0056] In other embodiments, a non-transient computer-readable medium is provided that includes one or more computer-executable instructions, wherein when the one or more computer-executable instructions are executed by at least one processor of a computing device, the computing device determines, for each of a plurality of different media items, a current time scale in which the media items are encoded for distribution, and for each of the plurality of media items, at least two of the plurality of media items are encoded at different frame rates, and for each of the plurality of media items, an integrated time scale that provides a constant frame interval for each of the plurality of media items, and changes at least one of the plurality of media items from the current time scale to the identified integrated time scale to provide a constant frame interval for at least one changed media item.

[0057] Next, with reference to FIG. 19, detailed descriptions of exemplary ecosystems will be provided in which content is served to end nodes and requests for content are directed to specific end nodes. The discussion corresponding to FIG. 20 and FIG. 21 presents an overview of an exemplary distribution infrastructure and an exemplary content player used during playback sessions, respectively. These exemplary ecosystems and distribution infrastructures are implemented in any of the embodiments described above with reference to FIG. 1 through 18.

[0058] FIG. 19 is a block diagram of a content distribution ecosystem (1900) comprising a distribution infrastructure (1910) communicating with a content player (1920). In some embodiments, the distribution infrastructure (1910) is configured to encode data at a specific data rate and transmit the encoded data to the content player (1920). The content player (1920) is configured to receive the encoded data through the distribution infrastructure (1910) and to decode the data for playback to a user. The data provided by the distribution infrastructure (1910) includes, for example, audio, video, text, images, animations, interactive content, haptic data, virtual or augmented reality data, location data, game data, or any other type of data provided via streaming.

[0059] The distribution infrastructure (1910) generally represents any services, hardware, software, or other infrastructure components configured to deliver content to end users. For example, the distribution infrastructure (1910) includes a content aggregation system, media transcoding and packaging services, network components, and / or various other types of hardware and software. In some cases, the distribution infrastructure (1910) is implemented as a highly complex distribution system, a single media server or device, or any in between. In some examples, regardless of size or complexity, the distribution infrastructure (1910) includes at least one physical processor (1912) and at least one memory device (1914). As discussed herein, one or more modules (1916) are stored or loaded in memory (1914) to enable adaptive streaming.

[0060] A content player (1920) generally represents any type or form of device or system capable of playing audio and / or video content provided through a distribution infrastructure (1910). Examples of content players (1920) include, without limitation, mobile phones, tablets, laptop computers, desktop computers, televisions, set-top boxes, digital media players, virtual reality headsets, augmented reality glasses, and / or any other type or form of device capable of rendering digital content. As with the distribution infrastructure (1910), the content player (1920) includes a physical processor (1922), memory (1924), and one or more modules (1926). Some or all of the adaptive streaming processes described in this specification are performed or enabled by modules (1926), and in some examples, modules (1916) of the distribution infrastructure (1910) cooperate with modules (1926) of the content player (1920) to provide adaptive streaming of digital content.

[0061] In certain embodiments, one or more of the modules (1916 and / or 1926) of FIG. 19 represent one or more software applications or programs that, when executed by a computing device, cause the computing device to perform one or more tasks. For example, as described in more detail below, one or more of the modules (1916 and 1926) represent modules stored and configured to be executed on one or more general-purpose computing devices. One or more of the modules (1916 and 1926) of FIG. 19 also represent all or part of one or more special-purpose computers configured to perform one or more tasks.

[0062] Additionally, one or more of the modules, processes, algorithms, or steps described herein convert data, physical devices, and / or representations of physical devices from one form to another. For example, one or more of the modules described herein receive audio data to be encoded, convert the audio data by encoding it, output the result of the encoding for use in an adaptive audio bitrate system, transmit the result of the conversion to a content player, and render the converted data to an end user for consumption. Additionally, or alternatively, one or more of the modules described herein execute on a computing device and convert a processor, volatile memory, non-volatile memory, and / or any other part of a physical computing device from one form to another by storing data on the computing device or interacting with the computing device in a different way.

[0063] Physical processors (1912 and 1922) generally represent any type or form of hardware implementation processing unit capable of interpreting and / or executing computer-readable instructions. In one example, physical processors (1912 and 1922) each access and / or modify one or more of the modules (1916 and 1926). Additionally or alternatively, physical processors (1912 and 1922) execute one or more of the modules (1916 and 1926) to facilitate adaptive streaming of digital content. Examples of physical processors (1912 and 1922) include, but are not limited to, microprocessors, microcontrollers, central processing units (CPUs), field programmable gate arrays (FPGAs) implementing softcore processors, application-specific integrated circuits (ASICs), parts of one or more of these, variations or combinations of one or more of these, and / or any other suitable physical processor.

[0064] Memory (1914 and 1924) generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, memory (1914 and / or 1924) stores, loads, and / or retains one or more of the modules (1916 and 1926). Examples of memory (1914 and / or 1924) include, without limitation, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drives (HDDs), solid-state drives (SSDs), optical disk drives, caches, variations or combinations of one or more of these, and / or any other suitable memory device or system.

[0065] FIG. 20 is a block diagram of exemplary components of a content distribution infrastructure (1910) according to specific embodiments. The distribution infrastructure (1910) includes a repository (2010), services (2020), and a network (2030). The repository (2010) generally represents any device, set of devices, and / or systems capable of storing content for delivery to end users. The repository (2010) includes a central repository having devices capable of storing several terabytes or several petabytes of data, and / or includes distributed storage systems (e.g., devices that mirror or cache content at Internet interconnection locations to provide faster access to mirrored content within specific regions). The repository (2010) is also configured in any other suitable manner.

[0066] As described, the repository (2010) may store various and different items, including content (2012), user data (2014), and / or log data (2016). Content (2012) includes television shows, movies, video games, user-generated content, and / or any other suitable type or form of content. User data (2014) includes personally identifiable information (PII), payment information, preference settings, language and accessibility settings, and / or any other information associated with a specific user or content player. Log data (2016) includes viewing history information, network throughput information, and / or any other metrics associated with a user's access to or interactions with the distribution infrastructure (1910).

[0067] Services (2020) include personalization services (2022), transcoding services (2024), and / or packaging services (2026). Personalization services (2022) personalize recommendations, content streams, and / or other aspects of the user's experience with the distribution infrastructure (1910). Encoding services (2024) compress media at different bitrates to enable real-time switching between different encodings, as described in more detail below. Packaging services (2026) package the encoded video before deploying it to a delivery network, such as a network (2030), for streaming.

[0068] A network (2030) generally represents any medium or architecture capable of facilitating communication or data transmission. A network (2030) facilitates communication or data transmission using wireless and / or wired connections. Examples of a network (2030) include, without limitation, an intranet, a wide area network (WAN), a local area network (LAN), a personal area network (PAN), the Internet, power line communications (PLC), a cellular network (e.g., a global system for mobile communications (GSM) network), parts of one or more of these, variations or combinations of one or more of these, and / or any other suitable network. For example, as illustrated in FIG. 20, a network (2030) includes an Internet backbone (2032), an Internet service provider (2034), and / or a local network (2036). As discussed in more detail below, bandwidth limits and bottlenecks within one or more of these network segments trigger video and / or audio bit rate adjustments.

[0069] FIG. 21 is a block diagram of an exemplary implementation of the content player (1920) of FIG. 19. The content player (1920) generally represents any type or form of computing device capable of reading computer-executable instructions. The content player (1920) includes, without limitation, laptops, tablets, desktops, servers, cellular phones, multimedia players, embedded systems, wearable devices (e.g., smart watches, smart glasses, etc.), smart vehicles, game consoles, Internet of Things (IoT) devices such as smart devices, one or more variations or combinations of these, and / or any other suitable computing device.

[0070] As illustrated in FIG. 21, in addition to the processor (1922) and memory (1924), the content player (1920) includes a communication interface (2122) and a communication infrastructure (2102) coupled to a network connection (2124). The content player (1920) also includes a graphics interface (2126) coupled to a graphics device (2128), an input interface (2134) coupled to an input device (2136), and a storage interface (2138) coupled to a storage device (2140).

[0071] The communication infrastructure (2102) generally represents any type or form of infrastructure capable of facilitating communication between one or more components of a computing device. Examples of the communication infrastructure (2102) include, without limitation, any type or form of communication bus (e.g., a peripheral component interconnect (PCI) bus, a PCIe (PCI Express) bus, a memory bus, a front bus, an integrated drive electronics (IDE) bus, a control or register bus, a host bus, etc.).

[0072] As mentioned, memory (1924) generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or other computer-readable instructions. In some examples, memory (1924) stores and / or loads an operating system (2108) for execution by a processor (1922). In one example, the operating system (2108) includes and / or represents software that manages computer hardware and software resources and / or provides common services to computer programs and / or applications on a content player (1920).

[0073] The operating system (2108) performs various system management functions, such as managing hardware components (e.g., a graphics interface (2126), an audio interface (2130), an input interface (2134), and / or a storage interface (2138)). The operating system (2108) also provides process and memory management models for the playback application (2110). Modules of the playback application (2110) include, for example, a content buffer (2112), an audio decoder (2118), and a video decoder (2120).

[0074] The playback application (2110) is configured to retrieve digital content through a communication interface (2122) and to play digital content through a graphic interface (2126). The graphic interface (2126) is configured to transmit a rendered video signal to a graphic device (2128). In normal operation, the playback application (2110) receives a request from a user to play a specific title or specific content. Then, the playback application (2110) identifies one or more encoded video and audio streams associated with the requested title. After the playback application (2110) finds the encoded streams associated with the requested title, the playback application (2110) downloads sequence header indices associated with each encoded stream associated with the requested title from the distribution infrastructure (1910). The sequence header index associated with the encoded content contains information related to the sequence of encoded data included in the encoded content.

[0075] In one embodiment, the playback application (2110) initiates the download of content associated with the requested title by downloading sequence data encoded at the lowest audio and / or video playback bitrates to minimize the startup time for playback. The requested digital content file is then downloaded to a content buffer (2112) configured to function as a first-in, first-out queue. In one embodiment, each unit of the downloaded data includes a unit of video data or a unit of audio data. As units of video data associated with the requested digital content file are downloaded to the content player (1920), the units of video data are pushed into the content buffer (2112). Similarly, as units of audio data associated with the requested digital content file are downloaded to the content player (1920), the units of audio data are pushed into the content buffer (2112). In one embodiment, units of video data are stored in the video buffer (2116) within the content buffer (2112), and units of audio data are stored in the audio buffer (2114) of the content buffer (2112).

[0076] The video decoder (2120) reads units of video data from the video buffer (2116) and outputs the units of video data as a sequence of video frames corresponding to a playback time range with a fixed duration. Reading units of video data from the video buffer (2116) effectively dequeues units of video data from the video buffer (2116). Subsequently, the sequence of video frames is rendered by the graphics interface (2126) and transmitted to the graphics device (2128) to be displayed to the user.

[0077] The audio decoder (2118) reads units of audio data from the audio buffer (2114) and outputs the units of audio data as a sequence of audio samples that is time-synchronized with a sequence of decoded video frames. In one embodiment, the sequence of audio samples is transmitted to an audio interface (2130), and the audio interface converts the sequence of audio samples into an electric audio signal. The electric audio signal is then transmitted to a speaker of an audio device (2132), and the speaker, in response, generates an audio output.

[0078] In situations where the bandwidth of the distribution infrastructure (1910) is limited and / or variable, the playback application (2110) downloads and buffers consecutive portions of video data and / or audio data from video encodings having different bitrates based on various factors (e.g., scene complexity, audio complexity, network bandwidth, device capabilities, etc.). In some embodiments, video playback quality is prioritized over audio playback quality. Audio playback and video playback quality are also balanced, and in some embodiments, audio playback quality is prioritized over video playback quality.

[0079] The graphics interface (2126) is configured to generate frames of video data and transmit the frames of video data to a graphics device (2128). In one embodiment, the graphics interface (2126) is included as part of an integrated circuit together with the processor (1922). Alternatively, the graphics interface (2126) is configured as a hardware accelerator separate from (i.e. not integrated within) the chipset containing the processor (1922).

[0080] The graphic interface (2126) generally represents any type or form of device configured to transmit images for display on the graphic device (2128). For example, the graphic device (2128) is manufactured using liquid crystal display (LCD) technology, cathode ray technology, and light-emitting diode (LED) display technology (organic or inorganic). In some embodiments, the graphic device (2128) also includes a virtual reality display and / or an augmented reality display. The graphic device (2128) includes any technically feasible means for generating images for display. That is, the graphic device (2128) generally represents any type or form of device capable of visually displaying information transmitted by the graphic interface (2126).

[0081] As illustrated in FIG. 21, the content player (1920) also includes at least one input device (2136) coupled to the communication infrastructure (2102) via an input interface (2134). The input device (2136) generally represents any type or form of computing device capable of providing computer or human-generated input to the content player (1920). Examples of the input device (2136) include, without limitation, a keyboard, a pointing device, a voice recognition device, a touch screen, a wearable device (e.g., a glove, a watch, etc.), a controller, variations or combinations of one or more of these, and / or any other type or form of electronic input mechanism.

[0082] The content player (1920) also includes a storage device (2140) coupled to the communication infrastructure (2102) via a storage interface (2138). The storage device (2140) generally represents any type or form of storage device or medium capable of storing data and / or other computer-readable instructions. For example, the storage device (2140) is a magnetic disk drive, a solid-state drive, an optical disk drive, a flash drive, etc. The storage interface (2138) generally represents any type or form of interface or device for transferring data between the storage device (2140) and other components of the content player (1920).

[0083] Exemplary embodiments:

[0084] 1. A method implemented by a computer, comprising the steps of: determining, for each of a plurality of different media items, a current time scale at which the media items are encoded for distribution (at least two of the plurality of media items are encoded at different frame rates); for the plurality of media items, identifying an integrated time scale that provides a constant frame interval for each of the plurality of media items; and changing at least one of the plurality of media items from the current time scale to the identified integrated time scale to provide a constant frame interval for at least one changed media item.

[0085] 2. A method implemented by a computer, wherein the media items include video media items, in accordance with claim 1.

[0086] 3. A method implemented by a computer, wherein, in paragraph 2, video media items are encoded at 23.97, 24, 25, 29.97, 30, 59.94, 60, 120, 240, or 300 frames per second.

[0087] 4. A method implemented by a computer, wherein the media items include audio media items, in accordance with claim 1.

[0088] 5. In paragraph 4, a method implemented by a computer, wherein the audio media items have a frame rate of 1024, 1536, or 2048 samples per frame.

[0089] 6. A method implemented by a computer, wherein, in paragraph 1, each of a plurality of media items within a designated media item group has a designated video frame rate and an audio frame rate, and an integrated time scale is calculated to optimize the designated video frame rate and the designated audio frame rate of the media items within the group.

[0090] 7. A method implemented by a computer, wherein the integrated time scale is implemented to generate one or more presentation time stamps (PTS) for multiple media items in accordance with claim 1.

[0091] 8. A method implemented by a computer in which, in paragraph 6, one or more PTSs are monotonically increasing, and the unit used in the integrated time scale is selected to maximize the wrap-around time for multiple media items.

[0092] 9. A method implemented by a computer, wherein the unit selected to maximize the wrap-around time for multiple media items is selected based on the video frame rate.

[0093] 10. A method implemented by a computer, wherein the identified integrated time scale includes a presentation time stamp (PTS) interval, and the PTS interval includes a minimum frame interval or a multiple of the minimum frame interval.

[0094] 11. A method implemented by a computer, wherein, in paragraph 10, the step of restoring the PTS interval to a specified resolution is further included.

[0095] 12. A system comprising at least one physical processor and physical memory including computer-executable instructions, wherein when the computer-executable instructions are executed by the physical processor, the physical processor determines, for each of a plurality of different media items, a current time scale in which the media items are encoded for distribution, wherein at least two of the plurality of media items are encoded at different frame rates, for the plurality of media items, an integrated time scale that provides a constant frame interval for each of the plurality of media items, and changes at least one of the plurality of media items from the current time scale to the identified integrated time scale to provide a constant frame interval for at least one changed media item.

[0096] 13. In paragraph 12, the identified integrated time scale is a system that allows multiple media items to be streamed at a variable frame rate while maintaining a constant frame interval.

[0097] 14. In paragraph 13, a system in which media items having different frame rates are streamed at a variable frame rate while maintaining a constant frame interval for each frame rate using an integrated time scale.

[0098] 15. In paragraph 12, a system wherein media items having video content captured using a variable refresh rate are streamed at a variable frame rate while maintaining a constant frame interval for each frame rate using an integrated time scale.

[0099] 16. A system according to claim 12, wherein identifying an integrated time scale comprises converting one or more input presentation time stamps (PTS) from a plurality of different media items having different time scales into PTSs based on an integrated time scale.

[0100] 17. In paragraph 16, a system that implements converted input PTSs to avoid PTS counter wrap-around.

[0101] 18. A system according to paragraph 12, wherein changing at least one of the multiple media items from the current time scale to an identified integrated time scale allows a single fixed V-synchronous interrupt to be implemented during the playback of the multiple media items.

[0102] 19. A system according to claim 12, further comprising optimizing PTSs for multiple media items such that scaled presentation time stamps match unique PTSs under an appropriate time scale without resulting rounding error.

[0103] 20. A non-transient computer-readable medium comprising one or more computer-executable instructions, wherein when the one or more computer-executable instructions are executed by at least one processor of a computing device, the computing device determines, for each of a plurality of different media items, a current time scale in which the media items are encoded for distribution, wherein at least two of the plurality of media items are encoded at different frame rates, for the plurality of media items, an integrated time scale that provides a constant frame interval for each of the plurality of media items, and changes at least one of the plurality of media items from the current time scale to the identified integrated time scale to provide a constant frame interval for at least one changed media item.

[0104] As described in detail above, the computing devices and systems described and / or illustrated in this specification broadly represent any type or form of computing device or system capable of executing computer-readable instructions, such as those included within the modules described in this specification. In the most basic configuration, these computing device(s) may each include at least one memory device and at least one physical processor.

[0105] In some examples, the term “memory device” generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, a memory device may store, load, and / or maintain one or more of the modules described herein. Examples of memory devices include, without limitation, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drives (HDDs), solid-state drives (SSDs), optical disk drives, caches, variations or combinations of one or more of these, or any other suitable storage memory.

[0106] In some examples, the term “physical processor” generally refers to any type or form of hardware implementation processing unit capable of interpreting and / or executing computer-readable instructions. In one example, a physical processor may access and / or modify one or more modules stored in the aforementioned memory device. Examples of physical processors include, but are not limited to, microprocessors, microcontrollers, central processing units (CPUs), field programmable gate arrays (FPGAs) implementing softcore processors, application-specific integrated circuits (ASICs), parts of one or more of these, variations or combinations of one or more of these, or any other suitable physical processor.

[0107] Although described as individual elements, the modules described and / or illustrated herein may represent parts of a single module or application. Additionally, in certain embodiments, one or more of these modules may represent one or more software applications or programs capable of enabling the computing device to perform one or more tasks when executed by the computing device. For example, one or more of the modules described and / or illustrated herein may represent modules stored and configured to be executed on one or more of the computing devices or systems described and / or illustrated herein. One or more of these modules may also represent all or part of one or more special-purpose computers configured to perform one or more tasks.

[0108] Additionally, one or more of the modules described herein may convert data, physical devices, and / or representations of physical devices from one form to another. For example, one or more of the modules described herein may receive data to be converted, convert the data, output the result of the conversion to determine an optimized time scale, generate an accurate presentation timestamp using the result of the conversion, and store the result of the conversion. Additionally or alternatively, one or more of the modules described herein may convert a processor, volatile memory, non-volatile memory, and / or any other part of a physical computing device from one form to another by executing on a computing device, storing data on the computing device, or interacting with the computing device in a different way.

[0109] In some embodiments, the term “computer-readable medium” generally refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media include, but are not limited to, transmission media such as carrier waves, and magnetic storage media (e.g., hard disk drives, tape drives, and floppy disks), optical storage media (e.g., CD (Compact Disk), DVD (Digital Video Disk), and BLU-RAY discs), electronic storage media (e.g., solid-state drives and flash media), and non-transient types of media such as other distribution systems.

[0110] The sequence of process parameters and steps described and / or illustrated in this specification is given merely as an example and may be varied as desired. For example, the steps described and / or illustrated in this specification may be shown or discussed in a specific order, but these steps do not necessarily need to be performed in the order described or discussed. Various exemplary methods described and / or illustrated in this specification may also omit one or more of the steps described or illustrated in this specification, or include additional steps in addition to those disclosed.

[0111] The foregoing description is provided to enable a person skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be comprehensive or limited to any exact form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the disclosure. The embodiments disclosed herein should be considered illustrative in all respects rather than restrictive. In determining the scope of the disclosure, the appended claims and their equivalents should be referenced.

[0112] Unless otherwise noted, the terms “connected to” and “combined to” (and their derivatives) shall be interpreted as enabling both direct and indirect (i.e., through other elements or components) connection, as used in the specification and claims. Additionally, the singular terms as used in the specification and claims shall be interpreted as meaning “at least one of”. Finally, for ease of use, the terms “including” and “having” (and their derivatives) as used in the specification and claims are interchangeable with and have the same meaning as the word “comprising”.

Claims

Claim 1 A method implemented by a computer, comprising: for each of a plurality of different media items, determining a current time scale in which the media items are encoded for distribution, wherein at least two of the plurality of media items are encoded at different frame rates; for the plurality of media items, identifying an integrated time scale that provides a constant frame interval for each of the plurality of media items; changing at least one of the plurality of media items from the current time scale to the identified integrated time scale to provide a constant frame interval for at least one changed media item; and selecting a minimum number of units in the integrated time scale that allows each of the plurality of media items to have a constant frame interval, thereby maximizing the wrap-around time for the plurality of media items. Claim 2 A method implemented by a computer, wherein the media items include video media items, in accordance with claim 1. Claim 3 A method implemented by a computer, wherein the video media items are encoded at 23.97, 24, 25, 29.97, 30, 59.94, 60, 120, 240, or 300 frames per second. Claim 4 A method implemented by a computer, wherein, in paragraph 1, the media items include audio media items. Claim 5 In paragraph 4, the method implemented by a computer, wherein the audio media items have a frame rate of 1024, 1536, or 2048 samples per frame. Claim 6 A method implemented by a computer according to claim 1, wherein each of the plurality of media items within a designated media item group has a designated video frame rate and an audio frame rate, and the integrated time scale is calculated to optimize the designated video frame rate and the designated audio frame rate of the media items within the group. Claim 7 A method implemented by a computer, wherein the integrated time scale is implemented to generate one or more presentation time stamps (PTS) for the plurality of media items in claim 1. Claim 8 In claim 7, a method implemented by a computer, wherein one or more PTSs are monotonically increasing, and the units used in the integrated time scale are selected to maximize the wrap-around time for the plurality of media items. Claim 9 A method implemented by a computer, wherein the unit selected to maximize the wrap-around time for the plurality of media items is selected based on the video frame rate. Claim 10 A method implemented by a computer, wherein the identified integrated time scale includes a presentation time stamp (PTS) interval, and the PTS interval includes a minimum frame interval or a multiple of the minimum frame interval. Claim 11 A method implemented by a computer, wherein, in paragraph 10, the above PTS interval further comprises the step of restoring to a specified resolution. Claim 12 A system comprising at least one physical processor; and physical memory containing computer-executable instructions, wherein when the computer-executable instructions are executed by the physical processor, the physical processor determines, for each of a plurality of different media items, a current time scale in which the media items are encoded for distribution, wherein at least two of the plurality of media items are encoded at different frame rates; for the plurality of media items, an integrated time scale that provides a constant frame interval for each of the plurality of media items; change at least one of the plurality of media items from the current time scale to the identified integrated time scale to provide a constant frame interval for at least one changed media item; and select a minimum number of units in the integrated time scale that allows each of the plurality of media items to have a constant frame interval, thereby maximizing the wrap-around time for the plurality of media items. Claim 13 In claim 12, the identified integrated time scale allows the plurality of media items to be streamed at a variable frame rate while maintaining the constant frame interval, a system. Claim 14 In paragraph 13, a system wherein media items having different frame rates are streamed at a variable frame rate while maintaining the constant frame interval for each frame rate using the integrated time scale. Claim 15 In claim 12, a system in which media items having video content captured using a variable refresh rate are streamed at a variable frame rate while maintaining a constant frame interval using the integrated time scale. Claim 16 A system according to claim 12, wherein identifying the integrated time scale comprises converting one or more input presentation time stamps from the plurality of different media items having different time scales into PTSs based on the integrated time scale. Claim 17 In paragraph 16, the system implementing the above-mentioned converted input PTSs avoids the PTS counter wrap-around. Claim 18 A system according to claim 12, wherein changing at least one of the plurality of media items from the current time scale to the identified integrated time scale allows a single fixed V-Synch interrupt to be implemented during the playback of the plurality of media items. Claim 19 A system according to claim 12, further comprising optimizing PTSs for the plurality of media items so that scaled presentation time stamps match unique PTSs without resulting rounding error. Claim 20 A non-transient computer-readable storage medium comprising one or more computer-executable instructions, wherein the one or more computer-executable instructions, when executed by at least one processor of a computing device, cause the computing device to determine, for each of a plurality of different media items, a current time scale in which the media items are encoded for distribution, wherein at least two of the plurality of media items are encoded at different frame rates; for the plurality of media items, identify an integrated time scale that provides a constant frame interval for each of the plurality of media items; change at least one of the plurality of media items from the current time scale to the identified integrated time scale to provide a constant frame interval for at least one changed media item; and select a minimum number of units in the integrated time scale that allows each of the plurality of media items to have a constant frame interval, thereby maximizing the wrap-around time for the plurality of media items.

Citation Information

Patent Citations

  • Data generation device and method, and program and computer readable storage medium

    JP2005223617A