Media stream data playing method, apparatus, device and medium

Abstract

The embodiments of the present specification provide a media stream data playing method, device, equipment and medium. The scheme comprises: monitoring a video buffer level of a video buffer and a display timestamp difference between audio playing data and video playing data; when the video buffer level is lower than a preset safety threshold, reducing a video playing speed and adjusting an audio playing speed according to the display timestamp difference; when the video buffer level is not lower than the preset safety threshold and an absolute value of the display timestamp difference exceeds a preset synchronization threshold, adjusting at least one of the audio playing speed and the video playing speed according to the display timestamp difference.

Description

Technical Field

[0001] This specification relates to the field of multimedia technology, and in particular to a method for playing media stream data. This specification also relates to a media stream data playback device, a computing device, and a computer-readable storage medium. Background Technology

[0002] Streaming media playback technology is widely used in scenarios such as live streaming and video-on-demand. During playback, the playback device receives audio and video data from the network and stores it in a buffer, then decodes and renders it for output. To ensure smooth playback, the playback device typically needs to dynamically adjust its playback strategy based on network conditions. Simultaneously, due to differences in encoding, transmission, and decoding between audio and video data, the playback device also needs to synchronize the playback progress of the audio and video to avoid issues such as stuttering or audio-visual asynchrony.

[0003] As users' demands for viewing experience continue to rise, achieving precise audio-visual synchronization while ensuring smooth playback has become a persistent concern in the field of streaming media playback technology. Therefore, a more optimized method for playing media streaming data is needed. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a method, apparatus, device, and computer-readable medium for playing media streaming data, in order to provide a method for playing media streaming data that is smoother, more synchronized, and with full picture quality.

[0005] According to a first aspect of one or more embodiments of this specification, a media stream data playback method is provided, comprising: Monitor the video buffer level in the video buffer and the difference in display timestamps between audio playback data and video playback data; When the video buffer level is lower than a preset safety threshold, the video playback speed is reduced, and the audio playback speed is adjusted according to the difference in the display timestamps. When the video buffer level is not lower than the preset safety threshold and the absolute value of the display timestamp difference exceeds the preset synchronization threshold, at least one of the audio playback speed and the video playback speed is adjusted according to the display timestamp difference.

[0006] According to a second aspect of one or more embodiments of this specification, a media streaming data playback device is provided, comprising: The playback monitoring module is used to monitor the video buffer level in the video buffer and the difference in display timestamps between audio playback data and video playback data; The playback adjustment module is used to: reduce the video playback speed and adjust the audio playback speed according to the display timestamp difference when the video buffer level is lower than a preset safety threshold; and adjust at least one of the audio playback speed and the video playback speed according to the display timestamp difference when the video buffer level is not lower than the preset safety threshold and the absolute value of the display timestamp difference exceeds a preset synchronization threshold.

[0007] According to a third aspect of one or more embodiments of this specification, a computing device is provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor, when executing the computer instructions, implements the steps of the media stream data playback method.

[0008] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions, which, when executed by a processor, implement the steps of the media stream data playback method.

[0009] One embodiment of this specification can achieve at least the following beneficial effects: by monitoring the video buffer level and the display timestamp difference between audio and video, when the video buffer level is lower than a preset safety threshold, the video playback speed is reduced and the audio playback speed is adjusted according to the display timestamp difference; when the video buffer level is not lower than the preset safety threshold and the absolute value of the display timestamp difference exceeds a preset synchronization threshold, at least one of the audio playback speed and video playback speed is adjusted according to the display timestamp difference; thus, the sufficiency of the video buffer level and the audio-visual synchronization deviation are used as two interrelated decision criteria to trigger differentiated control strategies under different states, thereby providing a smoother, more synchronized, and higher-quality viewing experience overall. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This diagram illustrates an application scenario of a media stream data playback method provided in the embodiments of this specification. Figure 2 A flowchart illustrating a media stream data playback method provided in an embodiment of this specification; Figure 3 This is a schematic diagram of the playback control process provided in one embodiment of this specification; Figure 4 This is a schematic diagram of a playback control process provided in another embodiment of this specification; Figure 5 This specification provides a flowchart of a phased strategy for media stream data playback in a practical application scenario, as illustrated in the embodiments of this specification. Figure 6 This specification provides a system block diagram for media stream data playback in a practical application scenario, as illustrated in the embodiments of this specification. Figure 7 The embodiments provided in this specification correspond to Figure 2 A schematic diagram of the structure of a media streaming data playback device; Figure 8 A structural block diagram of a computing device provided according to an embodiment of this specification is shown. Detailed Implementation

[0012] To enable users in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0013] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0014] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “an,” “an,” “the,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification includes any or all possible combinations of one or more associated listed items.

[0015] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded.

[0016] Although the terms "first," "second," etc., may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, "first" may also be referred to as "second," and similarly, "second" may also be referred to as "first," without departing from the scope of one or more embodiments of this specification. Ordinal numbers such as "first," "second," etc., do not necessarily indicate order; often they are used to facilitate the distinction of objects. For example, "first server" and "second server" usually refer to two servers. To distinguish these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0017] Depending on the context, the word "if" as used here can be interpreted as "when," "when," or "in response to determination."

[0018] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0019] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0020] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0021] The following explains the terms and concepts used in one or more embodiments of this specification.

[0022] Buffer: The amount of media data that a player downloads and stores in memory before playback. The buffer acts like a "reservoir" to handle network fluctuations and ensure playback continuity.

[0023] Presentation Time Stamp (PTS): A time stamp embedded in the audio and video data stream, used to indicate when a frame should be presented (played). The core of audio-visual synchronization is ensuring the alignment of the audio and video PTS.

[0024] Display timestamp difference (PTS difference, PTS Diff): refers to the time difference between the audio PTS and the video PTS, and is a key indicator for measuring the degree of audio-visual synchronization.

[0025] With the improvement of network bandwidth and the advancement of video encoding technology, live streaming has become an important form of internet content dissemination. Users have increasingly higher demands for the live streaming viewing experience, with "stuttering" and "audio-visual desynchronization" being two core issues affecting the experience. In live streaming scenarios, the playback end typically uses a buffer to combat network jitter: a certain amount of media data is pre-downloaded and stored in the buffer, and then data is read from the buffer at a constant rate for decoding and rendering. When network bandwidth decreases or fluctuates, the buffer level drops; if the level runs out, stuttering occurs. Simultaneously, because audio and video data packets may experience different delays during transmission and decoding, their display timestamps (PTS) are prone to discrepancies, leading to audio-visual desynchronization.

[0026] To address stuttering caused by network fluctuations, the industry commonly employs Adaptive Bitrate (ABR) technology. The basic idea is that the player continuously monitors network bandwidth and buffer levels. When the buffer level is low, it proactively switches to a lower bitrate video stream to reduce the amount of data per frame, thus lowering bandwidth requirements. When network conditions improve and the buffer level rises, it switches back to a higher bitrate video stream to restore image quality. However, ABR has several inherent limitations. In weak network environments, the player is forced to reduce the bitrate multiple times, resulting in blurry videos or noticeable quality fluctuations for the user. The bitrate switching decision itself has a certain delay, which may cause brief pauses or sudden changes in image quality during the switching process. Furthermore, ABR primarily focuses on video stream transmission and playback, with relatively independent processing of the audio stream, lacking a direct improvement effect on audio-visual synchronization issues.

[0027] On the other hand, traditional live streaming players often use frame skipping or silent insertion to handle audio-video desynchronization. When audio is detected to be ahead of video, the player will discard some audio frames; when video is ahead of audio, it will discard some video frames or insert silent frames. Although this type of operation can force the audio and video to realign, the cost is that there will be perceptible jumps or interruptions during playback, which will disrupt the continuity of the viewing experience.

[0028] To address the shortcomings of the existing technology, this specification provides a media streaming data playback method. This method constructs a coordinated control framework based on a comprehensive judgment condition: the video buffer level and the difference between audio and video display timestamps. Specifically, the playback device monitors the video buffer level and the difference between audio and video display timestamps in real time. When the video buffer level falls below a preset safety threshold, the playback device actively reduces the video playback speed and adjusts the audio playback speed accordingly based on the display timestamp difference. This operation does not maintain smoothness by reducing the bitrate, but rather by making a slight, almost imperceptible deceleration (e.g., 0.95x speed) on the timeline, allowing the buffer to quickly accumulate data (actively storing data), thus providing sufficient buffer reserves during subsequent network fluctuations. When the video buffer level is at a normal level, but the absolute value of the difference in displayed timestamps exceeds the preset synchronization threshold, the playback end selects to adjust at least one of the audio playback speed and / or video playback speed according to the direction of advance (audio advance or video advance). This adjustment prioritizes "preserving video and adjusting audio", giving priority to using the water level information of the audio buffer for smooth micro-perturbation speed adjustment (e.g., slowing down the audio when the audio is ahead, and speeding up the audio when the video is ahead and the audio water level is sufficient), avoiding the playback jumpiness caused by traditional frame skipping or mute filling.

[0029] Through the above methods, the solution in this embodiment of the specification can achieve at least the following technical effects: On the one hand, by using speed adjustment instead of bitrate reduction, it avoids sacrificing image quality in weak network environments, allowing the player to maintain a higher bitrate for a longer period, ensuring that the image viewed by the user remains clear; on the other hand, the linkage control based on buffer level and PTS difference achieves smooth and imperceptible audio-visual synchronization, eliminating perceptible interruptions caused by frame skipping and silent insertion. Therefore, without significantly reducing image quality, it is possible to simultaneously reduce stuttering and achieve precise and imperceptible audio-visual synchronization, thereby optimizing the user's overall live streaming viewing experience.

[0030] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0031] Figure 1 This is a schematic diagram illustrating an application scenario of a media stream data playback method provided in the embodiments of this specification.

[0032] like Figure 1 As shown, this scenario includes a live streamer 10 (e.g., a broadcaster), a live streaming client 20, a cloud server 30, a playback client 40, and a live stream viewer 50 (e.g., a viewer). The live streamer 10 collects audio and video data through the live streaming client 20 (e.g., a smartphone, camera, etc.) and streams the data to the cloud server 30; the live stream viewer 50 pulls the media stream from the cloud server 30 through the playback client 40 for viewing. The live streaming client 20 and the playback client 40 can connect to the cloud server 30 via a local area network, wide area network, mobile communication network, or other types of data network.

[0033] In some embodiments of this specification, a buffer linkage control architecture for live streaming media playback is provided. This architecture constructs a state machine-based audio and video buffer collaborative control mechanism. By actively slowing down and storing water in the early stage of playback, complementary micro-perturbation synchronization in the stable playback stage, and rapid recovery in the stuttering processing stage, the playback smoothness and audio-visual synchronization accuracy are optimized simultaneously without significantly reducing the image quality.

[0034] like Figure 1 As shown, the media stream data playback method provided in the embodiments of this specification can be applied to the playback terminal 40, which can run on a corresponding computing device. Exemplarily, the computing device may include a smartphone, tablet computer, laptop computer, desktop computer, smart TV, set-top box, in-vehicle entertainment system, smart wearable device (such as a smartwatch or smart glasses), virtual reality or augmented reality headset, or a distributed playback system composed of multiple devices; this specification does not specifically limit this.

[0035] In practical applications, each step of the media stream data playback method provided in the embodiments of this specification can be executed entirely on the client device (playback end), or partially on the client device and partially on the server, depending on the specific scenario. Considering the requirements of low latency and high real-time performance in live streaming scenarios, and the need for the control strategy to respond quickly based on the buffer state and audio-visual deviation, the method provided in the embodiments of this specification can be preferably applied to scenarios such as... Figure 1 The client-server collaboration scenario shown is not limited to this. In this scenario, the live stream user 10 pushes media data to the cloud server 30 through the live stream client 20. The cloud server 30 is responsible for pushing and distributing the media data, while the client (playback client) 40 used by the live stream user 50 is responsible for buffer monitoring, PTS difference calculation, and coordinated adjustment of playback speed.

[0036] To better understand the application scenarios of this disclosure, a specific user interaction example is provided below. Assume a streamer is broadcasting live through their device, and a viewer is watching the streamer's broadcast using a live streaming application (e.g., an e-commerce live streaming platform) installed on their smartphone. This application integrates the buffer linkage control module provided in the embodiments of this specification. When the user clicks to enter the streamer's live room, the mobile playback terminal immediately initiates the broadcast process: the server pushes the first frame of audio and video data to the mobile phone in accelerated sending mode; the mobile phone quickly completes the first frame display after receiving the data, simultaneously aligning the initial timestamps of the audio and video. The user sees the streamer's image almost instantly upon clicking, and the sound and lip movements are synchronized from the beginning. During the viewing process, assuming a brief fluctuation in the user's network environment, the mobile playback terminal detects a drop in the video buffer level and then slightly reduces the playback speed by a factor of 0.95, almost imperceptible to the user. The audio is also adjusted accordingly. Although the user feels the picture is slightly slower, the picture remains smooth without any stuttering. Subsequently, when the network returns to normal, the playback terminal automatically restores the speed to normal. Throughout the viewing process, users do not need to perform any additional operations. The playback device automatically optimizes smoothness and synchronization in the background, and users only experience a stable, clear, and synchronized live broadcast. In practical applications, users can also choose whether to enable "smoothness priority" or "synchronization priority" mode in the application's settings interface to adapt to viewing preferences for different content types (such as sports events, music performances, and language interviews). The control strategy disclosed herein can adjust the corresponding threshold parameters according to the user's set preferences.

[0037] This specification provides a method for playing media stream data in the embodiments. This application also relates to a media stream data playback device, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0038] Figure 2 This is a flowchart illustrating a media stream data playback method provided in an embodiment of this specification.

[0039] From a programming perspective, the entity executing the process can be a program mounted on the playback terminal. It can be understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities.

[0040] like Figure 2 As shown, the process may include the following steps: Step 202: Monitor the video buffer level in the video buffer and the difference in display timestamps between the audio playback data and the video playback data.

[0041] In the field of streaming media playback technology, "video buffer level" refers to the amount of media data stored in the video buffer, usually measured in playable duration (e.g., seconds) or frames. For example, if the data stored in the video buffer can be played continuously for 3 seconds at normal playback speed, then the video buffer level is said to be 3 seconds. The video buffer level reflects the current player's ability to withstand network fluctuations; the higher the level, the less likely playback is to stutter due to brief network interruptions. In some embodiments, the video buffer level can also be expressed as "the current amount of data in the buffer," "the current fill level of the buffer," or "the current occupancy level of the buffer."

[0042] The Presentation Time Stamp (PTS) is an expected value generated by the encoder based on the acquisition time and remains unchanged during transmission. The playback end uses the PTS to determine the rendering timing of each frame. The difference in presentation timestamps between audio and video playback data (hereinafter referred to as PTS difference or PTS_Diff) refers to the difference between the PTS of the audio frame and the PTS of the video frame at the same playback moment. For example, the calculation method is: PTS_Diff = PTS_audio - PTS_video. If the PTS of the currently playing audio frame is 1200 milliseconds, while the PTS of the video frame to be rendered is 1000 milliseconds, then PTS_Diff = +200 milliseconds, indicating that the audio playback progress is 200 milliseconds ahead of the video playback progress. Conversely, if PTS_Diff is negative, it means that the video is ahead of the audio.

[0043] In practical applications, PTS differences can be used not only to determine the current audio-visual synchronization status but also to predict future synchronization trends. By examining the PTS distribution of several future frames within the buffer, the playback device can anticipate potential desynchronization risks. For example, if a jump in PTS is detected in subsequent frames, it may indicate an anomaly in network transmission or decoding, allowing for proactive adjustments.

[0044] In some embodiments, the playback device continuously monitors the video buffer level and the display timestamp difference between the audio playback data and the video playback data. This method can be executed by the live streaming playback device, for example, when a user watches a live stream on a terminal device, the playback device (such as the player module of a live streaming application) acquires these two parameters in real time. The video buffer level reflects the sufficiency of the current video data reserve, and the PTS difference reflects the degree of audio-visual synchronization deviation; both parameters serve as the basis for subsequent playback control decisions.

[0045] For example, in an e-commerce live streaming scenario, while the host is introducing a product, the playback device checks the video buffer level every second (e.g., the current level is 2.5 seconds) and simultaneously calculates the PTS difference between the most recent audio and video frame (e.g., PTS_Diff is +150 milliseconds, meaning the audio is slightly faster). This data is recorded in real time and used to trigger subsequent slowdown or synchronization adjustments. In a sports live streaming scenario, due to potentially larger network fluctuations, the playback device can monitor these two parameters more frequently (e.g., every 200 milliseconds) for faster response.

[0046] In some embodiments, the audio playback data and the video playback data are received through independent transmission channels.

[0047] Specifically, audio and video data streams use different channels (e.g., different ports, different connections, or different transport protocol instances) during network transmission and do not share transmission resources. Due to potential differences in network latency, packet loss rate, and other characteristics between the two channels, audio and video frames often arrive at the playback end at different times, which is one of the main causes of audio-visual desynchronization. The monitoring method described in the embodiments of this specification can operate under this dual-channel independent transmission architecture, providing a data foundation for subsequent coordinated control by real-time monitoring of video buffer level and PTS differences.

[0048] For example, in a live streaming scenario, the server sends audio data through channel A and video data through channel B. The playback client receives audio packets from channel A and video packets from channel B. When channel B experiences a brief period of congestion, video data may arrive late, causing the video buffer level to drop; simultaneously, since the audio channel is functioning normally, audio data arrives normally, and the PTS (Pulse Time Slowdown) difference may manifest as audio being ahead. By monitoring these two parameters, the playback client can promptly detect this abnormal state and trigger corresponding control strategies (such as reducing video playback speed or adjusting audio playback speed).

[0049] In some embodiments, the media stream data is live streaming data.

[0050] In practical applications, live streaming scenarios are characterized by low latency, small buffer size, and high real-time requirements. Compared to video-on-demand scenarios (which can use larger buffers to pre-download large amounts of data), the buffer at the playback end in live streaming scenarios is typically smaller to control end-to-end latency. This makes live streaming playback more susceptible to network jitter, placing higher demands on buffer level monitoring and audio-visual synchronization adjustments.

[0051] The media stream data playback method provided in the embodiments of this specification is particularly suitable for live streaming media data playback scenarios. For example, in real-time interactive live broadcasts (such as e-commerce live broadcasts, sports live broadcasts, and game live broadcasts), the playback end needs to ensure smooth picture and audio-visual synchronization as much as possible without adding too much latency. By monitoring the video buffer level and PTS difference in real time, the playback end can make timely fine adjustments to avoid obvious stuttering or audio-visual misalignment, thereby improving the user's viewing experience.

[0052] Based on some embodiments of this specification, by monitoring the video buffer level and the difference in display timestamps between audio and video, the playback device can simultaneously obtain information on two dimensions: buffer reserve status and audio-visual synchronization status. The video buffer level reflects the player's ability to resist network fluctuations; a low level indicates a risk of stuttering. The difference in display timestamps reflects the degree of audio-visual alignment; a large difference means the user will perceive that the sound and picture are out of sync. Incorporating these two parameters into the same monitoring framework provides a complete data foundation for subsequent coordinated adjustments. The playback device can comprehensively judge the playback status based on the current values ​​of these two parameters (and even their changing trends), thereby making more reasonable control decisions. For example, when the video buffer level is low, the playback device can prioritize a slowdown and buffer replenishment strategy; when the level is normal but the audio-visual difference exceeds the standard, it prioritizes adjusting the synchronization deviation. This layered monitoring method helps maintain audio-visual synchronization while ensuring smooth playback, avoiding neglecting one aspect due to a single-dimensional judgment. At the same time, through continuous monitoring, the playback device can respond promptly to dynamic changes in the network environment, achieving fine-grained playback control, thereby providing users with a more stable viewing experience.

[0053] Step 204: When the video buffer level is lower than the preset safety threshold, reduce the video playback speed and adjust the audio playback speed according to the difference in the display timestamps.

[0054] According to step 204, when video data reserves are insufficient, the playback pace can be slowed down to increase buffer data accumulation (i.e., "water storage"), thereby preventing impending stuttering. At the same time, since the adjustment of audio playback speed needs to refer to the current audio-visual synchronization status, the playback end can maintain or improve audio-visual synchronization during the water storage process, avoiding the introduction of new desynchronization problems due to simple slowing down.

[0055] In some embodiments, the video buffer level being lower than a preset safety threshold includes: the amount of playable data stored in the video buffer being less than a preset data volume threshold. For example, this preset data volume threshold may correspond to approximately 2 to 3 seconds of audio and video playback data in frames, measured based on the rendering frame rate of the live streaming client. For example, if the live streaming client renders video at a rate of 25 frames per second, then 2 seconds corresponds to 50 frames, and 3 seconds corresponds to 75 frames. When the number of frames to be rendered stored in the video buffer is less than 50, the video buffer level is considered to be lower than the preset safety threshold. This threshold is a quantitative reference for determining whether the video buffer is in a safe state and for performing a slowdown and water storage operation.

[0056] In some embodiments, the video buffer includes a pre-decoding buffer and a post-decoding buffer; the pre-decoding buffer stores compressed video data (such as NAL units) received from the network but not yet decoded, and the post-decoding buffer stores raw video frames that have been decoded and are waiting to be rendered.

[0057] Furthermore, the monitoring of the video buffer level of the video buffer specifically includes: monitoring the post-decoding level of the post-decoding buffer and the pre-decoding level of the pre-decoding buffer; wherein, the post-decoding level is used as the main reference for judging the video buffer level, while the pre-decoding level is referenced to predict the trend of the post-decoding level in the future.

[0058] For example, the current water level after decoding may only be 0.5 seconds of playback (below the safety threshold), but if there is still a large amount of data to be decoded in the pre-decoding buffer (such as 2 seconds of playback), it means that the water level after subsequent decoding is expected to rise quickly, and there is no need to slow down significantly. Conversely, if the water level before decoding is also very low (for example, less than 0.5 seconds), it means that the data source is about to be exhausted, and a greater slowdown is needed to wait for the network data to be replenished.

[0059] In some embodiments, when the video buffer level is lower than a preset safety threshold, the specific operation of reducing the video playback speed is graded according to the combination of the pre-decoding and post-decoding levels.

[0060] In one scenario, if the decoded water level is lower than the decoded threshold (e.g., lower than the safety threshold), and the pre-decoding water level is higher than or equal to the pre-decoding threshold (e.g., sufficient data before decoding), then the video playback speed is reduced by a third deceleration magnitude. In this case, since there is still a significant amount of data stored before decoding, the playback end can implement a relatively gentle deceleration, for example, reducing the playback speed to 0.9 times the normal speed.

[0061] In another scenario, if the post-decoding water level is lower than the post-decoding threshold, and the pre-decoding water level is lower than the pre-decoding threshold (e.g., insufficient pre-decoding data volume), then the video playback speed is reduced by a fourth deceleration amplitude; the fourth deceleration amplitude is greater than the third deceleration amplitude. In this case, the data source is about to run out, and the playback end needs to more actively replenish the water, for example, by reducing the playback speed to 0.6 or 0.5 times the normal speed.

[0062] In some embodiments, the deceleration magnitude can be determined based on the current data volume in the pre-decoding buffer; the lower the data volume in the pre-decoding buffer, the greater the deceleration magnitude. This tiered design allows the playback device to take appropriate measures based on the actual risk level of the buffer, avoiding excessive deceleration that could negatively impact the user experience, and also avoiding insufficient deceleration that could cause stuttering.

[0063] In some embodiments, the playback device reduces the video playback speed while simultaneously adjusting the audio playback speed based on the difference in display timestamps. The adjustment method depends on the current audio-visual synchronization state.

[0064] The step of adjusting the audio playback speed based on the displayed timestamp difference specifically includes: In one scenario, if the displayed timestamp difference indicates that the audio is ahead (i.e., the audio playback is faster than the video), the audio playback speed is reduced by a first deceleration factor; and the first deceleration factor is greater than the deceleration factor used to reduce the video playback speed. For example, if the video playback speed is reduced to 0.9x, the audio playback speed is reduced to 0.8x or lower, causing the audio to actively "wait" for the video, gradually eliminating the audio lead.

[0065] In another scenario, if the displayed timestamp difference indicates that the video is ahead (i.e., the video playback is faster than the audio), the audio playback speed is reduced by a second deceleration factor, and this second deceleration factor is smaller than the deceleration factor used to reduce the video playback speed. Alternatively, the audio playback speed can be left unchanged or increased. For example, if the video playback speed is reduced to 0.9x, the audio playback speed can remain at normal speed (without adjustment), or be reduced to 0.95x (slight deceleration), or increased to 1.05x, depending on the actual degree of synchronization deviation and the buffer level. This flexibility allows the playback device to accumulate buffer space while avoiding further exacerbation of asynchrony due to the video being ahead.

[0066] In some embodiments, the goal of the above adjustment is not to make the audio speed and video speed completely consistent, but to keep them relatively synchronized during the water storage process, that is, to control the absolute value of the difference in display timestamps to not exceed a preset synchronization threshold.

[0067] For example, in a sports live streaming scenario, assume the normal playback speed is 1.0x, and the preset safety threshold is 2 seconds of playback. The current video buffer level is 1.5 seconds (below the safety threshold), the level after decoding is 1.5 seconds, and the level before decoding is 2.5 seconds (sufficient). The playback device determines to reduce the video playback speed by a third deceleration step, for example, to 0.9x. At this time, the displayed timestamp difference is -100 milliseconds (the video is 100 milliseconds ahead). Therefore, the playback device chooses not to adjust the audio playback speed (maintaining 1.0x) or only slightly reduces the audio speed to 0.95x, making the video slightly slower and the audio relatively faster, thus eliminating the video being ahead. Because the deceleration step is small, the user can hardly perceive the change in playback speed, while the buffer level gradually rises, and playback remains smooth even when there are subsequent network fluctuations.

[0068] For example, in an e-commerce live streaming scenario, assume the normal playback speed is 1.0x, and the preset safety threshold is 2 seconds of playback. The current video buffer level is 1.2 seconds (below the safety threshold), the post-decoding level is 1.2 seconds, and the pre-decoding level is 0.8 seconds. The playback device detects that the post-decoding level is below the post-decoding threshold (2 seconds), and the pre-decoding level is also below the pre-decoding threshold (e.g., 1 second). Therefore, it determines that the video playback speed needs to be reduced using the fourth deceleration factor, for example, to 0.6x. Simultaneously, it detects that the current timestamp difference is +180 milliseconds (audio is 180 milliseconds ahead). The playback device then reduces the audio playback speed using the first deceleration factor, for example, to 0.5x, causing the audio to decelerate more quickly, allowing the video to catch up. After a period of time, the video buffer level rises back to above 2 seconds, and the playback device gradually returns to normal speed.

[0069] Based on some embodiments of this specification, when the video buffer level falls below a preset safety threshold, the playback device can proactively "store water" when video data reserves are insufficient by reducing the video playback speed and adjusting the audio playback speed according to the display timestamp difference, thereby preventing stuttering. The reduction in video playback speed is determined according to the combined state of the water levels before and after decoding, ensuring that the deceleration strategy matches the actual risk level of the buffer: a gentle deceleration is used when the risk is low, and a larger deceleration is used when the risk is high, balancing user experience and anti-stuttering capability. In addition, the adjustment of audio playback speed can refer to the current audio-visual synchronization deviation: when the audio is ahead, the audio is decelerated more, achieving audio waiting for video; when the video is ahead, the audio is decelerated less, not decelerated, or appropriately accelerated, achieving video waiting for audio. This coordinated adjustment method ensures that the audio-visual synchronization state does not deteriorate during the water storage process, and may even be improved. Overall, the playback device can smoothly manage the buffer level and audio-visual synchronization through subtle adjustments on the timeline without discarding data or reducing image quality, providing users with a smoother and more synchronized viewing experience.

[0070] Step 206: When the video buffer level is not lower than the preset safety threshold and the absolute value of the display timestamp difference exceeds the preset synchronization threshold, adjust at least one of the audio playback speed and the video playback speed according to the display timestamp difference (so that the display timestamp difference does not exceed the preset synchronization threshold).

[0071] In some embodiments, when the playback device detects that the video buffer level is not lower than a preset safety threshold, it indicates that the current video data reserve is sufficient and the risk of playback smoothness is low. In addition, the playback device can also determine whether the absolute value of the display timestamp difference exceeds a preset synchronization threshold. If it does, it adjusts at least one of the audio playback speed and video playback speed according to the positive or negative direction of the display timestamp difference to restore the display timestamp difference to an acceptable range.

[0072] For example, the preset synchronization threshold can be set to ±200 milliseconds, meaning that a significant audio-visual desynchronization is considered to have occurred when the difference between the audio display timestamp and the video display timestamp is greater than 200 milliseconds or less than -200 milliseconds. This threshold can be dynamically adjusted according to the tolerance of the application scenario. For example, it can be set to ±150 milliseconds in high-requirement interactive live streaming scenarios, and relaxed to ±300 milliseconds in general content viewing scenarios.

[0073] Furthermore, different adjustment strategies can be adopted based on the positive or negative direction of the display timestamp difference. The adjustment of at least one of the audio playback speed and the video playback speed based on the display timestamp difference specifically includes: In one scenario, if the displayed timestamp difference indicates that the audio is ahead (i.e., the audio playback is faster than the video), then at least one of reducing the audio playback speed and increasing the video playback speed is executed. For example, the audio playback speed can be reduced to make the audio "wait" for the video; or, provided the video buffer has sufficient space, the video playback speed can be appropriately increased to make the video "catch up" with the audio. Both methods help to eliminate audio lead.

[0074] In another scenario, if the displayed timestamp difference indicates that the video is ahead (i.e., the video playback is faster than the audio), then at least one of increasing the audio playback speed and decreasing the video playback speed is executed. For example, the audio playback speed can be increased to allow the audio to "catch up" with the video; or the video playback speed can be appropriately decreased to allow the video to "wait" for the audio. Both methods help to eliminate video advance.

[0075] In some embodiments, the adjustment range can be controlled within a range imperceptible to the user, such as a speed adjustment range between 0.9 and 1.1 times the speed, to avoid significant changes in playback speed affecting the viewing experience.

[0076] In some embodiments, the playback device also monitors the audio buffer level of the audio buffer. The step of increasing the audio playback speed and decreasing the video playback speed if the displayed timestamp difference indicates that the video is ahead includes: if the displayed timestamp difference indicates that the video is ahead and the audio buffer level is higher than a preset audio level threshold, then increasing the audio playback speed.

[0077] In other words, when the timestamp difference indicates that the video is ahead, the playback device can further determine whether the audio buffer level is higher than a preset audio buffer level threshold. Typically, this preset audio buffer level threshold can be set to a safety threshold higher than the audio buffer level. If the audio buffer level is sufficient (e.g., higher than this threshold), the playback device chooses to increase the audio playback speed to catch up with the video, because audio data is usually small in size and decodes quickly, and accelerated playback is less likely to cause stuttering. If the audio buffer level is insufficient, the playback device may choose to reduce the video playback speed to avoid exhausting the audio buffer due to accelerated audio playback.

[0078] Similarly, in scenarios where audio is ahead of the video, the playback device can also refer to the video buffer level: if the video buffer level is higher than another preset level (e.g., much higher than the safety threshold), the playback speed can be appropriately increased to use the excess data in the video buffer to catch up with the audio, while consuming any video data that may be piling up.

[0079] Taking a game live streaming scenario as an example, assuming a preset safety threshold of 2 seconds of playback and a preset synchronization threshold of ±200 milliseconds. The current video buffer level is 2.5 seconds (not lower than the safety threshold), and the playback device detects a timestamp difference of +250 milliseconds (audio is 250 milliseconds ahead, exceeding the threshold). The playback device chooses to reduce the audio playback speed to 0.95x while keeping the video speed unchanged. After a brief adjustment, the audio PTS gradually converges to the video, the timestamp difference falls back to +150 milliseconds, and the playback device resumes normal audio speed. During this process, the user hardly notices the change in playback speed, but the audio-visual synchronization deviation is corrected.

[0080] In another example, the video buffer level is 3 seconds (sufficient), and the displayed timestamp difference is -300 milliseconds (video is 300 milliseconds ahead). The playback device checks the audio buffer level and finds that the audio buffer stores approximately 2 seconds of playable data (above the preset audio buffer level threshold), so it chooses to increase the audio playback speed to 1.05x. After the audio playback speed is accelerated, it gradually catches up with the video, and the displayed timestamp difference shrinks to -100 milliseconds. Subsequently, the playback device resumes the normal audio speed. The entire adjustment process is smooth and imperceptible, avoiding the stuttering caused by frame skipping or dropped frames. If the audio buffer level is low (e.g., only 0.5 seconds of playback time), the playback device chooses to reduce the video playback speed to 0.95x, slowing down the video slightly while maintaining the normal audio speed, waiting for the audio buffer data to accumulate before making further adjustments.

[0081] Based on some embodiments of this specification, when the video buffer level is normal, by monitoring the difference in display timestamps and adjusting at least one of the audio and video playback speeds accordingly, the playback device can actively correct audio-visual synchronization deviations while maintaining smooth playback. This control method takes advantage of the sufficient video buffer, allowing for flexible selection of various methods such as accelerating video, accelerating audio, slowing down video, or slowing down audio, without worrying about causing new stuttering. In particular, by simultaneously monitoring the audio and video buffer levels, the playback device can make more intelligent decisions: when one buffer has sufficient data, it prioritizes accelerating that side to catch up with the other, thereby achieving smooth synchronization without sacrificing data integrity; when one buffer has insufficient data, it slows down the other side to avoid data exhaustion. This linkage control based on dual buffer levels and PTS differences enables the playback device to maintain audio-visual synchronization in a subtle, user-unnoticed manner, avoiding the playback jumpiness and content loss caused by frame skipping or dropping frames in traditional solutions, providing users with a more immersive viewing experience.

[0082] Figure 3 This is a schematic diagram of the playback control process provided in one embodiment of this specification.

[0083] like Figure 3 As shown, in some embodiments, the playback device is controlled according to the following logic: When the video buffer level falls below a preset safety threshold, the playback device reduces the video playback speed to replenish the buffer and adjusts the audio playback speed accordingly based on the displayed timestamp difference. If the displayed timestamp difference indicates audio-video synchronization (PTS-DIFF not reaching the threshold), the audio playback speed can be reduced by the same magnitude as the video playback speed. If the displayed timestamp difference indicates audio is ahead, the audio playback speed is reduced by a larger magnitude (e.g., video reduced to 0.9x speed, audio reduced to 0.8x speed). If the displayed timestamp difference indicates video is ahead, the audio playback speed is reduced by a relatively small magnitude, left unadjusted, or even appropriately accelerated (e.g., video reduced to 0.9x speed, audio maintained at 1.0x speed or increased to 1.02x speed), provided that the audio buffer has sufficient data.

[0084] When the video buffer level is not lower than a preset safety threshold, but the absolute value of the displayed timestamp difference exceeds a preset synchronization threshold, the playback device adjusts according to the lead direction. If the displayed timestamp difference indicates that the audio is ahead, the audio playback speed is slowed down (e.g., the audio speed is reduced to 0.95x). If the displayed timestamp difference indicates that the video is ahead, the playback speed is adjusted based on the audio buffer level: if the audio buffer level is sufficient (above the preset audio level threshold), the audio playback speed is accelerated (e.g., the audio speed is increased to 1.05x); if the audio buffer level is insufficient, the video playback speed is reduced (e.g., the video speed is reduced to 0.95x).

[0085] Based on such Figure 3 Some embodiments shown in this specification use the video buffer level as the first priority criterion. When the buffer is sufficient, the audio playback speed is adjusted first to correct audio-visual synchronization deviations. When the buffer is insufficient, the video speed is reduced first to build up buffer space, and the audio is adjusted accordingly. The playback device can adopt differentiated control strategies under different playback states. This logic fully utilizes the video buffer level information as the basis for decision-making, avoiding stuttering caused by forcibly adjusting the video speed when the buffer is insufficient, and also avoiding excessive conservatism that allows audio-visual desynchronization to worsen when the buffer is sufficient. At the same time, by monitoring the audio buffer level and determining the adjustment method (accelerating audio or slowing down video) when the video is ahead, the playback device can choose a safer operation path, reducing the risk of buffer exhaustion due to accelerated playback. This hierarchical decision-making and conditional execution control mechanism helps to achieve smooth audio-visual synchronization while ensuring playback continuity.

[0086] Figure 4 This is a schematic diagram of the playback control process provided for another embodiment of this specification.

[0087] like Figure 4 As shown, in some embodiments, the playback device is controlled according to the following logic: The playback device first determines whether the absolute value of the displayed timestamp difference exceeds a preset synchronization threshold. If it does, adjustments are made according to the direction of the lead. If the displayed timestamp difference indicates that the audio is ahead, the audio playback speed is slowed down. If the displayed timestamp difference indicates that the video is ahead, the playback speed is adjusted based on the audio buffer level: if the audio buffer level is sufficient (above the preset audio buffer level threshold), the audio playback speed is accelerated; if the audio buffer level is insufficient, the video playback speed is reduced.

[0088] If the absolute value of the displayed timestamp difference does not exceed the preset synchronization threshold (i.e., audio and video synchronization is normal), but the video buffer level is lower than the preset safety threshold, the playback device will implement a buffering strategy when the buffer level is insufficient: reduce the video playback speed and adjust the audio accordingly based on the displayed timestamp difference (which is within the threshold). When reducing the video playback speed, the subsequent execution process can be referenced... Figure 3 .

[0089] Based on such Figure 4 Some embodiments shown in this specification, by using the display timestamp difference as the first priority criterion, allow the playback device to prioritize responses to audio-visual asynchrony issues directly perceptible to the user, intervening immediately when the deviation exceeds the tolerable range. This intervention primarily uses audio playback speed adjustment because audio data is small and decodes quickly; accelerating or decelerating audio has minimal impact on the buffer and is less likely to cause stuttering. When the audio buffer is insufficient, the playback device switches to reducing video playback speed as an alternative, ensuring that adjustment methods are available under various buffering conditions. When audio-visual synchronization is normal, the playback device then checks the video buffer level and performs a buffer replenishment operation to prevent future stuttering. This "synchronize first, replenish later" control sequence is suitable for scenarios with high audio-visual synchronization quality requirements. It proactively corrects deviations before the user perceives the asynchrony, while not neglecting the health of the buffer, thus balancing synchronization accuracy and playback smoothness.

[0090] While one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is merely one possible execution order among many steps and does not represent the only possible execution order. The order of some steps may be adjusted according to actual needs, or some steps may be omitted. When the claims involve method steps, changes in the order of such steps, or parallel execution between steps, are also within the scope of protection of the claims.

[0091] Figure 2The method described in this paper monitors both the video buffer level and the audio / video display timestamp difference simultaneously, incorporating both into a single decision-making framework. This allows for automatic selection of appropriate adjustment directions under different playback states. When the video buffer level is low, the playback device prioritizes reducing the video playback speed to accumulate data reserves. Simultaneously, it adjusts the audio speed based on the current audio-visual synchronization deviation, ensuring that deceleration is not introduced or exacerbated by simply slowing down the buffer during the buffer accumulation process. When the video buffer level is sufficient, the playback device focuses on the audio-visual synchronization deviation, flexibly adjusting the audio or video speed according to the direction of the deviation to restore harmony between the audio and video. This hierarchical decision-making mechanism allows the playback device to prioritize continuity when resources are scarce and prioritize synchronization accuracy when resources are abundant, smoothly switching between the two states without relying on aggressive measures such as image quality reduction or data discarding. Through subtle speed adjustments, the playback device can manage both the buffer health and audio-visual synchronization quality almost imperceptibly to the user, thereby providing a smoother, more synchronized, and higher-quality viewing experience overall.

[0092] based on Figure 2 In addition to the method described herein, this specification also provides some improved implementation methods, which will be described below.

[0093] In one or more embodiments of this specification, the playback method further includes an active buffering operation performed within a time period after playback begins. Specifically, after playback begins, the audio playback data and the video playback data are played at a coordinated deceleration until the video buffer level reaches a preset buffering threshold. This stage can be referred to as the pre-playback phase, and its goal is to proactively and quickly build up the data reserves of the video buffer without the user's awareness, forming a "reservoir" capable of resisting network jitter.

[0094] In some embodiments, coordinated slowdown playback refers to playing audio and video simultaneously at a rate slightly lower than the normal playback speed. For example, if the normal playback speed is 1.0x, the playback speed can be adjusted to 0.95x during the initial playback phase. This slight slowdown is usually beyond the user's perception range, and ordinary viewers are unlikely to notice the change in playback speed. However, from a data accumulation perspective, the amount of data consumed per second of playback is reduced by about 5% compared to normal, while the network receiving speed remains unchanged (or may even be higher due to accelerated transmission during the initial playback phase), thus the amount of data in the video buffer gradually increases.

[0095] In some embodiments, the deceleration amplitude used in the early stages of playback (e.g., 0.95x speed) is less than the deceleration amplitude corresponding to the deceleration parameters determined based on the display timestamp differences during the stable playback phase (e.g., 0.9x speed or 0.6x speed). This is because in the early stages of playback, the playback device does not face an urgent risk of stuttering and only needs to gently accumulate buffer data; however, if insufficient buffer data occurs during the stable playback phase, it usually means that the network condition is already poor, requiring a larger deceleration to urgently replenish the buffer. The difference in deceleration amplitude between the two scenarios reflects the playback device's approach of responding in stages according to the degree of risk.

[0096] Taking e-commerce live streaming as an example, after a user enters the live stream, the playback device begins receiving audio and video data. In the initial stage of playback, the playback device plays the audio and video at 0.95x speed. Assuming the network reception rate is stable at the data volume corresponding to 1.0x speed, the buffer increases by approximately 0.05 seconds of data per second. After about 20 seconds of accumulation, the buffer can increase its data reserve by about 1 second, thus significantly improving its ability to withstand network fluctuations. When the video buffer level reaches a preset threshold (e.g., corresponding to 2 seconds of playback data), the playback device gradually resumes normal playback speed.

[0097] In some embodiments, the entire playback process may include at least a pre-playback phase and a stable playback phase. In the pre-playback phase, the playback device plays audio and video data at a first speed lower than the normal playback speed to actively accumulate data in the buffer. Once the buffer level reaches a preset threshold, the playback device enters the stable playback phase and performs the aforementioned steps. Figure 2 The described process (i.e., dynamic adjustment based on the difference between buffer level and display timestamp) divides the playback device into two stages. This allows the playback device to proactively build up buffer reserves in the first stage and then use these reserves to cope with network fluctuations and synchronization deviations in the second stage, forming a complete control link from proactive prevention to dynamic response.

[0098] Taking a sports live stream as an example, a user enters the live stream room during a match. After completing the first frame display and initial synchronization calibration during the start-up phase, the playback device enters the early playback phase. At this time, the video buffer level is 0.3 seconds (far below the preset buffer threshold of 2 seconds). The playback device synchronously reduces the audio and video playback speed to 0.95x, while the network continues to receive data at the normal rate. After about 30 seconds of gentle slowdown playback, the video buffer level gradually rises to 2 seconds. The playback device detects that the level has reached the target and gradually restores the playback speed to 1.0x, switching to the stable playback phase. Even if there are brief network fluctuations (such as a 2-second bandwidth drop), the playback device can still maintain smooth playback because the buffer has a 2-second reserve, and the user will not perceive any stuttering.

[0099] Based on some embodiments of this specification, in the early stages after playback begins, by coordinating audio and video playback at a deceleration level imperceptible to the user, the playback device can proactively accumulate data in the video buffer without affecting the viewing experience. This proactive buffering strategy allows the playback device to establish sufficient buffer reserves before entering the stable playback stage, essentially providing a "stuttering prevention reservoir" for subsequent playback. When network jitter occurs, the playback device can utilize this reserve to continue outputting the image without degrading the image quality or experiencing frequent stuttering. Unlike passively waiting for stuttering to occur and then taking remedial measures, this strategy proactively improves the system's anti-jitter capability from the initial playback stage. Simultaneously, because the deceleration is controlled within a range imperceptible to the user (e.g., 0.95x speed), viewers hardly notice the change in playback speed, maintaining a natural and smooth experience. In this way, the playback device can more reliably maintain a higher bitrate for playback, without being forced to reduce image quality in weak network environments as with some traditional methods, thus providing users with higher average image quality and a more stable viewing experience.

[0100] In one or more embodiments of this specification, the playback method further includes a quick start and initial synchronization operation performed at the start of playback (i.e., the start-up phase). The goal of the start-up phase is to present the first frame to the user in the shortest possible time and ensure that the audio and video are synchronized at the start of playback, thus establishing a clean synchronization starting point for the subsequent early and stable playback phases.

[0101] In some embodiments, the playback process may include three consecutive time phases: the initiation phase, the pre-playback phase, and the stable playback phase. The initiation phase refers to the time period from when the user initiates a playback request to when the playback device completes the rendering of the first frame and begins normal playback; the pre-playback phase refers to the active buffering phase after the initiation phase but before the video buffer level reaches a preset buffering threshold; the stable playback phase refers to the phase after the video buffer level reaches a safety threshold, where the playback device dynamically adjusts the buffer based on the real-time status.

[0102] Specifically, the playback method further includes: receiving start-up media data transmitted by the server in accelerated sending mode when playback starts; aligning the display timestamps of the first frame audio data and the first frame video data in the start-up media data; and using the aligned first frame audio data and the first frame video data as the playback starting point.

[0103] In some embodiments, upon playback initiation, the playback client receives initiation media data transmitted by the server in accelerated transmission mode. Accelerated transmission mode refers to the server pushing data to the playback client at a transmission rate higher than the media data encoding playback rate. For example, if the video's encoding playback rate is 30 frames per second (corresponding to approximately 300KB of data per second), the server can send data at a rate of 600KB per second during the initiation phase, allowing the playback client to receive a sufficient amount of the first frame data in a shorter time. This accelerated transmission is typically achieved through a low-latency transmission network, such as a content delivery network optimized based on a real-time transport protocol, a WebRTC data channel, or the QUIC protocol. In some embodiments, the Global Real-Time Transport Network (GRTN) can be used to provide accelerated push capabilities during the initiation phase, but the embodiments described in this specification are not limited to this; any network transmission method capable of achieving similar accelerated transmission functionality is applicable.

[0104] It should be noted that the accelerated transmission mode is mainly used during the initial playback phase. In the subsequent early and stable playback phases, the playback client can continue to receive data through a regular content delivery network (such as a standard CDN based on RTMP or HLS protocols) to reduce transmission costs. Of course, using a low-latency transmission network throughout is also feasible, but from a cost-effectiveness perspective, the marginal benefit of its ultra-high real-time performance will decrease after the initial playback phase.

[0105] In some embodiments, after receiving the start-up media data, the playback terminal aligns the display timestamps of the first frame audio data and the first frame video data in the data.

[0106] Specifically, the playback device compares the display timestamp of the first frame of audio data with the display timestamp of the first frame of video data, and performs the following operations based on the comparison result: If the timestamp of the first audio frame is earlier than the timestamp of the first video frame (i.e., the scheduled playback time of the first audio frame is earlier than the first video frame), the playback device discards the earlier audio frame. Since audio frames are usually short and independently decodeable, discarding a small number of audio frames will not have a significant impact on subsequent playback, but it can ensure that the start time of the audio playback is aligned with the video.

[0107] If the display timestamp of the first frame of audio data lags behind the display timestamp of the first frame of video data (i.e., the scheduled playback time of the first frame of audio is later than the first frame of video), a silent frame is inserted at the playback end. A silent frame is a placeholder data that does not contain valid audio content but has the correct timestamp. Its insertion ensures that the audio channel has corresponding audio output when the first frame of the video is presented (although it is muted), thus avoiding the awkward situation of the video appearing on screen but without sound.

[0108] Through the above alignment operation, the playback end obtains a pair of first-frame audio and first-frame video with matching display timestamps, and uses the aligned first-frame audio data and first-frame video data as the playback starting point.

[0109] Taking e-commerce live streaming as an example, a user clicks to enter the live stream during a product introduction. The playback client sends a playback request to the server, which pushes the starting media data to the playback client via a low-latency transmission network in an accelerated sending mode (e.g., 1.5 times the normal rate). The playback client receives the first set of audio and video data packets within approximately 200 milliseconds. The playback client extracts the PTS (Programmable Time Slot) of the first frame of audio data as 1000 milliseconds and the PTS of the first frame of video data as 1100 milliseconds. Calculating PTS_Diff = 1000 - 1100 = -100 milliseconds, this means the audio lags behind the video by 100 milliseconds. The playback client generates a 100-millisecond silent frame and inserts it before the audio data, adjusting the audio's start playback time to 1100 milliseconds. Subsequently, the playback client begins rendering with the aligned first frame of audio (including the silent frame) and the first frame of video. The first frame the user sees has corresponding audio output (although it is silent), avoiding the phenomenon of video appearing first but without sound. The entire process, from user click to first frame display, can be controlled within 500 milliseconds, enabling rapid start-up.

[0110] In another example, the first frame audio PTS is 1200 milliseconds, the first frame video PTS is 1000 milliseconds, and PTS_Diff = +200 milliseconds, meaning the audio is 200 milliseconds ahead. The playback device can discard the ahead portion of the audio data (or discard the entire first frame audio), adjusting the audio's start playback time to 1000 milliseconds, aligning it with the video. The first sound the user hears and the first image they see are synchronized in time.

[0111] Based on some embodiments of this specification, during the start-up phase, by receiving start-up media data transmitted by the server in accelerated sending mode, the playback end can significantly shorten the acquisition time of the first frame data, allowing users to see the live broadcast faster. Specifically, the accelerated sending mode utilizes the characteristics of low-latency transmission networks, centrally pushing key data at the moment playback starts, compensating for the shortcomings of traditional playback methods that require waiting for buffer accumulation before rendering can begin during the start-up phase. Furthermore, by aligning the display timestamps of the first frame audio and the first frame video (discarding audio frames when ahead and inserting silent frames when behind), the playback end establishes a baseline state of audio-visual synchronization at the start of playback. This initial synchronization calibration eliminates the need to handle the abrupt transition from desynchronization to synchronization during the subsequent early and stable playback phases, reducing the burden of subsequent adjustments. Therefore, the accelerated reception and first frame alignment during the start-up phase work together to achieve both rapid first-screen presentation and accurate initial audio-visual relationships, providing a solid foundation for a good user experience from the very first second of viewing.

[0112] In one or more embodiments of this specification, the playback method further includes detecting and handling stuttering that may occur during playback. Stuttering refers to a pause in the video or interruption of the audio that is perceptible to the user, and technically it usually manifests as the player's inability to output the next frame of data on time.

[0113] In some embodiments, playback further includes pausing audio and video playback when the playback timestamp interval between adjacent video frames exceeds a preset stuttering threshold.

[0114] Specifically, the playback device detects the playback timestamp interval between adjacent video frames (such as the actual interval between the rendering timestamps of two adjacent frames). When this time interval exceeds a preset stuttering threshold, the playback device determines that stuttering has occurred and pauses audio and video playback (i.e., pauses rendering). For example, the preset stuttering threshold can be set to 200 milliseconds or 500 milliseconds, and the specific value can be configured according to the tolerance of the application scenario. For example, in live streaming scenarios with high interactivity requirements, a threshold of 200 milliseconds can be selected to more sensitively capture stuttering; in general content viewing scenarios, a threshold of 500 milliseconds can be selected to avoid triggering stuttering processing too frequently.

[0115] In practical applications, stuttering is usually caused by the video buffer running out of space (the buffer level drops to 0 or extremely low), meaning the playback device has no more data frames to render. However, the buffer level decreases gradually from "insufficient" to "exhausted." If the stuttering process is initiated prematurely (e.g., pausing rendering) when the buffer level falls below a certain intermediate threshold, it may cause premature pausing—at which point the user has not yet perceived any stuttering, but the playback device has already actively stopped outputting video, thus affecting the normal viewing experience. Therefore, using "user-perceived stuttering" (i.e., an abnormally increased frame interval) as the entry condition ensures that stuttering is only triggered when the user actually feels a pause in playback, avoiding unnecessary intervention.

[0116] When the playback device detects a stutter and pauses rendering, it stops retrieving data from the buffer for decoding and display, but continues to receive and store media data from the network into the buffer. At this point, the playback device is in a "waiting for data accumulation" state, with the goal of raising the video buffer level to a sufficiently safe level as quickly as possible.

[0117] In some embodiments, after the playback is paused, the audio and video playback is resumed in a decelerated manner when the video buffer level recovers to the recovery threshold; wherein the recovery threshold is lower than the preset safety threshold.

[0118] Specifically, after playback is paused, the playback device continuously monitors the video buffer level. When the video buffer level recovers to a preset recovery threshold, the playback device resumes audio and video playback at a slower pace. The recovery threshold is a value lower than a preset safety threshold. For example, the preset safety threshold might correspond to 2 seconds of playable data, while the recovery threshold might correspond to 1.5 seconds of playable data. The reason the recovery threshold is lower than the safety threshold is that the playback device doesn't need to wait for the buffer to fully fill to a safe level before resuming playback. Instead, it can start playing at a slower speed when the buffer level reaches a relatively reliable value (e.g., 1.5 seconds), while continuing to accumulate data during playback until the buffer level reaches the safety threshold before resuming normal speed. This helps to shorten the perceived stuttering time for the user.

[0119] Resuming playback at a slower speed means that the playback device starts rendering audio and video data at a rate lower than the normal playback speed (e.g., 0.9x speed). Slowing down playback reduces the amount of data consumed per unit of time, allowing the buffer level to gradually increase until it reaches a safe threshold. Once the video buffer level reaches the preset safe threshold, the playback device can gradually resume normal playback speed.

[0120] Taking a game live streaming scenario as an example, the preset stuttering threshold is 200 milliseconds, the preset safety threshold is 2 seconds of playback time, and the recovery threshold is 1.5 seconds of playback time. During normal playback, due to a sudden network interruption, the video buffer level drops rapidly from 2 seconds to 0. The playback client detects that the playback timestamp interval between two consecutive video frames is 250 milliseconds (exceeding 200 milliseconds), determines that stuttering has occurred, and immediately pauses audio and video rendering. At this time, the user sees the screen freeze, but the playback client continues to receive data from the network in the background. After about 0.5 seconds of network recovery and data download, the video buffer level rises to 1.5 seconds (reaching the recovery threshold). The playback client resumes playback at 0.9x speed, and the user sees the screen start moving again, but at a slightly slower speed than normal. As playback continues, the buffer level rises further to 2 seconds, and the playback client gradually adjusts the playback speed back to 1.0x speed. The entire stuttering process lasts about 1 second, and after a brief screen freeze, playback returns to smooth.

[0121] In another example, if network recovery is slow and the video buffer level fails to reach the recovery threshold for an extended period, the playback device remains paused until the buffer level reaches the threshold. This strategy avoids frequent short stutters (e.g., "stutter, play, stutter again") after a single stutter, allowing users to experience continuous smooth playback after a longer stutter, which is more acceptable than multiple short stutters.

[0122] Based on some embodiments of this specification, by using the time interval between playback timestamps of adjacent video frames exceeding a preset stuttering threshold as a stuttering detection condition, the playback terminal can accurately identify the moment of playback stagnation perceptible to the user and trigger a pause in rendering at that moment to accelerate data accumulation. This detection method, guided by actual user perception, can avoid interruptions caused by premature pauses. After a pause, the playback terminal uses a recovery threshold as a recovery condition. This threshold is lower than the safety threshold for normal playback, allowing the playback terminal to resume playback in a decelerated manner before the buffer is fully filled, thereby shortening the user's waiting time. Deceleration recovery further ensures that the buffer level can continue to grow during playback, gradually transitioning to a safe state. This differentiated design of the entry and exit conditions for the stuttering stage allows the playback terminal to replace multiple short, frequent stutters with a relatively long but controllable stutter. From the user's perspective, continuous smoothness after a single stutter is more acceptable than repeated stutters. Overall, this stuttering handling method takes into account the accuracy of detection, the timeliness of recovery, and the smoothness of transition, helping to optimize the user's viewing experience as much as possible under unavoidable network fluctuations.

[0123] In one or more embodiments of this specification, in addition to the aforementioned control mechanism for insufficient video buffer level, the playback terminal can also take corresponding measures to address situations where the video buffer level may be too high (i.e., data accumulation). When the network transmission rate is consistently higher than the playback consumption rate, the amount of data in the video buffer may continuously increase, approaching or exceeding the buffer capacity limit, posing a risk of buffer overflow, increased memory pressure, or even playback abnormalities. Therefore, the playback terminal can coordinate control from both the data inflow and outflow directions.

[0124] In some embodiments, when the playback device detects that the video buffer level exceeds a preset upper limit threshold, it sends a rate-limiting request to the server. This request instructs the server to reduce the video data transmission rate. Upon receiving the request, the server can adjust its streaming or forwarding strategy, such as reducing the transmission bitrate, extending the transmission interval, or switching to a lower bitrate video stream, thereby reducing the amount of data pushed to the playback device. Through this feedback mechanism, the playback device can proactively control the data inflow rate and prevent the buffer from being overfilled.

[0125] In some embodiments, the playback client can increase the video playback speed while sending a rate-limiting request to the server. Increasing the playback speed means increasing the amount of video data consumed per unit time, thereby accelerating the data outflow from the buffer. For example, the playback client can increase the video playback speed from the normal 1.0x speed to 1.05x or 1.1x speed, causing the buffer level to drop more quickly. It should be noted that the increase in playback speed should be controlled within a range imperceptible to the user to avoid affecting the viewing experience.

[0126] In some embodiments, if the video buffer level still exceeds a preset upper limit threshold after the playback speed is increased (i.e., the outflow acceleration fails to alleviate the backlog in time), the playback device can discard some video frames. Discarded video frames are typically non-critical frames (such as B-frames or non-reference P-frames) that have minimal impact on the user's visual experience, or frames are discarded in order of timestamp from oldest to newest to quickly reduce the buffer level.

[0127] In some embodiments, when the playback end increases the video playback speed to accelerate the outgoing stream, it can simultaneously increase the audio playback speed. Since audio and video need to maintain synchronization during playback, increasing only the video speed while keeping the audio speed constant may exacerbate audio-visual asynchrony. Therefore, the playback end can increase the audio playback speed to the same multiple as the video speed, allowing both to play at synchronized acceleration, thereby alleviating video buffer congestion while maintaining audio-visual synchronization. If the audio buffer itself also experiences congestion, synchronously accelerating the audio will also help consume the data in the audio buffer.

[0128] The aforementioned control strategies, implemented from both inbound (rate limiting requests) and outbound (accelerated playback, frame dropping) directions, can be combined with the previously mentioned slow-down and buffering strategy based on insufficient buffer level to form a two-way management of the video buffer level. When the buffer level is low, the playback device reduces the playback speed and may request the server to accelerate transmission; when the buffer level is high, the playback device increases the playback speed and may request the server to rate limit transmission. This two-way, closed-loop buffer management mechanism helps maintain the video buffer level within a reasonable range, avoiding stuttering due to excessively low buffer levels and preventing memory pressure or playback abnormalities due to excessively high buffer levels, thereby ensuring long-term playback stability.

[0129] Based on some embodiments of this specification, by monitoring the video buffer level and setting an upper limit threshold, the playback device can proactively take two-way control measures when data accumulates. Sending a rate-limiting request to the server controls the data inflow rate at the source, reducing the pressure on the buffer to continue growing; increasing the video playback speed accelerates data consumption at the playback device, thus mitigating the accumulation problem through a two-pronged approach. When accelerating playback is still insufficient to suppress the rising buffer level, frame dropping provides a final safeguard as a supplementary measure. Simultaneously, the synchronized adjustment of audio playback speed ensures that the accelerated outflow process does not disrupt the audio-visual synchronization. This hierarchical processing method, from inflow to outflow, from requesting rate limiting to proactive acceleration to selective frame dropping, provides a multi-layered and selectable solution to the buffer accumulation problem. The playback device can choose the most appropriate combination of measures based on the severity of the accumulation, helping to minimize the impact on user experience while avoiding buffer overflow.

[0130] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they have not been described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.

[0131] Figure 5 This is a flowchart illustrating a phased strategy for media stream data playback in a practical application scenario provided by an embodiment of this specification.

[0132] like Figure 5 As shown in the embodiments of this specification, an audio and video buffer linkage control method based on a state machine is provided. This method divides the entire playback process into four stages: playback start, pre-playback, stuttering handling, and stable playback, and designs targeted linkage strategies for each stage, forming a full-link staged state machine model.

[0133] During the initial playback phase, the playback device can receive the media stream sent by the server at double speed through low-latency transmission networks such as GRTN to quickly obtain the first frame data. Furthermore, it can implement a display timestamp (PTS) alignment strategy on the received first frame audio data and first frame video data (e.g., discard the leading audio frame when the audio is ahead, and insert a silent frame when the audio is lagging), to establish an initial synchronization benchmark for subsequent playback.

[0134] After playback begins, the system enters the pre-playback phase. During this phase, the playback device coordinates the slowdown of audio and video data (e.g., playing at 0.95x speed) to actively reduce playback speed and accumulate buffer data. Throughout this process, the playback device continuously checks whether the video buffer level has reached a preset stability threshold (e.g., higher than or equal to a preset safety threshold). If it hasn't, the coordinated slowdown playback continues, forming a loop until the buffer level meets the stability condition; if the stability threshold has been reached, the system transitions to the stable playback phase. This proactive slowdown and buffer accumulation strategy in the pre-playback phase, using a slight, imperceptible slowdown to rapidly increase the buffer level, is a proactive anti-lag measure.

[0135] During the stable playback phase, the playback device continuously monitors the status of the audio and video buffers and the audio-video display timestamp difference (PTS Diff). When the PTS Diff exceeds a preset synchronization threshold, the playback device performs a PTS alignment correction operation, specifically employing a complementary micro-perturbation synchronization method based on dual buffer levels: according to the respective buffer levels of audio and video, opposite, minute playback speed perturbations are applied (e.g., a higher video buffer level accelerates video playback, and a higher audio buffer level accelerates audio playback; conversely, a lower video buffer level slows down audio playback, and vice versa), thereby correcting synchronization deviations while maintaining buffer balance, achieving a smooth and imperceptible synchronization effect. Furthermore, during the stable playback phase, if stuttering occurs (e.g., the playback timestamp interval between two adjacent video frames exceeds a stuttering threshold), the playback device switches to a stuttering handling phase.

[0136] During the stuttering process, the playback device first pauses the audio and video synchronization, halting rendering while simultaneously accelerating data download from the network to accumulate buffer data. The playback device continuously checks whether the video buffer level has recovered to a preset recovery threshold (e.g., below or equal to a preset safety threshold). If it hasn't recovered, rendering remains paused and data accumulation continues. Once the buffer level reaches the recovery threshold, the playback device exits the stuttering process and re-enters the pre-playback phase or directly transitions to the stable playback phase (depending on the specific implementation). Playback can be resumed at a slower speed, thus completing a full stuttering recovery process.

[0137] The division and connection of the above four stages construct a complete audio and video buffer linkage control system. This system breaks away from the traditional isolated management of audio and video buffers, unifying the analysis and collaborative decision-making regarding the status of the audio and video buffers and their PTS (Programmable Streaming Time) differences. Through rapid startup and initial alignment in the playback initiation stage, proactive buffer building in the early playback phase, complementary micro-perturbation synchronization in the stable playback stage, and rapid recovery in the stuttering handling stage, it simultaneously optimizes playback smoothness and synchronization accuracy without significantly reducing image quality, aiming to provide users with an immersive live streaming viewing experience.

[0138] Figure 6 This is a system block diagram for media stream data playback in a practical application scenario provided by an embodiment of this specification.

[0139] like Figure 6 As shown in the embodiments of this specification, a live streaming optimization system is provided. The system includes a cloud server and a playback terminal. The playback terminal internally includes an audio processing link, a video processing link, and a buffer equalizer and an audio / video synchronization module for coordinating audio and video.

[0140] Audio Processing Link: The cloud server sends audio data packets to the playback end via Real-Time Transport Protocol (RTP). The playback end first stores the received audio RTP packets into an audio RTP packet buffer. This buffer supports latency and jitter resistance and supports packet loss correction. The buffered and corrected audio data is then sent to the audio decoder for decoding. The decoded audio Pulse Code Modulation (PCM) data is available for rendering and playback. In the audio processing link, the playback end can perform pause, resume, speed up, or slow down operations on the audio decoding or rendering process according to the acceleration / deceleration strategies from the buffer equalizer, in order to adjust the overall playback state.

[0141] Video processing chain: The cloud server sends video data packets to the playback end via RTP. The playback end stores the video RTP packets in a video RTP packet buffer. The video RTP packet buffer stores the raw RTP data packets received from the cloud before jitter processing. This buffer supports latency and jitter resistance and supports packet loss retransmission mechanisms. The buffered and retransmitted video data is sent to the video decoder for decoding. The decoded video frames are stored in the video frame buffer. Subsequently, the video rendering module retrieves frames from the video frame buffer for smooth rendering output. The video processing chain can also perform pause, resume, speed up, or slow down operations on the video decoding or rendering process based on the acceleration / deceleration strategies from the buffer equalizer.

[0142] The playback device monitors the status information of each level of audio and video buffers in real time, including the audio RTP packet buffer, video RTP packet buffer (jitter buffer), and video decoding frame buffer. This multi-level buffer status information is reported to the buffer equalizer. As the system's decision-making unit, the buffer equalizer receives the status information from the multi-level audio and video buffers and determines the audio / video acceleration / deceleration strategy. The buffer equalizer formulates acceleration / deceleration control commands for the audio and video (e.g., acceleration, deceleration, pause, resume) and issues them to the audio and video processing links for execution.

[0143] The audio-video synchronization module can determine the audio-video synchronization status based on the PTS difference between audio and video. This module feeds back the judgment result to the buffer equalizer, which serves as one of the bases for the buffer equalizer to formulate acceleration and deceleration strategies. Through the linkage between the buffer equalizer and the audio-video synchronization module, the system can achieve coordinated control based on multi-level buffer levels and PTS differences, maintaining audio-video synchronization while ensuring smooth playback.

[0144] During playback, the cloud continuously pushes audio and video RTP streams to the playback device. The playback device receives data through the audio RTP packet buffer and the video RTP packet buffer respectively, performs anti-jitter, packet loss correction or retransmission processing, and then sends it to their respective decoders for decoding. The decoded audio data is directly prepared for rendering, and the decoded video frames are stored in the video frame buffer. The buffer equalizer collects the status of each level of buffer (such as fill length) and PTS difference information provided by the audio and video synchronization module in real time, and dynamically generates acceleration and deceleration strategies. When water storage is needed to prevent stuttering, the equalizer instructs the audio and video links to decelerate together; when audio and video desynchronization needs to be corrected, the equalizer instructs the leading side to decelerate or the lagging side to accelerate; when stuttering occurs, the equalizer can instruct to pause rendering to accelerate data accumulation. Through the above multi-level buffer linkage and audio and video coordinated control, the system can simultaneously optimize playback smoothness and synchronization accuracy without significantly reducing image quality.

[0145] Furthermore, in some embodiments, the above-mentioned audio-video linkage control method can be adaptively adjusted according to the actual media type being played to cover pure video or pure audio playback scenarios, thereby improving the applicability of the solution. Specifically, when the playback content only contains video streams and not audio streams (i.e., pure video scenario), the playback end no longer performs audio-related monitoring and adjustment operations, but degenerates to only monitoring the real-time level of the video buffer and adjusting the video playback speed accordingly based on the level status (e.g., slowing down playback to store water when the level is too low). When the playback content only contains audio streams and not video streams (i.e., pure audio scenario), the playback end may no longer consider the synchronization relationship between audio and video, only monitor the real-time level of the audio buffer, and adjust the audio playback speed according to the level status (e.g., slowing down playback when the level is too low, accelerating playback when the level is too high, or notifying the server to limit the flow). Through the above-mentioned degradation control strategies for different media types, the playback method proposed in this disclosure can leverage the advantages of linkage control when audio and video coexist, and still maintain basic buffer management capabilities in single media type scenarios, thus making it suitable for more diverse streaming media playback needs.

[0146] Based on the same idea, embodiments of this specification also provide apparatus corresponding to the above methods.

[0147] Figure 7 The embodiments provided in this specification correspond to Figure 2 A schematic diagram of the structure of a media streaming data playback device.

[0148] like Figure 7 As shown, the device may include: The playback monitoring module 702 is used to monitor the video buffer level in the video buffer and the difference in display timestamps between audio playback data and video playback data. The playback adjustment module 704 is used to: reduce the video playback speed and adjust the audio playback speed according to the display timestamp difference when the video buffer level is lower than a preset safety threshold; and adjust at least one of the audio playback speed and the video playback speed according to the display timestamp difference when the video buffer level is not lower than the preset safety threshold and the absolute value of the display timestamp difference exceeds a preset synchronization threshold.

[0149] based on Figure 7 The embodiments of this specification also provide some specific implementation schemes of the method, which are described below.

[0150] Optionally, adjusting the audio playback speed based on the display timestamp difference specifically includes: If the displayed timestamp difference indicates that the audio is ahead, the audio playback speed is reduced by a first deceleration magnitude; If the displayed timestamp difference indicates that the video is ahead, reduce the audio playback speed with a second deceleration magnitude, or do not adjust the audio playback speed, or increase the audio playback speed; Wherein, the first deceleration amplitude is greater than the deceleration amplitude of reducing the video playback speed, and the second deceleration amplitude is less than the deceleration amplitude of reducing the video playback speed.

[0151] Optionally, the video buffer includes a pre-decoding buffer and a post-decoding buffer; The video buffer level of the monitored video buffer specifically includes: Monitor the post-decoding water level of the post-decoding buffer and the pre-decoding water level of the pre-decoding buffer; The step of reducing video playback speed when the video buffer level falls below a preset safety threshold specifically includes: If the water level after decoding is lower than the threshold after decoding, and the water level before decoding is higher than or equal to the threshold before decoding, then the video playback speed is reduced by the third deceleration amplitude. If the water level after decoding is lower than the threshold after decoding, and the water level before decoding is lower than the threshold before decoding, then the video playback speed is reduced by a fourth deceleration amplitude; the fourth deceleration amplitude is greater than the third deceleration amplitude.

[0152] Optionally, adjusting at least one of the audio playback speed and the video playback speed based on the display timestamp difference specifically includes: If the displayed timestamp difference indicates that the audio is ahead, then at least one of reducing the audio playback speed and increasing the video playback speed is performed; If the displayed timestamp difference indicates that the video is ahead, then at least one of increasing the audio playback speed and decreasing the video playback speed is performed.

[0153] Optionally, the media streaming data playback device is further configured to: monitor the audio buffer level of the audio buffer; If the displayed timestamp difference indicates that the video is ahead, then performing at least one of increasing the audio playback speed and decreasing the video playback speed includes: If the displayed timestamp difference indicates that the video is ahead, and the audio buffer level is higher than the preset audio level threshold, then the audio playback speed is increased.

[0154] Optionally, the video buffer level being lower than a preset safety threshold includes: The amount of playable data stored in the video buffer is less than a preset data volume threshold.

[0155] Optionally, the media streaming data playback device is further configured to: After playback is initiated, the audio playback data and the video playback data are played at a reduced speed in a coordinated manner until the video buffer level reaches a preset water storage threshold.

[0156] Optionally, the media streaming data playback device is further configured to: When playback starts, receive the start-up media data transmitted by the server in accelerated sending mode; The first frame of audio data and the first frame of video data in the initiation media data are aligned with the display timestamps. The first frame of audio data and the first frame of video data after alignment are used as the starting point for playback.

[0157] Optionally, aligning the display timestamps of the first frame audio data and the first frame video data in the initiation media data specifically includes: Compare the display timestamp of the first frame audio data with the display timestamp of the first frame video data; If the display timestamp of the first frame of audio data is earlier than the display timestamp of the first frame of video data, then the earlier audio frame is discarded. If the display timestamp of the first frame of audio data lags behind the display timestamp of the first frame of video data, a silent frame is inserted.

[0158] Optionally, the media streaming data playback device is further configured to: When the playback timestamp interval between adjacent video frames exceeds the preset stuttering threshold, audio and video playback will be paused.

[0159] Optionally, after the playback is paused, the audio and video playback is resumed in a decelerated manner when the video buffer level recovers to the recovery threshold; wherein the recovery threshold is lower than the preset safety threshold.

[0160] Optionally, the audio playback data and the video playback data are received through independent transmission channels.

[0161] Optionally, the media stream data is live streaming media data.

[0162] It is understood that the modules mentioned above refer to computer programs or program segments used to perform one or more specific functions. Furthermore, the distinction between these modules does not imply that the actual program code must also be separate.

[0163] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0164] The above is an illustrative scheme of a media streaming data playback device according to this embodiment. It should be noted that the technical solution of this media streaming data playback device and the technical solution of the media streaming data playback method described above belong to the same concept. For details not described in detail in the technical solution of the media streaming data playback device, please refer to the description of the technical solution of the media streaming data playback method described above.

[0165] Based on the same idea, this specification also provides devices corresponding to the above methods in its embodiments.

[0166] Figure 8 A structural block diagram of a computing device 800 provided according to an embodiment of this specification is shown.

[0167] The computing device 800 includes: Memory 810 and processor 820; The memory 810 is used to store computer programs / instructions, and the processor 820 is used to execute the computer programs / instructions, which, when executed by the processor 820, implement the steps of the media stream data playback method.

[0168] Specifically, the components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and the database 850 is used to store data.

[0169] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0170] In one embodiment of this specification, the aforementioned components of the computing device 800 and Figure 8 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 8 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0171] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 800 can also be a mobile or stationary server.

[0172] The processor 820 executes the computer instructions to implement the media stream data playback method.

[0173] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the media stream data playback method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the media stream data playback method described above.

[0174] An embodiment of this specification also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the media stream data playback method described above.

[0175] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the media streaming data playback method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the media streaming data playback method described above.

[0176] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the media stream data playback method described above.

[0177] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the media stream data playback method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the media stream data playback method described above.

[0178] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the apparatus and device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The apparatus, device and method provided in the embodiments of this specification are corresponding to each other, and therefore the apparatus and device also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatus and device will not be repeated here.

[0179] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0180] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0181] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0182] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0183] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0184] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0186] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0187] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0188] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0189] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0190] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital character versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0191] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0192] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for playing media stream data, comprising: Monitor the video buffer level in the video buffer and the difference in display timestamps between audio playback data and video playback data; When the video buffer level is lower than a preset safety threshold, the video playback speed is reduced, and the audio playback speed is adjusted according to the difference in the display timestamps. When the video buffer level is not lower than the preset safety threshold and the absolute value of the display timestamp difference exceeds the preset synchronization threshold, at least one of the audio playback speed and the video playback speed is adjusted according to the display timestamp difference.

2. The method as described in claim 1, wherein adjusting the audio playback speed according to the display timestamp difference specifically includes: If the displayed timestamp difference indicates that the audio is ahead, the audio playback speed is reduced by a first deceleration magnitude; If the displayed timestamp difference indicates that the video is ahead, reduce the audio playback speed with a second deceleration magnitude, or do not adjust the audio playback speed, or increase the audio playback speed; Wherein, the first deceleration amplitude is greater than the deceleration amplitude of reducing the video playback speed, and the second deceleration amplitude is less than the deceleration amplitude of reducing the video playback speed.

3. The method as described in claim 1, wherein the video buffer includes a pre-decoding buffer and a post-decoding buffer; The video buffer level of the monitored video buffer specifically includes: Monitor the post-decoding water level of the post-decoding buffer and the pre-decoding water level of the pre-decoding buffer; The step of reducing video playback speed when the video buffer level falls below a preset safety threshold specifically includes: If the water level after decoding is lower than the threshold after decoding, and the water level before decoding is higher than or equal to the threshold before decoding, then the video playback speed is reduced by the third deceleration amplitude. If the water level after decoding is lower than the threshold after decoding, and the water level before decoding is lower than the threshold before decoding, then the video playback speed is reduced by a fourth deceleration amplitude; the fourth deceleration amplitude is greater than the third deceleration amplitude.

4. The method as described in claim 1, wherein adjusting at least one of the audio playback speed and the video playback speed according to the display timestamp difference specifically includes: If the displayed timestamp difference indicates that the audio is ahead, then at least one of reducing the audio playback speed and increasing the video playback speed is performed; If the displayed timestamp difference indicates that the video is ahead, then at least one of increasing the audio playback speed and decreasing the video playback speed is performed.

5. The method of claim 4, further comprising: Monitor the audio buffer level in the audio buffer; If the displayed timestamp difference indicates that the video is ahead, then performing at least one of increasing the audio playback speed and decreasing the video playback speed includes: If the displayed timestamp difference indicates that the video is ahead, and the audio buffer level is higher than the preset audio level threshold, then the audio playback speed is increased.

6. The method as described in claim 1, wherein the video buffer level is lower than a preset safety threshold, comprising: The amount of playable data stored in the video buffer is less than a preset data volume threshold.

7. The method of claim 1, further comprising: After playback is initiated, the audio playback data and the video playback data are played at a reduced speed in a coordinated manner until the video buffer level reaches a preset water storage threshold.

8. The method of claim 1, further comprising: When playback starts, receive the start-up media data transmitted by the server in accelerated sending mode; The first frame audio data and the first frame video data in the start-up media data are aligned with the display timestamps. The first frame of audio data and the first frame of video data after alignment are used as the starting point for playback.

9. The method as described in claim 8, wherein aligning the display timestamps of the first frame audio data and the first frame video data in the broadcast media data specifically includes: Compare the display timestamp of the first frame audio data with the display timestamp of the first frame video data; If the display timestamp of the first frame of audio data is earlier than the display timestamp of the first frame of video data, then the earlier audio frame is discarded. If the display timestamp of the first frame of audio data lags behind the display timestamp of the first frame of video data, a silent frame is inserted.

10. The method of claim 1, further comprising: When the playback timestamp interval between adjacent video frames exceeds the preset stuttering threshold, audio and video playback will be paused.

11. The method as described in claim 10, wherein after pausing playback, when the video buffer level recovers to the recovery threshold, playback of audio and video is resumed in a decelerated manner; wherein, The recovery threshold is lower than the preset security threshold.

12. The method of claim 1, wherein the audio playback data and the video playback data are received through mutually independent transmission channels.

13. The method as described in claim 1, wherein the media stream data is live streaming media data.

14. A media streaming data playback device, comprising: The playback monitoring module is used to monitor the video buffer level in the video buffer and the difference in display timestamps between audio playback data and video playback data; The playback adjustment module is used to: reduce the video playback speed when the video buffer level is lower than a preset safety threshold, and adjust the audio playback speed according to the display timestamp difference; When the video buffer level is not lower than the preset safety threshold and the absolute value of the display timestamp difference exceeds the preset synchronization threshold, at least one of the audio playback speed and the video playback speed is adjusted according to the display timestamp difference.

15. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 13.

16. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

Images