Method and apparatus for playing streaming media data, and device and medium

Abstract

Provided in the embodiments of the present disclosure are a method and apparatus for playing streaming media data, and a device and a medium. The method comprises: on the basis of the size of an audio buffer data block in an audio buffer corresponding to audio content in streaming media data, presentation timestamp (PTS) information of the audio buffer data block, and the size of each audio frame in an audio frame sequence acquired from the audio buffer, determining PTS information for each audio frame in the audio frame sequence; furthermore, on the basis of the PTS information for each audio frame in the audio frame sequence, playing video content and audio content in the streaming media data. In this way, the accuracy of audio-video synchronization can be improved.

Description

Method, device, equipment and medium for playing streaming media data

[0001] This application claims priority to the Chinese invention patent application entitled “Method, apparatus, device and medium for playing streaming media data” and application number 202410249507.4, filed on March 5, 2024. The entire contents of that application are incorporated herein by reference. Technical Field

[0002] Embodiments of the present disclosure generally relate to the field of computers, and more particularly to methods, devices, equipment, and media for playing streaming media data. Background Art

[0003] In recent years, mobile terminals such as mobile phones and tablet computers have become important tools for people's daily life, study and work. Live audio and video broadcasting has become an increasingly popular way of online interaction. For example, people can use mobile terminals to watch live broadcasts, hold remote meetings, and so on. Real Time Media (RTM) live broadcasting is a live broadcasting solution that aims to enhance the user interaction experience. Compared with conventional live broadcasting solutions, RTM live broadcasting has a smaller end-to-end delay, for example, it can reach 1 second. In audio and video live broadcasting scenarios, accurate audio and video synchronization is often required, so audio and video synchronization is a key experience indicator in audio and video live broadcasting. How to improve the accuracy of audio and video synchronization has become an urgent problem to be solved. Summary of the Invention

[0004] In a first aspect of the present disclosure, a method for playing streaming media data is provided. In the method, based on the size of an audio buffer data block in an audio buffer corresponding to the audio content in the streaming media data, the presentation time stamp (PTS) information of the audio buffer data block, and the size of each audio frame in the audio frame sequence obtained from the audio buffer, PTS information for each audio frame in the audio frame sequence is determined. Furthermore, based on the PTS information for each audio frame in the audio frame sequence, video content and audio content in the streaming media data are played.

[0005] In a second aspect of the present disclosure, a device for playing streaming media data is provided. The device includes a determination module and a playback module. The determination module is configured to determine the PTS information of each audio frame in an audio frame sequence based on the size of an audio buffer data block in an audio buffer for audio content in the streaming media data, the presentation timestamp (PTS) information of the audio buffer data block, and the size of each audio frame in the audio frame sequence obtained from the audio buffer. The playback module is configured to play the video content and audio content in the streaming media data based on the PTS information of each audio frame in the audio frame sequence.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processor; and at least one memory, the at least one memory being coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions, when executed by the at least one processor, causing the electronic device to perform the method according to the first aspect of the present disclosure.

[0007] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which instructions are stored. When the instructions are executed by a processor, the processor is caused to implement the method according to the first aspect of the present disclosure.

[0008] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent hereinafter with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] FIG1 is a schematic diagram illustrating an example environment in which various embodiments of the present disclosure can be implemented;

[0011] FIG2A shows a schematic diagram of determining audio frame PTS information under normal circumstances according to some embodiments of the present disclosure;

[0012] FIG2B shows a schematic diagram of determining audio frame PTS information under acceleration according to some embodiments of the present disclosure;

[0013] FIG2C shows a schematic diagram of determining audio frame PTS information under a deceleration condition according to some embodiments of the present disclosure;

[0014] FIG2D shows a schematic diagram of determining audio frame PTS information under packet loss compensation according to some embodiments of the present disclosure;

[0015] FIG3 shows a flowchart of a video content playback process according to some embodiments of the present disclosure;

[0016] FIG4 shows a flowchart of a method for playing streaming media data according to some embodiments of the present disclosure;

[0017] FIG5 shows a block diagram of an example apparatus for playing streaming media data according to some embodiments of the present disclosure; and

[0018] FIG6 illustrates a block diagram of a device in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0019] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0020] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". The following may also include other explicit and implicit definitions. As used herein, the term "model" can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and / or to be developed in the future.

[0021] As used herein, the term "in response to" refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of executing a subsequent action executed in response to the event or condition is not necessarily strongly correlated with the time when the event occurs or the condition is satisfied. For example, in some cases, the subsequent action may be executed immediately when the event occurs or the condition is satisfied; in other cases, the subsequent action may be executed some time after the event occurs or the condition is satisfied.

[0022] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0023] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0024] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0025] As an optional but non-limiting embodiment, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also include a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0026] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the embodiments of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the embodiments of the present disclosure.

[0027] Sample Environment

[0028] FIG1 shows a schematic diagram of an example environment 100 in which various embodiments of the present disclosure can be implemented. The example environment 100 may generally include a terminal device 110, a server 120, and a user 130. The terminal device 110 may be communicatively coupled to the server 120. In some embodiments, the terminal device 110 may communicate directly with the server 120. In other embodiments, a content delivery network (CDN) may be deployed between the terminal device 110 and the server 120, so that the terminal device 110 can obtain the required content nearby. The embodiments of the present disclosure are not limited in this respect.

[0029] The terminal device 110 can be any type of mobile terminal or portable terminal, including a mobile phone, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the terminal device 110 can also support any type of interface for the user 130 (such as a "wearable" circuit, etc.).

[0030] As shown in Figure 1, the terminal device 110 may include a media engine 112, an intermediate adaptation layer 114, and a player kernel layer 116 for low-latency live broadcast. For example, the media engine 112 can communicate with the server 120 to send or receive streaming media data. For example, but not limitation, the streaming media data includes live broadcast data (e.g., RTM live broadcast data), online video data, and the like. The media engine 112 can also be responsible for framing and framing. The player kernel layer 116 is coupled to the media engine 112 via the intermediate adaptation layer 114. The intermediate adaptation layer 114 can transparently transmit the audio and video data obtained by the media engine 112 to the player kernel layer 116.

[0031] Furthermore, the player core layer 116 can decode and post-process the audio and video data, and render the audio and video data for presentation to the user 130. For example, the player core layer 116 can perform processing such as packet loss compensation (PLC), muting, acceleration, and deceleration on the decoded audio data. The processed audio data can be stored in an audio buffer. The player core layer 116 can read an audio frame sequence from the audio buffer to play the audio content of the streaming media data.

[0032] In some embodiments, the server 120 may be a device with computing and communication capabilities, such as a workstation or cloud server. It should be understood that the structure and functionality of the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. For example, the media engine 112, the intermediate adaptation layer 114, and the player core layer 116 may also be implemented in a distributed manner across different devices.

[0033] As briefly mentioned above, audio and video live broadcast technology is increasingly widely used. An important measure of the audio and video live viewing experience is audio and video synchronization. In a currently available solution, multiple audio frames obtained from an audio buffer data block in the audio buffer share the same presentation time stamp (PTS). For example, the starting PTS of multiple audio frames read from an audio buffer data block with a starting PTS of 46ms are all determined to be 46ms. This results in a large error in the PTS information of the audio frame, resulting in a low accuracy of audio and video synchronization, which in turn affects the viewing experience of user 130.

[0034] To this end, the various embodiments of the present disclosure propose a solution for finely determining the PTS information of each audio frame for achieving audio-visual synchronization. Specifically, according to the embodiments of the present disclosure, a solution for playing streaming media data is proposed. In this solution, the PTS information of each audio frame in the audio frame sequence is determined based on the size of the audio buffer data block in the audio buffer for the audio content in the streaming media data, the PTS information of the audio buffer data block, and the size of each audio frame in the audio frame sequence obtained from the audio buffer. Furthermore, the video content and audio content in the streaming media data are played based on the PTS information of each audio frame in the audio frame sequence.

[0035] It will be more clearly understood from the following description that according to an embodiment of the present disclosure, the PTS of an audio frame is determined based on the size and PTS information of the audio buffer data block and the size of the audio frame for playing audio and video content. In this way, on the one hand, the PTS corresponding to each audio frame in the audio frame sequence can be determined, thereby effectively improving the accuracy of the PTS information of each audio frame. On the other hand, by achieving audio and video synchronization based on the PTS information determined in this way, the accuracy of audio and video synchronization can be improved, thereby effectively improving the viewing experience of user 130.

[0036] Example PTS determination

[0037] As briefly mentioned above, the player core layer 116 can perform processing such as acceleration, deceleration, and packet loss compensation on the audio content in the streaming media data. The processed audio data can be stored in an audio buffer, and the player core layer 116 can read the audio frame sequence from the audio buffer for playback. In the context of this disclosure, the terms "play," "render," "present," and "display" can be used interchangeably to refer to providing the corresponding media content to the user 130.

[0038] In some embodiments, the PTS information of each audio frame in the audio frame sequence can be determined based on the size of the audio buffer data block, the PTS information of the audio buffer data block, and the size of each audio frame in the audio frame sequence obtained from the audio buffer. In one example, the PTS information of the audio buffer data block may include the start PTS and the end PTS of the audio buffer data block. In another example, the PTS information of the audio buffer data block may include the start PTS of the audio buffer data block. In this case, the start PTS of the next audio buffer data block immediately following the current audio buffer data block can be determined as the end PTS of the current audio buffer data block.

[0039] In another example, the PTS information of the audio buffer data block may include the end PTS of the audio buffer data block. In this case, the end PTS of the audio buffer data block immediately preceding the current audio buffer data block may be determined as the start PTS of the current audio buffer data block. It should be understood that the start PTS and end PTS of the audio buffer data block may also be embodied in any other suitable manner, such as by means of a combination of the start PTS and the PTS span, etc. The scope of the present disclosure is not limited in this respect.

[0040] For ease of explanation, the following description uses the first audio frame in an audio frame sequence as an example. It should be noted that, in the context of this disclosure, the term "first audio frame" does not indicate that the audio frame is the first in the display sequence; the first audio frame can be any audio frame in the audio frame sequence. The audio frame that is the first in the display sequence can be referred to as the first audio frame.

[0041] For example, the PTS information of the first audio frame may include the PTS span of the first audio frame. The PTS span of the first audio frame may be determined based on the result of the following equation (1):

[0042] Among them, PTS1 represents the start PTS of the audio buffer data block, PTS2 represents the end PTS of the audio buffer data block, buffer Indicates the size of the audio buffer data block, and L block represents the size of the first audio frame. In some embodiments, the result of equation (1) can be directly determined as the PTS span of the first audio frame. Alternatively, the PTS span of the first audio frame can be determined by performing rounding, rounding up, or rounding down on the result of equation (1). The scope of the present disclosure is not limited in this respect.

[0043] Additionally or alternatively, the PTS information of the first audio frame may include the starting PTS of the first audio frame. For example, the starting PTS of the first audio frame may be determined by accumulating the PTS spans of all audio frames preceding the first audio frame in the audio frame sequence. Since an audio frame may correspond to a minimum playback unit of audio data, for example, the size of each audio frame in the audio frame sequence may be the same. In this case, the PTS span of each audio frame is also the same. Therefore, the starting PTS of the first audio frame may be determined based on the following formula (2): PTS start =(N-1)×S (2)

[0044] PTS start represents the starting PTS of the first audio frame, N represents the position of the first audio frame in the audio frame sequence, and S represents the PTS span of each audio frame.

[0045] Additionally or alternatively, the PTS information of the first audio frame may include an ending PTS of the first audio frame. For example, the ending PTS of the first audio frame may be determined by accumulating the PTS spans of all audio frames preceding the first audio frame in the audio frame sequence and the PTS span of the first audio frame itself. In the case where the PTS span of each audio frame is the same, the ending PTS of the first audio frame may be determined based on the following formula (3): PTS end =N×S (3)

[0046] PTS end represents the ending PTS of the first audio frame, N represents the position of the first audio frame in the audio frame sequence, and S represents the PTS span of each audio frame.

[0047] An example method for determining audio frame PTS information according to some embodiments of the present disclosure will be described below with reference to FIG2A to FIG2D. FIG2A shows a schematic diagram 200 of determining audio frame PTS information under normal conditions (i.e., when there is no acceleration / deceleration and packet loss compensation) according to some embodiments of the present disclosure. As shown in FIG2A , the start PTS of the audio buffer data block 210 is 0, and the end PTS is 46. Taking the example that the size of the audio buffer data block 210 is 4096 bytes and the size of the first audio frame 212 read from the audio buffer data block 210 is 882 bytes, the result of formula (1) is Therefore, the PTS span of the first audio frame 212 can be determined to be 10, for example. Further, since the first audio frame 212 is located at the 3rd position in the audio frame sequence, the start PTS of the first audio frame 212 can be determined to be (3-1)×10=20, and the end PTS of the first audio frame 212 can be determined to be 3×10=30.

[0048] FIG2B shows a schematic diagram 201 of determining audio frame PTS information under acceleration according to some embodiments of the present disclosure. In the example shown in FIG2B , the acceleration rate is 2. Therefore, compared to the normal situation shown in FIG2A , in FIG2B , the PTS span corresponding to audio buffer data blocks of the same size is doubled. It should be understood that the acceleration rate may also be any other suitable value, and the scope of the present disclosure is not limited in this respect.

[0049] As shown in FIG2B , the start PTS of the audio buffer data block 220 is 0, and the end PTS is 92. Taking the case where the size of the audio buffer data block 220 is 4096 bytes and the size of the first audio frame 222 read from the audio buffer data block 220 is 882 bytes as an example, the result of equation (1) is Therefore, the PTS span of the first audio frame 222 can be determined to be 20, for example. Further, since the first audio frame 222 is located at the 3rd position in the audio frame sequence, the start PTS of the first audio frame 222 can be determined to be (3-1)×20=40, and the end PTS of the first audio frame 222 can be determined to be 3×20=60.

[0050] FIG2C shows a schematic diagram 202 illustrating determining audio frame PTS information under deceleration according to some embodiments of the present disclosure. In the example shown in FIG2C , the deceleration rate is 0.5. Therefore, compared to the normal situation shown in FIG2A , the PTS span corresponding to an audio buffer data block of the same size in FIG2C is halved. It should be understood that the deceleration rate may also be any other suitable value, and the scope of the present disclosure is not limited in this respect.

[0051] As shown in FIG2C , the start PTS of the audio buffer data block 230 is 0, and the end PTS is 23. Taking the size of the audio buffer data block 230 as 4096 bytes and the size of the first audio frame 232 read from the audio buffer data block 230 as 882 bytes, the result of equation (1) is Therefore, the PTS span of the first audio frame 232 can be determined as 5, for example. Further, since the first audio frame 232 is located at the 3rd position in the audio frame sequence, the start PTS of the first audio frame 232 can be determined as (3-1)×5=10, and the end PTS of the first audio frame 232 can be determined as 3×5=15.

[0052] It can be seen that by means of the method of determining the audio frame PTS information described above, the corresponding PTS of each audio frame in the audio frame sequence can be determined more finely, thereby effectively improving the accuracy of the PTS information of each audio frame.

[0053] FIG2D shows a schematic diagram 203 of determining audio frame PTS information under packet loss compensation according to some embodiments of the present disclosure. In the example shown in FIG2D , audio buffer data block 240 stores actual audio data, while audio buffer data block 241 stores audio data generated based on packet loss compensation when corresponding audio data is lost or delayed.

[0054] In some embodiments, for an audio frame obtained from an audio buffer data block storing real audio data (e.g., audio buffer data block 240), the PTS information of the corresponding audio frame can be determined based on the method described above. For example, the start PTS of the audio buffer data block 240 is 0, and the end PTS is 46. Taking the size of the audio buffer data block 240 as 4096 bytes and the size of the first audio frame 242 as 882 bytes as an example, the result of formula (1) is Therefore, the PTS span of the first audio frame 242 can be determined to be 10, for example. Further, since the first audio frame 242 is located at the 3rd position in the audio frame sequence, the start PTS of the first audio frame 242 can be determined to be (3-1)×10=20, and the end PTS of the first audio frame 242 can be determined to be 3×10=30.

[0055] In some embodiments, for an audio buffer data block storing audio data generated based on packet loss compensation (e.g., audio buffer data block 241), the start PTS of one or more audio frames obtained from the audio buffer data block can be determined to be equal to the start PTS of the audio buffer data block (i.e., the end PTS of the previous audio buffer data block). Referring to FIG2D , the start PTSs of audio frames 244, 245, 246, and 247 are all 46. In this case, when audio data between PTS=46 and PTS=92 arrives subsequently, the PTS of the corresponding audio frame can still be determined based on the method described above with reference to equations (1), (2), and / or (3).

[0056] In this way, it is possible to prevent the audio frames generated based on packet loss compensation from occupying the PTS of the audio frames corresponding to the real data, thereby more finely determining the PTS information of each audio frame, thereby further improving the audio-visual synchronization effect. It should be understood that the specific numerical values ​​mentioned in the context of this disclosure are for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0057] Example playback process

[0058] In some embodiments, the video content and audio content in the streaming media data (e.g., RTM live data, etc.) can be played based on the audio frame PTS information determined in the above manner. Exemplarily, audio frames can be read sequentially from the audio buffer, the audio frames can be played, and the audio master clock can be updated based on the audio frame PTS information. It should be understood that the playback of audio frames and the playback of video frames can be carried out in parallel. In other words, while obtaining audio frames, determining audio frame PTS information and playing audio frames, video frames can be played based on the existing audio frame PTS information to achieve audio and video synchronization. The playback process of video content is described below with reference to Figure 3.

[0059] FIG3 shows a flowchart 300 of a video content playback process according to some embodiments of the present disclosure. In some embodiments, the video content playback process may be implemented at the player core layer 116 as shown in FIG1 . It should be understood that the video content playback process may further include additional blocks not shown and / or may omit certain (or some) of the blocks shown, and the scope of the present disclosure is not limited in this respect.

[0060] The video content playback process starts at 310. At 315, it is determined whether the video stream has ended. If the video stream has ended, proceed to 370 and the entire video content playback process ends. If the video stream has not ended, proceed to 320 to obtain the video frame to be presented. Exemplarily and not restrictively, the video frame to be presented can be read from the video buffer. At 325, it is determined whether the video frame to be presented is the first frame (i.e., the first video frame). If the video frame to be presented is the first frame, proceed to 355 to update the video master clock. Exemplarily, for the first video frame, the video master clock can be updated with the help of the current real time (e.g., 13:40). At 360, the presentation duration of the video frame to be presented can be predicted, which will be described in further detail below. At 365, the first video frame is presented. In this way, the first video frame can be rendered quickly first to ensure that the video content can be presented to the user 130 as soon as possible, thereby effectively shortening the first frame time and improving the viewing experience of the user 130.

[0061] Then the process returns to 315 and continues to determine whether the video stream has ended. If it is determined that the video stream has not ended, proceed to 320 and continue to obtain the video frame to be presented. For the purpose of ease of explanation, the video frame to be presented obtained can also be referred to as the first video frame. The PTS of the first video frame is greater than the first video frame. In other words, the first video frame is displayed after the first video frame. At 325, since the first video frame is not the first frame, proceed to 330. At 330, determine whether the first audio frame has started to play. If it is determined that the first audio frame has not started to play, proceed to 335 and wait for a predetermined time (e.g., 5ms, 10ms, etc.). During the waiting period, keep presenting the first video frame. After waiting for the predetermined time, return to 330 and continue to determine whether the first audio frame has started to play. By waiting for the playback of the first audio frame in this polling manner, it is possible to wait for the playback of the first audio frame more efficiently, thereby saving computing resources.

[0062] The inventors have discovered through research that in some live broadcast scenarios (e.g., RTM live broadcast scenarios), the first video frame and the first audio frame may not arrive at the same time. To address this issue, by referring to the operations described in blocks 330 and 335 , it is possible to wait for the first audio frame to play when the first video frame and the first audio frame do not arrive at the same time, thereby effectively reducing lag and improving the viewing experience of user 130 .

[0063] If it is determined at block 330 that the first audio frame has begun playing, the process proceeds to block 340 where a wait time for presenting the first video frame is determined based on a difference between a current PTS of an audio master clock for playing the audio content and the PTS of the first video frame. The audio master clock may be determined based on PTS information of one or more audio frames in the audio frame sequence. The PTS information of the one or more audio frames may be determined based on the method described above with reference to FIG. 2A to FIG. 2D .

[0064] In one example embodiment, the difference between the current PTS of the audio master clock and the PTS of the first video frame can be directly determined as the waiting time for presenting the first video frame. In this way, strict alignment of video content playback with audio content playback can be achieved, thereby accurately achieving audio and video synchronization.

[0065] In another example embodiment, the waiting time may be determined based on the following equation (4): W=PTS audio -PTS video -G (4)

[0066] Where W represents the waiting time, PTS audio Indicates the current PTS of the audio master clock, PTS video represents the PTS of the first video frame, and G represents the frame interval of the video content. By way of example and not limitation, in the case of 50 frames per second, the frame interval is 1 / 50=20ms.

[0067] Based on the waiting time determined by formula (4), the video frame is presented. Then, the playback timing of the video frame does not need to be strictly aligned with the audio frame with the corresponding PTS. Instead, a certain error between the video playback progress and the audio playback progress can be allowed (for example, the error is less than or equal to one video frame interval). In this way, a good balance can be achieved between achieving good audio and video synchronization and reducing freezes, thereby further improving the viewing experience of user 130. It should be understood that the waiting time can also be determined in any other suitable manner, and the scope of the present disclosure is not limited in this respect.

[0068] Further, the first video frame can be presented based on the determined waiting time. At 345, determine whether the waiting time for presenting the first video frame is less than or equal to zero. If it is determined that the waiting time is greater than zero, then proceed to 350 and wait for a predetermined time (e.g., 2ms, 10ms, etc.). During the waiting period, keep presenting the previous video frame before the first video frame. After waiting for the predetermined time, update the waiting time by subtracting the waited predetermined time from the waiting time for presenting the first video frame, and return to 345 to continue to judge whether the waiting time is less than or equal to zero. By waiting for the playback opportunity of the video frame in this polling manner, the playback opportunity of the video frame can be waited for more efficiently, thereby saving computing resources.

[0069] If it is determined in 345 that the waiting time is less than or equal to zero, proceed to 355 to update the video master clock. Exemplarily, the video master clock can be updated by adding the difference between the PTS of the first video frame and the PTS of the previous video frame to the current video master clock. In box 360, the presentation duration of the first video frame can be predicted. Exemplarily and not restrictively, the predicted value of the presentation duration of the first video frame can be determined based on the frame interval of the video content. For example, the predicted value of the presentation duration of the first video frame can be determined as one frame interval. Alternatively, the predicted value of the presentation duration of the first video frame can be determined by floating a certain time on the basis of one frame interval. It should be understood that the predicted value of the presentation duration of the first video frame can also be determined in any other suitable manner, and the scope of the present disclosure is not limited in this respect.

[0070] At 365, the first video frame may be presented. Further, the process returns to 315 to continue determining whether the video stream has ended and perform subsequent operations. This is similar to the process described in detail above, so this disclosure will not be repeated here.

[0071] In some embodiments, situations such as audio freeze or audio data not yet arriving may cause the audio master clock to fail. If it is determined that the audio master clock has failed, the first video frame or the first video frame can be presented based on the predicted value of the presentation duration. For example, after the presentation duration of the first video frame has reached the predicted value of the presentation duration, the next video frame after the first video frame can be presented without further waiting for audio data. In this way, video freezes when the audio master clock fails can be effectively avoided, thereby ensuring the smoothness of video playback and improving the viewing experience of user 130.

[0072] As described above with reference to block 345, in some embodiments of the present disclosure, if the wait time for presenting the first video frame is determined to be less than zero, the first video frame is presented. In other words, even if the first video frame lags behind the audio master clock, the first video frame is still presented. In this way, compared to conventional solutions that directly discard video frames when they lag behind the audio master clock, unnecessary discarding of video frames can be avoided, thereby effectively preventing the degradation of the viewing experience of user 130 caused by video skipping.

[0073] In some embodiments, the PTS for audio content and / or the PTS for video content can be obtained based on the Real-time Transport Protocol (RTP). In one example, the PTS of each audio frame or video frame can be carried by an extension header of the data packet of each audio frame or video frame. In another example, for video content, the decoding time stamp (DTS) and composition time (CTS) of the video frame can be carried by an extension header of the data packet of each video frame, and the PTS can be determined as the sum of the DTS and the CTS. In this way, the PTS information can be transmitted more efficiently.

[0074] It can be seen from the above description in combination with Figures 1 to 3 that in the method for playing streaming media data according to each embodiment of the present disclosure, the PTS of the audio frame is determined based on the size and PTS information of the audio buffer data block and the size of the audio frame for playing audio and video content. In this way, on the one hand, the PTS corresponding to each audio frame in the audio frame sequence can be determined, thereby effectively improving the accuracy of the PTS information of each audio frame. On the other hand, by achieving audio and video synchronization based on the PTS information determined in this way, the accuracy of audio and video synchronization can be improved, thereby effectively improving the user's viewing experience.

[0075] Example Method

[0076] FIG4 illustrates a flow chart of a method 400 for playing streaming media data according to some embodiments of the present disclosure. In some embodiments, the method 400 may be executed at the player inner layer 116 as shown in FIG1 . It should be understood that the method 400 may include additional blocks not shown and / or may omit one (or more) of the blocks shown, and the scope of the present disclosure is not limited in this respect.

[0077] In box 402, PTS information of each audio frame in the audio frame sequence is determined based on the size of the audio buffer data block in the audio buffer for the audio content in the streaming media data, the presentation timestamp PTS information of the audio buffer data block, and the size of each audio frame in the audio frame sequence obtained from the audio buffer.

[0078] At block 404 , video content and audio content in the streaming media data are played based on the PTS information of each audio frame in the audio frame sequence.

[0079] In some embodiments, the PTS information of the audio buffer data block includes a start PTS and an end PTS of the audio buffer data block, and the PTS information of the first audio frame in the audio frame sequence includes a PTS span of the first audio frame. The PTS span of the first audio frame is determined based on a result of the following formula:

[0080] Among them, PTS1 represents the start PTS of the audio buffer data block, PTS2 represents the end PTS of the audio buffer data block, buffer Indicates the size of the audio buffer data block, and L block Indicates the size of the first audio frame.

[0081] In some embodiments, each audio frame in the audio frame sequence has the same size, and the PTS information of the first audio frame includes a start PTS of the first audio frame, which is determined based on the following formula: PTS start =(N-1)×S,

[0082] PTS start represents the starting PTS of the first audio frame, N represents the position of the first audio frame in the audio frame sequence, and S represents the PTS span of each audio frame.

[0083] In some embodiments, each audio frame in the audio frame sequence has the same size, and the PTS information of the first audio frame includes an end PTS of the first audio frame, which is determined based on the following formula: PTS end =N×S,

[0084] PTS end represents the ending PTS of the first audio frame, N represents the position of the first audio frame in the audio frame sequence, and S represents the PTS span of each audio frame.

[0085] In some embodiments, the audio buffer data blocks are generated based on an accelerated process or a decelerated process.

[0086] In some embodiments, method 400 further includes: presenting a first video frame in the video content; obtaining the first video frame in the video content, the PTS of the first video frame being greater than the first video frame; and in response to determining that the first audio frame in the audio content has not yet started playing, keeping presenting the first video frame.

[0087] In some embodiments, method 400 also includes: in response to determining that the first audio frame in the audio content has started playing, determining a waiting time for presenting the first video frame based on a difference between a current PTS of an audio master clock used to play the audio content and the PTS of the first video frame, wherein the audio master clock is determined based on PTS information of one or more audio frames in the audio frame sequence; and presenting the first video frame based on the waiting time.

[0088] In some embodiments, the waiting time is determined based on the following formula: W = PTS audi o-PTS vide oG,

[0089] Where W represents the waiting time, PTS audio Indicates the current PTS of the audio master clock, PTS video represents the PTS of the first video frame, and G represents the frame interval of the video content.

[0090] In some embodiments, presenting the first video frame based on the wait time includes: presenting the first video frame in response to determining that the wait time is less than or equal to zero; or presenting the first video frame after the wait time has elapsed in response to determining that the wait time is greater than zero.

[0091] In some embodiments, method 400 further includes determining a predicted value for a presentation duration of the first video frame based on a frame interval of the video content; and presenting the first video frame based on the predicted value for the presentation duration in response to determining that the audio master clock has failed.

[0092] In some embodiments, the PTS for audio content or the PTS for video content is acquired based on the Real-time Transport Protocol (RTP).

[0093] In some embodiments, the streaming media data includes low-latency RTM live data.

[0094] Example devices and equipment

[0095] Embodiments of the present disclosure also provide corresponding apparatuses and devices for implementing the above-described methods or processes. FIG5 shows a block diagram of an example apparatus 500 for playing streaming media data according to some embodiments of the present disclosure. The apparatus 500 can, for example, be used to implement the methods according to some embodiments of the present disclosure. In some embodiments, the apparatus 500 can be implemented at the player kernel layer 116 as shown in FIG1 .

[0096] As shown in FIG5 , apparatus 500 may include a determination module 502 and a playback module 504. Determination module 502 is configured to determine the PTS information of each audio frame in an audio frame sequence obtained from the audio buffer based on the size of an audio buffer data block in an audio buffer for audio content in the streaming media data, presentation timestamp (PTS) information of the audio buffer data block, and the size of each audio frame in the audio frame sequence obtained from the audio buffer. Playback module 504 is configured to play the video content and the audio content in the streaming media data based on the PTS information of each audio frame in the audio frame sequence.

[0097] In some embodiments, the PTS information of the audio buffer data block includes a start PTS and an end PTS of the audio buffer data block, and the PTS information of the first audio frame in the audio frame sequence includes a PTS span of the first audio frame. The PTS span of the first audio frame is determined based on a result of the following formula:

[0098] Among them, PTS1 represents the start PTS of the audio buffer data block, PTS2 represents the end PTS of the audio buffer data block, buffer Indicates the size of the audio buffer data block, and L block Indicates the size of the first audio frame.

[0099] In some embodiments, each audio frame in the audio frame sequence has the same size, and the PTS information of the first audio frame includes a start PTS of the first audio frame, which is determined based on the following formula: PTS start =(N-1)×S,

[0100] PTS start represents the starting PTS of the first audio frame, N represents the position of the first audio frame in the audio frame sequence, and S represents the PTS span of each audio frame.

[0101] In some embodiments, each audio frame in the audio frame sequence has the same size, and the PTS information of the first audio frame includes an end PTS of the first audio frame, which is determined based on the following formula: PTS end =N×S,

[0102] PTS end represents the ending PTS of the first audio frame, N represents the position of the first audio frame in the audio frame sequence, and S represents the PTS span of each audio frame.

[0103] In some embodiments, the audio buffer data blocks are generated based on an accelerated process or a decelerated process.

[0104] In some embodiments, the apparatus 500 further includes a first frame presentation module, an acquisition module, and a hold presentation module. The first frame presentation module is configured to present the first video frame in the video content. The acquisition module is configured to acquire the first video frame in the video content. The PTS of the first video frame is greater than that of the first video frame. The hold presentation module is configured to, in response to determining that the first audio frame in the audio content has not yet begun playing, hold present the first video frame.

[0105] In some embodiments, the apparatus 500 further includes a latency determination module and a video frame presentation module. The latency determination module is configured to, in response to determining that a first audio frame in the audio content has begun playing, determine a latency for presenting the first video frame based on a difference between a current PTS of an audio master clock used to play the audio content and the PTS of the first video frame. The audio master clock is determined based on PTS information of one or more audio frames in the audio frame sequence. The video frame presentation module is configured to present the first video frame based on the latency.

[0106] In some embodiments, the waiting time is determined based on the following formula: W = PTS audio -PTS video -G,

[0107] Where W represents the waiting time, PTS audio Indicates the current PTS of the audio master clock, PTS video represents the PTS of the first video frame, and G represents the frame interval of the video content.

[0108] In some embodiments, the video frame presentation module includes a first presentation module and / or a second presentation module. The first presentation module is configured to: present the first video frame in response to determining that the waiting time is less than or equal to zero. The second presentation module is configured to: present the first video frame after the waiting time has elapsed in response to determining that the waiting time is greater than zero.

[0109] In some embodiments, the apparatus 500 further includes a prediction value determination module and a first video frame presentation module. The prediction value determination module is configured to determine a prediction value for a presentation duration of the first video frame based on a frame interval of the video content. The first video frame presentation module is configured to present the first video frame based on the prediction value for the presentation duration in response to determining that the audio master clock has failed.

[0110] In some embodiments, the PTS for audio content or the PTS for video content is acquired based on the Real-time Transport Protocol (RTP).

[0111] In some embodiments, the streaming media data includes low-latency RTM live data.

[0112] The modules and / or units included in the device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units in the device 500 can be implemented at least in part by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0113] The modules and / or units shown in Figure 5 may be partially or entirely implemented as hardware modules, software modules, firmware modules, or any combination thereof. In particular, in some embodiments, the processes, methods, or procedures described above may be implemented by hardware in a storage system, a host corresponding to the storage system, or other computing devices independent of the storage system.

[0114] FIG6 illustrates a block diagram of a device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 600 shown in FIG6 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 600 shown in FIG6 may be used to implement the player core layer 116 shown in FIG1 and / or the methods described above.

[0115] As shown in FIG6 , electronic device 600 is in the form of a general electronic device. Components of electronic device 600 may include, but are not limited to, one or more processing units or processors 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processor 610 may be a real or virtual processor and is capable of executing various processes according to programs stored in memory 620. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of electronic device 600.

[0116] The electronic device 600 typically includes a plurality of computer storage media. Such media can be any available media accessible to the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and can be accessed within the electronic device 600.

[0117] The electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG6 , a disk drive for reading from or writing to a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 625 having one or more program modules configured to perform various methods or actions of various implementations of the present disclosure.

[0118] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0119] The input device 650 may be one or more input devices, such as a mouse, keyboard, or trackball. The output device 660 may be one or more output devices, such as a display, a speaker, or a printer. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with the electronic device 600, or with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0120] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0121] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0122] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0123] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0124] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0125] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for playing streaming media data, comprising: Determine PTS information of each audio frame in the audio frame sequence based on a size of an audio buffer data block in an audio buffer for audio content in the streaming media data, presentation timestamp (PTS) information of the audio buffer data block, and a size of each audio frame in the audio frame sequence obtained from the audio buffer; as well as Based on the PTS information of each audio frame in the audio frame sequence, the video content and the audio content in the streaming media data are played.

2. The method according to claim 1 , wherein the PTS information of the audio buffer data block includes a start PTS and an end PTS of the audio buffer data block, and the PTS information of the first audio frame in the audio frame sequence includes a PTS span of the first audio frame, and the PTS span of the first audio frame is determined based on a result of the following formula: Wherein PTS1 represents the start PTS of the audio buffer data block, PTS2 represents the end PTS of the audio buffer data block, buffer represents the size of the audio buffer data block, and L block Indicates the size of the first audio frame.

3. The method according to claim 2, wherein each audio frame in the audio frame sequence has the same size, the PTS information of the first audio frame includes a start PTS of the first audio frame, and the start PTS of the first audio frame is determined based on the following formula: PTS start =(N-1)×S, PTS start represents the starting PTS of the first audio frame, N represents the position of the first audio frame in the audio frame sequence, and S represents the PTS span of each audio frame.

4. The method according to claim 2, wherein each audio frame in the audio frame sequence has the same size, the PTS information of the first audio frame includes an end PTS of the first audio frame, and the end PTS of the first audio frame is determined based on the following formula: PTS end =N×S, PTS end represents the end PTS of the first audio frame, N represents the position of the first audio frame in the audio frame sequence, and S represents the PTS span of each audio frame. The method according to claim 1 , wherein the audio buffer data block is generated based on an accelerated process or a decelerated process.

6. The method according to claim 1, wherein the PTS information of the audio buffer data block includes a start PTS of the audio buffer data block, and if it is determined that the audio buffer data block stores audio data generated based on packet loss compensation, the start PTSs of multiple audio frames obtained from the audio buffer data block are the same.

7. The method according to claim 1, further comprising: presenting a first video frame of the video content; Acquire a first video frame in the video content, where the PTS of the first video frame is greater than that of the first video frame; as well as In response to determining that the first audio frame in the audio content has not yet begun playing, the first video frame remains presented.

8. The method according to claim 7, further comprising: In response to determining that a first audio frame in the audio content has started playing, determining a wait time for presenting the first video frame based on a difference between a current PTS of an audio master clock used to play the audio content and the PTS of the first video frame, wherein the audio master clock is determined based on the PTS information of one or more audio frames in the audio frame sequence; and Based on the wait time, the first video frame is presented.

9. The method of claim 8, wherein the waiting time is determined based on the following formula: W = PTS audio -PTS video -G, Where W represents the waiting time, PTS audio Indicates the current PTS of the audio master clock, PTS video represents the PTS of the first video frame, and G represents the frame interval of the video content.

10. The method of claim 8, wherein presenting the first video frame based on the latency comprises: In response to determining that the wait time is less than or equal to zero, presenting the first video frame; or In response to determining that the wait time is greater than zero, the first video frame is presented after the wait time has elapsed.

11. The method according to claim 8, further comprising: Determining a predicted value for a presentation duration of the first video frame based on a frame interval of the video content; as well as In response to determining that the audio master clock has failed, the first video frame is presented based on the predicted value of the presentation duration. 12 . The method according to claim 1 , wherein the PTS for the audio content or the PTS for the video content is acquired based on a real-time transport protocol (RTP). 13 . The method according to claim 12 , wherein the PTS of the video frame in the video content is determined based on the sum of the decoding time stamp (DTS) of the video frame and the composition time (CTS) of the video frame. 14 . The method according to claim 13 , wherein an extension header of the RTP data packet of the video frame carries at least one of the DTS or the CTS.

15. The method according to any one of claims 1 to 14, wherein the streaming media data comprises low-latency RTM live data.

16. A device for playing streaming media data, comprising: a determining module configured to: determine PTS information of each audio frame in the audio frame sequence based on a size of an audio buffer data block in an audio buffer for audio content in streaming media data, presentation timestamp (PTS) information of the audio buffer data block, and a size of each audio frame in the audio frame sequence obtained from the audio buffer; as well as The playing module is configured to play the video content and the audio content in the streaming media data based on the PTS information of each audio frame in the audio frame sequence.

17. An electronic device comprising: at least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 15 when executed by the at least one processor.

18. A computer-readable storage medium having instructions stored thereon, which, when executed by a processor, cause the processor to implement the method according to any one of claims 1 to 15.

19. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 15.

Images