A method for audio-visual synchronization in MSE playback scenarios

Abstract

This invention discloses a method for audio-visual synchronization in an MSE playback scenario. The method includes pre-synchronization: before the audio and video data downloaded to the browser is pushed to the player, preliminary alignment is performed to ensure that the audio and video data received by the player does not deviate too much; player buffering is optimized: the data volume of the audio stream and video stream are counted and monitored separately; the data volume of the stream with the lowest water level is taken as the final data water level reported; if the reported water level is lower than the low water level of the buffer threshold, playback is paused and buffering is initiated; after entering the buffering state, playback resumes and the buffering state ends only when both data streams are higher than the high water level of the buffer threshold. This invention ensures that audio and video segments remain approximately within the same time period, preventing playback abnormalities and preventing the player's pipeline from running idle without data.

Description

Technical Field

[0001] This invention relates to the field of audio and video playback technology, specifically, a method for audio-visual synchronization in an MSE playback scenario. Background Technology

[0002] MSE (MediaSourceExtensions) can be understood as an API that provides the functionality for implementing plugin-free, web-based streaming media. Through MSE, media streams can be created using JavaScript and can utilize HTML5. <audio>and <video>Playback is tagged. Compared to traditional online playback, MSE offers flexible buffer management and seamless bitrate switching, leading more and more video websites to choose it as their online video playback method in recent years. Therefore, support for this new playback method by the playback terminal is becoming increasingly important. Audio-video synchronization is a crucial component of MSE playback, requiring proper handling of audio and video synchronization to ensure a good user experience when watching online videos. For the playback terminal, a key characteristic of MSE playback is that the audio and video streams are independent. They no longer share common PCR (program clock reference) information, and their data downloads are separate. Therefore, the audio-video synchronization of MSE needs to be redesigned. Summary of the Invention

[0003] The purpose of this invention is to provide a method for audio-visual synchronization in MSE playback scenarios, so as to achieve synchronized playback of audio and video in MSE playback.

[0004] The present invention solves the above problems through the following technical solution:

[0005] A method for audio-visual synchronization in an MSE playback scenario includes:

[0006] Step S100: The browser downloads the audio and video data, performs preliminary alignment, and then pushes it to the player. This specifically includes:

[0007] Each time the browser downloads an audio fragment and a video fragment, it records the start and end times of the audio fragment and the video fragment, and updates the current highest time of the audio fragment to the end time of the last downloaded audio fragment, and updates the current highest time of the video fragment to the end time of the last downloaded video fragment.

[0008] When a browser pushes an audio / video fragment to a player, it checks the start time of that fragment. If the start time is earlier than or equal to the current highest time of the video stream / audio stream, the fragment is pushed directly to the player. Otherwise, the fragment is saved while the video / audio stream downloads the new fragment. The current highest time of the video stream / audio stream is updated to the end time of the new fragment. If the current highest time of the video stream / audio stream is later than the start time of the saved fragment, the saved fragment is pushed to the player.

[0009] Step S200: Optimize the player's buffering mechanism, specifically including:

[0010] Monitor and count the data volume of audio stream and video stream separately, and take the data volume with the lower data volume in audio stream and video stream as the final data volume reported;

[0011] If the final data level is not lower than the low level of the buffer threshold, playback will proceed normally. If it is lower than the low level of the buffer threshold, playback will be paused and the system will enter a buffering state. Once in the buffering state, playback will resume and the buffering state will end when the data volume of both the audio and video streams is higher than the high level of the buffer threshold.

[0012] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0013] The pre-synchronization mechanism of this invention performs preliminary synchronization before data is transmitted to the player, ensuring that audio and video fragments are approximately within the same time frame. This effectively prevents situations where only one channel in the player has data while the other is empty, thus preventing playback abnormalities. Furthermore, by utilizing an optimized buffering mechanism on the player side, audio and video data are statistically analyzed separately. This ensures that when audio and video are sent independently, the player can promptly enter a buffering state if either audio or video data is insufficient, preventing the player pipeline from running idle without data. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the pre-synchronization mechanism in this invention. Detailed Implementation

[0015] The present invention will be further described in detail below with reference to embodiments, but the implementation of the present invention is not limited thereto.

[0016] Example 1:

[0017] A method for audio-visual synchronization in an MSE playback scenario includes:

[0018] (I) Pre-synchronization mechanism:

[0019] Before the audio and video data downloaded by the browser is pushed to the player, preliminary alignment is required to ensure that the audio and video data received by the player is not too far off. The specific steps are as follows:

[0020] 1. For each downloaded audio / video fragment, obtain and record its start and end times (start time and end time);

[0021] 2. When pushing audio and video data to the player, check the start and end times of this data segment. If the start time of this data segment has exceeded the end time of the last data segment pushed from another data segment, save it and do not push it for the time being.

[0022] 3. The other data stream continues to update and download. After the end time of its pushed data is updated, the start time of this data is compared with the end time of the updated data from the other stream. When the end time of the data pushed by the other stream exceeds the start time of the current data fragment, the current fragment of this stream is pushed.

[0023] 4. Repeat this process for each downloaded fragment.

[0024] like Figure 1 As shown, audio path L1, video path L2, audio path is downloaded to audio fragment L1F1, and its start time ST1 and end time ET1 are recorded; video path is downloaded to video fragment L2F1, and its start time ST2 and end time ET2 are recorded.

[0025] Compare ST1 with the current highest time CT2 of the video path (at this time CT2 = ET2). If ST1 <= CT2, then L1F1 can be sent and the current highest time CT1 of the audio path is set to ET1. Otherwise, wait and check repeatedly.

[0026] Compare ST2 with the current highest time CT1 of the audio path (at this time CT1 = ET1). If ST2 <= CT1, then L2F1 can be sent and the current highest time CT2 of the video path is set to ET2. Otherwise, wait and check cyclically.

[0027] The same processing is performed on each downloaded audio and video fragment.

[0028] (II) Optimize the player's buffering mechanism:

[0029] 1. Separately count and monitor the data volume of audio and video streams.

[0030] 2. Take the data from the two data streams with the lowest water level as the final data water level report.

[0031] 3. If the reported water level is lower than the buffer threshold, playback will be paused and the device will enter buffer mode.

[0032] 4. Once in buffering mode, playback will resume and the buffering state will end only when both data streams are above the buffer threshold.

[0033] Although the present invention has been described herein with reference to illustrative embodiments, the above embodiments are merely preferred embodiments of the present invention, and the implementation of the present invention is not limited to the above embodiments. It should be understood that those skilled in the art can devise many other modifications and implementations, which will fall within the scope and spirit of the principles disclosed in this application.< / video> < / audio>

Claims

1. A method for audio-visual synchronization in an MSE playback scenario, characterized in that, include: Step S100: The browser downloads the audio and video data, performs preliminary alignment, and then pushes it to the player. This specifically includes: Each time the browser downloads an audio fragment and a video fragment, it records the start and end times of the audio fragment and the video fragment, and updates the current highest time of the audio fragment to the end time of the last downloaded audio fragment, and updates the current highest time of the video fragment to the end time of the last downloaded video fragment. When a browser pushes an audio / video fragment to a player, it checks the start time of that fragment. If the start time is earlier than or equal to the current highest time of the video stream / audio stream, the fragment is pushed directly to the player. Otherwise, the fragment is saved while the video / audio stream downloads the new fragment. The current highest time of the video stream / audio stream is updated to the end time of the new fragment. If the current highest time of the video stream / audio stream is later than the start time of the saved fragment, the saved fragment is pushed to the player. Step S200: Optimize the player's buffering mechanism, specifically including: Monitor and count the data volume of audio stream and video stream separately, and take the data volume with the lower data volume in audio stream and video stream as the final data volume reported; If the final data level is lower than the buffer threshold (low level), playback will pause and enter buffering mode. Once in buffering mode, playback will resume and the buffering mode will end when both the audio and video stream data levels are higher than the buffer threshold (high level).

Images