Audio and Video Synchronization Method and System for Media, and Electronic Device

Abstract

This application discloses a method and system for audio and video synchronization of media, and an electronic device. Among them, the method includes: respectively receiving video stream data from the same video source and audio stream data from the same audio source; determining a target audio frame in the audio stream data, and using the first timestamp corresponding to the target audio frame as the audio reference clock, where the first timestamp is used to indicate the start playback time of the target audio frame; determining the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data, where the second timestamp is used to indicate the start playback time of the target video frame; synchronizing the video stream data and the audio stream data according to the difference. This application solves the technical problem of out-of-sync audio and video playback on the client caused by the uncertain rendering time and linear difference in the playback of the streaming media client.

Description

Technical Field

[0001] This application relates to the field of multimedia technology, and in particular, to an audio - video synchronization method and system for media, and an electronic device. Background Art

[0002] In related scenarios of using streaming media services for audio - video streaming playback, if audio - video synchronization is not used, after a long - time playback, the situation where the audio and video do not correspond will occur, affecting the user's use and experience.

[0003] In the related art, when performing audio - video synchronization, the audio and video are mixed in a streaming media service. The audio - video data is synchronously collected at the server - side, and the playback client performs demultiplexing and then decodes and plays them separately. The problem is that there are uncertainties in the rendering time and linear differences during client - side playback, and it cannot ensure that the audio and video are accurately synchronized during client - side playback.

[0004] In view of the above situation where there are uncertainties in the rendering time and linear differences during client - side playback, resulting in the problem that the streaming media cannot achieve accurate synchronization between audio and video during client - side playback, no effective solution has been proposed yet. Summary of the Invention

[0005] Embodiments of this application provide an audio - video synchronization method and system for media, and an electronic device, so as to at least solve the technical problem of out - of - sync during audio - video playback on the client side due to uncertainties in the rendering time and linear differences during streaming media client - side playback.

[0006] According to one aspect of the embodiments of this application, an audio - video synchronization method for media is provided. This method is applied to a decoding node in a distributed system. The distributed system includes: an audio source, a video source, an encoding node, and a decoding node. The method includes: receiving video stream data from the same video source and audio stream data from the same audio source in a streaming media respectively; during the process of playing the audio stream data and the video stream data, the audio is directly played after being received and decoded. Starting from obtaining the first frame of the audio, the time is counted, and the corresponding timestamp at this time is recorded as the audio PTS. Starting from obtaining the first frame of the video, the time is counted, and the corresponding timestamp at this time is recorded as the video PTS. Determine the target audio frame in the audio stream data, and use the first timestamp corresponding to the target audio frame as the audio reference clock. Wherein, the first timestamp is used to indicate the start playback time of the target audio frame; determine the difference between the reference clock and the second timestamp corresponding to the target video frame in the video stream data. Wherein, the second timestamp is used to indicate the start playback time of the target video frame; synchronize the video stream data and the audio stream data according to the difference.

[0007] Optionally, when obtaining the PTS of audio and video, synchronous playback starts at the PTS moment obtained last. If the audio PTS is obtained first and then the video PTS, synchronous playback starts from the moment when the video PTS is obtained; if the video PTS is obtained first and then the audio PTS, synchronous playback starts from the moment when the audio PTS is obtained. However, regardless of whether the audio PTS or the video PTS is obtained first, the video is synchronized to the audio for synchronization.

[0008] Optionally, the target audio frame includes: the audio frame first received in the audio stream data; the target video frame includes: the video frame first received in the video stream data; or

[0009] The target audio frame includes: any audio frame in the audio stream data except the first received audio frame; the target video frame includes: any video frame in the video stream data except the first received video frame.

[0010] Optionally, the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data includes: the first difference between the second timestamp and the audio reference clock; the video stream data is adjusted to be synchronized with the audio stream data according to the first difference.

[0011] Optionally, synchronizing the video stream data and the audio stream data according to the difference includes: comparing the first difference with a first threshold, where the first difference is the timestamp difference between the target audio frame and the target video frame, including: the timestamp difference between the first received audio frame in the audio stream data and the first received video frame in the video stream data, or the timestamp difference between any audio frame in the audio stream data except the first received audio frame and any video frame in the video stream data except the first received video frame; performing delay or frame dropping processing on the video frames in the video stream data according to the comparison result until the first difference is less than a second threshold, where both the first threshold and the second threshold are positive numbers, and the second threshold is less than the first threshold.

[0012] Optionally, performing delay or frame dropping processing on the video frames in the video stream data includes: performing delay processing on the video frames in the video stream data when the first difference is positive and greater than the first threshold; performing frame dropping processing on the video frames in the video stream data when the first difference is negative.

[0013] Optionally, before determining the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data, the method further includes: determining whether the video stream contains B frames, and if the video stream contains B frames, triggering the calculation of the difference between the second timestamp and the reference clock.

[0014] Optionally, the method further includes: when the video stream does not include B frames, playing the audio stream data and the video stream data synchronously according to the decoding order of the audio frames in the audio stream data and the decoding order of the video frames in the video stream data.

[0015] Optionally, synchronizing the video stream data and the audio stream data according to the difference includes: obtaining the current system time and the system time obtained by the reference clock recorded in the previous frame; determining the current time relative to the reference clock according to the current system time, the system time obtained by the reference clock recorded in the previous frame, and the audio reference clock; comparing the playback time of the target video frame according to the current time relative to the reference clock.

[0016] Optionally, during the process of synchronizing the video stream data and the audio stream data, the method further includes: performing delay or frame dropping processing step by step according to the target adjustment amplitude of the target video frame until the audio frames in the audio stream data and the video frames in the video stream data are played synchronously, where synchronous playback means that the difference between the timestamps of the audio frames in the audio stream data and the video frames in the video stream data is less than the set second threshold.

[0017] Optionally, the target adjustment amplitude of the target video frame includes: counting a preset number of video frames before the target video frame, and counting a preset number of audio frames before the target audio frame; obtaining the timestamp difference between the preset number of audio frames and the preset number of video frames to obtain a preset number of timestamp differences; determining the average value of the preset number of timestamp differences, and determining the target adjustment amplitude of the target video frame according to the average value.

[0018] According to another aspect of the embodiments of the present application, there is also provided an audio-visual synchronization system for media, including: an audio source, a video source, an encoding node, and a decoding node, where the audio source is used to provide audio stream data; the video source is used to provide video stream data; the encoding node, connected to the audio source and the video source, is used to encode the audio stream data and the video stream data; the decoding node, connected to the encoding node, is used to respectively receive the video stream data from the same video source and the audio stream data from the same audio source; determining a target audio frame in the audio stream data, and using the first timestamp corresponding to the target audio frame as the reference clock, where the first timestamp is used to indicate the start playback time of the target audio frame; determining the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data, where the second timestamp is used to indicate the start playback time of the target video frame; synchronizing the video stream data and the audio stream data according to the difference.

[0019] According to another aspect of the embodiments of the present application, there is also provided an audio-video synchronization device for media, including: a receiving module, configured to receive video stream data from the same video source and audio stream data from the same audio source respectively; a first determination module, determining a target audio frame in the audio stream data, and using the first timestamp corresponding to the target audio frame as the audio reference clock, where the first timestamp is used to indicate the start playback time of the target audio frame; a second determination module, configured to determine the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data, where the second timestamp is used to indicate the start playback time of the target video frame; a synchronization module, configured to synchronize the video stream data and the audio stream data according to the difference.

[0020] According to another aspect of the embodiments of the present application, there is also provided an audio-video synchronization electronic device for media, including: a communication module, configured to receive video stream data from the same video source and audio stream data from the same audio source respectively; a processor, connected to the communication module, determining a target audio frame in the audio stream data, and using the first timestamp corresponding to the target audio frame as the reference clock, where the first timestamp is used to indicate the start playback time of the target audio frame; determining the difference between the reference clock and the second timestamp corresponding to the target video frame in the video stream data, where the second timestamp is used to indicate the start playback time of the target video frame; synchronizing the video stream data and the audio stream data according to the difference; a display, connected to the processor, configured to display the video stream data and the audio stream data.

[0021] According to another aspect of the embodiments of the present application, there is also provided a non-volatile storage medium, which includes a stored program, where when the program runs, it controls the device where the storage medium is located to execute the above audio-video synchronization method for streaming media.

[0022] In the embodiments of the present application, an audio-video separation technology is adopted. The client receives video stream data from the same video source and audio stream data from the same audio source. Taking the first timestamp corresponding to the target audio frame as the reference clock, and recording the start playback time of the target video frame as the second timestamp, the video is synchronized to the audio. By comparing the difference between the second timestamp corresponding to the target video frame and the reference clock, the purpose of synchronizing the audio data stream and the video data stream is achieved, thereby realizing the technical effect of audio-video synchronization when the streaming media client plays, and further solving the technical problem of out-of-sync when the client plays audio and video due to the uncertain rendering time and linear difference in the streaming media client playback. Description of the Drawings

[0023] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0024] Figure 1 It is a structural diagram of an audio - video synchronization electronic device for a medium according to an embodiment of the present application;

[0025] Figure 2 It is a flowchart of a method for audio - video synchronization of a medium according to an embodiment of the present application;

[0026] Figure 3 It is a flowchart of a method for processing B - frames in a video stream according to an embodiment of the present application;

[0027] Figure 4 It is a flowchart of a method for difference calculation according to an embodiment of the present application;

[0028] Figure 5 It is a flowchart of a method for processing video frame delay or frame loss according to an embodiment of the present application;

[0029] Figure 6 It is a flowchart of audio - video synchronization according to an embodiment of the present application;

[0030] Figure 7 It is a flowchart of a target adjustment amplitude method for a target video frame according to an embodiment of the present application;

[0031] Figure 8 It is a flowchart of a method for audio - video synchronization according to an embodiment of the present application;

[0032] Figure 9 It is a structural diagram of an audio - video synchronization system for a medium according to an embodiment of the present application.

[0033] Figure 10 It is a structural diagram of an audio - video synchronization device for a medium according to an embodiment of the present application. Detailed implementation manners

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0035] It should be noted that in the description, claims and the above drawings of the present application, terms such as "first" and "second" are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily limit to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0036] In the related art, to ensure the audio-visual data stream, the audio and video are mixed in a streaming media service. The audio and video data are synchronously collected at the server side, and the playback client performs demultiplexing and then decodes and plays them separately. There are three main reasons for the audio-visual asynchronization: 1. It is difficult to accurately control the playback time of a frame. The decoding and rendering time of the audio and video are different, which may cause slight differences in the output of each frame. Over time, the asynchronization will become more and more obvious; 2. The audio output is linear, while the video output can be either linear or non-linear, resulting in deviations; 3. There may be a certain difference between the audio and video themselves, that is, the starting points of the first frames of the audio and video are different, resulting in audio-visual asynchronization during playback.

[0037] When processing audio and video, there are two commonly used timestamps. The first is DTS (Decoding TimeStamp), which is the decoding timestamp used to tell the player when to decode the data of this frame; PTS (Presentation Time Stamp), which is the display timestamp used to tell the player when to display the data of this frame. When encoding a video, different processing methods for frames will generate different video frames. The B frame, which is located between the I frame and the P frame, affects the above two timestamps. The I frame is the key frame, and the P frame is used to repair based on the I frame. The presence of B frames in the video stream will increase the network latency, but it can ensure smooth pictures. When there are no B frames in the video stream, the order of DTS and PTS is usually the same; but when there are B frames in the video stream, the decoding order and the playback order are inconsistent, that is, the video output is non-linear at this time, thus causing the problem of audio-visual asynchronization.

[0038] To solve the audio - video synchronization problem, there are three most basic synchronization strategies: synchronizing video to audio, synchronizing audio to video, and synchronizing video and audio to an external clock. Considering that humans are more sensitive to sound than video, frequently adjusting the audio will bring a poor visual experience to users. Since the playback clock of audio increases linearly, in this application, the audio clock is selected as the reference clock. During encoding, timestamps are added to each audio - video data block based on the reference clock. During playback, based on the audio - video timestamps and the reference clock, the played video is adjusted to synchronize the video to the audio. The specific method has been described in detail in the embodiments.

[0039] Figure 1 is a structural diagram of an audio - video synchronization electronic device for a medium according to an embodiment of the present application, as Figure 1 shown, the device includes:

[0040] A communication module 10, configured to receive video stream data from the same video source and audio stream data from the same audio source respectively; for example, when using the LIVE555 video player on a computer with a Linux system to play a video on a web page, the client player will pull the video stream and audio stream of the played video from the streaming media server, and perform corresponding decoding operations on the obtained audio stream and video stream, or the client can receive audio stream data and video stream data forwarded by a forwarding device such as a router.

[0041] A processor 12, connected to the communication module 10, configured to directly play the audio after it is received and decoded during the process of playing the audio stream data and video stream data. Starting from the time when the first audio frame is obtained, the corresponding timestamp is recorded as the audio PTS. Starting from the time when the first video frame is obtained, the corresponding timestamp is recorded as the video PTS. Determine the target audio frame in the audio stream data, and use the first timestamp corresponding to the target audio frame as the audio reference clock. Among them, the target audio frame includes: the first - received audio frame in the audio stream data, and any audio frame other than the first - received audio frame during the process of playing the audio stream data. The first timestamp is used to indicate the start - playing time of the target audio frame; determine the difference between the reference clock and the second timestamp corresponding to the target video frame in the video stream data. Among them, the target video frame includes: the first - received video frame in the video stream data, and any video frame other than the first - received video frame during the process of playing the video stream data. The second timestamp is used to indicate the start - playing time of the target video frame; synchronize the video stream data and the audio stream data according to the difference between the reference clock and the second timestamp corresponding to the target video frame in the video stream data.

[0042] A display 14, connected to the processor 12, is used to display video stream data and audio stream data. After the client finishes decoding the audio stream data and video stream data, the user can see the synchronized audio and video on the display.

[0043] Specifically, the electronic device provided in the embodiments of the present application can be a mobile phone, iPad, computer, etc. In view of the problem of audio-video out-of-sync during client playback in the existing related technologies, the electronic device provided in this embodiment adjusts the pulling of audio-video streams and adopts an audio-video separation technology. Specifically, the client can pull the audio stream and video stream from the streaming media server respectively. Streaming media refers to Internet products that can watch audio and video online, including video players, audio players, and live broadcast software, etc. It can also receive audio stream data and video stream data forwarded by forwarding devices such as routers. Considering that people are more sensitive to sound than video, frequent adjustment of audio will bring a poor viewing experience. Since the playback clock of audio increases linearly, in this application, the audio clock is selected as the reference clock. During encoding, timestamps are added to each audio-video data block according to the reference clock. During playback, according to the audio-video timestamps and the reference clock, the playback video is adjusted to synchronize the video to the audio. By calculating the difference between the second timestamp corresponding to the target video frame and the audio reference clock, and slowly adjusting the difference between the audio reference clock and the second timestamp according to the target adjustment amplitude of the target video frame, the synchronization of the audio data stream and the video data stream is achieved.

[0044] Under the above operating environment, the embodiments of the present application provide an embodiment of an audio-video synchronization method for media. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0045] Figure 2 is a flowchart of an audio-video synchronization method for media according to an embodiment of the present application. As Figure 2 shown, the method includes the following steps:

[0046] Step S202, respectively receive video stream data from the same video source and audio stream data from the same audio source;

[0047] For example, when using the LIVE555 video player on a computer with a Linux system to play a video on a web page, the client player pulls the video stream and audio stream of the played video from the streaming media server and performs corresponding decoding operations on the obtained video stream and audio stream. Or, the client can also receive audio stream data and video stream data forwarded by forwarding devices such as routers.

[0048] Step S204, during the process of playing the audio stream data and the video stream data, the audio is directly played after being received and decoded. Timing starts from the moment when the first audio frame is obtained, and the corresponding timestamp at this moment is recorded as the audio PTS. Timing starts from the moment when the first video frame is obtained, and the corresponding timestamp at this moment is recorded as the video PTS. Determine the target audio frame in the audio stream data, and use the first timestamp corresponding to the target audio frame as the audio reference clock. Among them, the target audio frame includes: the first audio frame received for the first time in the audio stream data, and any audio frame other than the first audio frame received for the first time during the process of playing the audio stream data; the first timestamp is used to indicate the start playback time of the target audio frame;

[0049] Step S206, determine the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data. Among them, the target video frame includes: the first video frame received for the first time in the video stream data, and any video frame other than the first video frame received for the first time during the process of playing the video stream data; the second timestamp is used to indicate the start playback time of the target video frame;

[0050] Step S208, synchronize the video stream data and the audio stream data according to the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data.

[0051] Through the above steps, by using the audio-video separation technology, the client can receive the video stream data from the same video source and the audio stream data from the same audio source. Taking the first timestamp corresponding to the target audio frame as the reference clock, and recording the start playback time of the target video frame as the second timestamp, synchronize the video to the audio. By calculating the difference between the second timestamp corresponding to the target video frame and the reference clock, and slowly adjusting the difference between the reference clock and the second timestamp according to the target adjustment amplitude of the target video frame, the purpose of synchronizing the audio data stream and the video data stream is achieved, thus realizing the technical effect of audio-video synchronization when the streaming media client is playing, and further solving the technical problem of out-of-sync audio and video when the client plays due to the uncertain rendering time and linear difference in the streaming media client playback. Among them, the linear difference refers to the inconsistency between the video decoding order and the playback order due to the presence of B frames in the video stream.

[0052] Before step S204, when obtaining the PTS of the audio and video, start synchronous playback from the moment of the last obtained PTS. If the audio PTS is obtained first and then the video PTS is obtained, start synchronous playback from the moment of obtaining the video PTS; if the video PTS is obtained first and then the audio PTS is obtained, start synchronous playback from the moment of obtaining the audio PTS. However, whether the audio PTS or the video PTS is obtained first, the video is synchronized to the audio for synchronization, and the difference between the video PTS and the audio PTS at this time is calculated and recorded as d.

[0053] Before determining the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data in step S206, as Figure 3 shown, the above method further includes the following steps:

[0054] Step S302, determine whether the video stream includes B frames. B frames represent bidirectional difference frames. B frames are located between I frames and P frames. I frames are key frames, and P frames are used for repair based on I frames and record the differences between this frame and the previous and subsequent frames. The presence of B frames in the video stream will increase network latency but ensure smooth pictures. When decoding B frames, not only the previous cached pictures but also the subsequent pictures need to be decoded, and the final picture is obtained by superimposing the data of the previous and subsequent pictures with the data of this frame. Whether the video stream contains B frames can be detected according to the encoding method;

[0055] Step S304, when the video stream includes B frames, indicating that the video output is non-linear, trigger the calculation of the difference between the second timestamp and the reference clock;

[0056] Step S306, when the video stream does not include B frames, indicating that the video output is linear, synchronously play the audio stream data and the video stream data according to the decoding order of the audio frames in the audio stream data and the decoding order of the video frames in the video stream data.

[0057] In step S208, synchronize the video stream data and the audio stream data according to the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data, as Figure 4 shown, the difference includes the following steps:

[0058] Step S402, calculate the first difference between the second timestamp and the reference clock;

[0059] Step S404, compare the first difference with the first threshold. The first difference is the timestamp difference between the target audio frame and the target video frame, including: the timestamp difference between the first received audio frame in the audio stream data and the first received video frame in the video stream data, or the timestamp difference between any audio frame other than the first received audio frame in the audio stream data and any video frame other than the first received video frame in the video stream data;

[0060] Step S406: Perform latency or frame dropping processing on the video frames in the video stream data according to the comparison result until the first difference is less than the second threshold. Here, both the first threshold and the second threshold are positive numbers, and the second threshold is less than the first threshold. The first threshold is the maximum value of the allowable value range of the first difference, and the second threshold is the minimum value of the allowable value range of the first difference. The first threshold and the second threshold can be set by the target user themselves. For example, the first threshold is set to 200 ms and the second threshold is set to 20 ms.

[0061] Optionally, perform latency or frame dropping processing on the video frames in the video stream data, such as Figure 5 shown, including:

[0062] Step S502: When the first difference is positive and greater than the first threshold, perform latency processing on the video frames in the video stream data. For example, when the playback speed of the current video frame is faster than that of the current audio frame, perform latency processing on the current video frame;

[0063] Step S504: When the first difference is negative, perform frame dropping processing on the video frames in the video stream data. For example, when decoding the second frame of the audio stream data, the first frame of the video stream data has not been decoded yet. If frame dropping is not performed in time, the time difference between the target video frame and the target audio frame will become larger and larger, and the displayed picture will have a mosaic, and the target user will clearly notice the out-of-sync audio and video, resulting in a decline in the viewing experience.

[0064] In step S208, synchronize the video stream data and the audio stream data according to the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data, such as Figure 6 shown, specifically including the following steps:

[0065] Step S602: Obtain the current system time and the system time when the reference clock was recorded for the previous frame;

[0066] Step S604: Determine the current time relative to the reference clock based on the current system time, the system time when the reference clock was recorded for the previous frame, and the audio reference clock;

[0067] Step S606: Compare the playback time of the target video frame based on the current time relative to the reference clock.

[0068] The audio reference clock can refer to the initial audio reference clock, i.e., the PTS carried by the first packet of audio frames, or it can refer to the reference clock updated each time an audio stream is received. Depending on what the audio reference clock refers to, the above steps are divided into two cases: In the first case, when the audio reference clock refers to the initial audio reference clock, i.e., the audio reference clock is not updated, and each time the first audio clock is used as the reference, the current system time and the system time obtained when the reference clock of the previous frame was recorded are acquired. The current system time is the running time relative to the start of the system, and the system time obtained when the reference clock of the previous frame was recorded refers to the system time when the audio reference clock was first updated upon receiving the audio stream. The difference between the current system time and the system time when the audio reference clock was first updated is calculated, and this difference is denoted as c1. The sum of c1 and the audio reference clock is the current time relative to the audio reference clock, and this time is used to compare with the time of the target video frame, and frame dropping or delay processing of the video frame is performed according to the comparison value. For example, if the current system time is 8:30:30, the system time obtained when the reference clock of the previous frame was recorded is 8:30:10, and the audio reference clock is not updated, with the first audio reference clock as the reference, if the time of the first audio reference clock is 8:00:00, then the current time relative to the audio reference clock is 8:30:30 - 8:30:10 + 8:00:00 = 8:00:20, that is, the audio frame has 20s of audio stream data. The difference between the time that the current target video frame has been played and 20s is compared, and frame dropping or delay processing of the video frame is determined according to this difference. In the second case, when the audio reference clock refers to the reference clock updated each time an audio stream is received, the current system time and the system time obtained when the reference clock of the previous frame was recorded are acquired. The current system time is the running time relative to the start of the system, and the system time obtained when the reference clock of the previous frame was recorded refers to the system time when the audio reference clock was last updated upon receiving the audio stream. The difference between the current system time and the system time when the audio reference clock was last updated is calculated, and this difference is denoted as c2. The sum of c2 and the audio reference clock is the current time relative to the audio reference clock, and this time is used to compare with the time of the target video frame, and frame dropping or delay processing of the video frame is performed according to the comparison value. For example, if the current system time is 8:30:30, the system time obtained when the reference clock of the previous frame was recorded is 8:30:10, the audio reference clock is updated, with the previous audio reference clock as the reference, and the time of the previous audio reference clock is 8:30:10, then the current time relative to the audio reference clock is 8:30:30 - 8:30:10 + 8:30:10 = 8:30:20, that is, the audio frame has 20s of audio stream data. The difference between the time that the current target video frame has been played and 20s is compared, and frame dropping or delay processing of the video frame is determined according to this difference..

[0069] During the process of synchronizing the video stream data and the audio stream data in step S208, it specifically includes: gradually performing delay or frame dropping processing according to the target adjustment amplitude of the target video frame until the audio frames in the audio stream data and the video frames in the video stream data are played synchronously.

[0070] Optionally, the target adjustment amplitude of the target video frame is determined in the following manner, as Figure 7 shown, and specifically includes the following steps:

[0071] Step S702, count a preset number of video frames before the target video frame, and count a preset number of audio frames before the target audio frame;

[0072] Step S704, obtain the timestamp differences between the preset number of audio frames and the preset number of video frames, and obtain a preset number of timestamp differences;

[0073] Step S706, determine the average value of the preset number of timestamp differences, and determine the target adjustment amplitude of the target video frame based on the average value. For example, there are 6 video frames and 6 audio frames in the preset number. Calculate the timestamp differences between these 6 groups of preset number of video frames and preset number of audio frames, and sum up these 6 groups of timestamp differences. These 6 groups of timestamp differences are not necessarily the same. Assuming the sum of these 6 groups of timestamp differences is 72ms, then the target adjustment amplitude is 72÷6 = 12ms. Perform synchronous adjustment of video frames and audio frames in 6 times, and the adjustment amplitude each time is 12ms. This operation method avoids the problem of too large or too small adjustment amplitude each time, and can achieve the effect of slow convergence, making the synchronous operation not abrupt.

[0074] Figure 8 is a flowchart of an audio-video synchronization method according to an embodiment of the present application, as Figure 8As shown, when a streaming media client receives video stream data from the same video source and audio stream data from the same audio source, where the client can pull the video stream and audio stream of the played video from a streaming media server, or the client can also receive audio stream data and video stream data forwarded by a forwarding device such as a router. First, it is determined whether there is only an audio stream or a video stream in the audio source and the video source. If there is only an audio stream or a video stream, the client can directly decode and play when receiving the corresponding audio stream or data stream. If the stream pulled by the client contains both an audio stream and a video stream, it is then determined whether the video stream contains B frames. Since the decoding order and playback order of a video stream containing B frames are inconsistent, that is, the video output at this time is non-linear, which will cause the problem of audio-video asynchronization. If it is detected that the video stream does not contain B frames, the obtained audio stream data and video stream data are played synchronously according to the decoding order. If it is detected that the video stream contains B frames, they are played in the order of receiving the audio data stream and the video data stream. Starting from the time when the first audio frame is obtained, the corresponding timestamp at this time is recorded as the audio PTS, and the first timestamp corresponding to the target audio frame is used as the audio reference clock. The audio reference clock can refer to the initial audio reference clock, that is, the PTS carried by the first packet of audio frames, or it can refer to the reference clock updated each time the audio stream is received. Starting from the time when the first video frame is obtained, the corresponding timestamp at this time is recorded as the video PTS. The first difference between the audio reference clock and the second timestamp corresponding to the target video frame is compared, and the video stream data and the audio stream data are synchronized. Among them, the first difference is divided into two categories, including: the timestamp difference between the first received audio frame in the audio stream data and the first received video frame in the video stream data, or the timestamp difference between any audio frame other than the first received audio frame in the audio stream data and any video frame other than the first received video frame in the video stream data; the first difference is compared with the first threshold, and according to the comparison result, the video frames in the video stream data are delayed or dropped until the first difference is less than the second threshold, where both the first threshold and the second threshold are positive numbers, and the second threshold is less than the first threshold; when the first difference is positive and greater than the first threshold, the video frames in the video stream data are delayed; when the first difference is negative, the video frames in the video stream data are dropped.

[0075] It should be noted that Figure 8 The flowchart of the audio-video synchronization method shown is based on Figures 2 - 7 the audio-video synchronization method of the media shown, so the relevant explanations in the above audio-video synchronization method of the media also apply to this flowchart of the audio-video synchronization method, which will not be elaborated here.

[0076] Figure 9 is the structural diagram of an audio-video synchronization system of a media according to an embodiment of the present application, asFigure 9 As shown in the figure, the system includes:

[0077] An audio source 90 for providing audio stream data;

[0078] A video source 92 for providing video stream data;

[0079] An encoding node 94, connected to the audio source and the video source, for encoding the audio stream data and the video stream data;

[0080] A decoding node 96, connected to the encoding node, for respectively receiving the video stream data from the same video source and the audio stream data from the same audio source; determining a target audio frame in the audio stream data, and using the first timestamp corresponding to the target audio frame as a reference clock, where the first timestamp is used to indicate the start playback time of the target audio frame; determining the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data, where the second timestamp is used to indicate the start playback time of the target video frame; and synchronizing the video stream data and the audio stream data based on the difference.

[0081] It should be noted that Figure 9 the audio-visual synchronization system of the media shown is based on Figures 2 - 7 the audio-visual synchronization method of the media shown. Therefore, the relevant explanations in the above audio-visual synchronization method of the media also apply to the audio-visual synchronization system of the media, and will not be elaborated here.

[0082] Figure 10 is a structural diagram of an audio-visual synchronization device of a media according to an embodiment of the present application. As Figure 10 shown, the device includes:

[0083] A receiving module 100 for respectively receiving the video stream data from the same video source and the audio stream data from the same audio source;

[0084] A first determination module 102, during the playback of the audio stream data and the video stream data, the audio is directly played after being received and decoded. Timing starts from obtaining the first audio frame, and the corresponding timestamp at this time is recorded as the audio PTS. Timing starts from obtaining the first video frame, and the corresponding timestamp at this time is recorded as the video PTS. Determining a target audio frame in the audio stream data, and using the first timestamp corresponding to the target audio frame as the audio reference clock, and updating this reference clock each time an audio frame is played. Wherein, the target audio frame includes: the first audio frame first received in the audio stream data, and any audio frame other than the first received audio frame during the playback of the audio stream data. The first timestamp is used to indicate the start playback time of the target audio frame;

[0085] A second determination module 104, configured to determine a difference between an audio reference clock and a second timestamp corresponding to a target video frame in video stream data, where the target video frame includes: the first received video frame in the video stream data, and any video frame other than the first received video frame during the playback of the video stream data, and the second timestamp is used to indicate the start playback time of the target video frame; synchronize the video stream data and the audio stream data according to the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data;

[0086] A synchronization module 106, configured to synchronize the video stream data and the audio stream data according to the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data.

[0087] It should be noted that Figure 10 the audio-visual synchronization device of the media shown is used to execute Figures 2 - 7 the audio-visual synchronization method of the media shown. Therefore, the relevant explanations in the above audio-visual synchronization method of the media also apply to the audio-visual synchronization device of the media, and will not be elaborated here.

[0088] This embodiment of the specification also provides a non-volatile storage medium, where the non-volatile storage medium includes a stored program, and when the program runs, it controls the device where the storage medium is located to execute the following audio-visual synchronization method of the media:

[0089] Receive video stream data from the same video source and audio stream data from the same audio source respectively;

[0090] After the audio is received and decoded, it is directly played. Timing starts from obtaining the first audio frame, and the corresponding timestamp at this time is recorded as the audio PTS. Timing starts from obtaining the first video frame, and the corresponding timestamp at this time is recorded as the video PTS. Determine the target audio frame in the audio stream data, and use the first timestamp corresponding to the target audio frame as the audio reference clock. Where the target audio frame includes: the first received audio frame in the audio stream data, and any audio frame other than the first received audio frame during the playback of the audio stream data, and the first timestamp is used to indicate the start playback time of the target audio frame;

[0091] Determine the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data, where the target video frame includes: the first received video frame in the video stream data, and any video frame other than the first received video frame during the playback of the video stream data, and the second timestamp is used to indicate the start playback time of the target video frame; synchronize the video stream data and the audio stream data according to the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data;

[0092] Synchronize the video stream data and the audio stream data according to the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data.

[0093] The serial numbers of the embodiments of the present application above are only for description and do not represent the superiority or inferiority of the embodiments.

[0094] In the above embodiments of the present application, the descriptions of the respective embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0095] In the several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units can be a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the units or modules can be in an electrical or other form.

[0096] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0097] In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0098] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0099] The above are only the preferred embodiments of this application. It should be noted that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of this application.

Claims

1. A method for audio and video synchronization of a medium, characterized in that, The method is applied to a decoding node in a distributed system, where the distributed system includes: an audio source, a video source, an encoding node, and a decoding node. The method includes: Receiving video stream data from the same video source and audio stream data from the same audio source respectively; Determining a target audio frame in the audio stream data, and using the first timestamp corresponding to the target audio frame as an audio reference clock, where the first timestamp is used to indicate the start playback time of the target audio frame; Determining the difference between the audio reference clock and the second timestamp corresponding to a target video frame in the video stream data, where the second timestamp is used to indicate the start playback time of the target video frame; Synchronizing the video stream data and the audio stream data according to the difference; Wherein, the method further includes: in response to the audio reference clock being a non-updated audio reference clock, determining the difference between the current system time and the system time when the non-updated audio reference clock is recorded, and based on the comparison result between the sum of the difference and the non-updated audio reference clock and the played time of the target video frame, performing frame dropping processing or delay processing on the target video frame; where the non-updated audio reference clock is the audio reference clock when the audio stream data is first received; In response to the audio reference clock being an updated audio reference clock, determining the difference between the current system time and the system time when the updated audio reference clock was last recorded, where the updated audio reference clock is the audio reference clock updated each time the audio stream data is received; based on the comparison result between the sum of the difference and the previously recorded updated audio reference clock and the played time, performing the frame dropping processing or the delay processing on the target video frame.

2. The method according to claim 1, wherein The target audio frame includes: the first received audio frame in the audio stream data; the target video frame includes: the first received video frame in the video stream data; or The target audio frame includes: any audio frame in the audio stream data other than the first received audio frame; the target video frame includes: any video frame in the video stream data other than the first received video frame.

3. The method according to claim 1, wherein The difference includes: a first difference between the second timestamp and the audio reference clock; synchronizing the video stream data and the audio stream data according to the difference includes: Comparing the first difference with a first threshold, where the first difference is the timestamp difference between the target audio frame and the target video frame; performing the delay processing or the frame dropping processing on the video frames in the video stream data according to the comparison result until the first difference is less than a second threshold, where both the first threshold and the second threshold are positive numbers, and the second threshold is less than the first threshold.

4. The method according to claim 3, characterized in that, Performing the delay processing or the frame dropping processing on the video frames in the video stream data includes: When the first difference is positive and greater than the first threshold, perform the delay processing on the video frames in the video stream data; When the first difference is negative, perform the frame dropping processing on the video frames in the video stream data.

5. The method according to any one of claims 1 to 4, characterized in that The method further includes: During the process of synchronizing the video stream data and the audio stream data, perform the delay processing or the frame dropping processing step by step according to the target adjustment amplitude of the target video frame until the audio frames in the audio stream data and the video frames in the video stream data are played synchronously, where synchronous playing means that the difference between the timestamps of the audio frames in the audio stream data and the video frames in the video stream data is less than the set second threshold.

6. The method according to claim 5, wherein The target adjustment amplitude of the target video frame is determined by the following method: Count a preset number of video frames before the target video frame, and count the preset number of audio frames before the target audio frame; Obtain the timestamp difference between the preset number of audio frames and the preset number of video frames to obtain the preset number of timestamp differences; Determine the average value of the preset number of timestamp differences, and determine the target adjustment amplitude of the target video frame based on the average value.

7. An audio-video synchronization system for a medium, characterized in that, It includes: An audio source, a video source, an encoding node, and a decoding node, where The audio source is used to provide audio stream data; The video source is used to provide video stream data; The encoding node is connected to the audio source and the video source and is used to encode the audio stream data and the video stream data; The decoding node is connected to the encoding node and is used to respectively receive the video stream data from the same video source and the audio stream data from the same audio source, determine the target audio frame in the audio stream data, and use the first timestamp corresponding to the target audio frame as the reference clock, where the first timestamp is used to indicate the start playback time of the target audio frame; determine the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data, where the second timestamp is used to indicate the start playback time of the target video frame; synchronize the video stream data and the audio stream data based on the difference. Wherein, the system is further configured to perform the following steps: in response to the audio reference clock being the non-updated audio reference clock, determine the difference between the current system time and the system time when the non-updated audio reference clock is recorded, and based on the comparison result between the sum of the difference and the non-updated audio reference clock and the played time of the target video frame, perform frame dropping processing or delay processing on the target video frame; where the non-updated audio reference clock is the audio reference clock when the audio stream data is first received. In response to the audio reference clock being an updated audio reference clock, determine the difference between the current system time and the system time when the updated audio reference clock was last recorded, where the updated audio reference clock is the audio reference clock updated each time the audio stream data is received; based on the comparison result between the sum of the difference and the last recorded updated audio reference clock and the played time, perform the frame dropping process or the delay process on the target video frame.

8. An audio-video synchronization electronic device for a medium, characterized in that, Comprising: A communication module, configured to receive video stream data from the same video source and audio stream data from the same audio source respectively; A processor, connected to the communication module, configured to determine a target audio frame in the audio stream data, and use the first timestamp corresponding to the target audio frame as the audio reference clock, where the first timestamp is used to indicate the start playback time of the target audio frame; determine the difference between the audio reference clock and the second timestamp corresponding to the target video frame in the video stream data, where the second timestamp is used to indicate the start playback time of the target video frame; synchronize the video stream data and the audio stream data according to the difference. A display, connected to the processor, configured to display the video stream data and the audio stream data; Wherein, the processor is further configured to perform the following steps: in response to the audio reference clock being a non-updated audio reference clock, determine the difference between the current system time and the system time when the non-updated audio reference clock was recorded, and based on the comparison result between the sum of the difference and the non-updated audio reference clock and the played time of the target video frame, perform the frame dropping process or the delay process on the target video frame; where the non-updated audio reference clock is the audio reference clock when the audio stream data is first received. In response to the audio reference clock being an updated audio reference clock, determine the difference between the current system time and the system time when the updated audio reference clock was last recorded, where the updated audio reference clock is the audio reference clock updated each time the audio stream data is received; based on the comparison result between the sum of the difference and the last recorded updated audio reference clock and the played time, perform the frame dropping process or the delay process on the target video frame.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, where when the program runs, it controls the device where the storage medium is located to execute the audio-video synchronization method of the medium according to any one of claims 1 to 6.

Images