Audio and video synchronization method and device, electronic equipment, medium and program product

Abstract

Embodiments of the present specification disclose an audio-video synchronization method and device, electronic equipment, medium and program product. The method comprises: obtaining audio-video data, the audio-video data comprising audio packets and video packets; updating display timestamps corresponding to the audio packets and the video packets respectively by using absolute timestamps; inserting the audio packets into an audio buffer and inserting the video packets into a video buffer according to a first preset rule; taking out a target audio packet from the audio buffer according to a second preset rule; and taking out a corresponding target video packet from the video buffer based on the absolute timestamp of the target audio packet.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of computer, and particularly relates to an audio and video synchronization method and device, electronic equipment, medium and program product. BACKGROUND

[0002] With the development of life, the demand for real-time audio and video such as video chat and video conference is booming. In order to achieve faster transmission speed, udp is generally used. However, since udp transmission is unreliable, audio and video data packets are easy to be lost and out of order, so when using udp, the logic of packet retransmission and reordering is increased.

[0003] At present, there are mainly two kinds of audio and video synchronization schemes for real-time audio and video: one is to extract the neteq module of webrtc, ensure that the audio is output at a stable frame rate, and then align the video to the audio to realize real-time audio and video synchronization; the other is to expand the rtc protocol header field and add the ntp timestamp, and use the absolute timestamp alignment at the receiving end to realize real-time audio and video synchronization.

[0004] Therefore, there is an urgent need for an audio and video synchronization scheme which is more simple and has better compatibility. SUMMARY

[0005] The embodiments of the present specification provide an audio and video synchronization method, device, electronic equipment, medium and program product, which achieves the purpose of audio and video synchronization through audio buffer and video buffer and aligning the video to the audio, so that the audio and video synchronization is simpler and has better compatibility. The above technical scheme is as follows:

[0006] In a first aspect, the embodiments of the present specification provide an audio and video synchronization method, comprising:

[0007] Obtaining audio and video data; the audio and video data comprises audio packets and video packets;

[0008] Updating the display timestamps corresponding to the audio packets and the video packets respectively by using absolute timestamps;

[0009] According to a first preset rule, inserting the audio packets into an audio buffer and inserting the video packets into a video buffer;

[0010] According to a second preset rule, taking out target audio packets from the audio buffer;

[0011] Based on the absolute timestamp of the target audio packets, taking out corresponding target video packets from the video buffer.

[0012] In a possible implementation, before the display timestamps of the audio packet and the video packet are updated by using the absolute timestamps, the method further includes:

[0013] obtaining an SR packet corresponding to the audio-video data;

[0014] The display timestamps of the audio packet and the video packet are updated by using the absolute timestamps, including:

[0015] The display timestamps of the audio packet and the video packet are updated by using the absolute timestamps based on the SR packet.

[0016] In a possible implementation, the SR packet includes a first absolute timestamp and a first display timestamp corresponding to the audio, and a second absolute timestamp and a second display timestamp corresponding to the video;

[0017] The display timestamps of the audio packet and the video packet are updated by using the absolute timestamps based on the SR packet, including:

[0018] A first correspondence relationship between the absolute timestamp corresponding to the audio packet and the display timestamp corresponding to the audio packet is determined based on the first absolute timestamp and the first display timestamp corresponding to the audio, and a second correspondence relationship between the absolute timestamp corresponding to the video packet and the display timestamp corresponding to the video packet is determined based on the second absolute timestamp and the second display timestamp corresponding to the video;

[0019] The display timestamp corresponding to the audio packet is updated by using the absolute timestamp corresponding to the audio packet based on the first correspondence relationship, and the display timestamp corresponding to the video packet is updated by using the absolute timestamp corresponding to the video packet based on the second correspondence relationship.

[0020] In a possible implementation, the audio packet is inserted into the audio buffer and the video packet is inserted into the video buffer according to a first preset rule, including:

[0021] In a case where the display timestamp of the data packet is greater than the display timestamp of a tail packet in the buffer corresponding to the data packet, and there is no gap between the display timestamp of the data packet and the display timestamp of the tail packet, the data packet is inserted at the tail of the buffer corresponding to the data packet;

[0022] wherein the data packet is the audio packet, and the buffer corresponding to the data packet is the audio buffer; or

[0023] The data packet is the video packet, and the buffer corresponding to the data packet is the video buffer.

[0024] In a possible implementation, before the data packet is inserted into the buffer corresponding to the data packet, the method further includes:

[0025] determining whether the display timestamp of the data packet is less than the display timestamp of the tail packet in the buffer corresponding to the data packet.

[0026] If yes, the data packet is deleted.

[0027] In a possible implementation, after the display timestamp of the data packet is determined, the method further includes:

[0028] If no, it is determined whether there is a gap between the display timestamp of the data packet and the display timestamp of the tail packet in the buffer corresponding to the data packet.

[0029] If yes, a target data packet is filled in the buffer corresponding to the data packet based on the gap.

[0030] In a possible implementation, when the data packet is an audio packet, the target data packet is a mute packet.

[0031] When the data packet is a video packet, the target data packet is the tail packet in the video buffer.

[0032] In a possible implementation, after it is determined whether there is a gap between the display timestamp of the data packet and the display timestamp of the tail packet in the buffer corresponding to the data packet, the method further includes:

[0033] If no, a timeout packet in the buffer corresponding to the data packet is removed; the timeout packet is a packet in the buffer corresponding to the data packet, and a target time interval of the packet from the tail packet is greater than a preset buffering duration of the buffer corresponding to the data packet.

[0034] In a possible implementation, before the audio packet is inserted into the audio buffer and the video packet is inserted into the video buffer according to the first preset rule, the method further includes:

[0035] an invalid packet corresponding to a preset buffering duration of the audio buffer is inserted into the audio buffer.

[0036] In a possible implementation, the target audio packet is taken out of the audio buffer according to the second preset rule, including:

[0037] the target audio packet is taken out of the audio buffer at equal intervals based on a duration of the audio packet in the audio buffer.

[0038] In a possible implementation, the taking out of the target audio packet from the audio buffer at the equal interval timing based on the time length of the audio packet in the audio buffer comprises:

[0039] judging whether the taking packet time is reached at the equal interval timing;

[0040] If yes, judging whether the first packet at the head of the buffer queue in the audio buffer is valid in the case that the buffer queue in the audio buffer is not empty;

[0041] If yes, judging whether the last taken packet in the audio buffer is valid;

[0042] If no, taking out the first packet at the head of the buffer queue in the audio buffer as the target audio packet in the case that the buffer queue is full.

[0043] In a possible implementation, after the judging whether the taking packet time is reached at the equal interval timing, the method further comprises:

[0044] filling the buffer queue in the audio buffer with the silence packet in the case that the buffer queue in the audio buffer is empty.

[0045] In a possible implementation, after the judging whether the first packet at the head of the buffer queue in the audio buffer is valid, the method further comprises:

[0046] If no, taking out the first packet at the head of the buffer queue in the audio buffer.

[0047] In a possible implementation, after the judging whether the last taken packet in the audio buffer is valid, the method further comprises:

[0048] If no, filling the buffer queue with the silence packet in the case that the buffer queue is not full.

[0049] In a possible implementation, after the judging whether the last taken packet in the audio buffer is valid, the method further comprises:

[0050] If yes, judging whether the first packet at the head of the buffer queue in the audio buffer is continuous with the last taken packet in the audio buffer;

[0051] If yes, taking out the first packet at the head of the buffer queue in the audio buffer as the target audio packet;

[0052] If no, filling the first packet at the head of the buffer queue in the audio buffer with the silence packet of the corresponding time length.

[0053] In a possible implementation, the taking out of the corresponding target video packet from the video buffer based on the absolute timestamp of the target audio packet comprises:

[0054] determining whether a difference between the absolute timestamp corresponding to the first packet in the video buffer and the absolute timestamp corresponding to the target audio packet is less than a threshold value;

[0055] if yes, taking out the first packet in the video buffer as the target video packet from the video buffer.

[0056] In a second aspect, an audio-video synchronization apparatus is provided, comprising:

[0057] a first obtaining module configured to obtain audio-video data, wherein the audio-video data comprises audio packets and video packets;

[0058] an updating module configured to update display timestamps corresponding to the audio packets and the video packets respectively by using absolute timestamps;

[0059] a first inserting module configured to insert the audio packets into an audio buffer and insert the video packets into a video buffer according to a first preset rule;

[0060] a first packet taking-out module configured to take out a target audio packet from the audio buffer according to a second preset rule;

[0061] a second packet taking-out module configured to take out a corresponding target video packet from the video buffer based on an absolute timestamp of the target audio packet.

[0062] In a possible implementation, the audio-video synchronization apparatus further comprises:

[0063] a second obtaining module configured to obtain an SR packet corresponding to the audio-video data;

[0064] the updating module is specifically configured to:

[0065] update the display timestamps corresponding to the audio packets and the video packets respectively by using absolute timestamps based on the SR packet.

[0066] In a possible implementation, the SR packet comprises a first absolute timestamp and a first display timestamp corresponding to the audio, and a second absolute timestamp and a second display timestamp corresponding to the video;

[0067] the updating module comprises:

[0068] The determining unit is configured to determine a first correspondence relationship between an absolute timestamp corresponding to the audio packet and a display timestamp corresponding to the audio packet based on a first absolute timestamp corresponding to the audio and a first display timestamp, and determine a second correspondence relationship between an absolute timestamp corresponding to the video packet and a display timestamp corresponding to the video packet based on a second absolute timestamp corresponding to the video and a second display timestamp;

[0069] The updating unit is configured to update the display timestamp corresponding to the audio packet based on the absolute timestamp corresponding to the audio packet according to the first correspondence relationship, and update the display timestamp corresponding to the video packet based on the absolute timestamp corresponding to the video packet according to the second correspondence relationship.

[0070] In a possible implementation, the inserting module is specifically configured to:

[0071] In a case where the display timestamp of the data packet is greater than a display timestamp of a tail packet in a buffer corresponding to the data packet, and there is no hole between the display timestamp of the data packet and the display timestamp of the tail packet, the data packet is inserted at the tail of the buffer corresponding to the data packet.

[0072] In a possible implementation, the data packet is the audio packet, and the buffer corresponding to the data packet is an audio buffer; or

[0073] The data packet is the video packet, and the buffer corresponding to the data packet is a video buffer.

[0074] In a possible implementation, the audio-video synchronization device further includes:

[0075] The first determining module is configured to determine whether the display timestamp of the data packet is less than a display timestamp of a tail packet in a buffer corresponding to the data packet.

[0076] The first deleting module is configured to delete the data packet if the display timestamp of the data packet is less than the display timestamp of the tail packet in the buffer corresponding to the data packet.

[0077] In a possible implementation, the audio-video synchronization device further includes:

[0078] The second determining module is configured to determine whether there is a hole between the display timestamp of the data packet and a display timestamp of a tail packet in a buffer corresponding to the data packet if the display timestamp of the data packet is greater than the display timestamp of the tail packet in the buffer corresponding to the data packet.

[0079] The first filling module is configured to fill a target data packet in the buffer corresponding to the data packet based on the hole if there is a hole between the display timestamp of the data packet and the display timestamp of the tail packet in the buffer corresponding to the data packet.

[0080] In a possible implementation, when the data packet is an audio packet, the target data packet is a mute packet.

[0081] The data packet is a video packet, and the target data packet is a tail packet in the video buffer.

[0082] In a possible implementation, the audio-video synchronization device further includes:

[0083] The second deletion module is configured to remove a timeout packet in a buffer corresponding to the data packet if the answer is no; and the timeout packet is a packet in the buffer corresponding to the data packet, and a target time interval of the packet from a tail packet is greater than a preset buffer duration of the buffer corresponding to the data packet.

[0084] In a possible implementation, the audio-video synchronization device further includes:

[0085] The second insertion module is configured to insert an invalid packet corresponding to a preset buffer duration in the audio buffer into the audio buffer.

[0086] In a possible implementation, the first packet taking module is specifically configured to:

[0087] The target audio packet is taken from the audio buffer at an equal interval based on a duration of an audio packet in the audio buffer.

[0088] In a possible implementation, the first packet taking module includes:

[0089] The first judging unit is configured to judge whether a packet taking time is reached at an equal interval.

[0090] The second judging unit is configured to judge whether a first packet at a head of a buffer queue in the audio buffer is valid if the answer is yes, and the buffer queue in the audio buffer is not empty.

[0091] The third judging unit is configured to judge whether a last taken packet in the audio buffer is valid if the answer is yes.

[0092] The first packet taking unit is configured to take the first packet at the head of the buffer queue in the audio buffer as the target audio packet if the answer is no, and the buffer queue is filled.

[0093] In a possible implementation, the audio-video synchronization device further includes:

[0094] The second filling module is configured to fill the buffer queue in the audio buffer with the mute packet if the buffer queue is empty.

[0095] In a possible implementation, the audio-video synchronization device further includes:

[0096] The third packet taking module is configured to take the first packet at the head of the buffer queue in the audio buffer if the answer is no.

[0097] In a possible implementation, the audio-video synchronization device further includes:

[0098] The third filling module is configured to, if no, fill the buffer queue with the mute packets if the buffer queue is not full.

[0099] In a possible implementation, the audio-video synchronization device further includes:

[0100] The third determining module is configured to, if yes, determine whether the first packet at the head of the buffer queue in the audio buffer and the last taken packet in the audio buffer are continuous.

[0101] The fourth packet taking module is configured to, if yes, take the first packet at the head of the buffer queue in the audio buffer as the target audio packet.

[0102] The fourth filling module is configured to, if no, fill the corresponding time length of the mute packets before the first packet at the head of the buffer queue in the audio buffer.

[0103] In a possible implementation, the second packet taking module includes:

[0104] The fourth determining unit is configured to determine whether a difference between an absolute timestamp corresponding to the first packet in the video buffer and an absolute timestamp corresponding to the target audio packet is less than a threshold.

[0105] The second packet taking unit is configured to, if yes, take the first packet in the video buffer as a target video packet from the video buffer.

[0106] In a third aspect, an electronic device is provided, including a processor and a memory.

[0107] The processor is connected to the memory.

[0108] The memory is configured to store executable program codes.

[0109] The processor runs a program corresponding to the executable program codes by reading the executable program codes stored in the memory, to execute the method provided in the first aspect or any possible implementation of the first aspect.

[0110] In a fourth aspect, a computer storage medium is provided, which stores a plurality of instructions. The instructions are adapted to be loaded by a processor and executed to perform the method provided in the first aspect or any possible implementation of the first aspect.

[0111] In a fifth aspect, an embodiment of the present specification provides a computer program product containing instructions, which, when the computer program product is executed on a computer or a processor, causes the computer or the processor to perform the audio-video synchronization method provided by the first aspect of the present specification or any possible implementation manner of the first aspect.

[0112] After the audio-video data including the audio packets and the video packets are acquired, the display timestamps corresponding to the audio packets and the video packets are updated by using the absolute timestamps, then the audio packets are inserted into the audio buffer and the video packets are inserted into the video buffer according to the first preset rule, the target audio packet is taken out from the audio buffer according to the second preset rule, and the corresponding target video packet is taken out from the video buffer based on the absolute timestamp of the target audio packet, so that the purpose of audio-video synchronization is achieved by the audio buffer and the video buffer and the alignment of the video to the audio, and the audio-video synchronization is simpler and has better compatibility. BRIEF DESCRIPTION OF DRAWINGS

[0113] In order to more clearly illustrate the technical solutions in the embodiments of the present specification, the drawings required to be used in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present specification, and other drawings can also be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0114] Figure 1 An architecture schematic diagram of an audio-video synchronization system provided by an exemplary embodiment of the present specification;

[0115] Figure 2 A flowchart schematic diagram of an audio-video synchronization method provided by an exemplary embodiment of the present specification;

[0116] Figure 3 An implementation process schematic diagram of updating the display timestamps corresponding to the audio packets and the video packets provided by an exemplary embodiment of the present specification;

[0117] Figure 4 A specific implementation flowchart schematic diagram of inserting the audio packets into the audio buffer and inserting the video packets into the video buffer provided by an exemplary embodiment of the present specification;

[0118] Figure 5 A specific implementation flowchart schematic diagram of taking out the target audio packet from the audio buffer provided by an exemplary embodiment of the present specification;

[0119] Figure 6 An implementation flowchart schematic diagram of taking out the corresponding target video packet from the video buffer based on the absolute timestamp of the target audio packet provided by an exemplary embodiment of the present specification;

[0120] Figure 7 FIG. 1 is a structural schematic diagram of an audio-video synchronization device according to an example embodiment of the present specification.

[0121] Figure 8 FIG. 2 is a structural schematic diagram of an electronic device according to an example embodiment of the present specification. DETAILED DESCRIPTION

[0122] The technical solutions in the example embodiments of the present specification will be described clearly and completely below with reference to the drawings in the example embodiments of the present specification.

[0123] The terms "first", "second", "third", and the like in the present specification and claims and the above drawings are used to distinguish different objects, and are not used to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device including a series of steps or units is not limited to the listed steps or units, but can optionally further include steps or units not listed or can optionally further include other steps or units inherent to the process, method, product, or device.

[0124] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.), and signals involved in the example embodiments of the present specification are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data need to comply with relevant laws, regulations, and standards in relevant countries and regions. For example, the audio-video data, absolute timestamp, etc. involved in the present specification are obtained under sufficient authorization.

[0125] Next, please refer to Figure 1 which is a structural schematic diagram of an audio-video synchronization system according to an example embodiment of the present specification. As shown in Figure 1 the audio-video synchronization system includes a sending end 110 and a receiving end 120. Among them:

[0126] The sending end 110 can include one or more terminals corresponding to users. A user version of software can be installed in the sending end 110 to realize functions such as collecting audio-video data. The sending end 110 can establish a data relationship with a network and establish a data connection relationship with the receiving end 120 through the network, such as sending collected audio-video data to the receiving end 120. Any sending end 110 can be but is not limited to a mobile phone, tablet computer, notebook computer, etc. device installed with user version software.

[0127] The receiving end 120 can be a server capable of providing various audio-video synchronization, and can receive audio-video data including audio packets and video packets from the sending end 110 through a network. After obtaining the audio data, the receiving end 120 can first update the display timestamps of the audio packets and the video packets in the audio data corresponding to the audio packets and the video packets respectively using the absolute timestamps, and then insert the audio packets into an audio buffer and insert the video packets into a video buffer according to a first preset rule, and take out a target audio packet from the audio buffer and a target video packet corresponding to the target audio packet from the video buffer based on the absolute timestamp of the target audio packet according to a second preset rule. It can be understood that the receiving end 120 can be, but is not limited to, a hardware server, a virtual server, a cloud server, etc. The above receiving end 120 is not limited to a server, but can also be any terminal, and the embodiments of the present specification do not make specific limitations thereon.

[0128] Optionally, after taking out the target audio packet and the target video packet corresponding to the target audio packet, the receiving end 120 can further render and play the target audio packet and the target video packet corresponding to the target audio packet, or send the target audio packet and the target video packet corresponding to the target audio packet to other devices for rendering and playing.

[0129] The network can be a medium providing a communication link between the receiving end 120 and any one of the sending ends 110, or can be the Internet including network devices and transmission media, without being limited thereto. The transmission medium can be a wired link such as, but not limited to, a coaxial cable, an optical fiber, and a digital subscriber line (DSL), etc., or a wireless link such as, but not limited to, wireless fidelity (WIFI), Bluetooth, and a mobile device network, etc.

[0130] It can be understood that Figure 1 It can be understood that the number of the sending ends 110 and the receiving ends 120 in the audio-video synchronization system shown is only an example, and in a specific implementation, the audio-video synchronization system can include any number of sending ends and receiving ends, and the embodiments of the present specification do not make specific limitations thereon. For example, but not limited to, the sending end 110 can be a sending end cluster composed of multiple sending ends, and the receiving end 120 can be a receiving end cluster composed of multiple receiving ends.

[0131] Next, the audio-video synchronization method provided by the embodiments of the present specification will be introduced in combination with Figure 1 For example, taking the execution of the audio-video synchronization by the receiving end 120 as an example, the audio-video synchronization method provided by the embodiments of the present specification will be introduced. For details, please refer to Figure 2 , which is a flowchart of an audio-video synchronization method provided by an example embodiment of the present specification. As shown in Figure 2 , the audio-video synchronization method includes the following steps:

[0132] S202, acquire audio and video data, the audio and video data comprising audio packets and video packets.

[0133] Specifically, in various real-time streaming scenarios such as online video, video conference, and live broadcast, the sending end can send the audio and video data to the receiving end through the network after collecting the audio and video data. The receiving end needs to synchronize the audio and video data before ensuring the smoothness and display effect of subsequent audio and video.

[0134] S204, update the display timestamps corresponding to the audio packets and the video packets respectively by using the absolute timestamps.

[0135] Specifically, the absolute (NTP) timestamp can be the network time corresponding to the time when the sending end collects the audio and video data, which is the number of seconds since January 1, 1900 00:00:00. The display timestamp can be the play time corresponding to the audio and video data. In order to ensure the accuracy of the display timestamps corresponding to the audio packets and the video packets respectively, the absolute timestamps can be used to update the display timestamps corresponding to the audio packets and the video packets respectively after acquiring the audio and video data, so as to correct the errors of the display timestamps corresponding to the audio packets and the video packets respectively acquired by the sending end. The display timestamp can also be the sending time or the collection time corresponding to the audio and video data, which is not limited in the embodiments of the present application. Before updating the display timestamps corresponding to the audio packets and the video packets respectively by using the absolute timestamps, the receiving end can also acquire the SR packets corresponding to the audio and video data. When updating the display timestamps corresponding to the audio packets and the video packets respectively by using the absolute timestamps, the absolute timestamps can be used to update the display timestamps corresponding to the audio packets and the video packets respectively based on the SR packets.

[0136] Further, the SR packet comprises a first absolute timestamp and a first display timestamp corresponding to the audio, and a second absolute timestamp and a second display timestamp corresponding to the video. The implementation process of updating the display timestamps corresponding to the audio packets and the video packets respectively by using the absolute timestamps based on the SR packet is specifically shown in Figure 3 The first corresponding relationship between the absolute timestamp corresponding to the audio packet and the display timestamp corresponding to the audio packet can be determined based on the first absolute timestamp and the first display timestamp corresponding to the audio, and the second corresponding relationship between the absolute timestamp corresponding to the video packet and the display timestamp corresponding to the video packet can be determined based on the second absolute timestamp and the second display timestamp corresponding to the video. Then, the display timestamp corresponding to the audio packet can be updated by using the absolute timestamp corresponding to the audio packet based on the first corresponding relationship, and the display timestamp corresponding to the video packet can be updated by using the absolute timestamp corresponding to the video packet based on the second corresponding relationship.

[0137] S206, inserting the audio packet into the audio buffer and inserting the video packet into the video buffer according to a first preset rule.

[0138] Specifically, the receiving end creates an audio buffer and a video buffer. After updating the respective display timestamps of the audio packet and the video packet by using the absolute timestamp, the audio packet is inserted into the audio buffer and the video packet is inserted into the video buffer according to a first preset rule based on the respective updated display timestamps of the audio packet and the video packet.

[0139] Specifically, as shown in Figure 4 the implementation process of S206 can include:

[0140] S402, determining whether the display timestamp of the data packet is less than the display timestamp of the tail packet in the buffer corresponding to the data packet.

[0141] Specifically, after updating the respective display timestamps of the audio packet and the video packet by using the absolute timestamp, in order to ensure the continuity of the display timestamps of the data packets cached in the buffer corresponding to the data packet, it is first determined whether the display timestamp of the data packet is less than the display timestamp of the tail packet in the buffer corresponding to the data packet. When the data packet is an audio packet, the buffer corresponding to the data packet is an audio buffer. When the data packet is a video packet, the buffer corresponding to the data packet is a video buffer.

[0142] S404, if yes, deleting the data packet.

[0143] Specifically, if the display timestamp of the data packet currently acquired by the receiving end is less than the display timestamp of the tail packet in the buffer corresponding to the data packet in the receiving end, it can be considered that the data packet currently acquired is a packet that needs to be displayed before the tail packet in the buffer corresponding to the data packet, and the data packet currently acquired is already expired, which can be directly deleted.

[0144] S406, if no, determining whether there is a gap between the display timestamp of the data packet and the display timestamp of the tail packet in the buffer corresponding to the data packet.

[0145] Specifically, after determining whether the display timestamp of the data packet is less than the display timestamp of the tail packet in the buffer corresponding to the data packet at S402, if the display timestamp of the data packet currently acquired by the receiving end is not less than the display timestamp of the tail packet in the buffer corresponding to the data packet in the receiving end, in order to ensure the continuity of the display timestamps of the data packets cached in the buffer corresponding to the data packet, it is further determined whether there is a gap between the display timestamp of the data packet and the display timestamp of the tail packet in the buffer corresponding to the data packet, i.e., whether the display timestamp of the data packet currently acquired by the receiving end is continuous with the display timestamp of the tail packet in the buffer corresponding to the data packet.

[0146] At S408, if yes, the corresponding target data packet is filled in the buffer corresponding to the data packet based on the gap.

[0147] Specifically, if there is a gap between the display timestamp of the data packet and the display timestamp of the tail packet in the buffer corresponding to the data packet, it is considered that the display timestamp of the data packet currently acquired by the receiving end is not continuous with the display timestamp of the tail packet in the buffer corresponding to the data packet, and then the target data packet of the corresponding duration can be filled according to the time interval between the display timestamp of the data packet currently acquired and the display timestamp of the tail packet in the buffer corresponding to the data packet, so as to ensure the continuity of the display timestamps of the data packets in the buffer (video buffer and audio buffer).

[0148] Further, when the data packet is an audio packet, the target data packet is a mute packet, i.e., when the audio packet is inserted into the audio buffer, if there is a gap between the display timestamp of the audio packet and the display timestamp of the tail packet in the audio buffer, the target data packet of the corresponding duration can be filled according to the time interval between the display timestamp of the audio packet currently acquired and the display timestamp of the tail packet in the audio buffer, so as to ensure the continuity of the display timestamps of the audio packets cached in the audio buffer, and facilitate subsequent alignment of the video packet to the audio packet to achieve audio-video synchronization. When the data packet is a video packet, the target data packet is the tail packet in the video buffer, i.e., when the video packet is inserted into the video buffer, if there is a gap between the display timestamp of the video packet and the display timestamp of the tail packet in the video buffer, the target data packet of the corresponding duration can be filled according to the time interval between the display timestamp of the video packet currently acquired and the display timestamp of the tail packet in the video buffer, so as to ensure the continuity of the display timestamps of the video packets cached in the video buffer, and facilitate subsequent alignment of the video packet to the audio packet to achieve audio-video synchronization.

[0149] At S406, if no, after determining whether there is a gap between the display timestamp of the data packet and the display timestamp of the tail packet in the buffer corresponding to the data packet, it further includes:

[0150] S410, if not, remove the timeout packet from the buffer corresponding to the data packet.

[0151] Specifically, if there is no gap between the display timestamp of a data packet and the display timestamp of the tail packet in the corresponding buffer, a timeout packet in the buffer corresponding to the data packet will be removed. The aforementioned timeout packet is a packet in the buffer corresponding to the data packet whose target time interval with the tail packet is greater than the preset buffer duration of the buffer corresponding to the data packet. The aforementioned target time interval is the time interval between the display timestamps of each data packet in the buffer corresponding to the data packet and the display timestamp of the tail packet in that buffer. The aforementioned preset buffer duration can be, but is not limited to, 3 seconds, 2 seconds, etc.

[0152] Understandably, since real-time streaming transmission does not need to retain audio and video data that is too old, when encountering abnormal situations such as network lag causing a long period of no audio and video data to be obtained, and then suddenly obtaining a lot of audio and video data, expired timeout packets cached in the video buffer and audio buffer can be directly deleted to avoid invalid display problems where old audio or video is still being displayed.

[0153] Please continue to refer to the following. Figure 4 ,like Figure 4 As shown, in S406, if not, after determining whether there is a gap between the display timestamp of the data packet and the display timestamp of the tail packet in the corresponding buffer, the following steps are also included:

[0154] S412, if not, insert the data packet into the tail of the buffer corresponding to the data packet.

[0155] Specifically, if the display timestamp of a data packet is greater than the display timestamp of the tail packet in the corresponding buffer, and there is no gap between the display timestamp of the data packet and the tail packet, then the display timestamp of the data packet currently acquired by the receiving end can be considered continuous with the display timestamp of the tail packet in the corresponding buffer. In this case, the data packet can be directly inserted into the tail of the buffer corresponding to that type of data packet. Specifically, when the data packet is an audio packet, the corresponding buffer is an audio buffer; when the data packet is a video packet, the corresponding buffer is a video buffer.

[0156] Optionally, in order to ensure that each audio and video data obtained from the network can stay in the buffer for a period of time before being retrieved, and to ensure that the audio packets in the audio buffer and the video packets in the video buffer can be synchronized, invalid packets corresponding to the preset buffering time of the audio buffer can be inserted into the audio buffer before inserting the audio packets into the audio buffer according to the first preset rule.

[0157] Please continue to refer to the following. Figure 2 ,likeFigure 2 As shown, the audio-video synchronization method further includes:

[0158] S208, taking out a target audio packet from the audio buffer according to a second preset rule.

[0159] Specifically, since each audio packet includes one frame of audio, and the time length corresponding to each frame of audio is a fixed value, the taking out of the target audio packet from the audio buffer according to the second preset rule can be based on the time length of the audio packet in the audio buffer, i.e., setting a timer according to the time length of the audio frame to take out the target audio packet from the audio buffer at equal intervals. The time interval for taking out the target audio packet from the audio buffer can be the time length of each frame of audio or the time length of the target audio packet. In order to avoid the situation that the audio cannot be constantly taken out due to sudden network jam, the time interval for taking out the target audio packet from the audio buffer can also be less than the time length of the audio frame.

[0160] It can be understood that in the real-time streaming media scenario, S208 and S206 can be executed synchronously, i.e., the receiving end takes out the target audio packet from the audio buffer at the same time as it inserts the currently acquired audio-video packet and video packet into the corresponding buffer, which is not specifically limited in the embodiments of the present specification.

[0161] Specifically, as shown in the above S208, the specific implementation process of taking out the target audio packet from the audio buffer according to the second preset rule can include: Figure 5

[0162] S502, determining whether the packet taking time is reached at equal intervals.

[0163] Specifically, a timer can be set according to the time length of the audio frame to determine whether the packet taking time is reached at equal intervals.

[0164] For example, if the time length of the audio frame is 20 ms, in order to ensure the continuity of the audio, the packet taking operation can be performed from the audio buffer every 20 ms, and it can be determined every 20 ms whether the time from the last packet taking operation reaches 20 ms. If yes, it means that the packet taking time is reached, and if no, it means that the packet taking time is not reached.

[0165] S504, if the packet taking time is reached, determining whether the buffer queue in the audio buffer is empty.

[0166] S506, in the case that the buffer queue in the audio buffer is empty, filling the buffer queue with silence packets.

[0167] ​Specifically, if the packet taking time is reached, in order to ensure constant output of the audio, in the case that the buffer queue in the audio buffer is empty, for example, in the case that the sender transmits no audio frame in the mute state in the scene, since there can be transmission and playing of video packets, the buffer queue needs to be filled with mute packets to ensure that after the video is aligned to the audio, the video frame can be continuously and stably output.

[0168] S508, if the buffer queue in the audio buffer is not empty, it is determined whether the first packet at the head of the buffer queue in the audio buffer is valid.

[0169] Specifically, if the packet taking time is reached and the buffer queue in the audio buffer is not empty, it can be further determined whether the first packet at the head of the buffer queue in the audio buffer is valid.

[0170] S510, if the first packet at the head of the buffer queue in the audio buffer is invalid, the first packet at the head of the buffer queue in the audio buffer is taken out.

[0171] Specifically, if the first packet at the head of the buffer queue in the audio buffer is not the audio packet obtained by the receiving end through the network, but the invalid packet inserted in advance, it is considered that the first packet at the head of the buffer queue in the audio buffer is invalid, and the invalid packet at the head of the buffer queue in the audio buffer can be directly taken out.

[0172] S512, if the first packet at the head of the buffer queue in the audio buffer is valid, it is determined whether the last taken packet in the audio buffer is valid.

[0173] Specifically, if the first packet at the head of the buffer queue in the audio buffer is not the audio packet obtained by the receiving end through the network, it is considered that the first packet at the head of the buffer queue in the audio buffer is valid, and it is further determined whether the last taken packet in the audio buffer is valid.

[0174] S514, if the last taken packet in the audio buffer is invalid, it is determined whether the buffer queue is full.

[0175] Specifically, if the first packet at the head of the buffer queue in the audio buffer is valid and the last taken packet in the audio buffer is invalid, it is further determined whether the buffer queue is full.

[0176] S516, if the buffer queue is full, the first packet at the head of the buffer queue in the audio buffer is taken out as the target audio packet.

[0177] Specifically, if the first packet at the head of the buffer queue in the audio buffer is valid, the last taken packet in the audio buffer is invalid, and the buffer queue is full, it indicates that the first packet at the head of the buffer queue in the audio buffer is an audio packet obtained from the network, and there is no hole in the buffer queue in the audio buffer. Therefore, the first packet at the head of the buffer queue in the audio buffer can be directly taken as the target audio packet.

[0178] In S518, if the buffer queue is not full, the buffer queue is filled with silence packets.

[0179] Specifically, if the first packet at the head of the buffer queue in the audio buffer is valid, the last taken packet in the audio buffer is invalid, and the buffer queue is not full, in order to avoid the existence of a hole in the buffer queue in the audio buffer, which causes the audio frame to be taken unstably and affects the effect of audio-video alignment, the buffer queue in the audio buffer can be filled with silence packets.

[0180] Next, please continue to refer to Figure 5 In S512, if the first packet at the head of the buffer queue in the audio buffer is valid, after judging whether the last taken packet in the audio buffer is valid, the method further includes:

[0181] In S520, if the last taken packet in the audio buffer is valid, it is judged whether the first packet at the head of the buffer queue in the audio buffer and the last taken packet in the audio buffer are continuous.

[0182] In S522, if the first packet at the head of the buffer queue in the audio buffer and the last taken packet in the audio buffer are continuous, the first packet at the head of the buffer queue in the audio buffer is taken as the target audio packet.

[0183] Specifically, if the last taken packet in the audio buffer is valid, and the first packet at the head of the buffer queue in the audio buffer and the last taken packet in the audio buffer are continuous, it indicates that there is no hole between the first packet at the head of the buffer queue in the audio buffer and the last taken packet in the audio buffer. Therefore, the first packet at the head of the buffer queue in the audio buffer can be directly taken as the target audio packet.

[0184] In S524, if the first packet at the head of the buffer queue in the audio buffer and the last taken packet in the audio buffer are not continuous, silence packets corresponding to the time length are filled before the first packet at the head of the buffer queue in the audio buffer.

[0185] Specifically, if the last taken packet in the audio buffer is valid, but the first packet in the head of the buffer queue in the audio buffer is not continuous with the last taken packet in the audio buffer, it indicates that there is a hole between the first packet in the head of the buffer queue in the audio buffer and the last taken packet in the audio buffer. In order to ensure the continuity of the audio packet taken in the audio buffer, a corresponding duration of silence packet needs to be filled before the first packet in the head of the buffer queue in the audio buffer.

[0186] Next, please continue to refer to Figure 2 As shown in S208, after the target audio packet is taken out from the audio buffer according to the second preset rule, the audio-video synchronization method further includes: Figure 2 S210, taking out the corresponding target video packet from the video buffer based on the absolute timestamp of the target audio packet.

[0187] Specifically, as shown in S210, the implementation process of taking out the corresponding target video packet from the video buffer based on the absolute timestamp of the target audio packet can include:

[0188] Figure 6 S602, judging whether the difference between the absolute timestamp corresponding to the first packet in the video buffer and the absolute timestamp corresponding to the target audio packet is less than a threshold value.

[0189] Specifically, after the target audio packet is taken out from the audio buffer according to the second preset rule, it can be first judged whether the difference between the absolute timestamp corresponding to the first packet in the video buffer and the absolute timestamp corresponding to the target audio packet is less than a threshold value, that is, whether the absolute timestamp corresponding to the first packet in the video buffer is synchronized with the absolute timestamp corresponding to the target audio packet.

[0190] S604, if yes, taking out the first packet in the video buffer as the target video packet from the video buffer.

[0191] Specifically, if the difference between the absolute timestamp corresponding to the first packet in the video buffer and the absolute timestamp corresponding to the target audio packet is less than a threshold value, it can be considered that the absolute timestamp corresponding to the first packet in the video buffer is synchronized with the absolute timestamp corresponding to the target audio packet, and then the first packet in the video buffer can be taken out as the target video packet corresponding to the target audio packet from the video buffer, and the target audio packet and the target video packet corresponding to the target audio packet can be rendered and played, or sent to other terminals for rendering and playing, etc.

[0192] S606, if no, judging whether the absolute timestamp corresponding to the first packet in the video buffer is less than the absolute timestamp corresponding to the target audio packet.

[0193] S606, if no, judging whether the absolute timestamp corresponding to the first packet in the video buffer is less than the absolute timestamp corresponding to the target audio packet.

[0194] ​Specifically, if the difference between the absolute timestamp corresponding to the first packet in the video buffer and the absolute timestamp corresponding to the target audio packet is not less than the threshold value, it is considered that the absolute timestamp corresponding to the first packet in the video buffer is not synchronized with the absolute timestamp corresponding to the target audio packet, and it is further needed to determine whether the absolute timestamp corresponding to the first packet in the video buffer is less than the absolute timestamp corresponding to the target audio packet.

[0195] In S608, if the absolute timestamp corresponding to the first packet in the video buffer is less than the absolute timestamp corresponding to the target audio packet, the first packet in the video buffer is deleted.

[0196] Specifically, if the difference between the absolute timestamp corresponding to the first packet in the video buffer and the absolute timestamp corresponding to the target audio packet is not less than the threshold value, and the absolute timestamp corresponding to the first packet in the video buffer is less than the absolute timestamp corresponding to the target audio packet, it is considered that the first packet in the video buffer is an early video packet, in order to avoid invalid display of the video and ensure real-time synchronization of the video and the audio, the first packet in the video buffer needs to be deleted, and S602 is executed again to determine whether the difference between the absolute timestamp corresponding to the first packet in the video buffer and the absolute timestamp corresponding to the target audio packet is less than the threshold value.

[0197] In S610, if the absolute timestamp corresponding to the first packet in the video buffer is greater than the absolute timestamp corresponding to the target audio packet, the previous packet of the first packet in the video buffer is taken out from the video buffer as a target video packet.

[0198] Specifically, if the difference between the absolute timestamp corresponding to the first packet in the video buffer and the absolute timestamp corresponding to the target audio packet is not less than the threshold value, and the absolute timestamp corresponding to the first packet in the video buffer is greater than the absolute timestamp corresponding to the target audio packet, it is considered that the first packet in the video buffer should be a video packet corresponding to synchronization of an audio packet after the target audio packet, in order to ensure synchronization of subsequent audio and video, the previous packet of the first packet in the video buffer can be taken out from the video buffer as a target video packet, that is, the previous frame of picture is repeatedly played.

[0199] In the embodiments of the present specification, after the audio and video data including the audio packet and the video packet are acquired, the display timestamps corresponding to the audio packet and the video packet are updated by using the absolute timestamps, then the audio packet is inserted into the audio buffer and the video packet is inserted into the video buffer according to the first preset rule, the target audio packet is taken out from the audio buffer according to the second preset rule, and the corresponding target video packet is taken out from the video buffer based on the absolute timestamp of the target audio packet, so that the purpose of audio and video synchronization is achieved by the audio buffer and the video buffer and the alignment of the video to the audio, and the audio and video synchronization is simpler and has better compatibility.

[0200] Next, please refer to Figure 7Fig. 7 is a structural schematic diagram of an audio-video synchronization device according to an example embodiment of the present specification. As shown in Fig. 7, the audio-video synchronization device 700 includes: Figure 7

[0201] A first obtaining module 710 is configured to obtain audio-video data. The audio-video data includes audio packets and video packets.

[0202] An updating module 720 is configured to update display timestamps corresponding to the audio packets and the video packets respectively by using absolute timestamps.

[0203] A first inserting module 730 is configured to insert the audio packets into an audio buffer and insert the video packets into a video buffer according to a first preset rule.

[0204] A first packet obtaining module 740 is configured to obtain target audio packets from the audio buffer according to a second preset rule.

[0205] A second packet obtaining module 750 is configured to obtain target video packets corresponding to the target audio packets from the video buffer based on absolute timestamps of the target audio packets.

[0206] In a possible implementation, the audio-video synchronization device 700 further includes:

[0207] A second obtaining module is configured to obtain SR packets corresponding to the audio-video data.

[0208] The updating module 720 is specifically configured to:

[0209] update the display timestamps corresponding to the audio packets and the video packets respectively by using the absolute timestamps based on the SR packets.

[0210] In a possible implementation, the SR packets include first absolute timestamps and first display timestamps corresponding to the audio packets, and second absolute timestamps and second display timestamps corresponding to the video packets.

[0211] The updating module 720 includes:

[0212] A determining unit is configured to determine a first correspondence relationship between an absolute timestamp corresponding to the audio packets and a display timestamp corresponding to the audio packets based on the first absolute timestamps and the first display timestamps corresponding to the audio packets, and determine a second correspondence relationship between an absolute timestamp corresponding to the video packets and a display timestamp corresponding to the video packets based on the second absolute timestamps and the second display timestamps corresponding to the video packets.

[0213] ​The updating unit is configured to update the display timestamp of the audio packet based on the absolute timestamp corresponding to the audio packet according to the first correspondence relationship, and update the display timestamp of the video packet based on the absolute timestamp corresponding to the video packet according to the second correspondence relationship.

[0214] In a possible implementation, the first inserting module 730 is specifically configured to:

[0215] insert the data packet at the tail of the buffer corresponding to the data packet, in a case where the display timestamp of the data packet is greater than the display timestamp of the tail packet in the buffer corresponding to the data packet, and there is no gap between the display timestamp of the data packet and the display timestamp of the tail packet;

[0216] wherein the data packet is the audio packet, and the buffer corresponding to the data packet is the audio buffer; or

[0217] the data packet is the video packet, and the buffer corresponding to the data packet is the video buffer.

[0218] In a possible implementation, the audio-video synchronization apparatus 700 further includes:

[0219] The first judging module is configured to judge whether the display timestamp of the data packet is less than the display timestamp of the tail packet in the buffer corresponding to the data packet.

[0220] The first deleting module is configured to delete the data packet if the display timestamp of the data packet is less than the display timestamp of the tail packet in the buffer corresponding to the data packet.

[0221] In a possible implementation, the audio-video synchronization apparatus 700 further includes:

[0222] The second judging module is configured to judge whether there is a gap between the display timestamp of the data packet and the display timestamp of the tail packet in the buffer corresponding to the data packet if the display timestamp of the data packet is greater than the display timestamp of the tail packet in the buffer corresponding to the data packet.

[0223] The first filling module is configured to fill a target data packet in the buffer corresponding to the data packet based on the gap if there is a gap between the display timestamp of the data packet and the display timestamp of the tail packet in the buffer corresponding to the data packet.

[0224] In a possible implementation, when the data packet is an audio packet, the target data packet is a mute packet.

[0225] When the data packet is a video packet, the target data packet is the tail packet in the video buffer.

[0226] In a possible implementation, the audio-video synchronization apparatus 700 further includes:

[0227] The second deleting module is configured to remove a timeout packet in the buffer corresponding to the data packet if the answer is no; the timeout packet is a packet in the buffer corresponding to the data packet, and a target time interval of the packet from a tail packet is greater than a preset buffer duration of the buffer corresponding to the data packet.

[0228] In a possible implementation, the audio-video synchronization device 700 further includes:

[0229] The second inserting module is configured to insert an invalid packet corresponding to a preset buffer duration in the audio buffer into the audio buffer.

[0230] In a possible implementation, the first packet taking module 740 is specifically configured to:

[0231] The target audio packet is taken from the audio buffer at an equal interval based on a duration of an audio packet in the audio buffer.

[0232] In a possible implementation, the first packet taking module 740 includes:

[0233] The first judging unit is configured to judge whether a packet taking time is reached at an equal interval;

[0234] The second judging unit is configured to judge whether a first packet at a head of a buffer queue in the audio buffer is valid if the answer is yes and the buffer queue in the audio buffer is not empty.

[0235] The third judging unit is configured to judge whether a last taken packet in the audio buffer is valid if the answer is yes.

[0236] The first packet taking unit is configured to take the first packet at the head of the buffer queue in the audio buffer as the target audio packet if the answer is no and the buffer queue is full.

[0237] In a possible implementation, the audio-video synchronization device 700 further includes:

[0238] The second filling module is configured to fill the buffer queue in the audio buffer with the mute packet if the buffer queue is empty.

[0239] In a possible implementation, the audio-video synchronization device 700 further includes:

[0240] The third packet taking module is configured to take the first packet at the head of the buffer queue in the audio buffer if the answer is no.

[0241] In a possible implementation, the audio-video synchronization device 700 further includes:

[0242] The third filling module is configured to fill the buffer queue with the silence packet if the buffer queue is not filled.

[0243] In a possible implementation, the audio-video synchronization device 700 further includes:

[0244] The third determining module is configured to determine whether the first packet at the head of the buffer queue in the audio buffer is continuous with the last taken packet in the audio buffer if yes.

[0245] The fourth packet taking module is configured to take the first packet at the head of the buffer queue in the audio buffer as the target audio packet if yes.

[0246] The fourth filling module is configured to fill the first packet at the head of the buffer queue in the audio buffer with the silence packet of the corresponding time length if no.

[0247] In a possible implementation, the second packet taking module 750 includes:

[0248] The fourth determining unit is configured to determine whether the difference between the absolute timestamp corresponding to the first packet in the video buffer and the absolute timestamp corresponding to the target audio packet is less than a threshold.

[0249] The second packet taking unit is configured to take the first packet in the video buffer as a target video packet from the video buffer if yes.

[0250] The division of the modules in the audio-video synchronization device is merely for example, and in other embodiments, the audio-video synchronization device can be divided into different modules as needed to complete all or part of the functions of the audio-video synchronization device. The implementation of each module in the audio-video synchronization device provided in the embodiments of the present specification can be in the form of a computer program. The computer program can run on a terminal or a server. The program modules formed by the computer program can be stored in the memory of the terminal or the server. When the computer program is executed by the processor, all or part of the steps of the audio-video synchronization method described in the embodiments of the present specification are implemented.

[0251] Next, please refer to Figure 8 which is a structural schematic diagram of an electronic device provided by an example embodiment of the present specification. As Figure 8 shown, the electronic device 800 can include at least one processor 810, at least one communication bus 820, a user interface 830, at least one network interface 840, and a memory 850.

[0252] The communication bus 820 can be used to realize the connection and communication of the above-mentioned components.

[0253] The user interface 830 can include a display and a camera. Optionally, the user interface 830 can further include a standard wired interface, a wireless interface.

[0254] The network interface 840 can optionally include a Bluetooth module, a near field communication (NFC) module, a wireless fidelity (Wi-Fi) module, and the like.

[0255] The processor 810 can include one or more processing cores. The processor 810 connects various parts in the entire electronic device 800 through various interfaces and lines, executes various functions of the routing electronic device 800 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 850, and calling data stored in the memory 850. Optionally, the processor 810 can be implemented in at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 810 can be integrated with a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes operating systems, user interfaces, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 810, but can be realized by a separate chip.

[0256] The memory 850 can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory 850 includes a non-transitory computer-readable medium. The memory 850 can be used to store instructions, programs, codes, code sets, or instruction sets. The memory 850 can include a program storage area and a data storage area, where the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a retrieval function, an update function, a package taking function, etc.), instructions for implementing the various method embodiments described above, and the like; the data storage area can store data involved in the various method embodiments described above, and the like. The memory 850 can optionally be at least one storage device located away from the aforementioned processor 810. As shown in FIG. 8, the memory 850 as a computer storage medium can include an operating system, a network communication module, a user interface module, and program instructions. Figure 8

[0257] Specifically, the processor 810 can be configured to invoke the program instructions stored in the memory 850 and specifically perform the following operations:

[0258] Obtain audio and video data; the audio and video data includes audio packets and video packets.

[0259] Update the display timestamps corresponding to the audio packets and the video packets respectively using absolute timestamps.

[0260] According to a first preset rule, insert the audio packets into an audio buffer and insert the video packets into a video buffer.

[0261] According to a second preset rule, take out target audio packets from the audio buffer.

[0262] Based on the absolute timestamp of the target audio packets, take out corresponding target video packets from the video buffer.

[0263] In some possible embodiments, before the processor 810 updates the display timestamps corresponding to the audio packets and the video packets respectively using absolute timestamps, the processor 810 is further configured to perform the following operations:

[0264] Obtain an SR packet corresponding to the audio and video data.

[0265] When the processor 810 updates the display timestamps corresponding to the audio packets and the video packets respectively using absolute timestamps, the processor 810 is specifically configured to perform the following operations:

[0266] Update the display timestamps corresponding to the audio packets and the video packets respectively using absolute timestamps based on the SR packet.

[0267] ​In some possible embodiments, the SR packet includes a first absolute timestamp and a first display timestamp corresponding to the audio, and a second absolute timestamp and a second display timestamp corresponding to the video.

[0268] When the processor 810 updates the display timestamps corresponding to the audio packet and the video packet respectively based on the absolute timestamps in the SR packet, the processor 810 is specifically configured to perform the following operations:

[0269] determining a first correspondence relationship between the absolute timestamp corresponding to the audio packet and the display timestamp corresponding to the audio packet based on the first absolute timestamp and the first display timestamp corresponding to the audio, and determining a second correspondence relationship between the absolute timestamp corresponding to the video packet and the display timestamp corresponding to the video packet based on the second absolute timestamp and the second display timestamp corresponding to the video.

[0270] updating the display timestamp corresponding to the audio packet based on the absolute timestamp corresponding to the audio packet based on the first correspondence relationship, and updating the display timestamp corresponding to the video packet based on the absolute timestamp corresponding to the video packet based on the second correspondence relationship.

[0271] In some possible embodiments, when the processor 810 inserts the audio packet into the audio buffer and inserts the video packet into the video buffer according to a first preset rule, the processor 810 is specifically configured to perform the following operations:

[0272] inserting the data packet at the tail of the buffer corresponding to the data packet, in a case where the display timestamp of the data packet is greater than the display timestamp of a tail packet in the buffer corresponding to the data packet, and there is no gap between the display timestamp of the data packet and the display timestamp of the tail packet.

[0273] wherein the data packet is the audio packet, and the buffer corresponding to the data packet is the audio buffer; or

[0274] the data packet is the video packet, and the buffer corresponding to the data packet is the video buffer.

[0275] In some possible embodiments, before the processor 810 inserts the data packet into the buffer corresponding to the data packet, the processor 810 is further configured to perform the following operation:

[0276] determining whether the display timestamp of the data packet is less than the display timestamp of a tail packet in the buffer corresponding to the data packet.

[0277] If yes, the data packet is deleted.

[0278] In some possible embodiments, after determining whether the display timestamp of the data packet is less than the display timestamp of the tail packet in the buffer corresponding to the data packet, the processor 810 is further configured to:

[0279] If not, it is determined whether there is a gap between the display timestamp of the data packet and the display timestamp of the tail packet in the buffer corresponding to the data packet.

[0280] If yes, the corresponding target data packet is filled in the buffer corresponding to the data packet based on the gap.

[0281] In some possible embodiments, when the data packet is an audio packet, the target data packet is a mute packet.

[0282] When the data packet is a video packet, the target data packet is a tail packet in the video buffer.

[0283] In some possible embodiments, after determining whether there is a gap between the display timestamp of the data packet and the display timestamp of the tail packet in the buffer corresponding to the data packet, the processor 810 is further configured to:

[0284] If not, a timeout packet in the buffer corresponding to the data packet is removed; the timeout packet is a packet in the buffer corresponding to the data packet, and a target time interval of the packet from the tail packet is greater than a preset buffering duration of the buffer corresponding to the data packet.

[0285] In some possible embodiments, before inserting the audio packet into the audio buffer and inserting the video packet into the video buffer according to the first preset rule, the processor 810 is further configured to:

[0286] An invalid packet corresponding to a preset buffering duration of the audio buffer is inserted into the audio buffer.

[0287] In some possible embodiments, when the processor 810 takes out a target audio packet from the audio buffer according to the second preset rule, the processor 810 is specifically configured to:

[0288] The target audio packet is taken out from the audio buffer at equal intervals based on the duration of the audio packet in the audio buffer.

[0289] In some possible embodiments, when the processor 810 takes out a target audio packet from the audio buffer based on the duration of the audio packet in the audio buffer, the processor 810 is specifically configured to:

[0290] It is determined whether a packet taking time is reached at equal intervals.

[0291] If yes, it is judged whether the first packet at the head of the buffer queue in the audio buffer is valid in the case that the buffer queue in the audio buffer is not empty.

[0292] If yes, it is judged whether the last taken packet in the audio buffer is valid.

[0293] If no, the first packet at the head of the buffer queue in the audio buffer is taken as the target audio packet in the case that the buffer queue is full.

[0294] In some possible embodiments, after the processor 810 judges whether the packet taking time is reached according to the equal-interval timing, the processor 810 is further configured to perform:

[0295] In the case that the buffer queue in the audio buffer is empty, the buffer queue is filled with the silence packet.

[0296] In some possible embodiments, after the processor 810 judges whether the first packet at the head of the buffer queue in the audio buffer is valid, the processor 810 is further configured to perform:

[0297] If no, the first packet at the head of the buffer queue in the audio buffer is taken.

[0298] In some possible embodiments, after the processor 810 judges whether the last taken packet in the audio buffer is valid, the processor 810 is further configured to perform:

[0299] If no, the buffer queue is filled with the silence packet in the case that the buffer queue is not full.

[0300] In some possible embodiments, after the processor 810 judges whether the last taken packet in the audio buffer is valid, the processor 810 is further configured to perform:

[0301] If yes, it is judged whether the first packet at the head of the buffer queue in the audio buffer is continuous with the last taken packet in the audio buffer.

[0302] If yes, the first packet at the head of the buffer queue in the audio buffer is taken as the target audio packet.

[0303] If no, the silence packet corresponding to the time length is filled before the first packet at the head of the buffer queue in the audio buffer.

[0304] In some possible embodiments, when the processor 810 takes the target video packet corresponding to the target audio packet from the video buffer according to the absolute timestamp of the target audio packet, the processor 810 is specifically configured to perform:

[0305] determine whether a difference between an absolute timestamp corresponding to the first packet in the video buffer and an absolute timestamp corresponding to the target audio packet is less than a threshold value.

[0306] If yes, the first packet in the video buffer is taken out from the video buffer as a target video packet.

[0307] The embodiments of the present specification further provide a computer-readable storage medium, which stores instructions, when the instructions are executed on a computer or a processor, causing the computer or the processor to perform one or more steps in the above embodiments. The constituent modules of the above audio and video synchronization device, if realized in the form of software function units and sold or used as independent products, can be stored in the above computer-readable storage medium.

[0308] In the above embodiments, all or part of them can be realized by software, hardware, firmware, or any combination thereof. When realized by software, all or part of them can be realized in the form of a computer program product. The above computer program product includes one or more computer instructions. When the above computer program instructions are loaded and executed on a computer, all or part of the above processes or functions according to the embodiments of the present specification are generated. The above computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above computer instructions can be stored in a computer-readable storage medium or transmitted by the above computer-readable storage medium. The above computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The above computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The above available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital versatile disc (DVD)), or a semiconductor medium (for example, a solid state disk (SSD)) and the like.

[0309] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by a computer program instructing relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, the processes of the above-mentioned embodiments of the methods can be included. The storage medium mentioned above includes ROM, RAM, magnetic or optical disks and other various storage media that can store program codes. In the case of no conflict, the technical features in the embodiments and the implementation forms can be combined arbitrarily.

[0310] The above-described embodiments are merely preferred embodiments of the present specification and do not limit the scope of the present specification. Without departing from the design spirit of the present specification, various modifications and improvements of the technical solutions of the present specification made by those skilled in the art shall fall within the protection scope determined by the claims.

[0311] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims and the specification can be executed in an order different from that in the embodiments recorded in the specification and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

Claims

1. An audio-video synchronization method, the method comprising: Acquire audio and video data; the audio and video data includes audio packets and video packets; Update the display timestamps corresponding to the audio and video packets using absolute timestamps; According to the first preset rule, the audio packet is inserted into the audio buffer and the video packet is inserted into the video buffer; The target audio packet is retrieved from the audio buffer according to the second preset rule; The corresponding target video packet is retrieved from the video buffer based on the absolute timestamp of the target audio packet; The step of inserting the audio packet into the audio buffer and the video packet into the video buffer according to the first preset rule includes: If the display timestamp of the data packet is greater than or equal to the display timestamp of the tail packet in the buffer corresponding to the data packet, and there is no gap between the display timestamp of the data packet and the display timestamp of the tail packet, the data packet is inserted into the tail of the buffer corresponding to the data packet; wherein the data packet is the audio packet and the buffer corresponding to the data packet is the audio buffer; or, the data packet is the video packet and the buffer corresponding to the data packet is the video buffer. If the display timestamp of the data packet is less than the display timestamp of the tail packet in the buffer corresponding to the data packet, delete the data packet; If the display timestamp of the data packet is greater than or equal to the display timestamp of the tail packet in the buffer corresponding to the data packet, and there is a gap between the display timestamp of the data packet and the display timestamp of the tail packet, the corresponding target data packet is filled into the buffer corresponding to the data packet based on the gap; The step of retrieving the target audio packet from the audio buffer according to the second preset rule includes: If the packet retrieval time has been determined at equal intervals, and the buffer queue in the audio buffer is not empty, the first packet at the head of the buffer queue in the audio buffer is valid, the last packet retrieved in the audio buffer is invalid, and the buffer queue is full, then the first packet at the head of the buffer queue in the audio buffer will be retrieved as the target audio packet. If the packet retrieval time has been determined at equal intervals, and the buffer queue in the audio buffer is not empty, the first packet at the head of the buffer queue in the audio buffer is valid, the last retrieved packet in the audio buffer is invalid, and the buffer queue is not full, then the buffer queue is filled with a mute packet. If the packet retrieval time has been determined at equal intervals, and the buffer queue in the audio buffer is not empty, and the first packet at the head of the buffer queue in the audio buffer is invalid, then the first packet at the head of the buffer queue in the audio buffer is retrieved. If the packet retrieval time has been determined at equal intervals, and the buffer queue in the audio buffer is not empty, the first packet at the head of the buffer queue in the audio buffer is valid, and the first packet at the head of the buffer queue in the audio buffer is consecutive to the previously retrieved packet in the audio buffer, then the first packet at the head of the buffer queue in the audio buffer will be retrieved as the target audio packet. If the packet retrieval time has been reached at equal intervals, then the buffer queue in the audio buffer is not empty, the first packet at the head of the buffer queue in the audio buffer is valid, and the first packet at the head of the buffer queue in the audio buffer is not consecutive with the previous packet retrieved in the audio buffer, then the silence packet of the corresponding duration is filled before the first packet at the head of the buffer queue in the audio buffer. If the packet retrieval time has been determined at equal intervals, then if the buffer queue in the audio buffer is empty, the buffer queue will be filled with the mute packet.

2. The method as described in claim 1, wherein before updating the display timestamps corresponding to the audio packet and the video packet respectively using absolute timestamps, the method further comprises: Obtain the SR packet corresponding to the audio and video data; The step of updating the display timestamps corresponding to the audio packet and the video packet using absolute timestamps includes: Based on the SR package, the display timestamps corresponding to the audio package and the video package are updated using absolute timestamps.

3. The method as described in claim 2, wherein the SR packet includes a first absolute timestamp and a first display timestamp corresponding to the audio, and a second absolute timestamp and a second display timestamp corresponding to the video; The step of updating the display timestamps corresponding to the audio packet and the video packet respectively using absolute timestamps based on the SR packet includes: Based on the first absolute timestamp and the first display timestamp corresponding to the audio, a first correspondence is determined between the absolute timestamp corresponding to the audio packet and the display timestamp corresponding to the audio packet; and based on the second absolute timestamp and the second display timestamp corresponding to the video, a second correspondence is determined between the absolute timestamp corresponding to the video packet and the display timestamp corresponding to the video packet. Based on the first correspondence, the display timestamp corresponding to the audio package is updated using the absolute timestamp corresponding to the audio package, and based on the second correspondence, the display timestamp corresponding to the video package is updated using the absolute timestamp corresponding to the video package.

4. The method as described in claim 1, wherein when the data packet is an audio packet, the target data packet is a mute packet; and when the data packet is a video packet, the target data packet is the tail packet in the video buffer.

5. The method as described in claim 1, wherein the timeout packet is a packet in the buffer corresponding to the data packet whose target time interval with the tail packet is greater than the preset buffer duration of the buffer corresponding to the data packet.

6. The method as described in claim 1, wherein before inserting the audio packet into the audio buffer and the video packet into the video buffer according to the first preset rule, the method further comprises: An invalid packet corresponding to a preset buffer duration is inserted into the audio buffer.

7. The method of claim 1, wherein retrieving the corresponding target video packet from the video buffer based on the absolute timestamp of the target audio packet comprises: Determine whether the difference between the absolute timestamp corresponding to the first packet in the video buffer and the absolute timestamp corresponding to the target audio packet is less than a threshold. If so, the first packet in the video buffer is taken out of the video buffer as the target video packet.

8. An audio-visual synchronization device, the device comprising: The first acquisition module is used to acquire audio and video data; the audio and video data includes audio packets and video packets. The update module is used to update the display timestamps corresponding to the audio package and the video package respectively using absolute timestamps; The first insertion module is used to insert the audio packet into the audio buffer and the video packet into the video buffer according to a first preset rule; The first packet retrieval module is used to retrieve the target audio packet from the audio buffer according to the second preset rule; The second packet retrieval module is used to retrieve the corresponding target video packet from the video buffer based on the absolute timestamp of the target audio packet; The first insertion module is specifically used for: If the display timestamp of the data packet is greater than or equal to the display timestamp of the tail packet in the buffer corresponding to the data packet, and there is no gap between the display timestamp of the data packet and the display timestamp of the tail packet, the data packet is inserted into the tail of the buffer corresponding to the data packet; wherein the data packet is the audio packet and the buffer corresponding to the data packet is the audio buffer; or, the data packet is the video packet and the buffer corresponding to the data packet is the video buffer. If the display timestamp of the data packet is less than the display timestamp of the tail packet in the buffer corresponding to the data packet, delete the data packet; If the display timestamp of the data packet is greater than or equal to the display timestamp of the tail packet in the buffer corresponding to the data packet, and there is a gap between the display timestamp of the data packet and the display timestamp of the tail packet, the corresponding target data packet is filled into the buffer corresponding to the data packet based on the gap; The first packet retrieval module is specifically used for: If the packet retrieval time has been determined at equal intervals, and the buffer queue in the audio buffer is not empty, the first packet at the head of the buffer queue in the audio buffer is valid, the last packet retrieved in the audio buffer is invalid, and the buffer queue is full, then the first packet at the head of the buffer queue in the audio buffer will be retrieved as the target audio packet. If the packet retrieval time has been determined at equal intervals, and the buffer queue in the audio buffer is not empty, the first packet at the head of the buffer queue in the audio buffer is valid, the last retrieved packet in the audio buffer is invalid, and the buffer queue is not full, then the buffer queue is filled with a mute packet. If the packet retrieval time has been determined at equal intervals, and the buffer queue in the audio buffer is not empty, and the first packet at the head of the buffer queue in the audio buffer is invalid, then the first packet at the head of the buffer queue in the audio buffer is retrieved. If the packet retrieval time has been determined at equal intervals, and the buffer queue in the audio buffer is not empty, the first packet at the head of the buffer queue in the audio buffer is valid, and the first packet at the head of the buffer queue in the audio buffer is consecutive to the previously retrieved packet in the audio buffer, then the first packet at the head of the buffer queue in the audio buffer will be retrieved as the target audio packet. If the packet retrieval time has been reached at equal intervals, then the buffer queue in the audio buffer is not empty, the first packet at the head of the buffer queue in the audio buffer is valid, and the first packet at the head of the buffer queue in the audio buffer is not consecutive with the previous packet retrieved in the audio buffer, then the silence packet of the corresponding duration is filled before the first packet at the head of the buffer queue in the audio buffer. If the packet retrieval time has been determined at equal intervals, then if the buffer queue in the audio buffer is empty, the buffer queue will be filled with the mute packet.

9. An electronic device, comprising: Processor and memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to perform the method as described in any one of claims 1-7.

10. A computer storage medium storing a plurality of instructions adapted for loading by a processor and performing the method steps of any one of claims 1-7.

11. A computer program product comprising instructions that, when executed on a computer or processor, cause the computer or processor to perform the audio-video synchronization method as described in any one of claims 1-7.

Images