Audio-video synchronization method and apparatus, device, and computer-readable storage medium

Abstract

The present application discloses an audio-video synchronization method and apparatus, a device, and a computer-readable storage medium. The method is applied to a sending device, and comprises: on the basis of communication with a plurality of receiving devices, determining a target communication delay applicable to the plurality of receiving devices; on the basis of the target communication delay, determining a playback delay applicable to the plurality of receiving devices; and sending a media data frame carrying the playback delay to the plurality of receiving devices, so that the plurality of receiving devices can implement, on the basis of the playback delay, audio-video synchronization during playback of the media data frame. In this way, audio-video synchronization across a plurality of devices can be controlled, thereby improving the audio-video synchronization effect.

Description

Audio and video synchronization method, device, equipment and computer-readable storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on March 5, 2024, with application number 202410254076.0 and invention name “Audio and video synchronization method, device, equipment and computer-readable storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of audio and video technology, and in particular to an audio and video synchronization method, apparatus, device, and computer-readable storage medium. Background Art

[0004] With the development and popularization of various smart terminals and high-speed wireless networks, multi-screen interaction and data sharing technologies between devices based on network transmission are gradually being developed and widely used.

[0005] One-to-many audio and video sharing between devices is a new application scenario, where a source sends audio and video information to multiple sinks, and the sinks display and play the information synchronously. Although many audio and video synchronization solutions already exist, these solutions are not comprehensive, resulting in suboptimal audio and video synchronization across multiple devices and poor results. Summary of the Invention

[0006] The technical solution of this application is achieved as follows:

[0007] In a first aspect, an embodiment of the present application provides an audio and video synchronization method, applied to a sending device, the method comprising:

[0008] Determining target communication delays applicable to the plurality of receiving devices based on communications with the plurality of receiving devices;

[0009] Determine the playback delay suitable for multiple receiving devices based on the target communication delay;

[0010] The media data frames carrying the playback delay are sent to multiple receiving devices, so that the multiple receiving devices can achieve audio and video synchronization of playing the media data frames according to the playback delay.

[0011] In a second aspect, an embodiment of the present application provides an audio and video synchronization method, applied to a receiving device, the method comprising:

[0012] Receive media data frames with playback delay sent by the sending device;

[0013] Determine the decoding time of the media data frame according to the playback delay;

[0014] Control the media data frame to be decoded and played when it reaches the decoding time to achieve audio and video synchronization when playing the media data frame.

[0015] In a third aspect, an embodiment of the present application provides an audio and video synchronization device, which is applied to a sending device. The audio and video synchronization device includes a first determination unit and a first transceiver unit, wherein:

[0016] A first determining unit is configured to determine a target communication delay applicable to the plurality of receiving devices based on communications with the plurality of receiving devices; and determine a playback delay applicable to the plurality of receiving devices based on the target communication delay;

[0017] The first transceiver unit is configured to send media data frames carrying a playback delay to multiple receiving devices, so that the multiple receiving devices can achieve audio and video synchronization of playing the media data frames according to the playback delay.

[0018] In a fourth aspect, an embodiment of the present application provides an audio and video synchronization device, which is applied to a receiving device. The audio and video synchronization device includes a second transceiver unit and a playback unit, wherein:

[0019] a second transceiver unit configured to receive a media data frame carrying a playback delay sent by a sending device;

[0020] The playback unit is configured to determine the decoding time of the media data frame according to the playback delay; and control the media data frame to be decoded and played when the decoding time is reached, so as to achieve audio and video synchronization when playing the media data frame.

[0021] In a fifth aspect, an embodiment of the present application provides an electronic device, the electronic device including a memory and a processor, wherein:

[0022] a memory for storing computer programs capable of running on the processor;

[0023] A processor is configured to execute the method described in the first aspect or the method described in the second aspect when running a computer program.

[0024] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect or the method described in the second aspect.

[0025] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the method described in the first aspect or the method described in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG1 is a flow chart of a method for synchronizing audio and video according to an embodiment of the present invention;

[0027] FIG2 is a second flow chart of an audio and video synchronization method provided in an embodiment of the present application;

[0028] FIG3 is a schematic diagram of a clock synchronization principle between a transmitting end and a receiving end provided in an embodiment of the present application;

[0029] FIG4 is a schematic diagram of a detailed flow chart of an audio and video synchronization method provided in an embodiment of the present application;

[0030] FIG5 is a schematic diagram of the first structure of an audio and video synchronization system provided in an embodiment of the present application;

[0031] FIG6 is a schematic diagram of the first structure of an audio and video synchronization device provided in an embodiment of the present application;

[0032] FIG7 is a second schematic diagram of the structure of an audio and video synchronization device provided in an embodiment of the present application;

[0033] FIG8 is a schematic diagram of a specific hardware structure of an electronic device provided in an embodiment of the present application;

[0034] FIG9 is a second schematic diagram of the composition structure of an audio and video synchronization system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In a first aspect, an embodiment of the present application provides an audio and video synchronization method, applied to a sending device, the method comprising:

[0036] Determining target communication delays applicable to the plurality of receiving devices based on communications with the plurality of receiving devices;

[0037] Determine the playback delay suitable for multiple receiving devices based on the target communication delay;

[0038] The media data frames carrying the playback delay are sent to multiple receiving devices, so that the multiple receiving devices can achieve audio and video synchronization of playing the media data frames according to the playback delay.

[0039] In some embodiments, determining target communication delays applicable to the plurality of receiving devices based on communications with the plurality of receiving devices includes:

[0040] Within a preset time period, statistics are collected on the detection communication delays between the sending device and the multiple receiving devices to determine the average communication delay of the multiple receiving devices;

[0041] A target communication delay is determined based on the average communication delays of the multiple receiving devices.

[0042] In some embodiments, determining a target communication delay based on average communication delays of a plurality of receiving devices includes:

[0043] Determine the maximum value of multiple average communication delays and use the maximum value as the target communication delay; or,

[0044] Determine the median value of multiple average communication delays and use the median value as the target communication delay; or,

[0045] An average of multiple average communication delays is determined, and the average is used as the target communication delay.

[0046] In some embodiments, the method further comprises:

[0047] Sending a first detection packet to multiple receiving devices;

[0048] receiving first response information returned by each of the multiple receiving devices;

[0049] Based on the sending time of the first detection packet and the receiving time of the first response information, the detection communication delays of the sending device and the multiple receiving devices are determined.

[0050] In some embodiments, determining a playback delay applicable to multiple receiving devices based on a target communication delay includes:

[0051] Determine the maximum audio delay and preset processing delay for multiple receiving devices;

[0052] The target communication delay, the maximum audio delay, and the preset processing delay are summed to determine a playback delay suitable for multiple receiving devices.

[0053] In some embodiments, the method further comprises:

[0054] The variation range of the playback delay within a preset time interval is set to not exceed a first threshold.

[0055] In a second aspect, an embodiment of the present application provides an audio and video synchronization method, applied to a receiving device, the method comprising:

[0056] Receive media data frames with playback delay sent by the sending device;

[0057] Determine the decoding time of the media data frame according to the playback delay;

[0058] Control the media data frame to be decoded and played when it reaches the decoding time to achieve audio and video synchronization when playing the media data frame.

[0059] In some embodiments, determining a decoding time of a media data frame based on a playback delay includes:

[0060] Get the current latency, decoding time, and rendering time of the media data frame;

[0061] Determine the decoding time of the media data frame based on the playback delay, current delay, decoding time and rendering time.

[0062] In some embodiments, determining a decoding time of a media data frame based on a playback delay, a current delay, a decoding time, and a rendering time includes:

[0063] Based on the playback delay, the current delay, decoding time, and rendering time are subtracted to determine the decoding time of the media data frame.

[0064] In some embodiments, obtaining a current delay of a media data frame includes:

[0065] Get the absolute acquisition time carried in the media data frame;

[0066] The current delay of the media data frame is determined based on the absolute acquisition time, the local time of the receiving device, and the clock synchronization difference between the sending device and the receiving device.

[0067] In some embodiments, the method further comprises:

[0068] Determining a detection communication delay corresponding to the second detection packet;

[0069] When the detection communication delay is lower than the second threshold, receiving a second detection packet sent by the sending device; wherein the second detection packet carries the sending time of the sending device;

[0070] The clock synchronization difference between the sending device and the receiving device is determined according to the receiving time and the sending time of the second detection packet received by the receiving device.

[0071] In some embodiments, the decoding time of the media data frame includes a first decoding time of the video data frame and a second decoding time of the audio data frame;

[0072] Controls the decoding and playback of media data frames at the decoding time, including:

[0073] The video data frame is controlled to be decoded and played when the first decoding time is reached, and the audio data frame is controlled to be decoded and played when the second decoding time is reached, so as to achieve audio and video synchronization when playing the media data frame.

[0074] In some embodiments, the method further comprises:

[0075] Obtain the video decoding time and video rendering time of the video data frame, and obtain the audio decoding time and audio rendering time of the audio data frame;

[0076] Perform a subtraction operation based on the playback delay, the current delay, the video decoding time, and the video rendering time to determine the first decoding time of the video data frame;

[0077] A subtraction operation is performed according to the playback delay, the current delay, the audio decoding time, and the audio rendering time to determine the second decoding time of the audio data frame.

[0078] In some embodiments, the method further comprises:

[0079] When the current delay is greater than the playback delay, determine the timeout period between the current delay and the playback delay;

[0080] The processing method of the media data frame is determined based on the timeout period.

[0081] In some embodiments, determining a processing method for a media data frame based on a timeout period includes:

[0082] When the timeout period is greater than a third threshold, performing frame drop processing on the media data frame;

[0083] When the timeout period is less than the third threshold, accelerated playback processing is performed on the media data frame.

[0084] In a third aspect, an embodiment of the present application provides an audio and video synchronization device, which is applied to a sending device. The audio and video synchronization device includes a first determination unit and a first transceiver unit, wherein:

[0085] A first determining unit is configured to determine a target communication delay applicable to the plurality of receiving devices based on communications with the plurality of receiving devices; and determine a playback delay applicable to the plurality of receiving devices based on the target communication delay;

[0086] The first transceiver unit is configured to send media data frames carrying a playback delay to multiple receiving devices, so that the multiple receiving devices can achieve audio and video synchronization of playing the media data frames according to the playback delay.

[0087] In a fourth aspect, an embodiment of the present application provides an audio and video synchronization device, which is applied to a receiving device. The audio and video synchronization device includes a second transceiver unit and a playback unit, wherein:

[0088] a second transceiver unit configured to receive a media data frame carrying a playback delay sent by a sending device;

[0089] The playback unit is configured to determine the decoding time of the media data frame according to the playback delay; and control the media data frame to be decoded and played when the decoding time is reached, so as to achieve audio and video synchronization when playing the media data frame.

[0090] In a fifth aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein:

[0091] a memory for storing computer programs capable of running on the processor;

[0092] A processor is configured to execute the method described in the first aspect or the method described in the second aspect when running a computer program.

[0093] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect or the method described in the second aspect.

[0094] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the method described in the first aspect or the method described in the second aspect.

[0095] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0096] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0097] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0098] It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0099] Understandably, audio and video synchronization is a crucial factor in the user's viewing experience. With the rapid development and continuous advancement of image processing technology, audio processing technology, and terminal technology, electronic devices such as mobile phones and tablets now have audio and video playback capabilities, becoming commonly used audio and video playback devices for users.

[0100] Audio and video devices can not only play audio and video, but also use cameras and microphones to enable video calls or online live broadcasts. During this process, audio and video playback delays may deviate due to network jitter, inconsistent anti-jitter strategies, or other reasons. When the deviation exceeds a certain range, audio and video synchronization adjustment is required.

[0101] In one possible implementation, the start time of the audio and video frame groups is synchronized based on the logical clock. The audio device uses the logical clock as a reference to adaptively adjust the processing frequency within the group; the video device uses the logical clock and audio clock to adaptively adjust the processing frequency within the frame group. Without the need to introduce third-party equipment and professional transmission lines, multi-device synchronization within the local area network can be achieved. Smoothing the audio playback delay can effectively reduce the probability of frame loss or fast-forwarding, and keep it consistent with the video. However, in this implementation, synchronization between multiple devices is performed at the frame group granularity, which reduces the overall communication volume. The frequency is adaptively adjusted within the frame group to spread the time over multiple frames, further effectively reducing the probability of fast-forwarding or screen distortion.

[0102] In the embodiment of the present application, the best effect of audio and video synchronization is frame synchronization. Otherwise, the user will immediately feel that the audio and video are out of sync, and the optimal effect cannot be achieved. However, the above implementation method synchronizes according to frame groups, and the audio and video frames within the frame group do not refer to the playback progress of other devices. Therefore, when the first frame of the same frame group on different devices is out of sync, this implementation method is difficult to adjust in time, that is, the synchronization phenomenon of different devices will last for a long time. In actual scenarios, its synchronization effect is poor.

[0103] In another possible implementation, audio and video data units carrying timestamps and synchronization report instructions carrying reporting periods can be sent to multiple receiving ends; synchronization reports fed back by each receiving end based on the timestamps in the audio and video data units and the synchronization report instructions are received; synchronization adjustment parameters are generated based on the timestamps of the sent audio and video data units and the synchronization reports fed back by each receiving end, and sent to each receiving end; thereby, multiple receiving ends can play the audio and video data units synchronously.

[0104] This implementation method is to have the receiving end feedback the current playback progress, and the sending end will calculate and then dynamically adjust the playback speed of all receiving ends. This implementation method obviously has a control lag (the lag time is the statistical period of the synchronization report + the synchronization report transmission delay + the control parameter transmission delay), that is, the asynchrony phenomenon of the previous synchronization cycle needs to wait until the next cycle to be adjusted.

[0105] In other words, although there are already many audio and video synchronization solutions in the relevant technology, the existing solutions are not comprehensive and still have some defects. For example, audio and video synchronization between multiple devices in a local area network is a difficult problem. The current synchronization effect is poor, and many solutions can only barely meet acceptable standards or are even impractical. In addition, most related technologies do not consider the impact of audio latency. Audio latency refers to the delay from the time audio data is sent to the player until the sound is produced by the speaker, which is generally around 110ms. Video rendering (from writing data to the display system until the display system actually renders the picture) generally takes about 10ms, and there is an error of about 100ms between the two. Moreover, most related technologies still adjust the playback speed according to the playback progress of the receiving end, which leads to control lag problems, etc., resulting in unsatisfactory audio and video synchronization on multiple devices and poor results.

[0106] Based on this, an embodiment of the present application provides an audio and video synchronization method, which detects the communication delay in real time based on the communication between the sending device and multiple receiving devices, determines the target communication delay suitable for multiple receiving devices, and then covers the communication delay with the determined playback delay, which can stably control the audio and video synchronization, and the playback is smooth, thereby improving the ability to resist network jitter; in addition, the media data frame sent by the sending device directly carries the playback delay, so that the regulation of multiple receiving devices is timely, and the audio and video synchronization of multiple devices can be achieved. Moreover, there is no need for the receiving devices to rely on each other, and the interaction / computation amount will not increase significantly with the increase in the number of receiving devices; this technical solution also does not require the introduction of additional hardware design, is simple to implement, and has low cost. While supporting multiple devices to play audio and video at the same time, it improves the audio and video synchronization effect.

[0107] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0108] In one embodiment of the present application, FIG1 is a flow chart of an audio and video synchronization method provided by the embodiment of the present application. As shown in FIG1 , the method may include:

[0109] S101 : Determine target communication delays applicable to the multiple receiving devices based on communication between a transmitting device and the multiple receiving devices.

[0110] In an embodiment of the present application, the method can be applied to an audio and video synchronization device, or an electronic device integrated with the device. Here, the electronic device can be a sending device, which can also be referred to as a "sending end" or a "source end" (source end). Correspondingly, the electronic device that interacts with the sending end can be a receiving device, which can also be referred to as a "receiving end" or a "destination end" (sink end).

[0111] In an embodiment of the present application, in order to achieve the best effect of the current audio and video synchronization standard, for example, the user cannot perceive the standard: -100ms~20ms, at this time the communication delay detection module can detect the communication delay in real time.

[0112] It is understood that the communication delay can be measured by sending a first probe packet. In some embodiments, the method may include: sending the first probe packet to multiple receiving devices; receiving first response information returned by each of the multiple receiving devices; and determining the probe communication delay between the sending device and the multiple receiving devices based on the time the first probe packet was sent and the time the first response information was received.

[0113] In the embodiment of the present application, the first detection packet is a communication delay detection packet. The detection communication delay can be represented by the round-trip time (RTT), which is the time from the time the sender sends the communication delay detection packet to the time the sender receives the response information from the receiver.

[0114] In addition, in the embodiments of the present application, RTT is one of the important indicators for measuring network transmission performance, which can reflect the speed and stability of media data transmission in the network. Generally speaking, the shorter the RTT, the faster the network transmission speed, and vice versa. Therefore, in order to improve the ability to resist network jitter, it is necessary to detect the communication delay in real time so that the interference of different communication delays between the sender and different receivers on audio and video synchronization can be offset by subsequent playback delay.

[0115] In this way, the sender first sends a communication delay detection packet to multiple receivers, and then the multiple receivers immediately reply with response information to the sender; at this time, the time difference between the time the sender receives the response information and the time the detection packet is sent is the detection communication delay between the sender and the multiple receivers.

[0116] It is also understandable that in order to obtain a target communication delay applicable to multiple receiving terminals, it is also possible to calculate the target communication delay by statistically analyzing the probing communication delay over a period of time. In some embodiments, determining the target communication delay applicable to multiple receiving devices may include: calculating the probing communication delay between the transmitting device and multiple receiving devices within a preset time period to determine the average communication delay of each of the multiple receiving devices; and determining the target communication delay based on the average communication delay of each of the multiple receiving devices.

[0117] In an embodiment of the present application, the preset time period may be a pre-set statistical time period for determining the average communication delay of each receiving end. For example, the preset time period may be set to 1 second.

[0118] In the embodiment of the present application, the average communication delay can be represented by rtt_sink_aver_n, and the target communication delay can be represented by rtt_target. If there are N receiving ends, then n = 1, 2, ... N. That is, after obtaining the average communication delay rtt_sink_aver_n of each of the N receiving ends, the target communication delay rtt_target applicable to all receiving ends can be determined based on the N rtt_sink_aver_n.

[0119] Exemplarily, the transmitting end may count the detection communication delays between the transmitting end and multiple receiving ends within the most recent second. Taking a certain receiving end as an example, the transmitting end counts the detection communication delays between the transmitting end and the receiving end within the most recent second, and then calculates the average value of the multiple detection communication delays obtained to obtain the average communication delay of the receiving end; similarly, the average communication delays of multiple receiving ends may be obtained, and then the target communication delay rtt_target may be determined based on these multiple average communication delays.

[0120] It should be noted that the target communication delay rtt_target is compatible with the communication delay of all receiving ends (in fact, the communication delay for one-way data transmission is approximately equal to rtt / 2, but redundant processing is done here to deal with network jitter).

[0121] In some embodiments, determining the target communication delay based on the average communication delays of multiple receiving devices may include: determining the maximum value of multiple average communication delays and using the maximum value as the target communication delay; or determining the median value of multiple average communication delays and using the median value as the target communication delay; or determining the average value of multiple average communication delays and using the average value as the target communication delay.

[0122] That is to say, in the embodiment of the present application, the determination of the target communication delay can be the maximum value or median value among these multiple average communication delays, or the average value of these multiple average communication delays, or even the percentage value of these multiple average communication delays, etc., without any limitation.

[0123] In addition, the target communication delay can be set according to actual needs. For example, if the user wants all receiving terminals to keep up with the synchronous playback, the target communication delay can be set to the maximum value; if the user wants the playback delay of all receiving terminals to be in a compromise state, the target communication delay can be set to the average value.

[0124] In one possible implementation, for the first media data frame, the target communication delay can be the maximum value of rtt_sink_aver_n; for subsequent media data frames, the average value / median value / maximum value / percentage value of rtt_sink_aver_n can be selected according to actual needs.

[0125] S102: Determine a playback delay suitable for multiple receiving devices based on the target communication delay.

[0126] In the embodiment of the present application, by dynamically setting the size of the playback delay, the interference of different communication delays between the sending end and different receiving ends on the audio and video synchronization can be offset.

[0127] In some embodiments, based on the target communication delay, determining the playback delay applicable to multiple receiving devices, the method may include: determining the maximum audio delay and preset processing delay corresponding to the multiple receiving devices; summing the target communication delay, the maximum audio delay and the preset processing delay to determine the playback delay applicable to the multiple receiving devices.

[0128] In the embodiment of the present application, the playout delay can be represented by playout_delay, the maximum audio delay can be represented by audio_latency_max, and the preset processing delay can be represented by process_time. The playout delay playout_delay is calculated as follows: playout_delay = rtt_target + audio_latency_max + process_time (1)

[0129] Here, the maximum audio latency, audio_latency_max, indicates the maximum audio_latency among all receivers. Audio_latency refers to the delay from when audio data is sent to the player until sound is produced by the speaker, typically around 110ms. If the receiving device cannot obtain the exact audio_latency, an estimate can be provided based on the chip platform type / model of the receiving device.

[0130] The preset processing delay process_time represents a redundant value provided for the media data processing flow, decoding time, etc. For example, the default value is 50ms.

[0131] It is understandable that in order to avoid network fluctuations causing excessive fluctuations in the playback delay playout_delay, which in turn causes audio and video playback to become stuck, in some embodiments, the method may also include: setting the change range of the playback delay within a preset time interval to not exceed a first threshold.

[0132] In this embodiment of the present application, assuming the preset time interval is 1 second and the first threshold is 100ms, the variation of playout_delay should be less than 100ms / s. In other words, to prevent excessive fluctuations in the playout_delay, the transmitter can limit the variation of the playout_delay. That is, the variation of playout_delay needs to be capped, for example, the variation per second should be less than 100ms / s.

[0133] S103: Sending media data frames carrying a play delay to multiple receiving devices, so that the multiple receiving devices can achieve audio and video synchronization of playing the media data frames according to the play delay.

[0134] In an embodiment of the present application, after determining the playback delay, when the transmitting end sends a media data frame to multiple receiving ends, the media data frame carries the playback delay, so that the multiple receiving ends can achieve audio and video synchronization when playing the media data frame. It should be noted that if there is only one receiving device (i.e., receiving end), then after determining the playback delay of the receiving end based on the communication delay between the transmitting end and the receiving end, sending the media data frame carrying the playback delay to the receiving end can also achieve audio and video synchronization when the receiving end plays the media data frame.

[0135] In the embodiments of the present application, in addition to carrying the playback delay, the media data frame also carries the absolute capture time of the media data frame (represented by abs-capture-time), so that each receiving end can calculate the interval between the current frame and the capture time in real time, thereby accurately controlling the rendering time of the frame to enable simultaneous playback of audio and video on multiple devices. The absolute capture time here is the time when the sending end captures (acquires) the raw media data (before encoding), based on the sending end's clock.

[0136] In an embodiment of the present application, the playback delay and absolute capture time can be specifically written in the header extension fields of the media data packet based on the Real-Time Transport Protocol (RTP). For example, two RTP extension fields are used here: one is abs-capture-time, which is used to add the absolute capture time to the RTP message. It provides a timestamp to help the receiving end calculate network delay and clock synchronization; the other is playout_delay, which is used to transmit playback delay information to help the receiving end synchronize when playing audio and video.

[0137] The embodiment of the present application provides an audio and video synchronization method, in which, at the transmitting end, based on the communication with multiple receiving devices, a target communication delay applicable to multiple receiving devices is determined; based on the target communication delay, a playback delay applicable to multiple receiving devices is determined; and media data frames carrying the playback delay are sent to multiple receiving devices, so that multiple receiving devices can achieve audio and video synchronization of the playback media data frames according to the playback delay. In this way, based on the communication between the transmitting end and the multiple receiving ends, the communication delay is detected in real time, the target communication delay applicable to the multiple receiving ends is determined, and then the communication delay is covered by the determined playback delay, so that the audio and video synchronization can be stably controlled, the playback is smooth, and the ability to resist network jitter is improved; in addition, the media data frames sent by the transmitting end directly carry the playback delay, so that the regulation of the multiple receiving ends is timely, the audio and video synchronization of multiple devices can be achieved, and the audio and video synchronization effect can also be improved.

[0138] In another embodiment of the present application, FIG2 is a flow chart of a second method for synchronizing audio and video provided in an embodiment of the present application. As shown in FIG2 , the method may include:

[0139] S201: Receive a media data frame carrying a playback delay sent by a sending device.

[0140] In an embodiment of the present application, the method can be applied to an audio and video synchronization device, or an electronic device integrated with the device. Here, the electronic device can be a receiving device, which can also be referred to as a "receiving end" or a "destination end" (sink end). Correspondingly, the electronic device that interacts with the receiving end can be a sending device, which can also be referred to as a "sending end" or a "source end" (source end).

[0141] In an embodiment of the present application, multiple receiving devices can independently calculate and render, without relying on or influencing each other. Taking one of the receiving devices as an example, the receiving end can receive a media data frame sent by the transmitting end, and the media data frame carries a playback delay. Based on the playback delay, the decoding time of the media data frame can be calculated in real time so that playback can be performed immediately after the playback delay is reached.

[0142] S202: Determine a decoding time of the media data frame according to the playback delay.

[0143] S203: Control the media data frame to be decoded and played when the decoding time is reached, so as to achieve audio and video synchronization when playing the media data frame.

[0144] In an embodiment of the present application, since decoded data in Web Real-Time Communication (WebRTC) is immediately sent for playback, the specific implementation in WebRTC can be to control the time when the media data frame is sent for decoding. In some embodiments, determining the decoding time of the media data frame based on the playback delay can include: obtaining the current delay, decoding time, and rendering time of the media data frame; and determining the decoding time of the media data frame based on the playback delay, current delay, decoding time, and rendering time.

[0145] It should be noted that the current delay can be represented by cur_delay, the decoding time can be represented by decode_time, and the rendering time can be represented by render_time. Therefore, the calculation of the decoding time of the media data frame is as follows: Decoding time = playout_delay - cur_delay - decode_time - render_time (2)

[0146] Here, cur_delay represents the current real-time delay of the media data frame, which can be calculated in real time based on the abs-capture-time in the RTP media data packet. decode_time represents the predicted decoding time of the media data frame. Here, the 95% percentile time of all decoded audio data frames or video data frames can be selected. render_time represents the rendering time of the media data frame, that is, the time from when the frame is written to the system interface to when the system actually plays / sounds out. For example, the audio data frame is audio_latency, for example 110ms; the video data frame can default to 10ms.

[0147] In some embodiments, obtaining the current delay of the media data frame may include: obtaining the absolute acquisition time carried in the media data frame; and determining the current delay of the media data frame based on the absolute acquisition time, the local time of the receiving device, and the clock synchronization difference between the sending device and the receiving device.

[0148] In an embodiment of the present application, since the clocks of the sending end and the receiving end may not be synchronized, it is also necessary to determine the clock synchronization difference between the sending device and the receiving device; then, based on the absolute acquisition time, the local time of the receiving device and the clock synchronization difference between the sending device and the receiving device, the interval between the current moment of the media data frame and the acquisition moment can be determined, that is, the current delay of the media data frame can be determined.

[0149] It is understood that time synchronization between the transmitting and receiving ends can be achieved by a soft clock synchronization module. In some embodiments, the method may further include: determining a probe communication delay corresponding to a second probe packet; when the probe communication delay is lower than a second threshold, receiving a second probe packet sent by the transmitting device; wherein the second probe packet carries the transmit time of the transmitting device; and determining a clock synchronization difference between the transmitting and receiving devices based on the receive time and transmit time of the second probe packet received by the receiving device.

[0150] In the embodiment of the present application, the second detection packet can be a clock synchronization detection packet, where a high-precision clock is relied upon to synchronize the device time. Figure 3 is a schematic diagram of a clock synchronization principle between a transmitter and a receiver provided in the embodiment of the present application.

[0151] As shown in Figure 3, the sender (device A) sends a second detection packet carrying the sending time to the receiver (device B). After receiving the second detection packet, the receiver (device B) immediately replies with a second response message to the sender (device A).

[0152] As shown in Figure 3, the time at the receiving end can be expressed as local time, and the time at the sending end can be expressed as remote time. The relationship between the two is as follows: local time1 + A = remote time1 (3) local time2 - local time1 = rtt / 2 (4)

[0153] According to equations (3) and (4), we can get: A=remote time1-local time2+rtt / 2 (5)

[0154] Here, A represents the clock synchronization difference between the sender and the receiver. Thus, for any time, the corresponding time of the sender calculated in real time on the receiver can be: local time + A.

[0155] The maximum clock synchronization error can be RTT / 2. To improve clock accuracy, it's necessary to continuously perform probes using clock synchronization probe packets to determine the corresponding probe communication delay. Specifically, a second probe packet is sent to the receiver; a second response message is received from the receiver; and the RTT (probe communication delay) between the sender and receiver is determined based on the time the second probe packet is sent and the time the second response message is received.

[0156] For example, assuming that the second threshold is 20ms, that is, the clock accuracy error is less than 10ms, then in order to meet the accuracy of 10ms, it is necessary to continue the detection calculation until the detection communication delay RTT of a certain time is less than 20ms.

[0157] It should be noted that although the soft clock solution is not the focus of audio and video synchronization, it will affect the effect of audio and video synchronization. Therefore, the embodiment of the present application can also set a higher precision soft clock synchronization solution, so as to more accurately control the playback delay / playback time of multiple terminals.

[0158] It is also understandable that the media data frame may include a video data frame and an audio data frame, so the decoding time of the media data frame may include a first decoding time of the video data frame and a second decoding time of the audio data frame. In some embodiments, for controlling the media data frame to be decoded and played upon reaching the decoding time, the method may include: controlling the video data frame to be decoded and played upon reaching the first decoding time, and controlling the audio data frame to be decoded and played upon reaching the second decoding time, so as to achieve audio and video synchronization when playing the media data frame.

[0159] In an embodiment of the present application, the video decoding time and video rendering time of the video data frame can be obtained, and then the first decoding time of the video data frame can be determined based on the playback delay, current delay, video decoding time and video rendering time.

[0160] In a specific embodiment, a subtraction operation can be performed based on the playback delay, the current delay, the video decoding time, and the video rendering time to determine the first decoding time of the video data frame, as shown in formula (2).

[0161] In an embodiment of the present application, the audio decoding time and audio rendering time of the audio data frame can be obtained, and then the second decoding time of the audio data frame can be determined based on the playback delay, current delay, audio decoding time and audio rendering time.

[0162] In a specific embodiment, a subtraction operation can be performed based on the playback delay, the current delay, the audio decoding time, and the audio rendering time to determine the second decoding time of the audio data frame, as shown in formula (2).

[0163] In this way, in the adaptive playback module at the receiving end, the receiving end calculates the delay of a certain media data frame in real time, and sends it for playback immediately after the playback delay is reached; specifically, the media data frame can be controlled to decode and play the video data frame when it reaches the first decoding time, and decode and play the audio data frame when it reaches the second decoding time, thereby achieving synchronous playback of audio and video.

[0164] It should also be noted that in the embodiment of the present application, for playback at the receiving end, it is possible to wait first, then decode and play immediately after the decoding time is reached; or it is possible to decode first, then wait until the playback delay is reached and then play immediately. There is no limitation here.

[0165] It is also understandable that when network fluctuations cause the current frame delay cur_delay to be greater than the play delay playout_delay, that is, the media data frame should have been played but was not played. Two simple processing methods are provided here: frame dropping or accelerated playback.

[0166] In some embodiments, the method may include: when the current delay is greater than the playback delay, determining a timeout period between the current delay and the playback delay; and determining a processing method for the media data frame based on a comparison between the timeout period and a third threshold.

[0167] In a specific embodiment, when the timeout period is greater than a third threshold, the media data frame is subjected to frame drop processing; when the timeout period is less than the third threshold, the media data frame is subjected to accelerated playback processing.

[0168] That is to say, in the embodiment of the present application, a timeout threshold (ie, a third threshold) may be set, which is used to measure the extent to which the current delay cur_delay exceeds the playout delay playout_delay.

[0169] For example, when the timeout is less than the threshold, acceleration is performed until a subsequent frame can be played on time; when the timeout is greater than the threshold, frame dropping is performed. If the timeout is equal to the threshold, then accelerated playback or frame dropping can be performed.

[0170] In a specific embodiment, if the third threshold is set to 0, it means that only frame drop processing is performed; if the third threshold is set to infinity, it means that only accelerated playback is performed. For example, the default value of the third threshold can be 1000ms.

[0171] The embodiment of the present application provides an audio and video synchronization method, which includes receiving a media data frame with a playback delay sent by a sending device; determining the decoding time of the media data frame based on the playback delay; and controlling the media data frame to be decoded and played when the decoding time is reached, so as to achieve audio and video synchronization when playing the media data frame. In this way, the media data frame sent by the sending device directly carries the playback delay, so that the regulation of multiple receiving devices is timely, and the audio and video synchronization of multiple devices can be achieved. Moreover, there is no need for the receiving devices to rely on each other, and the interaction / computation amount will not increase significantly as the number of receiving devices increases. In addition, the technical solution does not require the introduction of additional hardware design, is simple to implement, and has low cost. While supporting multiple devices to play audio and video at the same time, it can also improve the audio and video synchronization effect.

[0172] In another embodiment of the present application, FIG4 is a detailed flowchart of an audio and video synchronization method provided in an embodiment of the present application. As shown in FIG4 , the detailed flowchart may include:

[0173] S401: The sending end sends a first detection packet to the receiving end.

[0174] S402: The receiving end returns first response information to the sending end.

[0175] S403: The sending end determines a detection communication delay between the sending end and the receiving end based on the sending time of the first detection packet and the receiving time of the first response information.

[0176] S404: The transmitting end determines a target communication delay applicable to multiple receiving ends.

[0177] In this embodiment of the present application, the transmitting end may count the detection communication delays obtained by executing steps S401 to S403 within a preset time period and calculate the corresponding average communication delay. Then, the target communication delay applicable to the multiple receiving ends is determined based on the average communication delays of the multiple receiving ends.

[0178] Exemplarily, the target communication delay applicable to the multiple receiving ends may be determined based on the maximum value / median value / average value / percentage value, etc. of the average communication delays of the multiple receiving ends.

[0179] S405: The transmitting end determines a playback delay applicable to multiple receiving ends.

[0180] In the embodiment of the present application, after determining the target communication delay, the target communication delay, the maximum audio delay, and the preset processing delay can be summed to determine the playback delay applicable to multiple receiving devices, as shown in formula (1). In this way, by dynamically setting the size of the playback delay, the interference of different communication delays between the transmitter and different receivers on audio and video synchronization can be offset.

[0181] S406: The sending end sends a media data frame carrying a playback delay to the receiving end.

[0182] S407: The receiving end determines a decoding time of the media data frame according to the playback delay.

[0183] S408: The receiving end controls the media data frame to be decoded and played when the decoding time is reached, so as to achieve audio and video synchronization when playing the media data frame.

[0184] In the embodiment of the present application, the receiving end calculates the delay of the media data frame in real time and determines the decoding time for controlling the media data frame to be sent for decoding, as shown in formula (2). In this way, the decoded audio and video data are immediately sent for playback, thereby achieving audio and video synchronization and achieving the best effect of the current audio and video synchronization standard. Here, multiple receiving ends independently calculate and render, without relying on or affecting each other.

[0185] FIG5 is a schematic diagram of the structure of an audio and video synchronization system provided in an embodiment of the present application. As shown in FIG5 , the audio and video synchronization system 50 may include a soft clock synchronization module 501 , a communication delay detection module 502 , a playback delay calculation module 503 , and a receiving-end adaptive playback module 504 .

[0186] In the embodiment of the present application, the soft clock synchronization module 501 can be provided on the transmitting end and / or the receiving end, the communication delay detection module 502 and the playback delay calculation module 503 can be provided on the transmitting end, and the receiving end adaptive playback module 504 can be provided on the receiving end. These four modules are described below:

[0187] In the embodiment of the present application, the soft clock synchronization module 501 is used for all receiving ends to synchronize the time of the sending end, and the clock accuracy error is <10ms.

[0188] In the embodiment of the present application, the communication delay detection module 502 is used for the transmitting end to detect the communication delay in real time so as to set the playback delay of the audio and video frames.

[0189] In an embodiment of the present application, for the playback delay calculation module 503, the playback delay refers to how long it takes for a certain media frame to be allowed to be played starting from the capture moment. The sending end calculates the playback delay of the media data frame applicable to all receiving ends based on the communication delay RTT with different receiving ends, and writes it in the header extension field of the RTP media data packet. Two RTP extension fields are provided here: one is for abs-capture-time, which is used to carry the absolute capture time of the audio and video frames, so that each receiving end can calculate the current moment and the interval between the frame and the capture moment in real time to accurately control the rendering time of the frame; the other is for playout_delay, which is used to carry the playback delay of the audio and video frames, that is, the frame will not be rendered until x milliseconds have passed after the capture moment.

[0190] In the embodiment of the present application, for the receiving end adaptive playback module 504, the receiving end calculates the delay of a certain media frame in real time and sends it for playback immediately after the playback delay is reached (each receiving end calculates and renders independently and does not depend on / affect each other).

[0191] Understandably, the International Telecommunication Union's ITU-R BT.1359-1 standard for audio-video synchronization describes the following as imperceptible: -100ms to 25ms; perceptible: -125ms to 45ms; and acceptable: -185ms to 90ms. Negative values ​​indicate the picture comes before the audio, while positive values ​​indicate the picture comes before the audio.

[0192] In conjunction with FIG4 and FIG5, the specific process of this technical solution is described as follows:

[0193] (1) Soft clock synchronization module.

[0194] This technical solution relies on a high-precision clock to synchronize device time. Here's a brief introduction to one implementation (as shown in Figure 3): Device A sends a probe packet with the send time to Device B. Upon receiving the probe packet, Device B immediately sends a response back to Device A.

[0195] For any time, the corresponding time of the sender calculated in real time on the receiver is: local time + A.

[0196] It should be noted that the maximum error of this technical solution is rtt / 2. In order to achieve an accuracy of 10ms, continuous detection and calculation are required until the RTT of a certain detection packet is less than 20ms.

[0197] (2) Communication delay detection module.

[0198] This technical solution offsets the interference of different communication delays between the sending end and different receiving ends on audio and video synchronization by dynamically setting the delay playback size.

[0199] Here, the sender will count the average communication delay rtt_sink_aver_n with each of the N receivers in the last second, and calculate the rtt_target applicable to all receivers based on the N rtt_sink_aver_n, and then dynamically adjust the playout_delay.

[0200] Among them, rtt_target is the communication delay compatible with all receiving ends (in fact, the one-way data transmission delay is approximately equal to rtt / 2, and redundant processing is done here for network jitter). For example, for the first audio and video frame, this value can take the maximum value of rtt_sink_aver_n; for subsequent audio and video frames, the average value / median value / maximum value / percentage value can be selected as needed, or the maximum value can be selected for the first frame, and the value can be gradually reduced for subsequent frames to achieve smooth playback.

[0201] (3) Playback delay calculation module.

[0202] Playout_delay = rtt_target + audio_latency_max + process_time.

[0203] Among them, audio_latency_max is the maximum audio_latency of all sink ends (audio_latency refers to the delay from sending audio data to the player to the speaker to produce sound, which is generally around 110ms. If the device cannot obtain the exact audio_latency, you can write an estimated value based on the chip platform type / model of this device); process_time is a redundant value for the media data processing flow and decoding time, and defaults to 50ms. It is important to note that there is no video rendering delay here because the video rendering delay is 10ms, while audio_latency is approximately 110ms. For playback delay, audio and video can be played synchronously after 110ms, without waiting another 10ms (i.e., playing after 120ms).

[0204] In addition, it is also important to note that in order to avoid extreme network fluctuations causing excessive fluctuations in the playback delay playout_delay, which in turn causes audio and video playback to freeze, the change range of playout_delay should be set to a maximum value, for example, the change per second should be less than 100ms / s.

[0205] (4) Adaptive playback module at the receiving end.

[0206] The receiving end calculates the delay of a media data frame in real time and sends it for playback immediately after the playback delay is reached. Each receiving end calculates and renders independently without relying on or affecting each other.

[0207] In the embodiment of the present application, since decoded data in WebRTC is immediately sent for playback, the specific implementation of this technical solution in WebRTC is to control the time when the media data frame is sent for decoding (i.e., the decoding time): decoding time = playout_delay - cur_delay - decode_time - render_time. It should be noted that for WebRTC, WebRTC, as web instant messaging, can refer to the WebRTC standard, and this technical solution specifically refers to the WebRTC standard and its open source technology framework.

[0208] In the embodiment of the present application, cur_delay represents the current real-time delay of the frame, which can be calculated in real time based on the abs-capture-time in the RTP data packet. decode_time represents the predicted decoding time of the frame, which can be the 95th percentile time of all decoded audio or video frames. render_time represents the rendering time, that is, the time from when the frame is written to the system interface to when the system actually plays / sounds it. For example, the audio frame is audio_latency, which can be 110ms; the video frame can default to 10ms.

[0209] In addition, when network fluctuations cause the current delay cur_delay of the media data frame to be greater than the play delay playout_delay, that is, the media data frame should have been played long ago, this technical solution provides two simple processing methods: frame drop or acceleration.

[0210] Specifically, you can set a timeout threshold (the amount by which the current delay cur_delay exceeds the playout delay playout_delay). When the timeout is less than the threshold, acceleration is performed until a subsequent frame can be played on time. When the timeout exceeds the threshold, frame dropping is performed. Setting the threshold to 0 means only frame dropping is performed; setting it to infinity means only acceleration is performed. The default value is 1000ms.

[0211] At present, most audio and video synchronization solutions can only barely reach the acceptable standard (-185ms ~ 90ms). In the embodiment of the present application, since the playout_delay of all audio and video frames is directly specified from the sending end, the communication delay between multiple devices is offset by a small amount of delay increase, thereby ensuring the optimal cross-end audio and video synchronization effect between multiple devices (unable to perceive the gear: -100ms ~ 25ms), which can meet the use requirements of actual scenarios in the near field; and this technical solution directly controls the playback delay of each frame, thereby controlling the audio and video synchronization effect of multiple devices in advance, and does not require feedback on the playback status of the receiving end, and can achieve the best effect of the current audio and video synchronization standard (users cannot perceive the standard).

[0212] It can also be understood that the Selective Forwarding Unit (SFU) is a streaming media forwarding server suitable for multiple devices. Quality of Experience (QoE) refers to the user's subjective feeling about the quality and performance of devices, networks and systems, applications or services. In addition, the link layer is the second layer of the TCP / IP network protocol model. The protocol model has five layers from low to high: physical layer, link layer, network layer, transport layer, and application layer. The RTT of the related technology is a network transmission indicator of the fourth layer transport layer, which is generally a single network transmission socket in a single application; all sockets of all applications will pass through the link layer, which can represent the transmission quality of the entire network.

[0213] It is also understandable that the soft clock is not the focus of this technical solution, but it will affect the effect of audio and video synchronization; a higher-precision soft clock synchronization solution can also be designed here, so as to more accurately control the playback delay / playback time of multiple terminals.

[0214] An embodiment of the present application provides an audio and video synchronization method. The specific implementation of the aforementioned embodiment is elaborated in detail through the above embodiment. It can be seen that by dynamically controlling the playback delay of the media data frame and thus controlling the audio and video synchronization of multiple terminals, it truly supports multiple receiving terminals to play audio and video at the same time, and does not cause a large increase in interaction / computation as the number of devices increases; by real-time detection of the communication delay RTT and then covering the communication delay through playback delay, the audio and video synchronization can be stably controlled, the playback is smooth, and the ability to resist network jitter is improved. That is to say, through the technical solution of the aforementioned embodiment, the best effect of the current audio and video synchronization standard can be achieved (the user cannot perceive the standard: -100ms~20ms); it truly supports multiple sink ends to play audio and video at the same time, and will not cause a large increase in interaction / computation as the number of devices increases, and each sink end does not need to rely on / reference each other; the regulation is timely, and when the sending end sends the media data frame, it directly carries the playback delay of the frame to control the audio and video synchronization of each receiving end; in addition, this technical solution is a pure software solution, which is simple to implement, low in implementation cost, and does not require the introduction of additional hardware; and by detecting the communication delay in real time, and then covering the communication delay through the playback delay, the audio and video synchronization can be stably controlled, and the playback is smooth, which improves the ability to resist network jitter, and thus improves the audio and video synchronization effect.

[0215] In yet another embodiment of the present application, based on the same inventive concept as the aforementioned embodiment, FIG6 is a schematic diagram of the first structure of an audio and video synchronization device provided in an embodiment of the present application. As shown in FIG6 , the audio and video synchronization device 60 is applied to a sending device, and the audio and video synchronization device 60 may include a first determining unit 601 and a first transceiver unit 602, wherein:

[0216] The first determining unit 601 is configured to determine a target communication delay applicable to the multiple receiving devices based on communications with the multiple receiving devices; and determine a playback delay applicable to the multiple receiving devices based on the target communication delay;

[0217] The first transceiver unit 602 is configured to send media data frames carrying a playback delay to multiple receiving devices, so that the multiple receiving devices can achieve audio and video synchronization of playing the media data frames according to the playback delay.

[0218] In some embodiments, the first determination unit 601 is further configured to collect statistics on the detection communication delays between the sending device and multiple receiving devices within a preset time period to determine the average communication delay of each of the multiple receiving devices; and determine the target communication delay based on the average communication delay of each of the multiple receiving devices.

[0219] In some embodiments, the first determination unit 601 is further configured to determine the maximum value of multiple average communication delays and use the maximum value as the target communication delay; or, determine the median value of multiple average communication delays and use the median value as the target communication delay; or, determine the average value of multiple average communication delays and use the average value as the target communication delay.

[0220] In some embodiments, the first transceiver unit 602 is further configured to send a first detection packet to multiple receiving devices; and receive first response information returned by each of the multiple receiving devices; the first determination unit 601 is further configured to determine the detection communication delay between the sending device and the multiple receiving devices based on the sending time of the first detection packet and the receiving time of the first response information.

[0221] In some embodiments, the first determination unit 601 is further configured to determine the maximum audio delay and preset processing delay corresponding to multiple receiving devices; and to sum the target communication delay, the maximum audio delay and the preset processing delay to determine the playback delay applicable to multiple receiving devices.

[0222] In some embodiments, the first determining unit 601 is further configured to set a variation range of the playback delay within a preset time interval not to exceed a first threshold.

[0223] In yet another embodiment of the present application, based on the same inventive concept as the aforementioned embodiment, FIG7 is a second schematic diagram of the structure of an audio and video synchronization device provided in an embodiment of the present application. As shown in FIG7 , the audio and video synchronization device 70 is applied to a receiving device, and the audio and video synchronization device 70 may include a second transceiver unit 701, a playback unit 702, and a second determination unit 703, wherein:

[0224] The second transceiver unit 701 is configured to receive a media data frame carrying a playback delay sent by a sending device;

[0225] The playback unit 702 is configured to determine the decoding time of the media data frame according to the playback delay; and control the media data frame to be decoded and played when the decoding time is reached, so as to achieve audio and video synchronization when playing the media data frame.

[0226] In some embodiments, the second determination unit 703 is further configured to obtain the current delay, decoding time and rendering time of the media data frame; and determine the decoding time of the media data frame based on the playback delay, current delay, decoding time and rendering time.

[0227] In some embodiments, the second determination unit 703 is further configured to obtain the absolute acquisition time carried in the media data frame; and determine the current delay of the media data frame based on the absolute acquisition time, the local time of the receiving device, and the clock synchronization difference between the sending device and the receiving device.

[0228] In some embodiments, the second determination unit 703 is further configured to determine the detection communication delay corresponding to the second detection packet; the second transceiver unit 701 is further configured to receive the second detection packet sent by the sending device when the detection communication delay is lower than the second threshold; wherein the second detection packet carries the sending time of the sending device; the second determination unit 703 is further configured to determine the clock synchronization difference between the sending device and the receiving device based on the receiving time and sending time of the second detection packet received by the receiving device.

[0229] In some embodiments, the decoding time of the media data frame includes a first decoding time of the video data frame and a second decoding time of the audio data frame; accordingly, the playback unit 702 is also configured to control the video data frame to perform video decoding and playback when the first decoding time is reached, and control the audio data frame to perform audio decoding and playback when the second decoding time is reached, so as to achieve audio and video synchronization when playing the media data frame.

[0230] In some embodiments, the second determination unit 703 is further configured to obtain the video decoding time and video rendering time of the video data frame, and obtain the audio decoding time and audio rendering time of the audio data frame; and perform a subtraction operation based on the playback delay, current delay, video decoding time and video rendering time to determine the first decoding moment of the video data frame; and perform a subtraction operation based on the playback delay, current delay, audio decoding time and audio rendering time to determine the second decoding moment of the audio data frame.

[0231] In some embodiments, the second determination unit 703 is further configured to determine the timeout period between the current delay and the playback delay when the current delay is greater than the playback delay; the playback unit 702 is further configured to perform frame drop processing on the media data frame when the timeout period is greater than a third threshold; and to perform accelerated playback processing on the media data frame when the timeout period is less than the third threshold.

[0232] It is understood that in this embodiment, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular system. Furthermore, the various components in this embodiment can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The aforementioned integrated units can be implemented in the form of hardware or software functional modules.

[0233] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0234] Therefore, this embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method described in any one of the aforementioned embodiments are implemented.

[0235] Based on the composition of the audio and video synchronization device 60 or the audio and video synchronization device 70 and the computer-readable storage medium, Figure 8 is a schematic diagram of the specific hardware structure of an electronic device provided in an embodiment of the present application. As shown in Figure 8, the electronic device 80 may include a communication interface 801, a memory 802 and a processor 803; each component is coupled together through a bus system 804. It can be understood that the bus system 804 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 804 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as bus systems 804 in Figure 8. Among them, the communication interface 801 is used to receive and send signals in the process of sending and receiving information between other external network elements;

[0236] Memory 802, used to store computer programs that can be run on processor 803;

[0237] The processor 803 is configured to execute the steps of the method described in any one of the aforementioned embodiments when running the computer program.

[0238] In one possible implementation, when the electronic device 80 is a sending device (transmitting end), the processor 803 is used to execute the following when running the computer program: based on the communication with multiple receiving devices, determine the target communication delay applicable to multiple receiving devices; based on the target communication delay, determine the playback delay applicable to multiple receiving devices; and send media data frames carrying the playback delay to multiple receiving devices, so that the multiple receiving devices can achieve audio and video synchronization of the playback media data frames according to the playback delay.

[0239] In one possible implementation, when the electronic device 80 is a receiving device (receiving end), the processor 803 is used to execute the following when running the computer program: receiving a media data frame carrying a playback delay sent by a sending device; determining a decoding time of the media data frame based on the playback delay; and controlling the media data frame to be decoded and played when the decoding time is reached, so as to achieve audio and video synchronization when playing the media data frame.

[0240] It is understood that the memory 802 of the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DRRAM). The memory 802 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0241] The processor 803 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the processor 803. The processor 803 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 802, and the processor 803 reads the information in the memory 802 and, in conjunction with its hardware, completes the steps of the above method.

[0242] It is also understood that the embodiments described herein may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or a combination thereof.

[0243] For software implementation, the techniques described herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0244] In yet another embodiment of the present application, FIG9 is a second schematic diagram of the structure of an audio and video synchronization system provided in an embodiment of the present application. As shown in FIG9 , the audio and video synchronization system 50 may include a transmitting device 901 and N receiving devices ( 902 - 1 , 902 - 2 , ..., 902 -N), where N is a positive integer.

[0245] In the embodiment of the present application, the sending device 901 may be an electronic device 80 or a device integrated with the audio and video synchronization device 60, and each receiving device may also be an electronic device 80 or a device integrated with the audio and video synchronization device 70. The sending device may interact with N receiving devices.

[0246] In an embodiment of the present application, the sending device 901 sends a media data frame to N receiving devices, and controls the audio and video synchronization of multiple device ends by dynamically controlling the playback delay of the media data frame, truly supporting multiple receiving ends to play audio and video at the same time, and does not cause a large increase in interaction / computation as the number of devices increases; by real-time detection of the communication delay RTT, and then covering the communication delay with the playback delay, the audio and video synchronization can be stably controlled, the playback is smooth, and the ability to resist network jitter is improved.

[0247] In yet another embodiment of the present application, the embodiment of the present application further provides a computer program product, including a computer program or instructions, which implement the steps of the method as described in any one of the aforementioned embodiments when executed by a processor.

[0248] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, equipment, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0249] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0250] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0251] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0252] It should be noted that, in this application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0253] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0254] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0255] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0256] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0257] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims. Industrial Applicability

[0258] In an embodiment of the present application, at the transmitting end, based on the communication with multiple receiving devices, a target communication delay applicable to the multiple receiving devices is determined; based on the target communication delay, a playback delay applicable to the multiple receiving devices is determined; and media data frames carrying the playback delay are sent to the multiple receiving devices, so that the multiple receiving devices can achieve audio and video synchronization when playing the media data frames based on the playback delay. At the receiving end, media data frames carrying the playback delay sent by the transmitting device are received; the decoding time of the media data frames is determined based on the playback delay; and the media data frames are controlled to be decoded and played when the decoding time is reached, so as to achieve audio and video synchronization when playing the media data frames. In this way, based on the communication with multiple receiving devices, the communication delay is detected in real time, and the target communication delay suitable for multiple receiving devices is determined. Then, the communication delay is covered by the determined playback delay, which can stably control the audio and video synchronization, and the playback is smooth, thereby improving the ability to resist network jitter; in addition, the media data frame sent by the sending device directly carries the playback delay, so that the regulation of multiple receiving devices is timely, and the audio and video synchronization of multiple devices can be achieved. Moreover, there is no need for the receiving devices to rely on each other, and the interaction / computation amount will not increase significantly with the increase in the number of receiving devices; this technical solution does not require the introduction of additional hardware design, is simple to implement, and has low cost. While supporting multiple devices to play audio and video at the same time, it can also improve the audio and video synchronization effect.

Claims

1. A method for synchronizing audio and video, applied to a sending device, the method comprising: determining, based on communications with a plurality of receiving devices, target communication delays applicable to the plurality of receiving devices; determining, based on the target communication delay, a playback delay applicable to the plurality of receiving devices; The media data frame carrying the playback delay is sent to the multiple receiving devices, so that the multiple receiving devices can achieve audio and video synchronization of playing the media data frame according to the playback delay.

2. The method according to claim 1, wherein The determining, based on communications with a plurality of receiving devices, target communication delays applicable to the plurality of receiving devices, includes: Within a preset time period, statistics are collected on the detection communication delays between the sending device and the multiple receiving devices to determine the average communication delays of the multiple receiving devices; The target communication delay is determined according to the average communication delays of the multiple receiving devices.

3. The method according to claim 2, wherein: The determining the target communication delay according to the average communication delays of the plurality of receiving devices includes: determining a maximum value of the multiple average communication delays, and using the maximum value as the target communication delay; or, Determine a median value of the multiple average communication delays, and use the median value as the target communication delay; or, An average value of the multiple average communication delays is determined, and the average value is used as the target communication delay.

4. The method according to claim 2, wherein: The method further comprises: Sending a first detection packet to the multiple receiving devices; receiving first response information returned by each of the multiple receiving devices; Based on the sending time of the first probe packet and the receiving time of the first response information, a probe communication delay between the sending device and the plurality of receiving devices is determined.

5. The method according to claim 1, wherein The determining, based on the target communication delay, a playback delay applicable to the plurality of receiving devices, includes: Determining maximum audio delays and preset processing delays corresponding to the multiple receiving devices; A sum operation is performed on the target communication delay, the maximum audio delay, and the preset processing delay to determine a playback delay applicable to the multiple receiving devices.

6. The method according to any one of claims 1 to 5, wherein: The method further comprises: The variation range of the playback delay within a preset time interval is set to not exceed a first threshold.

7. A method for synchronizing audio and video, applied to a receiving device, the method comprising: Receive media data frames with playback delay sent by the sending device; Determining a decoding time of the media data frame according to the playback delay; The media data frame is controlled to be decoded and played when the decoding time is reached, so as to achieve audio and video synchronization when playing the media data frame.

8. The method according to claim 7, wherein: Determining the decoding time of the media data frame according to the playback delay includes: Obtaining the current delay, decoding time, and rendering time of the media data frame; Determine a decoding time of the media data frame according to the playback delay, the current delay, the decoding time, and the rendering time.

9. The method according to claim 8, wherein The determining, according to the playback delay, the current delay, the decoding time, and the rendering time, of the decoding time of the media data frame includes: A subtraction operation is performed on the current delay, the decoding time, and the rendering time based on the playback delay to determine a decoding time of the media data frame.

10. The method according to claim 8, wherein The obtaining of the current delay of the media data frame includes: Obtaining the absolute acquisition time carried in the media data frame; The current delay of the media data frame is determined according to the absolute acquisition time, the local time of the receiving device, and the clock synchronization difference between the sending device and the receiving device.

11. The method according to claim 10, wherein: The method further comprises: Determining a detection communication delay corresponding to the second detection packet; When the detection communication delay is lower than a second threshold, receiving a second detection packet sent by the sending device; wherein the second detection packet carries the sending time of the sending device; A clock synchronization difference between the sending device and the receiving device is determined according to a receiving time of the second detection packet received by the receiving device and the sending time.

12. The method according to claim 8, wherein The decoding time of the media data frame includes a first decoding time of the video data frame and a second decoding time of the audio data frame; The controlling the media data frame to be decoded and played when reaching the decoding time includes: The video data frame is controlled to be decoded and played when the first decoding time is reached, and the audio data frame is controlled to be decoded and played when the second decoding time is reached, so as to achieve audio and video synchronization when playing the media data frame.

13. The method according to claim 12, wherein: The method further comprises: Obtaining the video decoding time and the video rendering time of the video data frame, and obtaining the audio decoding time and the audio rendering time of the audio data frame; Perform a subtraction operation according to the playback delay, the current delay, the video decoding time, and the video rendering time to determine a first decoding time of the video data frame; A subtraction operation is performed according to the playback delay, the current delay, the audio decoding time and the audio rendering time to determine a second decoding time of the audio data frame.

14. The method according to claim 8, wherein The method further comprises: If the current delay is greater than the playback delay, determining a timeout period between the current delay and the playback delay; A processing method for the media data frame is determined according to the timeout period.

15. The method according to claim 14, wherein The determining, according to the timeout period, a processing method for the media data frame includes: When the timeout period is greater than a third threshold, performing frame drop processing on the media data frame; When the timeout period is less than a third threshold, accelerated playback processing is performed on the media data frame.

16. An audio and video synchronization device, applied to a sending device, the audio and video synchronization device comprising a first determination unit and a first transceiver unit, wherein: The first determining unit is configured to determine target communication delays applicable to the multiple receiving devices based on communications with the multiple receiving devices; and determine playback delays applicable to the multiple receiving devices based on the target communication delays; The first transceiver unit is configured to send the media data frame carrying the playback delay to the multiple receiving devices, so that the multiple receiving devices can achieve audio and video synchronization of playing the media data frame according to the playback delay.

17. An audio and video synchronization device, applied to a receiving device, comprising a second transceiver unit and a playback unit, wherein: The second transceiver unit is configured to receive a media data frame carrying a playback delay sent by a sending device; The playback unit is configured to determine the decoding time of the media data frame according to the playback delay; and control the media data frame to be decoded and played when the decoding time is reached, so as to achieve audio and video synchronization when playing the media data frame.

18. An electronic device comprising a memory and a processor, wherein: The memory is used to store a computer program that can be run on the processor; The processor is configured to execute the method according to any one of claims 1 to 6, or the method according to any one of claims 7 to 15, when running the computer program.

19. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 or the method according to any one of claims 7 to 15 is implemented.

20. A computer program product comprising a computer program or instructions, wherein: When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 6 or the method according to any one of claims 7 to 15 is implemented.

Images