Audio and video synchronization method, sending end, system, computer device and storage medium

Abstract

This application relates to an audio-video synchronization method, a transmitting end, a system, a computer device, and a storage medium. The method involves acquiring target audio data from an audio queue and target video data from a video queue; determining the current audio time and current video time; and, based on the current audio time, current video time, and a preset synchronization threshold, sending the target audio data and / or target video data to a playback end when the transmission conditions are met. By controlling the transmission of audio and video data according to the transmission conditions, the time difference between audio and video data with the same timestamp received by the playback end is kept within the synchronization threshold range. This avoids the problem that when the time difference between audio and video data with the same timestamp arriving at the playback end exceeds the synchronization threshold due to independent transmission and differences in generation speed, the playback end cannot synchronously play the audio and video data.

Description

Technical Field

[0001] This application relates to the field of audio and video processing technology, and in particular to an audio and video synchronization method, a transmitting end, a system, a computer device, and a storage medium. Background Technology

[0002] In Web Real-Time Communication (WebRTC) scenarios, the audio and video timestamp synchronization mechanism typically involves the sending end assigning the same or aligned Presentation Time Stamp (PTS) to audio and video frames of the same time sequence, and the receiving end playing them in PTS order.

[0003] However, in WebRTC scenarios, audio and video are transmitted on separate tracks, each using different encoders, transmission paths, and buffering mechanisms. Even if audio and video frames are assigned the same or aligned timestamps, factors such as network jitter, encoding latency, and decoding latency can cause audio and video frames to arrive at the playback end at inconsistent times. Furthermore, the time deviation between audio and video frames arriving at the playback end accumulates. When the accumulated time deviation becomes too large, the playback end cannot achieve precise synchronization of audio and video during playback. Summary of the Invention

[0004] Therefore, it is necessary to provide an audio-video synchronization method, a transmitting end, a system, a computer device, and a storage medium to address the aforementioned technical problems and improve the synchronization accuracy of audio and video.

[0005] In a first aspect, this application provides an audio-video synchronization method, which is applied at a sending end and includes:

[0006] Acquire the target audio data and the target video data;

[0007] Determine the current audio time and the current video time;

[0008] Based on the current audio time, the current video time, and a preset synchronization threshold, the target audio data and / or the target video data are sent to the playback terminal when the sending conditions are met; wherein, the sending conditions are used to limit the difference between the sending progress of the target video data and the sending progress of the target audio data to within the preset synchronization threshold range.

[0009] In one embodiment, the method further includes:

[0010] Place the initial audio and initial video data output by the renderer into the buffer;

[0011] The initial audio data in the buffer is corrected to obtain target audio data, and the initial video data is corrected to obtain target video data.

[0012] In one embodiment, the target audio data includes a target audio frame and a target audio timestamp, and the initial audio data includes at least an initial audio frame. The target audio data is obtained by correcting the initial audio data in the buffer, including:

[0013] The initial audio frame is merged with the audio frames in the buffer to obtain a merged audio frame;

[0014] If the first data length of the merged audio frame is greater than or equal to the second data length of a unit audio frame, then the target number of target audio frames are extracted from the merged audio frame.

[0015] Based on the audio frame interval, determine the target audio timestamp corresponding to each target audio frame.

[0016] In one embodiment, determining the target audio timestamp corresponding to each target audio frame based on the audio frame interval includes:

[0017] For each non-first target audio frame, the sum of the target audio timestamp of the previous target audio frame adjacent to the target audio frame and the audio frame interval is used as the target audio timestamp of the target audio frame.

[0018] For the first target audio frame, the audio frame interval is used as the target audio timestamp of the target audio frame.

[0019] In one embodiment, the target video data includes target video frames and target video timestamps; the initial video data includes initial video frames and initial video timestamps; and the initial video data is corrected to obtain the target video data, including:

[0020] If the initial video frame is not the first frame, then the reference timestamp of the initial video frame is determined based on the target video timestamp and video frame interval of the previous initial video frame.

[0021] The later timestamp between the reference timestamp and the initial timestamp of the initial video frame is used as the target video timestamp of the initial video frame, and the initial video frame is used as the target video frame.

[0022] If the initial video frame is the first frame, then the initial video timestamp is used as the target video timestamp, and the initial video frame is used as the target video frame.

[0023] In one embodiment, the synchronization threshold is less than or equal to the video frame interval; wherein the video frame interval is determined based on the frame rate of the video frame.

[0024] In one embodiment, determining the current audio time and the current video time includes:

[0025] Get the number of audio frames sent and the sampling rate of the audio frames;

[0026] The current audio time is determined based on the number of audio frames and the sampling rate of the audio frames;

[0027] Get the number of video frames sent and the frame rate of the video frames;

[0028] The current video time is determined based on the number of video frames and the frame rate of the video frames.

[0029] Secondly, this application also provides a sending end, including:

[0030] The audio transmission track module is used to acquire target audio data;

[0031] The video transmission track module is used to acquire target video data;

[0032] The renderer buffer module is used to determine the current audio time and the current video time, and to determine whether the sending conditions are met based on the current audio time, the current video time, and a preset synchronization threshold.

[0033] An audio transmission track module is used to send the target audio data to the playback end when transmission conditions are met; and / or,

[0034] The video transmission track module is used to send the target video data to the playback terminal when the transmission conditions are met.

[0035] In one embodiment, the transmitter further includes:

[0036] The renderer module is used to output initial video and initial audio data;

[0037] The renderer buffer module is also used to put the initial audio data and initial video data output by the renderer into a buffer, and to correct the initial audio data in the buffer to obtain target audio data, and to correct the initial video data to obtain target video data.

[0038] Thirdly, this application also provides an audio-visual synchronization system, the system comprising:

[0039] The sending end is used to acquire target audio data and target video data; and determine the current audio time and current video time; based on the current audio time, the current video time, and a preset synchronization threshold, and when it is determined that the sending conditions are met, the sending end sends the target audio data and / or the target video data to the playback end; wherein, the sending conditions are used to limit the difference between the sending progress of the target video data and the sending progress of the target audio data to be within the preset synchronization threshold range;

[0040] The playback device is used to synchronize the received audio and video data.

[0041] Fourthly, this application also provides an audio-visual synchronization device, which is applied at a transmitting end and includes:

[0042] The acquisition module is used to acquire target audio data and target video data;

[0043] The determination module is used to determine the current audio time and the current video time;

[0044] The sending module is used to send the target audio data and / or the target video data to the playback terminal based on the current audio time, the current video time, and a preset synchronization threshold, provided that the sending conditions are met; wherein, the sending conditions are used to limit the difference between the sending progress of the target video data and the sending progress of the target audio data to within the preset synchronization threshold range.

[0045] Fifthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0046] Acquire the target audio data and the target video data;

[0047] Determine the current audio time and the current video time;

[0048] Based on the current audio time, the current video time, and a preset synchronization threshold, the target audio data and / or the target video data are sent to the playback terminal when the sending conditions are met; wherein, the sending conditions are used to limit the difference between the sending progress of the target video data and the sending progress of the target audio data to within the preset synchronization threshold range.

[0049] Sixthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0050] Acquire the target audio data and the target video data;

[0051] Determine the current audio time and the current video time;

[0052] Based on the current audio time, the current video time, and a preset synchronization threshold, the target audio data and / or the target video data are sent to the playback terminal when the sending conditions are met; wherein, the sending conditions are used to limit the difference between the sending progress of the target video data and the sending progress of the target audio data to within the preset synchronization threshold range.

[0053] In a seventh aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0054] Acquire the target audio data and the target video data;

[0055] Determine the current audio time and the current video time;

[0056] Based on the current audio time, the current video time, and a preset synchronization threshold, the target audio data and / or the target video data are sent to the playback terminal when the sending conditions are met; wherein, the sending conditions are used to limit the difference between the sending progress of the target video data and the sending progress of the target audio data to within the preset synchronization threshold range.

[0057] The aforementioned audio and video synchronization method, transmitting end, system, computer equipment, and storage medium determine whether the transmission conditions are met based on the current audio time, current video time, and a preset synchronization threshold. Only when the transmission conditions are met are the audio data and / or video data sent to the playback end. By controlling the transmission progress of audio and video data through the transmission conditions, the time difference between video frames and audio frames with the same timestamp received by the playback end is kept within the synchronization threshold range. This avoids the problem that when the cumulative time deviation of audio and video data with the same timestamp arriving at the playback end exceeds the synchronization threshold due to independent transmission and generation speed differences, the playback end cannot play the audio and video synchronously. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart illustrating an audio-video synchronization method in one embodiment;

[0060] Figure 2 This is a schematic diagram of the process for correcting audio and video data in one embodiment;

[0061] Figure 3 This is a schematic diagram of the process for correcting initial audio data in one embodiment;

[0062] Figure 4 This is a flowchart illustrating the audio-video synchronization method in another embodiment;

[0063] Figure 5 This is a schematic diagram of the transmitting end in one embodiment;

[0064] Figure 6 This is a structural block diagram of an audio-video synchronization device in one embodiment;

[0065] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0067] The audio-video synchronization method provided in this application can be applied to application scenarios involving audio-video fusion presentation. This method can be executed by a sending end, which can be a server or a terminal. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.

[0068] In Web Real-Time Communication (WebRTC) scenarios, the audio and video timestamp synchronization mechanism typically involves the sending end assigning the same or aligned Presentation Time Stamp (PTS) to audio and video frames of the same time sequence, and the receiving end playing them in PTS order.

[0069] However, in WebRTC scenarios, audio and video are transmitted on separate tracks, each using different encoders, transmission paths, and buffering mechanisms. Even if audio and video frames are assigned the same or aligned timestamps, factors such as network jitter, encoding latency, and decoding latency can cause audio and video frames to arrive at the playback end at inconsistent times. Furthermore, the time deviation between audio and video frames arriving at the playback end accumulates. When the accumulated time deviation becomes too large, the playback end cannot achieve precise synchronization of audio and video during playback.

[0070] However, if the time difference between the arrival of audio and video data at the same timestamp at the playback end exceeds the waiting delay, the playback end will not wait for the unreceived video or audio data, resulting in the inability to achieve precise synchronization of audio and video during playback.

[0071] Therefore, this application provides an audio-video synchronization method. It determines whether a transmission condition is met based on the current audio time, the current video time, and a preset synchronization threshold. Only when the transmission condition is met is the audio data and / or video data sent to the playback end. By controlling the transmission progress of audio and video data through the transmission condition, the time difference between video frames and audio frames with the same timestamp received by the playback end is kept within the synchronization threshold range. This avoids the problem that when the cumulative time deviation of audio and video data with the same timestamp arriving at the playback end exceeds the synchronization threshold due to independent transmission and differences in generation speed, the playback end cannot synchronously play the audio and video.

[0072] In one exemplary embodiment, such as Figure 1 As shown, an audio-video synchronization method is provided. Taking the application of this method to the sending end as an example, the method includes the following steps:

[0073] S101, acquire the target audio data and the target video data.

[0074] In this context, the sending end refers to the device or module that generates and sends audio and video data, such as a server or terminal device integrating a renderer; the target audio data is the audio data to be sent, which can be the audio data generated by the renderer and stored in an audio queue. The audio queue is an ordered data structure for temporarily storing the target audio data; the target video data is the video data to be sent, which can be the video data generated by the renderer and stored in a video queue. The video queue is an ordered data structure for temporarily storing the target video data.

[0075] For example, the target audio data and its corresponding timestamp can be retrieved from the head of the audio queue according to the first-in-first-out (FIFO) principle; and the target video data and its corresponding timestamp can be retrieved from the head of the video queue according to the FIFO principle.

[0076] Furthermore, the target audio data can be converted into a standard audio frame format, and the target video data can be converted into a standard video frame format for the encoder to recognize; and parameters such as PTS, time base, and sampling rate can be set so that the playback end can align and play the target audio data and target video data by comparing the PTS and time base.

[0077] S102, determine the current audio time and the current video time.

[0078] The current audio time refers to the cumulative playback time of all audio data successfully sent by the sending end, reflecting the progress of audio data transmission; the current video time refers to the cumulative playback time of all video data successfully sent by the sending end, reflecting the progress of video data transmission.

[0079] For example, before sending the target audio data and target video data, it is necessary to determine whether the sending conditions are met. This can be done by first determining the current audio time and the current video time, and then determining whether the sending conditions are met based on the current audio time and the current video time.

[0080] For example, the number of audio frames and video frames sent can be counted. Then, the current audio time can be determined based on the number of audio frames sent and the time interval between sending two adjacent audio frames, and the current video time can be determined based on the number of video frames sent and the time interval between sending two adjacent video frames. For instance, assuming the time interval between two adjacent audio frames is 20ms, and 3 audio frames have been sent, the current audio time is 60ms, meaning the cumulative playback time corresponding to all successfully sent audio data is 60ms. As another example, assuming the time interval between two adjacent video frames is 40ms, and 4 video frames have been sent, the current video time is 160ms, meaning the cumulative playback time corresponding to all successfully sent video data is 160ms.

[0081] For example, when determining the current audio time and the current video time, it is necessary to acquire locks on the audio time variables and the video time variables to prevent them from being updated during the process of determining the current audio time and the current video time. After determining the current audio time and the current video time, the locks on the audio time variables and the video time variables are released.

[0082] In some optional implementations, the number of audio frames sent and the sampling rate of the audio frames can be obtained; and the current audio time can be determined based on the number of audio frames and the sampling rate of the audio frames.

[0083] The number of audio frames sent refers to the total number of unit audio frames successfully sent from the sending end to the playback end through the audio transmission track, i.e., one unit audio frame is one audio frame; the sampling rate of the audio frame is the sampling frequency of the audio data, i.e., the number of audio samples collected per unit time, such as 22050Hz means 22050 samples are collected per second.

[0084] For example, the sending end can maintain an audio frame transmission counter. The counter is incremented by 1 for each successful transmission of a unit audio frame. The number of audio frames transmitted can be obtained by reading the counter value. The sampling rate of the audio frames is a system preset parameter that can be read through a configuration file.

[0085] Therefore, the calculation formula "T" can be used. 当前音频时间 The current audio time T is calculated using the formula: (Number of audio frames sent × Audio frame size) / Sampling rate. 当前音频时间 The unit can be seconds. The audio frame size is the preset number of samples contained in a unit audio frame, which can be calculated based on the sampling rate. The calculation formula is the sampling rate divided by 50. For example, a sampling rate of 22050Hz corresponds to an audio frame size of 441 samples.

[0086] For example, the number of video frames sent and the frame rate of the video frames can also be obtained; and the current video time can be determined based on the number of video frames and the frame rate of the video frames.

[0087] The number of video frames sent refers to the total number of video frames successfully sent from the sending end to the playback end via the video transmission track; the frame rate of the video frame is the frame rate of the video data, that is, the number of video frames generated per unit time, such as 25fps means that 25 frames of video are generated per second.

[0088] For example, the sending end can maintain a video frame transmission counter. Each time a video frame is successfully transmitted, the counter is incremented by 1. The number of video frames transmitted can be obtained by reading the counter value. The frame rate of the video frame is a system preset parameter that can be read through a configuration file.

[0089] Therefore, the calculation formula "T" can be used. 当前视频时间 The current video duration T is calculated using the formula: "Number of video frames sent / Frame rate". 当前视频时间 The unit can be seconds.

[0090] For example, assuming the audio sampling rate is 22050Hz, the audio frame size is 441 samples, and the number of audio frames sent is 10, T 当前视频时间= (10 × 441) / 22050 = 4410 / 22050 = 0.2 seconds = 200ms; the video frame rate is 25fps, and the number of video frames sent is 5. T 当前视频时间 =5 / 25 = 0.2 seconds = 200ms. If the number of audio frames sent increases to 11, T 当前视频时间 = (11 × 441) / 22050 = 4851 / 22050 = 0.22 seconds = 220 ms.

[0091] In the above embodiments, through a clear calculation formula and data acquisition method, the current audio time and current video time can be accurately calculated, ensuring the quantitative accuracy of audio and video transmission progress and avoiding synchronization deviations caused by progress calculation errors.

[0092] S103: Based on the current audio time, the current video time, and the preset synchronization threshold, and if the sending conditions are met, send the target audio data and / or target video data to the playback end.

[0093] The preset synchronization threshold is a preset time threshold used to determine whether the sending conditions are met, and is used to control the error range of audio and video synchronization; the sending conditions are the criteria for allowing the sending of audio or video data, and are used to limit the difference between the sending progress of video data and the sending progress of audio data to be within the preset synchronization threshold range.

[0094] For example, the preset synchronization threshold can be set according to the buffering capacity of the playback device. For instance, if the playback device can buffer a time difference of 40ms, the preset synchronization threshold can be set to a value less than or equal to 40ms.

[0095] In some optional implementations, to avoid excessive waiting time when sending audio and video data, the synchronization threshold can be set to be less than or equal to the video frame interval, thus adapting to the generation rhythm of video frames. The video frame interval is determined based on the video frame rate. Specifically, the video frame interval is the ideal time interval between two adjacent video frames, determined by the video frame rate; for example, a video frame interval of 1 / frame rate reflects the generation and playback rhythm of video frames.

[0096] In the above embodiments, the synchronization threshold is limited to within the video frame interval, ensuring that the audio and video synchronization error does not exceed the playback interval of a single video frame. This avoids screen stuttering or audio lag caused by an excessively large synchronization threshold. At the same time, it adapts to the generation rhythm of video frames, enabling the synchronization judgment to respond promptly to fluctuations in the audio and video generation speed. While ensuring synchronization accuracy, it avoids increased latency caused by excessive waiting.

[0097] For example, the current audio time, current video time, and a preset synchronization threshold can be used to determine whether the sending conditions are met. The sending conditions can include both audio and video sending conditions. This allows for separate determination of whether the audio and video sending conditions are met. If only the audio sending condition is met, only the target audio data is sent to the playback device; or, if only the video sending condition is met, only the target video data is sent to the playback device; or, if both the audio and video sending conditions are met, both the target audio and target video data are sent to the playback device.

[0098] Based on this, it can be determined whether the audio transmission conditions are met by considering the current video time and the first difference, where the first difference is the difference between the current audio time and a preset synchronization threshold. For example, if the current video time is greater than or equal to the first difference, it means that the difference between the video transmission progress and the audio transmission progress is within the preset synchronization threshold range. In this case, the audio transmission conditions are met, and the target audio data is sent to the playback device.

[0099] Once the audio transmission conditions are met, a lock on the audio time variable can be acquired, the audio time variable updated to the current audio time, and the threads waiting for the audio time variable notified. The lock on the audio time variable is then released. The target audio data can then be sent to the playback device using an audio transmission track.

[0100] The system can also determine whether the video sending conditions are met based on the current audio time and the second difference, which is the difference between the current video time and a preset synchronization threshold. For example, if the current audio time is greater than or equal to the second difference, it means that the difference between the video sending progress and the audio sending progress is within the preset synchronization threshold range. In this case, the video sending conditions are met, and the target video data is sent to the playback device.

[0101] Once the video transmission conditions are met, the lock on the video time variable is acquired, the video time variable is updated to the current video time, the threads waiting for the video time variable are notified, and the lock on the video time variable is released. The target video data can then be sent to the playback end using the video transmission track.

[0102] For example, assume the preset synchronization threshold is 40ms. The current audio time is 120ms, and the current video time is 90ms. Calculate the first difference as 120ms - 40ms = 80ms, and the second difference as 90ms - 40ms = 50ms. Determine if the current video time (90ms) ≥ the first difference (80ms), satisfying the audio sending condition; and if the current audio time (120ms) ≥ the second difference (50ms), satisfying the video sending condition. That is, at this point, the difference between the video data sending progress and the audio data sending progress is within the preset synchronization threshold range, and both the target audio data and the target video data can be sent to the playback end.

[0103] If the current audio duration is 120ms and the current video duration is 70ms, the first difference is calculated as 120ms - 40ms = 80ms, and the second difference is 70ms - 40ms = 30ms. If the current video duration (70ms) is less than or equal to the first difference (80ms), the audio sending condition is not met. If the current audio duration (120ms) is greater than or equal to the second difference (30ms), the video sending condition is met. This means that the video data sending progress is slower than the audio data sending progress, so audio data cannot be sent at this time. Video data needs to be sent to reduce the progress difference between the video and audio data sending progress. Therefore, the target video data is sent to the playback end. The target audio data is sent only after the video duration reaches the specified value, i.e., after the audio sending condition is met.

[0104] If the current audio duration is 60ms and the current video duration is 110ms, the first difference is calculated as 60ms - 40ms = 20ms, and the second difference is 110ms - 40ms = 70ms. If the current video duration (110ms) is greater than or equal to the first difference (20ms), the audio sending condition is met. If the current audio duration (60ms) is less than or equal to the second difference (70ms), the video sending condition is not met. This means that the audio data transmission progress is slower than the video data transmission progress, so video data cannot be sent at this time. Audio data needs to be sent to reduce the progress difference between the audio and video data transmission progress. Therefore, the target audio data is sent to the playback end. The target video data is sent only after the audio duration meets the specified time, i.e., after the video sending condition is met.

[0105] If the audio transmission conditions are not met, the system enters a waiting state until the conditions are met. For example, during the waiting process, the locks on the audio and video time variables can be acquired again, and the audio and video time variables can be read to determine if they meet the audio transmission conditions. Once the audio transmission conditions are met, the audio time variable is updated to the current audio time, the threads waiting for the audio time variable are notified, and the locks on the audio and video time variables are released. Then, the target audio data is sent to the playback end using the audio transmission track.

[0106] If the video sending conditions are not met, the system will enter a waiting state until the conditions are met. For example, during the waiting process, the locks on the audio and video time variables can be acquired again, and the audio and video time variables can be read to determine if they meet the video sending conditions. Once the video sending conditions are met, the video time variable is updated to the current video time, the threads waiting for the video time variable are notified, and the locks on the audio and video time variables are released. Then, the target video data is sent to the playback end using the video sending track.

[0107] This clarifies the specific judgment logic for audio and video transmission conditions, forming a two-way waiting synchronization mechanism for audio and video. Audio transmission must wait for the video progress to reach the target, and video transmission must wait for the audio progress to reach the target. This two-way constraint can dynamically compensate for the difference in audio and video generation speeds, avoiding synchronization deviations caused by data leading in one direction. At the same time, the accuracy of synchronization judgment is ensured by quantitative difference comparison, overcoming the synchronization defects of independent audio and video transmission, and further improving the audio and video synchronization accuracy in low-latency scenarios.

[0108] When the sending conditions are not met, the system simply blocks and waits until the sending conditions are met or a notification is received, instead of pre-caching audio and video data of a fixed duration. This avoids the fixed delay caused by the fixed buffer scheme, which requires "filling the fixed buffer before outputting audio and video data".

[0109] In the above embodiments, the system determines whether the sending conditions are met based on the current audio time, the current video time, and a preset synchronization threshold. Only when the sending conditions are met is the audio and video data to be sent to the playback end. By controlling the sending of audio and video data according to the sending conditions, the time difference between the video data and audio data with the same timestamp received by the playback end is kept within the synchronization threshold range. This avoids the problem that when the time difference between audio data and video data with the same timestamp arriving at the playback end exceeds the synchronization threshold due to independent transmission and differences in generation speed, the playback end cannot synchronously play the audio and video data.

[0110] In some alternative implementations, the audio and video data can be time-corrected before being added to the queue to further improve the synchronization accuracy of the audio and video.

[0111] For example, see Figure 2 , Figure 2 A flowchart illustrating the process of correcting audio and video data is provided, specifically including the following steps:

[0112] S201, put the initial audio data and initial video data output by the renderer into the buffer.

[0113] For example, the initial audio and video data generated by the renderer can be obtained through a data transfer interface. The initial audio data generated by the renderer may have issues such as inconsistent audio samples in each frame, leading to discontinuous timestamps; the initial video data generated by the renderer may also have discontinuous timestamps or rollback issues. Therefore, the initial audio and video data output by the renderer can be placed in a buffer to correct the initial audio and video data in the buffer.

[0114] S202, correct the initial audio data in the buffer to obtain the target audio data, and correct the initial video data to obtain the target video data.

[0115] For example, the initial audio data in the buffer can be modified to obtain the target audio data. For instance, the number of audio samples in the initial audio data can be adjusted so that each audio frame contains the same number of audio samples, thus making the audio frames a fixed size and ensuring that the playback duration corresponding to each audio frame is a fixed duration, thereby making the timestamps of the audio frames continuous.

[0116] It can also modify the initial video data to obtain the target video data. For example, it can adjust the timestamps of the initial video data so that the timestamps corresponding to the video frames monotonically increase.

[0117] In the above embodiments, by placing the initial audio data and initial video data output by the renderer into a buffer and correcting the initial audio data and video data in the buffer, the problem of the playback end being unable to play the audio data and video data synchronously due to the time stamps of the initial audio data and initial video data being out of sync is avoided from the source.

[0118] In some optional implementations, the target audio data includes target audio frames and target audio timestamps, and the initial audio data includes at least initial audio frames. The initial audio frames can be merged or segmented to ensure that the audio frames are of standard size and that the audio timestamps are continuous.

[0119] Based on this, see Figure 3 , Figure 3 A flowchart illustrating the process of correcting initial audio data is provided, specifically including the following steps:

[0120] S301, merge the initial audio frame with the audio frames in the buffer to obtain a merged audio frame.

[0121] The buffer can include an audio buffer and a video buffer. The audio buffer is a temporary storage area used to accumulate and merge incomplete audio data, and its format is a tuple of audio data and timestamp.

[0122] For example, the audio frames stored in the audio buffer can be read, and the initial audio frame can be concatenated with the audio frames in the buffer in sequence to maintain data continuity and obtain a merged audio frame. At the same time, the start timestamp of the merged data is recorded, that is, the start timestamp of the merged audio frame.

[0123] S302, if the first data length of the merged audio frame is greater than or equal to the second data length of the unit audio frame, then extract the target number of target audio frames from the merged audio frame.

[0124] Wherein, the first data length is the total number of audio samples contained in the merged audio frame; the unit audio frame is a single audio data block divided according to the preset audio frame size, and its data length is the second data length; the second data length is the preset number of samples in a single audio data block, which can be calculated from the audio sampling rate. For example, when the sampling rate is 22050Hz, the second data length is 22050 / 50=441 samples; the target number is the number of unit audio frames that can be extracted from the merged audio frame, that is, the integer part of the first data length divided by the second data length.

[0125] For example, the first data length and the second data length can be compared. If the first data length is greater than or equal to the second data length, the integer part of the first data length divided by the second data length is taken as the target quantity. Then, unit audio frames are extracted sequentially from the starting position of the merged audio frames according to the second data length. The extracted unit audio frames are taken as the target audio frames, and the number of extractions is the target quantity.

[0126] S303, determine the target audio timestamp corresponding to each target audio frame based on the audio frame interval.

[0127] The audio frame interval can be determined based on the number of samples and the sampling rate in a unit audio frame. For example, if the number of samples in a unit audio frame is 441 and the sampling rate is 22050Hz, then the audio frame interval is 441 samples / 22050Hz = 0.02 seconds = 20ms.

[0128] For example, the target audio timestamp of each target audio frame can be calculated sequentially based on the audio frame interval. For instance, the audio frame intervals can be accumulated one by one to obtain the target audio timestamp corresponding to each target audio frame.

[0129] In the above embodiments, by merging audio frames through a buffer and dividing the merged audio frames into target audio frames of fixed size, the continuity of audio data is ensured, and timestamp calculation errors caused by irregular audio frame sizes are avoided. At the same time, dividing the target audio frames into fixed sizes and uniformly calculating the target audio frame timestamps gives the audio frame timestamps a unified time reference, further improving the accuracy and continuity of audio timestamps. This provides a reliable audio time basis for audio and video synchronization and solves the problem of synchronization deviation caused by inconsistent audio frame sizes.

[0130] In some alternative implementations, for each non-first target audio frame, the sum of the target audio timestamp of the previous target audio frame adjacent to the target audio frame and the audio frame interval can be used as the target audio timestamp of the target audio frame. For the first target audio frame, the audio frame interval is used as the target audio timestamp of the target audio frame.

[0131] For example, assuming the audio frame interval is 20ms, the first data length of the merged audio frames is 900 samples, the second data length is 441 samples, and the target number is 2. The target audio timestamp of the first unit audio frame A4-1 is 20ms; the target audio timestamp of the non-first unit audio frame A4-2 is 20ms + 20ms = 40ms. After extraction, 18 samples remain (900-882), and the buffer start timestamp is updated to 20ms + 20ms = 40ms, which is used as the new target audio timestamp of the previous target audio frame.

[0132] In the above embodiments, the calculation method of target audio timestamps for target audio frames at different locations is clarified. The timestamp of the first target audio frame is the audio frame interval, which ensures the consistency of the time base. For non-first target audio frames, the timestamp is increased by adding the audio frame interval to the timestamp of the previous target audio frame, which ensures the continuous increment of the timestamp and avoids the situation of discontinuous or disordered audio frame timestamps. This further improves the accuracy of audio timestamps and provides a precise time basis for audio and video synchronization.

[0133] Then, each target audio frame and its corresponding target audio timestamp can be added to the audio queue.

[0134] For example, each extracted target audio frame can be bound to its corresponding target audio timestamp and added to the end of the audio queue in the order of extraction, waiting to be sent.

[0135] For example, the initial audio data may include an initial audio frame and an initial audio timestamp. The initial audio timestamp of the initial audio frame may include the start timestamp and the end timestamp of the initial audio frame. Taking a sampling rate of 22050Hz and 441 samples in a single audio frame as an example, the audio frame interval is calculated as 441 samples / 22050Hz = 0.02 seconds = 20ms.

[0136] Ideally, each initial audio frame generated by the renderer should contain 441 samples, with a fixed audio frame interval. However, in the actual process of the renderer generating the initial audio frame, there may be problems such as irregular initial audio frame size and inconsistent audio frame interval. Therefore, it is necessary to put the generated initial audio frame and initial audio timestamp into a buffer for correction in order to obtain a target audio frame with regular size and fixed audio frame interval.

[0137] Taking the first initial audio frame generated by the renderer as an example, assuming that the number of samples in the first initial audio frame is 430, the start timestamp in the initial audio timestamp of the first initial audio frame is 0ms, and the end timestamp is the timestamp of the 430th sample, that is, 430 / 22050 seconds.

[0138] At this point, there are no audio frames in the buffer. The first initial audio frame is placed into the buffer, and the start timestamp of the first initial audio frame is 0ms as the start timestamp of the audio data in the buffer. Since the number of samples in the first initial audio frame is less than 441, it is necessary to wait for the next initial audio frame to be placed into the buffer before dividing the target audio frame.

[0139] The renderer generates the second initial audio frame. Assuming the number of samples in the second initial audio frame is 460, the start timestamp in the initial audio timestamp of the second initial audio frame is 430 / 22050 seconds, and the end timestamp is the timestamp of the 430+460th sample, i.e. (430+460) / 22050 seconds.

[0140] The initial audio frame of the second frame is placed in the buffer and merged with the audio frames in the buffer to obtain a merged audio frame. At this time, the first data length of the merged audio frame is 430 + 460, and the second data length of the unit audio frame is greater than or equal to 441. Two target audio frames can be extracted from the merged audio frame. The number of samples in each target audio frame is 441. The start timestamp of the first target audio frame is 0ms and the end timestamp is 20ms (derived from 441 / 22050 seconds). The start timestamp of the second target audio frame is 20ms and the end timestamp is 40ms (derived from 882 / 22050 seconds). Then, these two target audio frames and their corresponding target timestamps can be placed into the audio queue.

[0141] At this point, there are 430 + 460 - 882 = 8 samples remaining in the buffer. The start timestamp of the audio data in the buffer is updated to 40ms, and 40ms is used as the start timestamp of the next target audio frame.

[0142] To retrieve the target audio data from the audio queue, before sending the target audio data, you can first determine whether the audio sending conditions are met based on the current audio time, the current video time, and the preset synchronization threshold.

[0143] For example, taking the second frame of target audio data in the audio queue as an example, since the audio queue has already sent one target audio frame with an interval of 20ms, the current audio time is 20ms. Assume the video queue has already sent one target video frame with an interval of 40ms, and the current video time is 40ms. At this point, the audio sending condition can be determined based on the current video time and a first difference (the difference between the current audio time and a preset synchronization threshold). For example, if the current video time is greater than or equal to the first difference, it means the difference between the video sending progress and the audio sending progress is within the preset synchronization threshold range, thus the audio sending condition is met, and the target audio data is sent to the playback end. For instance, if the first difference is the difference between the current audio time of 20ms and the preset synchronization threshold of 40ms - 20ms, and the current video time of 40ms is greater than the first difference - 20ms, the audio sending condition is met, and the second target audio frame and the target audio timestamp can be sent to the playback end.

[0144] In some alternative implementations, the initial timestamp of the initial video frame can be modified to make the timestamp of the initial video frame incremented, thus avoiding the problem of timestamp rollback.

[0145] For example, if the initial video frame is not the first frame, a reference timestamp can be determined based on the target video timestamp of the previous initial video frame and the video frame interval; then, the later timestamp between the reference timestamp and the initial video timestamp is used as the target video timestamp, the initial video frame is used as the target video frame, and the initial video frame and the target video timestamp are added to the video queue.

[0146] If the initial video frame is the first frame, then the initial video timestamp is used as the target video timestamp, and the initial video frame is used as the target video frame. The initial video frame and the initial video timestamp are then added to the video queue.

[0147] The reference timestamp is a timestamp calculated by adding the target video timestamp of the previous initial video frame to the initial video frame interval for non-first initial video frames, used to ensure the continuous increment of the initial video frame timestamp; the video frame interval is the ideal time interval between two adjacent video frames, which can be calculated from the video frame rate. For example, the video frame interval is 1 / frame rate. If the frame rate is 25fps, then the video frame interval is 40ms.

[0148] For example, if the initial video frame is not the first frame, the target video timestamp of the previous initial video frame can be read, and the sum of the target video timestamp of the previous initial video frame and the video frame interval can be used as the reference timestamp. Then, the reference timestamp and the initial video timestamp of the initial video frame are compared, and the timestamp with the larger value is selected as the target video timestamp. The initial video frame is then used as the target video frame. After the target video frame is bound to the target video timestamp, it is added to the end of the video queue in sequence.

[0149] If the initial video frame is the first frame, the initial video frame is directly used as the target video frame, the initial video timestamp is used as the target video timestamp, and the target video frame is bound to the target video timestamp without correction. It is then added to the video queue, and the target video timestamp is recorded as the target video timestamp of the previous initial video frame.

[0150] For example, assuming a video frame rate of 25fps, a video frame interval of 40ms, and the target video timestamp of the previous initial video frame V3 is 400ms, the currently acquired initial video frame V4 is not the first frame, and its initial video timestamp is 430ms. The reference timestamp is calculated as 400ms + 40ms = 440ms. Comparing the reference timestamp 440ms with the initial video timestamp 430ms, the later 440ms is selected as the target video timestamp for V4. V4 is then bound to 440ms and added to the video queue, and the target video timestamp of the previous initial video frame is updated to 440ms. If the current initial video frame is the first frame, with an initial video timestamp of 40ms, then this initial video frame is directly bound to 40ms and added to the video queue, and the target video timestamp of the previous initial video frame is recorded as 40ms.

[0151] Before sending the target video data, the target video data in the video queue is obtained. Before sending the target video data, the current audio time, the current video time, and the preset synchronization threshold can be used to determine whether the video sending conditions are met.

[0152] For example, let's take the target video frame V4 in the video sending queue as an example. The current video duration is 400ms, and the current audio duration is assumed to be 420ms.

[0153] At this point, the system can determine whether the video transmission conditions are met based on the current audio time and the second difference, where the second difference is the difference between the current video time and a preset synchronization threshold. For example, if the current audio time is greater than or equal to the second difference, it means that the difference between the video transmission progress and the audio transmission progress is within the preset synchronization threshold range. In this case, the video transmission conditions are met, and the target video frame V4 and its corresponding target video timestamp can be sent to the playback device.

[0154] For example, the second difference is 360ms, which is the difference between the current video time of 400ms and the preset synchronization threshold of 40ms. The current audio time of 420ms is greater than the second difference of 360ms, which meets the video sending conditions. The target video frame V4 and the corresponding target video timestamp can be sent to the playback end.

[0155] In the above embodiments, the differentiated timestamp correction strategy for the first frame and non-first frames ensures the rationality of the starting reference by directly using the initial timestamp of the first frame. For non-first frames, the later timestamp is selected as the target video timestamp by comparing the reference timestamp with the initial timestamp. This avoids the problem of timestamp rollback or discontinuity caused by rendering delay and system time calibration, ensuring that the video frame timestamp always increases monotonically. This provides an accurate time reference for calculating the video transmission progress, thereby improving the accuracy of audio and video synchronization.

[0156] Furthermore, by correcting the timestamps of audio and video frames, and setting a two-way waiting mechanism where audio transmission waits for video progress to reach a certain threshold and video transmission waits for audio progress to reach a certain threshold, audio and video synchronization can be maintained even if the renderer's processing speed fluctuates.

[0157] In some alternative implementations, see [link to relevant documentation]. Figure 4 , Figure 4 Another method for audio and video synchronization is provided, including the following steps:

[0158] S401, acquire the target audio data and the target video data.

[0159] S402: The initial audio data and initial video data output by the renderer are placed into the buffer, and the initial audio data in the buffer is corrected to obtain the target audio data, and the initial video data is corrected to obtain the target video data.

[0160] S403, obtain the number of audio frames sent and the sampling rate of the audio frames, and determine the current audio time based on the number of audio frames and the sampling rate of the audio frames.

[0161] S404: Obtain the number of video frames sent and the frame rate of the video frames, and determine the current video time based on the number of video frames and the frame rate of the video frames.

[0162] S405, obtain the first difference between the current audio time and the preset synchronization threshold, and obtain the second difference between the current video time and the preset synchronization threshold.

[0163] S406, based on the first difference, if the sending conditions are met, the target audio data is sent to the playback end; and / or based on the second difference, if the sending conditions are met, the target video data is sent to the playback end.

[0164] The specific processes of S401 to S406 described above can be found in the description of the above method embodiments. Their implementation principles and technical effects are similar, and will not be repeated here.

[0165] Furthermore, the execution order of the above steps is merely illustrative and is not intended to limit the execution steps. The execution order of other steps is within the protection scope of the embodiments of this application.

[0166] In some alternative implementations, a sender is provided, including:

[0167] The audio transmission track module is used to acquire target audio data;

[0168] The video transmission track module is used to acquire target video data;

[0169] The renderer buffer module is used to determine the current audio time and the current video time, and based on the current audio time, the current video time, and a preset synchronization threshold, to determine whether the sending conditions are met.

[0170] An audio transmission track module is used to send target audio data to the playback end when transmission conditions are met; and / or,

[0171] The video sending track module is used to send the target video data to the playback terminal when the sending conditions are met.

[0172] In one embodiment, the transmitter further includes:

[0173] The renderer module is used to output initial video and initial audio data;

[0174] The renderer buffer module is also used to put the initial audio data and initial video data output by the renderer into the buffer, and to correct the initial audio data in the buffer to obtain the target audio data, and to correct the initial video data to obtain the target video data.

[0175] In some alternative implementations, see [link to relevant documentation]. Figure 5 , Figure 5A schematic diagram of a transmitter is provided. The transmitter includes a renderer module, a renderer buffer module, an audio transmission track, a video transmission track, and a WebRTC / RTMP transmission module.

[0176] The renderer module is used to generate raw video frames (video_frame) and audio frames (audio_frame) in real time, and to attach an initial timestamp to each set of simultaneous audio and video frames.

[0177] The renderer buffer module is used for timestamp correction, audio pending processing, audio and video queue management, and condition variable management.

[0178] Timestamp correction includes monotonically increasing the original timestamps of video frames. The rule is to take the larger value between the sum of the timestamp of the previous frame and the interval between video frames for non-first frames, and the initial timestamp of the current frame. This solves the problems of rendering delay, timestamp rollback caused by system time jumps, and discontinuity, and avoids synchronization failure at the playback end from the root.

[0179] Audio pending processing (audio buffering) involves merging scattered and irregular input audio frames into a temporary buffer, dividing them into blocks of fixed size (e.g., 441 samples / block for a sampling rate of 22050Hz), and assigning a continuously increasing timestamp to each audio block to resolve timestamp calculation errors caused by uneven audio frame sizes.

[0180] Audio and video queue management includes storing corrected video frames into the video queue (video_queue) and segmented audio frames into the audio queue (audio_queue), and outputting them to the downstream transmission track according to the first-in-first-out rule to ensure the correct timing of the frames.

[0181] Condition variable management includes maintaining global state variables for sent audio time and sent video time, as well as condition variables for audio time and video time.

[0182] The AudioSendTrack is used to retrieve audio frames to be sent sequentially from the audio queue, convert them to the WebRTC-compatible standard audio frame format, and set the correct PTS (display timestamp), time base, sampling rate, and other parameters. Then, following the rule that video_time >= audio_time - 0.04, audio frames are allowed to be sent if the time of the sent video is greater than or equal to the time of the current audio to be sent - 40ms (synchronization threshold). If the condition is not met, the system blocks and waits until the video sending progress is updated to the required level, preventing audio from leading and causing audio-visual desynchronization. After a successful audio frame transmission, the global sent audio time variable is updated, and the video send track is notified via the condition variable to trigger synchronization verification on the video end. Finally, the verified and standardized audio frame is transmitted to the WebRTC / RTMP transmission module.

[0183] The VideoSendTrack is used to retrieve video frames to be sent sequentially from the video queue, convert them to the WebRTC-compatible standard video frame format, and set the correct PTS, time base, frame rate, and other parameters. Following the rule that audio_time >= video_time - 0.04, that is, when the time of the sent audio is greater than or equal to the time of the current video to be sent - 40ms (synchronization threshold), video frames are allowed to be sent. If the condition is not met, the system blocks and waits via a condition variable until the audio sending progress is updated to the required level, preventing audio-visual asynchrony caused by video being ahead or behind. After a video frame is successfully sent, the global sent video time variable is updated, and the audio send track is notified via a condition variable to trigger synchronization verification at the audio end. Finally, the verified and standardized video frames are transmitted to the WebRTC / RTMP transmission module.

[0184] The WebRTC / RTMP transmission module is used to receive standardized audio and video frames sent from the audio and video transmission tracks, encapsulate, encode, and package them according to the WebRTC (real-time interactive scenarios, such as video calls and real-time digital human interaction) or RTMP (live streaming scenarios) protocols, and send them to the playback end over the network.

[0185] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0186] Based on the same inventive concept, this application also provides an audio-video synchronization device for implementing the audio-video synchronization method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more audio-video synchronization device embodiments provided below can be found in the limitations of the audio-video synchronization method described above, and will not be repeated here.

[0187] In one exemplary embodiment, such as Figure 6 As shown, an audio-video synchronization device is provided. This device is applied at the transmitting end and includes:

[0188] The acquisition module 10 is used to acquire target audio data and target video data;

[0189] Module 20 is used to determine the current audio time and the current video time;

[0190] The sending module 30 is used to send target audio data and / or target video data to the playback end based on the current audio time, the current video time, and a preset synchronization threshold, provided that the sending conditions are met; wherein, the sending conditions are used to limit the difference between the sending progress of the video data and the sending progress of the audio data to within the preset synchronization threshold range.

[0191] In the above embodiments, the system determines whether the transmission conditions are met based on the current audio time, the current video time, and a preset synchronization threshold. Only when the transmission conditions are met is the audio data and / or video data sent to the playback end. By controlling the transmission progress of audio and video data according to the transmission conditions, the time difference between video frames and audio frames receiving the same timestamp at the playback end is kept within the synchronization threshold range. This avoids the problem that when the cumulative time deviation of audio and video data arriving at the playback end at the same timestamp exceeds the synchronization threshold due to independent transmission and differences in generation speed, the playback end cannot synchronously play the audio and video.

[0192] In one embodiment, the device further includes a buffer module for:

[0193] The initial audio and video data output by the renderer are placed into a buffer; the initial audio data in the buffer is corrected to obtain the target audio data, and the initial video data is corrected to obtain the target video data.

[0194] In one embodiment, the target audio data includes a target audio frame and a target audio timestamp, the initial audio data includes at least an initial audio frame, and the buffer module is specifically used for:

[0195] The initial audio frame is merged with the audio frames in the buffer to obtain a merged audio frame. If the first data length of the merged audio frame is greater than or equal to the second data length of the unit audio frame, the target number of target audio frames is extracted from the merged audio frame. The target audio timestamp corresponding to each target audio frame is determined according to the audio frame interval.

[0196] In one embodiment, the buffer module is specifically used for:

[0197] For each non-first target audio frame, the sum of the target audio timestamp of the previous target audio frame adjacent to the target audio frame and the audio frame interval is used as the target audio timestamp of the target audio frame; for the first target audio frame, the audio frame interval is used as the target audio timestamp of the target audio frame.

[0198] In one embodiment, the target video data includes a target video frame and a target video timestamp; the initial video data includes an initial video frame and an initial video timestamp; the buffer module is specifically used for:

[0199] If the initial video frame is not the first frame, the reference timestamp of the initial video frame is determined based on the target video timestamp and video frame interval of the previous initial video frame.

[0200] The later timestamp between the reference timestamp and the initial timestamp of the initial video frame is used as the target video timestamp of the initial video frame, and the initial video frame is used as the target video frame.

[0201] If the initial video frame is the first frame, then the initial video timestamp is used as the target video timestamp, and the initial video frame is used as the target video frame.

[0202] In one embodiment, the synchronization threshold is less than or equal to the video frame interval; wherein the video frame interval is determined based on the frame rate of the video frame.

[0203] In one embodiment, the determining module 20 is specifically used for:

[0204] Obtain the number of audio frames sent and the sampling rate of the audio frames; determine the current audio time based on the number of audio frames and the sampling rate of the audio frames; obtain the number of video frames sent and the frame rate of the video frames; determine the current video time based on the number of video frames and the frame rate of the video frames.

[0205] Each module in the aforementioned audio-visual synchronization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0206] In one exemplary embodiment, an audio-video synchronization system is provided, the system comprising:

[0207] The sending end is used to acquire target audio data and target video data; and determine the current audio time and current video time; based on the current audio time, the current video time, and a preset synchronization threshold, and when it is determined that the sending conditions are met, the sending end sends the target audio data and / or the target video data to the playback end; wherein, the sending conditions are used to limit the difference between the sending progress of the target video data and the sending progress of the target audio data to be within the preset synchronization threshold range;

[0208] The playback device is used to synchronize the received audio and video data.

[0209] In the above embodiments, the system determines whether the transmission conditions are met based on the current audio time, the current video time, and a preset synchronization threshold. Only when the transmission conditions are met is the audio data and / or video data sent to the playback end. By controlling the transmission progress of audio and video data according to the transmission conditions, the time difference between video frames and audio frames receiving the same timestamp at the playback end is kept within the synchronization threshold range. This avoids the problem that when the cumulative time deviation of audio and video data arriving at the playback end at the same timestamp exceeds the synchronization threshold due to independent transmission and differences in generation speed, the playback end cannot synchronously play the audio and video.

[0210] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores audio and video data to be transmitted. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an audio and video synchronization method.

[0211] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0212] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0213] Acquire the target audio data and the target video data;

[0214] Determine the current audio time and the current video time;

[0215] Based on the current audio time, the current video time, and a preset synchronization threshold, the target audio data and / or target video data are sent to the playback device when the sending conditions are met. The sending conditions are used to limit the difference between the sending progress of the video data and the sending progress of the audio data to within the preset synchronization threshold range.

[0216] In one embodiment, the target audio data includes a target audio frame and a target audio timestamp, the initial audio data includes at least an initial audio frame, and the processor, when executing the computer program, further implements the following steps:

[0217] The initial audio frame is merged with the audio frames in the buffer to obtain a merged audio frame. If the first data length of the merged audio frame is greater than or equal to the second data length of the unit audio frame, the target number of target audio frames is extracted from the merged audio frame. The target audio timestamp corresponding to each target audio frame is determined according to the audio frame interval.

[0218] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0219] For each non-first target audio frame, the sum of the target audio timestamp of the previous target audio frame adjacent to the target audio frame and the audio frame interval is used as the target audio timestamp of the target audio frame; for the first target audio frame, the audio frame interval is used as the target audio timestamp of the target audio frame.

[0220] In one embodiment, the target video data includes a target video frame and a target video timestamp; the initial video data includes an initial video frame and an initial video timestamp; the processor, when executing the computer program, further implements the following steps:

[0221] If the initial video frame is not the first frame, then the reference timestamp of the initial video frame is determined based on the target video timestamp of the previous initial video frame and the video frame interval; the later timestamp between the reference timestamp and the initial timestamp of the initial video frame is taken as the target video timestamp of the initial video frame, and the initial video frame is taken as the target video frame; if the initial video frame is the first frame, then the initial video timestamp is taken as the target video timestamp, and the initial video frame is taken as the target video frame.

[0222] In one embodiment, the synchronization threshold is less than or equal to the video frame interval; wherein the video frame interval is determined based on the frame rate of the video frame.

[0223] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0224] Get the number of audio frames sent and their sampling rate; determine the current audio time based on the number of audio frames and their sampling rate; get the number of video frames sent and their frame rate; determine the current video time based on the number of video frames and their frame rate.

[0225] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0226] Acquire the target audio data and the target video data;

[0227] Determine the current audio time and the current video time;

[0228] Based on the current audio time, the current video time, and a preset synchronization threshold, the target audio data and / or target video data are sent to the playback device when the sending conditions are met. The sending conditions are used to limit the difference between the sending progress of the video data and the sending progress of the audio data to within the preset synchronization threshold range.

[0229] In one embodiment, the target audio data includes a target audio frame and a target audio timestamp, the initial audio data includes at least an initial audio frame, and the computer program, when executed by a processor, further implements the following steps:

[0230] The initial audio frame is merged with the audio frames in the buffer to obtain a merged audio frame. If the first data length of the merged audio frame is greater than or equal to the second data length of the unit audio frame, the target number of target audio frames is extracted from the merged audio frame. The target audio timestamp corresponding to each target audio frame is determined according to the audio frame interval.

[0231] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0232] For each non-first target audio frame, the sum of the target audio timestamp of the previous target audio frame adjacent to the target audio frame and the audio frame interval is used as the target audio timestamp of the target audio frame; for the first target audio frame, the audio frame interval is used as the target audio timestamp of the target audio frame.

[0233] In one embodiment, the target video data includes target video frames and target video timestamps; the initial video data includes initial video frames and initial video timestamps; and the computer program, when executed by a processor, further implements the following steps:

[0234] If the initial video frame is not the first frame, then the reference timestamp of the initial video frame is determined based on the target video timestamp of the previous initial video frame and the video frame interval; the later timestamp between the reference timestamp and the initial timestamp of the initial video frame is taken as the target video timestamp of the initial video frame, and the initial video frame is taken as the target video frame; if the initial video frame is the first frame, then the initial video timestamp is taken as the target video timestamp, and the initial video frame is taken as the target video frame.

[0235] In one embodiment, the synchronization threshold is less than or equal to the video frame interval; wherein the video frame interval is determined based on the frame rate of the video frame.

[0236] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0237] Get the number of audio frames sent and their sampling rate; determine the current audio time based on the number of audio frames and their sampling rate; get the number of video frames sent and their frame rate; determine the current video time based on the number of video frames and their frame rate.

[0238] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0239] Acquire the target audio data and the target video data;

[0240] Determine the current audio time and the current video time;

[0241] Based on the current audio time, the current video time, and a preset synchronization threshold, the target audio data and / or target video data are sent to the playback device when the sending conditions are met. The sending conditions are used to limit the difference between the sending progress of the video data and the sending progress of the audio data to within the preset synchronization threshold range.

[0242] In one embodiment, the target audio data includes a target audio frame and a target audio timestamp, the initial audio data includes at least an initial audio frame, and the computer program, when executed by a processor, further implements the following steps:

[0243] The initial audio frame is merged with the audio frames in the buffer to obtain a merged audio frame. If the first data length of the merged audio frame is greater than or equal to the second data length of the unit audio frame, the target number of target audio frames is extracted from the merged audio frame. The target audio timestamp corresponding to each target audio frame is determined according to the audio frame interval.

[0244] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0245] For each non-first target audio frame, the sum of the target audio timestamp of the previous target audio frame adjacent to the target audio frame and the audio frame interval is used as the target audio timestamp of the target audio frame; for the first target audio frame, the audio frame interval is used as the target audio timestamp of the target audio frame.

[0246] In one embodiment, the target video data includes target video frames and target video timestamps; the initial video data includes initial video frames and initial video timestamps; and the computer program, when executed by a processor, further implements the following steps:

[0247] If the initial video frame is not the first frame, then the reference timestamp of the initial video frame is determined based on the target video timestamp of the previous initial video frame and the video frame interval; the later timestamp between the reference timestamp and the initial timestamp of the initial video frame is taken as the target video timestamp of the initial video frame, and the initial video frame is taken as the target video frame; if the initial video frame is the first frame, then the initial video timestamp is taken as the target video timestamp, and the initial video frame is taken as the target video frame.

[0248] In one embodiment, the synchronization threshold is less than or equal to the video frame interval; wherein the video frame interval is determined based on the frame rate of the video frame.

[0249] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0250] Get the number of audio frames sent and their sampling rate; determine the current audio time based on the number of audio frames and their sampling rate; get the number of video frames sent and their frame rate; determine the current video time based on the number of video frames and their frame rate.

[0251] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0252] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0253] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0254] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An audio-video synchronization method, characterized in that, The method is applied at the sending end, and the method includes: Acquire the target audio data and the target video data; Determine the current audio time and the current video time; Based on the current audio time, the current video time, and a preset synchronization threshold, the target audio data and / or the target video data are sent to the playback terminal when the sending conditions are met; wherein, the sending conditions are used to limit the difference between the sending progress of the target video data and the sending progress of the target audio data to within the preset synchronization threshold range.

2. The method according to claim 1, characterized in that, The method further includes: Place the initial audio and initial video data output by the renderer into the buffer; The initial audio data in the buffer is corrected to obtain target audio data, and the initial video data is corrected to obtain target video data.

3. The method according to claim 2, characterized in that, The target audio data includes a target audio frame and a target audio timestamp. The initial audio data includes at least an initial audio frame. The initial audio data in the buffer is corrected to obtain the target audio data, including: The initial audio frame is merged with the audio frames in the buffer to obtain a merged audio frame; If the first data length of the merged audio frame is greater than or equal to the second data length of a unit audio frame, then the target number of target audio frames are extracted from the merged audio frame. Based on the audio frame interval, determine the target audio timestamp corresponding to each target audio frame.

4. The method according to claim 3, characterized in that, The step of determining the target audio timestamp corresponding to each target audio frame based on the audio frame interval includes: For each non-first target audio frame, the sum of the target audio timestamp of the previous target audio frame adjacent to the target audio frame and the audio frame interval is used as the target audio timestamp of the target audio frame. For the first target audio frame, the audio frame interval is used as the target audio timestamp of the target audio frame.

5. The method according to claim 2, characterized in that, The target video data includes target video frames and target video timestamps; The initial video data includes initial video frames and initial video timestamps; The initial video data is corrected to obtain the target video data, including: If the initial video frame is not the first frame, then the reference timestamp of the initial video frame is determined based on the target video timestamp and video frame interval of the previous initial video frame. The later timestamp between the reference timestamp and the initial timestamp of the initial video frame is used as the target video timestamp of the initial video frame, and the initial video frame is used as the target video frame. If the initial video frame is the first frame, then the initial video timestamp is used as the target video timestamp, and the initial video frame is used as the target video frame.

6. The method according to any one of claims 1-5, characterized in that, The synchronization threshold is less than or equal to the video frame interval; wherein the video frame interval is determined according to the frame rate of the video frame.

7. The method according to any one of claims 1-5, characterized in that, Determining the current audio time and current video time includes: Get the number of audio frames sent and the sampling rate of the audio frames; The current audio time is determined based on the number of audio frames and the sampling rate of the audio frames; Get the number of video frames sent and the frame rate of the video frames; The current video time is determined based on the number of video frames and the frame rate of the video frames.

8. A transmitter, characterized in that, include: The audio transmission track module is used to acquire target audio data; The video transmission track module is used to acquire target video data; The renderer buffer module is used to determine the current audio time and the current video time, and to determine whether the sending conditions are met based on the current audio time, the current video time, and a preset synchronization threshold. An audio transmission track module is used to send the target audio data to the playback end when the transmission conditions are met; And / or, The video transmission track module is used to send the target video data to the playback terminal when the transmission conditions are met.

9. The transmitting end according to claim 8, characterized in that, The transmitting end also includes: The renderer module is used to output initial video and initial audio data; The renderer buffer module is also used to put the initial audio data and initial video data output by the renderer into a buffer, and to correct the initial audio data in the buffer to obtain target audio data, and to correct the initial video data to obtain target video data.

10. An audio-visual synchronization system, characterized in that, The system includes: The sending end is used to acquire target audio data and target video data; and determine the current audio time and current video time; based on the current audio time, the current video time, and a preset synchronization threshold, and when it is determined that the sending conditions are met, the sending end sends the target audio data and / or the target video data to the playback end; wherein, the sending conditions are used to limit the difference between the sending progress of the target video data and the sending progress of the target audio data to be within the preset synchronization threshold range; The playback device is used to synchronize the received audio and video data.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images