Local area network audio and video synchronization method and system, storage medium and electronic equipment

Abstract

The invention discloses a local area network audio and video synchronization method and system, a storage medium and electronic equipment. The method comprises the steps that a sending end adds an original timestamp based on the same clock source to an audio data packet and a video frame and then sends the audio data packet and the video frame; a receiving end establishes an absolute playing moment sPTS0 by taking an arrival moment of a first audio packet as a reference, an audio playing timeline is recursively reconstructed according to an audio timestamp difference value, a video timestamp is mapped to the timeline to obtain a video playing moment, finally synchronous output is realized, and the system comprises a display terminal and a mobile control terminal for realizing the functions. According to the method, the playing time sequence is reconstructed by taking the audio stream as the leading part, the problem of audio and video asynchronization caused by network delay difference in a local area network environment is effectively solved, and the method has the advantages of high synchronization precision, high network jitter resistance and low implementation complexity, and is suitable for audio and video return scenes among equipment such as smart televisions and mobile terminals.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and specifically to a method, system, storage medium, and electronic device for synchronizing audio and video on a local area network. Background Technology

[0002] With the widespread adoption of wireless LAN technology and the improvement of smart terminal performance, transmitting audio and video content from large-screen display devices such as smart TVs and projectors back to mobile control terminals such as mobile phones and tablets in real time for operation or auxiliary display has become a common requirement in smart homes, corporate presentations, and interactive education scenarios. In such applications, audio and video data is typically streamed from the display terminal to the mobile control terminal via a local area network.

[0003] However, in existing technical solutions, audio and video data are typically encoded and encapsulated independently at the sending end and transmitted through different network channels or data packets. During transmission, they experience varying network latency, jitter, and processing paths. When they reach the mobile control terminal, although the audio and video streams may remain internally continuous, their relative timing relationship often drifts, leading to audio-visual desynchronization during playback and severely impacting the user experience.

[0004] To address the synchronization problem, existing technologies have proposed several solutions. For example, a common approach is to require the sending end to insert a high-precision, unified global timestamp during audio and video encapsulation, which the receiving end uses for synchronization during playback. However, this method requires extremely high system clock synchronization accuracy between the sending and receiving ends, making it difficult to implement in local area networks composed of ordinary consumer devices, and it cannot effectively combat random latency caused by network transmission. Another solution, such as Chinese Patent Publication No. CN111131917A, provides a real-time audio spectrum synchronization method. It analyzes audio PCM data frames, calculates the duration of each frame, and recursively generates the timestamp of the spectrum graph to achieve synchronization between the spectrum graph and audio playback. However, this solution mainly addresses the problem of synchronized rendering of audio and derived visual elements within a single device, with its synchronization benchmark derived from the processing time of the first frame of data within the device. It does not address, and cannot solve, the synchronization problem between independent audio and video streams caused by network uncertainties during cross-network and cross-device transmission. Its technical concept and application scenario are fundamentally different from the audio and video cross-device network synchronization problem that this invention aims to solve.

[0005] Therefore, the industry urgently needs a technical solution that is suitable for ordinary local area network environments, has no strict requirements for device clock synchronization, and can effectively combat network latency and jitter, thereby achieving high-precision, cross-device real-time audio and video synchronization. Summary of the Invention

[0006] This invention provides a method, system, storage medium, and electronic device for local area network audio and video synchronization. This solution achieves high-precision, real-time audio and video synchronization across devices in a normal local area network environment without stringent requirements for device clock synchronization and can effectively combat network latency and jitter.

[0007] To solve the above problems, the present invention is achieved through the following technical solution: First aspect: A local area network audio and video synchronization method, applied to a communication system including a display terminal and a mobile control terminal, comprising: The sending end processing steps are executed by the display terminal: The audio data stream is encapsulated into several numbered audio data packets, and each audio data packet is assigned a raw audio timestamp aPTS_n based on the sender's clock. The video data stream is split into several numbered video frames, and each video frame is assigned an original video timestamp vPTS_m based on the same sender clock. The audio data packet and the video frame are sent to the mobile control terminal via a local area network; The receiving end synchronization step is performed by the mobile control terminal: Taking the arrival event of the first successfully received audio data packet as the time origin, record the local system clock value when it enters the local buffer, and use it as the absolute playback reference time sPTS_0; Based on the difference between the absolute playback reference time sPTS_0 and the original audio timestamp aPTS_n, the audio playback timeline at the receiving end is reconstructed, wherein the playback time sPTS_n of the nth audio data packet satisfies: sPTS_n = sPTS_{n-1} + (aPTS_n - aPTS_{n-1}); Based on the absolute playback reference time sPTS_0, the original audio timestamp aPTS_0 of the first audio data packet, and the original video timestamp vPTS_m of the video frame, the original timestamp of the video frame is mapped to the reconstructed audio playback timeline, wherein the playback time sPTS_m of the m-th video frame satisfies: sPTS_m = sPTS_0 + (vPTS_m - aPTS_0); Based on the playback times sPTS_n and sPTS_m, audio data packets and video frames are output synchronously.

[0008] In a preferred embodiment of the present invention, the local area network is a low-latency, non-time-guaranteed IP network, and the audio data packets and video frames are transmitted in the form of independent UDP data streams or RTP streams, and out-of-order arrival is allowed.

[0009] A preferred embodiment of the present invention: if the playback time sPTS_m of a video frame is earlier than the current audio playback timeline, and the difference Δt falls within a preset advance threshold window T_pre (Δt∈[–T_pre, 0)), then the video frame is temporarily stored in the pre-rendering buffer and output when its playback time arrives; If Δt ≥ 0, then wait for playback normally; If Δt < –T_pre, the video frame is determined to be expired, and a discard operation is performed and the frame drop event is recorded, or a frame replacement strategy is triggered to insert a duplicate frame.

[0010] A preferred embodiment of the present invention: The original audio timestamp aPTS_n and the original video timestamp vPTS_m are generated by the encoder or encapsulator of the display terminal during the acquisition, encoding or encapsulation stage according to the same system clock counter, thereby ensuring that the original audio timestamp sequence {aPTS_n} and the original video timestamp sequence {vPTS_m} are under the same time base and clock frequency.

[0011] A preferred embodiment of the present invention: The mobile control terminal maintains a logical clock locally corresponding to the audio playback timeline. The advancement of this logical clock during playback depends only on the difference in the original audio timestamps carried in the continuous audio data packets. The arrival or absence of video frames does not affect the continuity and rate of the logical clock. In the event of network jitter or temporary loss of audio packets, the logical clock can remain continuous based on local clock interpolation and be smoothly corrected after subsequent audio packets arrive.

[0012] The second aspect: a local area network audio and video synchronization system, comprising: The display terminal is equipped with: The audio encapsulation module is used to generate audio data packets with sequence numbers and original audio timestamps aPTS_n from the audio data stream; The video splitting module is used to generate video frames with sequence numbers and original video timestamps vPTS_m from the video data stream; The network transmission module is used to transmit the audio data packets and the video frames via a local area network; The mobile control terminal is equipped with: A network receiving module is used to receive the audio data packets and the video frames; The clock reference module is configured to be triggered by the arrival of the first successfully received audio data packet, and record the local system clock value as the absolute playback reference time sPTS_0; The audio clock module is used to reconstruct and maintain the audio playback timeline based on the difference in the original audio timestamps of consecutive audio data packets, starting from sPTS_0. The video synchronization module is used to map the original video timestamp of each video frame to the audio playback timeline according to the formula sPTS_m = sPTS_0 + (vPTS_m – aPTS_0) to obtain its playback time sPTS_m. The caching and rendering module is used to synchronously output audio and video at the corresponding playback time.

[0013] A preferred embodiment of the present invention: the caching and rendering module includes: The pre-rendered buffer is used to store video frames whose playback time is earlier than the current audio playback timeline but are within the advance threshold window; The frame management unit is used to perform frame dropping or frame replacement operations when video frames expire, and to record synchronization quality statistics. The clock maintenance unit is used to smooth the audio playback timeline when the audio stream resumes after an interruption.

[0014] In a preferred embodiment of the present invention, the audio clock module and the video synchronization module are implemented by hardware logic circuits, firmware, or software instruction sets running on the processing core; or, part of their timestamp calculation and mapping functions are deployed on edge computing nodes or cloud servers, and synchronization instructions are sent to the local machine through a low-latency control link.

[0015] Third aspect: A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0016] Fourth aspect: An electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0017] Beneficial effects: Compared with the prior art, the solution provided by the present invention has the following beneficial effects: 1. Improve synchronization accuracy By reconstructing the playback timeline based on the first audio data packet received at the receiving end and driving the clock forward with the audio timestamp difference, the audio and video outputs are kept strictly synchronized in relative timing, effectively improving the audio-visual misalignment problem caused by network path differences in traditional solutions.

[0018] 2. Enhance anti-interference capabilities Using a synchronization mechanism based on timestamp differences rather than absolute packet arrival times can, to some extent, resist network latency fluctuations and brief jitters, thus improving synchronization stability under non-ideal network conditions.

[0019] 3. Reduce implementation complexity The core of the synchronization algorithm is recursive addition and difference calculation, which has low computational complexity and does not require complex global clock synchronization or large-scale buffer management, making it easy to deploy and implement on mobile terminals with limited resources.

[0020] 4. Clearly define the application's target audience. The solution is specifically designed for low-latency, non-time-guaranteed IP networks and UDP / RTP transmission scenarios, solving the practical problem of achieving reliable audio and video synchronization in such LAN environments.

[0021] 5. Possesses a sound processing mechanism By employing mechanisms such as pre-rendering caching, frame management, and clock correction, the system can effectively handle abnormal situations such as premature or expired video frames and interrupted audio streams, thereby improving the overall robustness of the system. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will describe the technical solutions used in the embodiments. The accompanying drawings are briefly described below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a local area network audio and video synchronization method according to the present invention; Figure 2 This is a hardware architecture diagram of a local area network audio and video synchronization system according to the present invention. Detailed Implementation

[0024] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] Example 1: Basic Method Implementation In advertising applications, when a display terminal (such as a smart TV) is playing an advertisement, the user may want the current audio and video feed to be transmitted in real time to their handheld mobile control terminal (such as a smartphone) for operation or as a second screen viewing. The audio and video synchronization method of this invention can be applied to this scenario.

[0026] Step S101: Processing steps at the sending end The display terminal, acting as the sending end, performs the following operations: First, the raw audio data stream is processed. The audio encoder encapsulates the compressed audio data or uncompressed PCM data according to the network transmission MTU size or fixed time slices, such as 20ms packets, generating a series of numbered audio data packets. Simultaneously, the audio encapsulation module obtains the current time from the display terminal's system clock and assigns an original audio timestamp aPTS_n to each generated audio data packet. This timestamp reflects the theoretical playback time of the data within that audio data packet on the original playback timeline.

[0027] Secondly, the original video data stream is processed. The video encoder splits the compressed video data into frames, resulting in a series of video frames. The video splitting module, also based on the same system clock counter as the audio module, assigns an original video timestamp vPTS_m to each video frame. For example, vPTS_m of a video frame can represent the display time of that frame in the original playback sequence. Since the audio and video timestamps originate from the same clock source, the consistency of the two sequences {aPTS_n} and {vPTS_m} in terms of time base and frequency is guaranteed.

[0028] Finally, the network sending module sends the encapsulated audio data packets and video frames, each with its own timestamp and sequence number, out over the local area network as independent UDP packets or RTP streams. To reduce latency, the UDP protocol is typically used; therefore, the network does not guarantee the reliable and ordered arrival of data packets.

[0029] Step S102: Receiver synchronization steps, such as... Figure 1 As shown: The mobile control terminal, acting as the receiving end, performs the following operations: 1. Establishing a Reference: The network receiving module receives data packets from the network. When the first audio data packet successfully arrives and is recognized, the clock reference module is immediately triggered. It records the local system clock value of the mobile control terminal at this moment and sets this value as the absolute playback reference time sPTS_0. This sPTS_0 is the absolute starting point for all subsequent playback timings at the receiving end.

[0030] 2. Reconstructing the Audio Playback Timeline: The audio clock module begins operation. Starting with sPTS_0, it recursively reconstructs an audio playback timeline using the difference between aPTS_n carried by consecutively arriving audio packets as the "scale." Specifically, for the first audio packet, its playback time is sPTS_0. For the nth audio packet (n>1), its playback time sPTS_n = sPTS_{n-1} + (aPTS_n - aPTS_{n-1}). This formula means that the playback time interval of the nth packet relative to the (n-1)th packet is equal to their interval on the original timeline at the sending end. In this way, the receiving end reconstructs a playback plan based on local absolute time that strictly corresponds to the original audio timing at the sending end. Importantly, the advancement of this timeline depends only on the timestamp information carried by the audio packets themselves and is unaffected by network fluctuations.

[0031] 3. Mapping Video Playback Time: For each received video frame (with vPTS_m), the video synchronization module performs a mapping calculation. Its playback time sPTS_m = sPTS_0 + (vPTS_m - aPTS_0). Here, aPTS_0 is the original timestamp of the first audio packet. This formula maps the original timestamp vPTS_m of the video frame to the audio playback timeline reconstructed in the previous step, based on its relative relationship with the audio reference aPTS_0. Essentially, it ensures that the relative playback timing of video frames and audio packets at the receiving end remains consistent with their original relative timing at the sending end.

[0032] 4. Synchronous Output: The caching and rendering module maintains the audio and video playback queues. When the system's current time reaches the spTS_n of an audio packet or the spTS_m of a video frame, it retrieves the data from the corresponding queue and calls the audio driver or graphics rendering interface for playback / display, thereby achieving synchronous audio and video output.

[0033] Example 2: Anomaly Handling and Optimized Implementation Based on the basic method, this invention also provides an optimized handling mechanism for network anomalies and playback anomalies.

[0034] Synchronization processing for video frames: In real-world networks, video frames may arrive at a time that deviates from the calculated playback time due to encoding complexity, network congestion, or other factors. The frame management unit within the caching and rendering module handles this issue. Slightly Early Arrival: When calculations reveal that the playback time of a video frame, sPTS_m, is slightly earlier than the current audio playback timeline (i.e., Δt = sPTS_m - current audio time falls within the interval [-T_pre, 0)), where T_pre is, for example, set to 30ms, it is determined to be a "waitable" frame. This frame will be stored in the pre-rendering buffer. Once the current audio time advances to a point equal to sPTS_m, this frame is immediately retrieved and rendered. This avoids playback jitter that may be caused by slightly early video frame arrival.

[0035] Normal or delayed arrival: If Δt ≥ 0, it means that the video frame playback time has not arrived, so wait normally.

[0036] Severely delayed arrival: If Δt < -T_pre, it means that the video frame is severely delayed and has missed its playback window. In this case, the frame management unit can choose to discard the frame directly or trigger a frame interpolation strategy, such as repeating the previous frame, to maintain the continuity of the picture and avoid video playback stuttering due to waiting for expired frames.

[0037] Logical clock maintenance for audio stream interrupts: In some cases, audio data packets may be temporarily lost or severely delayed due to network problems. In such situations, the "logical clock" maintained in the audio clock module must not be idle. The advancement of this logical clock should ideally rely on a continuous (aPTS_n - aPTS_{n-1}) cycle. When no new audio packets are received within the expected timeframe, the clock maintenance unit is activated. It interpolates based on the mobile terminal's local, highly stable hardware clock, advancing the logical clock at a predicted, smooth rate to maintain playback continuity. When subsequent audio packets arrive again, the module compares the interpolated logical clock time with the theoretical time calculated based on the timestamp of the newly arrived audio packet. If a discrepancy exists, a smoothing correction algorithm is employed—for example, gradually adjusting the clock frequency over a short period rather than abruptly jumping—to slowly bring the logical clock back on track, avoiding perceptible abrupt changes in playback speed or tone.

[0038] Example 3: System Hardware and Deployment Implementation Figure 2 This is a hardware architecture diagram of a local area network audio and video synchronization system according to the present invention.

[0039] Display terminals, such as smart TVs, have software or firmware running on their internal processors that include audio encapsulation modules, video splitting modules, and network transmission modules. These modules can be implemented using general-purpose CPU cores or dedicated multimedia encoding hardware accelerators.

[0040] A mobile control terminal, such as a smartphone, has a network receiving module implemented using a Wi-Fi chip and driver. The clock reference module, audio clock module, video synchronization module, and buffering and rendering module can be implemented in various forms. Software implementation: As part of the application, it runs on the phone's application processor and uses the multimedia framework provided by the operating system for final rendering.

[0041] Hardware-assisted implementation: Some computationally intensive or low-latency-requirement tasks, such as timestamp difference calculation and mapping calculation, can be performed by the digital signal processor or dedicated multimedia processing unit on the mobile phone.

[0042] Hybrid Implementation: Some complex timestamp calculations and global synchronization strategies can be deployed on edge computing nodes in the home network or on cloud servers. The mobile control terminal uploads the received initial timestamp information through a low-latency control channel. After the cloud completes the complex synchronization calculations, it sends precise playback commands to the terminal. This approach can reduce the computational load on the terminal and potentially leverage the more powerful computing capabilities of the cloud to achieve a better global synchronization strategy.

[0043] Example 4: Implementation of Storage Media and Electronic Devices This invention can also be implemented through a program product. A computer-readable storage medium, such as a USB flash drive, solid-state drive, or mobile phone flash memory, stores a computer program thereon. When the storage medium is connected to an electronic device, such as the aforementioned smartphone, tablet computer, or even smart TV itself, the processor of the electronic device reads and executes the program from the memory, thereby controlling the electronic device to implement all or part of the method steps described in Embodiments 1 and 2, thus enabling it to possess the audio and video synchronization function of this invention.

[0044] The computer-readable storage medium described in this invention is a tangible entity containing computer-executable instructions. Its core lies in storing specific program code so that when this code is read and executed by the processor of a computing device, it can fully implement all the steps and effects of the video image angle adjustment method described in the first aspect of this invention. The computer-readable storage medium can take common forms, including but not limited to: universal serial bus flash drives, portable hard drives, read-only memory, random access memory, optical discs, magnetic disks, and various memory cards based on flash memory technology. Furthermore, in specific application scenarios, the medium can also be a storage chip specifically designed for embedded systems, such as erasable programmable read-only memory or flash memory. All these media possess non-transitory characteristics, enabling them to stably store the computer program they carry for a long period. The medium stores a specific computer program, which consists of a series of instructions that can be directly or indirectly executed by a processor. When the medium is connected to or installed in a computing device, i.e., the electronic terminal described in the third aspect of this invention, the stored program is loaded into the device's memory. By storing and executing the aforementioned program, the computer-readable storage medium transforms a general-purpose computing device into a dedicated device capable of executing the specific method of this invention. In summary, the computer-readable storage medium of the present invention is a functional product that differs from traditional media. By storing specific computer programs, it provides a reliable and convenient carrier for implementing the method of the present invention.

[0045] The illustrative, for example, modular division is merely a logical functional division; in actual implementation, there may be other division methods. For instance, multiple modules or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through interfaces, devices, apparatuses, or modules, or they may be electrical, mechanical, or other forms of connection.

[0046] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the objectives of the embodiments of the present invention, depending on actual needs.

[0047] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0048] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0049] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A local area network audio and video synchronization method, applied to a communication system including a display terminal and a mobile control terminal, characterized in that, include: The sending end processing steps are executed by the display terminal: The audio data stream is encapsulated into several numbered audio data packets, and each audio data packet is assigned a raw audio timestamp aPTS_n based on the sender's clock. The video data stream is split into several numbered video frames, and each video frame is assigned an original video timestamp vPTS_m based on the same sender clock. The audio data packet and the video frame are sent to the mobile control terminal via a local area network; The receiving end synchronization step is performed by the mobile control terminal: Taking the arrival event of the first successfully received audio data packet as the time origin, record the local system clock value when it enters the local buffer, and use it as the absolute playback reference time sPTS_0; Based on the difference between the absolute playback reference time sPTS_0 and the original audio timestamp aPTS_n, the audio playback timeline at the receiving end is reconstructed, wherein the playback time sPTS_n of the nth audio data packet satisfies: sPTS_n = sPTS_{n-1} + (aPTS_n - aPTS_{n-1}); Based on the absolute playback reference time sPTS_0, the original audio timestamp aPTS_0 of the first audio data packet, and the original video timestamp vPTS_m of the video frame, the original timestamp of the video frame is mapped to the reconstructed audio playback timeline, wherein the playback time sPTS_m of the m-th video frame satisfies: sPTS_m = sPTS_0 + (vPTS_m - aPTS_0); Based on the playback times sPTS_n and sPTS_m, audio data packets and video frames are output synchronously.

2. The method according to claim 1, characterized in that, The local area network is a low-latency, non-time-guaranteed IP network. The audio data packets and video frames are transmitted in the form of independent UDP data streams or RTP streams, and out-of-order arrival is allowed.

3. The method according to claim 1, characterized in that, If the playback time of the video frame sPTS_m is earlier than the current audio playback timeline, and the difference Δt falls within the preset advance threshold window T_pre (Δt∈[–T_pre, 0)), then the video frame is temporarily stored in the pre-rendering buffer and output when its playback time arrives. If Δt ≥ 0, then wait for playback normally; If Δt < –T_pre, the video frame is determined to be expired, and a discard operation is performed and the frame drop event is recorded, or a frame replacement strategy is triggered to insert a duplicate frame.

4. The method according to claim 1, characterized in that, The original audio timestamp aPTS_n and the original video timestamp vPTS_m are generated by the encoder or encapsulator of the display terminal during the acquisition, encoding or encapsulation stage according to the same system clock counter, thereby ensuring that the original audio timestamp sequence {aPTS_n} and the original video timestamp sequence {vPTS_m} are under the same time base and clock frequency.

5. The method according to claim 1, characterized in that, The mobile control terminal maintains a logical clock locally corresponding to the audio playback timeline. The advancement of this logical clock during playback depends only on the difference in the original audio timestamps carried in the continuous audio data packets. The arrival or absence of video frames does not affect the continuity and rate of the logical clock. In the event of network jitter or temporary loss of audio packets, the logical clock can remain continuous based on local clock interpolation and be smoothly corrected after subsequent audio packets arrive.

6. A local area network audio and video synchronization system, characterized in that, include: The display terminal is equipped with: The audio encapsulation module is used to generate audio data packets with sequence numbers and original audio timestamps aPTS_n from the audio data stream; The video splitting module is used to generate video frames with sequence numbers and original video timestamps vPTS_m from the video data stream; The network transmission module is used to transmit the audio data packets and the video frames via a local area network; The mobile control terminal is equipped with: A network receiving module is used to receive the audio data packets and the video frames; The clock reference module is configured to be triggered by the arrival of the first successfully received audio data packet, and record the local system clock value as the absolute playback reference time sPTS_0; The audio clock module is used to reconstruct and maintain the audio playback timeline based on the difference in the original audio timestamps of consecutive audio data packets, starting from sPTS_0. The video synchronization module is used to map the original video timestamp of each video frame to the audio playback timeline according to the formula sPTS_m = sPTS_0 + (vPTS_m – aPTS_0) to obtain its playback time sPTS_m. The caching and rendering module is used to synchronously output audio and video at the corresponding playback time.

7. The system according to claim 6, characterized in that, The caching and rendering module includes: The pre-rendered buffer is used to store video frames whose playback time is earlier than the current audio playback timeline but are within the advance threshold window; The frame management unit is used to perform frame dropping or frame replacement operations when video frames expire, and to record synchronization quality statistics. The clock maintenance unit is used to smooth the audio playback timeline when the audio stream resumes after an interruption.

8. The system according to claim 6, characterized in that, The audio clock module and the video synchronization module are implemented using hardware logic circuits, firmware, or software instruction sets running on the processing core; or, some of their timestamp calculation and mapping functions are deployed on edge computing nodes or cloud servers, and synchronization instructions are sent to the local machine through a low-latency control link.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 5.

Images