Audio and video dynamic synchronization system and method based on playing sequence and resource index

Abstract

The invention discloses an audio and video dynamic synchronization system and method based on a playing sequence and a resource index. The audio and video dynamic synchronization system comprises a receiving module, a serial number generation module, a resource index setting module, a cache module, a synchronization judgment module and a clock switching module. The receiving module is used for receiving audio and video packets from multiple paths of audio and video data streams; the serial number generation module is used for distributing a globally unique and continuously increasing 64-bit serial number for each received audio and video packet; the resource index setting module is used for setting a resource index according to the source of the audio and video packet, the resource index increases progressively from 0, and the smaller the value is, the higher the priority is; and the cache module is used for respectively storing the audio and video packets into corresponding cache queues according to the resource indexes, and sorting the packets in each queue according to serial numbers, so as to construct a logic playing time axis at a receiving end.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of audio and video dynamic synchronization, and particularly relates to an audio and video dynamic synchronization system based on a play sequence and a resource index and a method thereof. BACKGROUND

[0002] The existing audio and video synchronization scheme (such as timestamp alignment-based, fixed threshold adjustment) has the following limitations: Stream discontinuity failure: when a certain audio / video stream suddenly stops or suddenly resumes or the timestamp is unreliable, the reference clock is lost / disordered, leading to the collapse of the synchronization reference; Poor dynamic frame rate adaptability: when the frame rate fluctuates in a small to large range (for example, 0.01~160fps, small fps for static pictures, large fps for dynamic pictures), the fixed threshold cannot adapt to the change of the rendering interval; Resolution switching and Seek (jump) operation destroy the timeline: user intervention in the playback process, codec time consumption mutation, and timestamp resetting will destroy the time sequence continuity; In the prior art, the audio and video synchronization scheme is mostly based on timestamp alignment. For example, the RTP protocol in RFC 3550 uses a timestamp for media synchronization, but this method is prone to failure when the stream is discontinuous. In addition, some schemes use sequence numbers for packet ordering within a single stream, such as PCR and PTS in MPEG-TS, but do not solve the cross-stream synchronization problem. US20180007316A1 proposes a synchronization method based on sequence numbers, but it is only applicable to continuous streams and does not involve dynamic priority in a multi-device scenario. Therefore, the prior art cannot effectively handle the synchronization problems in the scenarios of random stream interruption, frame rate fluctuation, and multi-device. SUMMARY

[0003] In view of the problems in the prior art, the present application provides an audio and video dynamic synchronization system based on a play sequence and a resource index.

[0004] To achieve the above purpose, the present application provides an audio and video dynamic synchronization system based on a play sequence and a resource index, comprising: a receiving module, a sequence number generation module, a resource index setting module, a cache module, a synchronization judgment module, and a clock switching module. The receiving module is configured to receive audio and video packets from multiple audio and video data streams. The sequence number generation module is configured to assign a globally unique and continuously increasing 64-bit sequence number to each received audio and video packet. The resource index setting module is configured to set a resource index according to the source of the audio and video packet, and the resource index starts from 0 and increases, and the smaller the value is, the higher the priority is. The cache module is used for storing audio and video packets into corresponding cache queues according to the resource index, and sorting the packets in each queue according to the sequence number, so as to construct a logical playing timeline at the receiving end; The synchronization judgment module is used for determining the audio and video packets to be rendered based on a three-level decision matrix, which compares the priority in the order of sequence number first, resource index second, and timestamp third. The clock switching module is used for switching between the audio dominant mode and the video dominant mode according to the length fluctuation of the current video frame, and calculating the rendering time of the audio and video frames according to the currently selected mode.

[0005] Preferably, the 64-bit counter in the sequence number generation module is looped back after reaching the maximum value, and a loop comparison algorithm is used when comparing the sequence numbers, and when the absolute value of the difference between two sequence numbers exceeds 2^63, it is determined that the sequence number is looped back and the sequence number with smaller difference is processed preferentially.

[0006] Preferably, in the audio dominant mode, the rendering time of the video frame = current audio time + audio frame average length × (video frame sequence number Audio frame sequence); in the video dominant mode, the video frame is rendered according to the average frame interval calculated dynamically, and the video playing rate is adjusted by the frame skipping or delay strategy to synchronize with the audio clock.

[0007] Preferably, the clock switching module calculates the frame rate fluctuation threshold based on the length of the last N video frames, and calculates the average frame length using the exponential moving average algorithm, and when the change rate of the current video frame length and the average frame length exceeds the preset threshold, the switching between the audio dominant mode and the video dominant mode is triggered.

[0008] Preferably, the resource index setting module assigns different resource indexes to audio and video data streams from different sources to realize the shunting of multiple audio and video data streams; the cache module stores the data packets into corresponding queues according to the different resource indexes, and when the sequence numbers are the same, the data packet with smaller resource index value is processed preferentially under the three-level decision matrix.

[0009] The application also provides an audio and video dynamic synchronization method based on playing sequence and resource index, comprising the following steps: Step 1: receiving audio packets and video packets of multiple audio and video data streams; Step 2: assigning a globally unique and continuously increasing 64-bit sequence number to each of the audio packets and video packets, and setting a resource index according to the data packet source information, the resource index being incremented from 0, and the smaller the value, the higher the priority; Step 3: The audio and video packets are sorted according to the sequence numbers, a logical playback timeline is constructed at the receiving end, and the audio and video packets are respectively placed into corresponding cache queues according to the resource index, and the packets in each cache queue are arranged in ascending order of sequence number; Step 4: The audio and video packets to be rendered are determined according to a three-level decision matrix each time rendering is performed, and the decision matrix performs priority comparison in sequence according to sequence number, resource index, and timestamp, that is, the packet with the smallest sequence number is selected first, the packet with the smaller resource index is selected when the sequence numbers are the same, and the packet with the earliest timestamp is selected when the first two levels are the same; Step 5: The clock mode is dynamically switched according to the current frame rate fluctuation, and the clock mode includes an audio dominant mode and a video dominant mode; in the audio dominant mode, the rendering time of a video frame is obtained by adding the current audio time and the product of the average duration of the audio frame and the difference between the sequence numbers of the video frame and the audio frame; in the video dominant mode, the video frame is rendered according to the average interframe interval dynamically calculated, and the video playback rate is adjusted by frame skipping or delay to follow the audio clock; when the change rate of the duration of the current video frame and the average duration of the last N frames exceeds a preset threshold, the switching of the clock mode is triggered.

[0010] Preferably, the globally unique and continuously increasing 64-bit sequence number is generated by a 64-bit counter, and the counter is wrapped around when reaching the maximum value; when the sequence numbers are compared, a wrap-around comparison algorithm is used, and when the absolute value of the difference between two sequence numbers exceeds 2^63, it is judged that wrap-around has occurred, and the sequence number with the smaller difference is processed preferentially.

[0011] Preferably, the resource index is used to split a plurality of audio and video data streams, and each audio and video stream is assigned a different resource index; a plurality of cache queues are set according to the resource index, each queue corresponding to a resource index, and the cache queues store the data packets of each audio and video stream in isolation to avoid timing conflicts caused by resolution switching or Seek operation.

[0012] Preferably, the dynamic switching of the audio dominant mode and the video dominant mode includes the following processing: in the audio dominant mode, the rendering time of a video frame = current audio time + audio frame average duration × (video frame sequence number In the video dominant mode, the average frame duration of the last N video frames is used as the rendering interval target, and the video rendering is dynamically adjusted by frame skipping or delay strategy to keep the video playback rate synchronized with the audio clock.

[0013] Preferably, the frame rate fluctuation threshold is calculated based on the average frame duration of the last N frames of video, and when the change rate of the duration of the current video frame and the average frame duration exceeds a preset threshold, the switching between the audio dominant mode and the video dominant mode is triggered; the average frame duration is calculated by an exponential moving average method to smooth the duration fluctuation.

[0014] The technical scheme of the present application has the following beneficial effects: The present application proposes a dynamic priority synchronization strategy based on a play sequence (Sequence) and a resource index (ResIdx), replaces the traditional timestamp dependence by constructing a cross-stream logical time axis, and introduces a double-mode clock dynamic switching mechanism and a three-level decision matrix. This method solves the synchronization failure problem of the traditional synchronization scheme under the conditions of random interruption of audio and video streams, severe fluctuation of frame rate, resolution switching and frequent Seek operation, significantly improves the synchronization robustness and adaptability; the core innovation of the present application is that a uniform and continuously increasing'sequence number' (play sequence) is marked on each data packet, and the resource index (ResIdx) corresponding to the source of the data packet is marked. In this way, even if the clock of a certain camera suddenly fails, the system can also sort out the order according to these internal numbers to realize correct synchronization.

[0015] A unified logical playback timeline is constructed to ensure cross-stream comparability and sequence: a globally unique and continuously increasing 64-bit sequence number is assigned to each audio / video packet, and the sequence numbers are sorted in the corresponding cache queue of each resource index to construct a logical playback timeline at the receiving end, reducing the interference of out-of-order / dithering on synchronization. The sequence number comparison is oriented to the wrap-around, and the bit width cycle is avoided to cause ambiguity: a 64-bit counter wrap-around and cycle comparison algorithm is used, when the absolute value of the difference between two sequence numbers exceeds 2^63, it is considered as wrap-around, and the sequence number with smaller difference value is processed preferentially, to ensure the reliability of sequence judgment under long-time operation. The resource index is used to realize the shunting, isolation and priority of multiple data streams: according to the source, each stream is assigned a resource index (from 0, the smaller the value, the higher the priority), an independent cache queue is established according to the index, and under the condition that the sequence numbers are the same, the one with smaller resource index is given priority, reducing the cross-stream conflict caused by resolution switching or Seek (jump). Three-level decision matrix, rendering selection is determinable and reproducible: sequence number, resource index and timestamp are compared in sequence to ensure stable selection of data packets to be rendered in the concurrent queue. Dual-mode clock and adaptive switching improve synchronization convergence and stability: provide audio-dominated and video-dominated rendering clocks; based on the change rate of the last N frame durations and the exponential moving average (EMA) criterion, when the change rate exceeds the preset threshold, switch between the two modes, balancing sensitivity and anti-dithering, and adapting to frame duration fluctuations / VFR and other situations. Robustness in typical complex scenarios: in scenarios such as stream interruption, Seek, resolution switching, packet out-of-order / dithering, frame duration fluctuation, etc., the synchronization error can be reduced and the convergence speed can be improved (the specific effect is subject to the test results of the embodiments). DETAILED DESCRIPTION

[0016] The application is further described below in combination with specific embodiments.

[0017] The application provides an audio / video dynamic synchronization system based on playback sequence and resource index, comprising: a receiving module, a sequence number generation module, a resource index setting module, a cache module, a synchronization judgment module and a clock switching module. The receiving module is used for receiving audio / video packets from multiple audio / video data streams, for example, the system can take a server, gateway or player as a receiving end, when one or more cameras, microphones and other devices send audio / video packets, the receiving module is responsible for receiving these audio / video packets and transmitting them to the subsequent processing unit, and the receiving module provides input data for subsequent sequence number generation and resource index setting.

[0018] The sequence number generation module assigns a globally unique and continuously increasing 64-bit sequence number to each received audio / video packet. Specifically, the module uses a 64-bit counter to count each data packet, ensuring that the assigned sequence number is globally unique and continuously increasing throughout the system. When the counter reaches its maximum value, the sequence numbers are used cyclically according to modulo 2^64 logic, and a cyclic comparison algorithm is used to handle sequence number wrapping: if the difference between two sequence numbers exceeds 2^63, wrapping is considered to have occurred, and the sequence number with the smaller difference should be processed first. This module ensures that the sequence order of each data packet can still be correctly compared and identified during system restarts or sequence number cyclication.

[0019] The resource index setting module is used to set resource indexes according to the source of audio and video packets. The resource indexes start from 0 and increment, with smaller values ​​indicating higher priority. This module assigns different resource indexes to audio and video data streams from different sources to achieve multi-channel audio and video data stream splitting. The caching module isolates and stores data packets in corresponding queues based on the different resource indexes, and under the three-level decision matrix, when sequence numbers are the same, prioritizes processing data packets with smaller resource index values. Specifically, the system pre-assigns different resource indexes to devices or data streams from different sources, with index values ​​incrementing from 0; smaller values ​​indicate higher synchronization priority for that data stream. For example, resource indexes can be automatically set based on camera or stream identifiers. The resource index setting module appends this index value to the corresponding data packet for subsequent modules to use for priority judgment and queue classification.

[0020] The caching module stores audio and video packets into corresponding caching queues based on the resource indexes, and sorts the packets by sequence number within each queue to construct a logical playback timeline at the receiving end. Specifically, the caching module maintains an independent queue for each resource index. When the receiving module sends a new data packet, it places it into the queue corresponding to the resource index and sorts the packets in the queue in ascending order according to their sequence number. This constructs a cross-stream logical timeline, replacing the traditional timestamp-dependent approach and avoiding timeline conflicts caused by Seek operations or clock discontinuities. For example, when the audio stream is interrupted, the video stream packets can still be rendered sequentially according to their sequence numbers, without relying on the interrupted audio timestamps.

[0021] The synchronization judgment module is used to determine the audio and video packets to be rendered based on a three-level decision matrix. The three-level decision matrix compares priorities in the following order: sequence number first, resource index second, and timestamp last. Specifically, when a new frame needs to be rendered, the system compares the buffer queues of all streams in the following order of priority: First priority – Sequence number: First, compare the global sequence number of the first packet in each queue, and select the data packet with the smallest sequence number to ensure the continuity of the global playback order.

[0022] Second priority – Resource index: If the sequence numbers are the same, the packet with the smaller resource index (ResIdx) is selected. A smaller resource index indicates that the flow has a higher priority.

[0023] Third priority – timestamp: If the sequence number and resource index are the same when compared in the first two levels, then the traditional timestamp is compared further, and the data packet with the earliest timestamp is selected.

[0024] Through the above three-level comparison, the next frame's rendering object can still be accurately selected even in cases of stream discontinuity or frame loss, enhancing synchronization robustness. The synchronization decision module sends the selected data packet to the rendering pipeline based on the decision result, while simultaneously removing it from the corresponding buffer queue.

[0025] The clock switching module switches between audio-dominated and video-dominated modes based on the duration fluctuations of the current video frames, and calculates the rendering times of audio and video frames according to the currently selected mode. This module includes two rendering clock modes and determines when to switch by monitoring frame duration changes. Audio-Dominated Mode: Video rendering is based on an audio clock. In audio-dominated mode, the rendering time of a video frame is calculated by the difference between the current audio time and its sequence number. Video rendering time = Current audio time + (Video sequence number - Audio sequence number) x Average audio frame duration. The average audio frame duration can be obtained from the codec or calculated based on historical data. In this mode, audio frames play at a normal rate, while the video adjusts its rendering interval according to the audio sequence to maintain synchronization with the audio.

[0026] Video-driven mode: Rendering is based on the video clock. In video-driven mode, the rendering interval is calculated based on the actual duration of the video frame, and the video rendering timing is dynamically adjusted through mechanisms such as frame skipping or waiting to synchronize with audio playback. For example, when the video frame sequence numbers differ significantly, some frames can be skipped or rendered later to keep the video progress consistent with the audio progress and avoid audio pitch distortion.

[0027] The switching trigger condition is determined by the frame rate fluctuation threshold. The system calculates the average frame duration based on the most recent N frames (where N is a positive integer, e.g., N is 3 to 10, without limitation), and calculates the rate of change of the current frame duration relative to the average. If the rate of change exceeds a preset threshold (e.g., 10%), a clock mode switch is triggered. The average frame duration is smoothed using an exponential moving average (EMA) algorithm to avoid frequent switching due to instantaneous fluctuations. The average frame duration is estimated using the exponential moving average (EMA) of the codec header information or historical samples. Through the above dynamic switching, this invention can flexibly select audio or video as the master clock in different scenarios, improving the system's adaptability to situations such as drastic frame rate changes and resolution switching.

[0028] This invention also provides a method for dynamic audio-video synchronization based on playback sequence and resource index, comprising the following steps: Step 1: Receive audio and video packets from multiple audio and video data streams; Step 2: Assign a globally unique and continuously increasing 64-bit sequence number to each audio and video packet, and set a resource index according to the data packet source information. The resource index starts from 0 and increments, with smaller values ​​indicating higher priority. Step 3: Sort the audio and video packets according to the sequence number, construct a logical playback timeline at the receiving end, and put the audio and video packets into the corresponding cache queues according to the resource index. The packets in each cache queue are arranged in ascending order of sequence number. Step 4: During each rendering, the audio and video packets to be rendered are determined based on the three-level decision matrix. The decision matrix compares the priority of the sequence number, resource index, and timestamp in sequence. That is, the packet with the smallest sequence number is selected first. If the sequence numbers are the same, the packet with the smaller resource index is selected. If the first two levels are the same, the packet with the earliest timestamp is selected. Step 5: Dynamically switch the clock mode based on the current frame rate fluctuation. The clock mode includes an audio-dominated mode and a video-dominated mode. In the audio-dominated mode, the rendering time of the video frame is obtained by adding the current audio time, the average duration of the audio frame, and the difference between the sequence numbers of the video frame and the audio frame. In the video-dominated mode, the video frame is rendered according to the dynamically calculated average frame interval, and the video playback rate is adjusted by skipping frames or delaying to make it follow the audio clock. When the rate of change between the duration of the current video frame and the average duration of the most recent N frames exceeds a preset threshold, the clock mode switching is triggered.

[0029] Furthermore, the globally unique and continuously increasing 64-bit sequence number is generated by a 64-bit counter, which loops around after reaching its maximum value. A loop comparison algorithm is used when comparing sequence numbers. When the absolute value of the difference between two sequence numbers exceeds 2^63, it is determined that a loop has occurred, and the sequence number with the smaller difference is processed first.

[0030] Furthermore, the resource index is used to split multiple audio and video data streams, with each audio and video stream being assigned a different resource index; multiple cache queues are set up according to the resource index, with each queue corresponding to a resource index, and the cache queues store data packets of each audio and video stream in an isolated manner to avoid timing conflicts caused by resolution switching or Seek operations.

[0031] Furthermore, the dynamic switching between the audio-dominated mode and the video-dominated mode includes the following processing: In the audio-dominated mode, the rendering time of the video frame = the current audio time + the average duration of the audio frame × (video frame sequence number - audio frame sequence number); In the video-dominated mode, the average frame duration of the most recent N video frames is used as the rendering interval target, and the video rendering is dynamically adjusted through frame skipping or delay strategies to keep the video playback rate synchronized with the audio clock.

[0032] Furthermore, the frame rate fluctuation threshold is calculated based on the average frame duration of the most recent N video frames. When the rate of change between the duration of the current video frame and the average frame duration exceeds a preset threshold, the switching between the audio-dominated mode and the video-dominated mode is triggered. The average frame duration is calculated using the exponential moving average method to smooth out duration fluctuations.

[0033] As described in the above embodiments, this invention proposes a dynamic priority synchronization strategy based on playback sequences and resource indexes (ResIdx). It replaces the traditional timestamp dependency by constructing a cross-stream logical timeline and introduces a dual-mode clock dynamic switching mechanism and a three-level decision matrix. This method solves the synchronization failure problem of traditional synchronization schemes under conditions of random interruption of audio and video streams, drastic frame rate fluctuations, resolution switching, and frequent Seek operations, significantly improving synchronization robustness and adaptability. The core innovation of this scheme lies in assigning each data packet an internally unified, continuously increasing 'sequence number' (playback sequence) and labeling the data packet with the resource index (ResIdx) corresponding to its source. Thus, even if the clock of a certain camera suddenly malfunctions, the system can still clarify the order based on these internal numbers and achieve correct synchronization.

[0034] To ensure comparability and orderliness across streams, a unified logical playback timeline is constructed: Each audio / video packet is assigned a globally unique and continuously increasing 64-bit sequence number, which is then sorted by sequence number within the corresponding buffer queue for each resource index. This constructs a logical playback timeline at the receiving end, reducing interference from out-of-order / jitter on synchronization. For wraparound-oriented sequence number comparison, ambiguity caused by bit-width loops is avoided: A 64-bit counter wraparound and loop comparison algorithm is used. When the absolute value of the difference between two sequence numbers exceeds 2^63, it is considered a wraparound, and sequence numbers with smaller differences are prioritized, ensuring reliable order judgment under long-term operation. Resource indexes are used to achieve multi-stream splitting, isolation, and prioritization: Resource indices are assigned to each stream based on its source (starting from 0, with smaller values ​​taking precedence). Independent buffer queues are established based on the index. In the case of identical sequence numbers, the stream with the smaller resource index takes priority, reducing cross-stream conflicts caused by resolution switching or Seek (jump). A three-level decision matrix ensures deterministic and reproducible rendering selection: sequentially comparing sequence numbers, resource indexes, and timestamps to ensure stable selection of data packets to be rendered from the concurrent queue. Dual-mode clocking and adaptive switching improve synchronization convergence and stability: providing both audio-dominated and video-dominated rendering clocks; based on the rate of change of the most recent N frame durations and the exponential moving average (EMA) criterion, switching between the two modes when the rate of change exceeds a preset threshold, balancing sensitivity and jitter resistance, adapting to frame duration fluctuations / VFR and other situations. Robustness in typical complex scenarios: reducing synchronization errors and improving convergence speed in scenarios such as stream interruptions, Seek, resolution switching, packet out-of-order / jitter, and frame duration fluctuations (specific effects are subject to test results in the implementation examples).

[0035] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. An audio-video dynamic synchronization system based on play sequence and resource index, comprising: The receiving module, the serial number generating module, the resource index setting module, the buffer module, the synchronization judging module and the clock switching module; The receiving module is configured to receive audio and video packets from multiple audio and video data streams; The serial number generating module is configured to assign a globally unique and continuously increasing 64-bit serial number to each received audio and video packet; The resource index setting module is configured to set a resource index according to the source of the audio and video packet, with the resource index starting from 0 and increasing, and a smaller value indicating a higher priority; The buffer module is configured to store the audio and video packets in corresponding buffer queues according to the resource index, and sort the packets in each queue according to the serial number to build a logical playback timeline at the receiving end; The synchronization judging module is configured to determine the audio and video packets to be rendered based on a three-level decision matrix, which compares the priorities in the order of serial number first, resource index second and timestamp third; The clock switching module is configured to switch between an audio-dominated mode and a video-dominated mode according to the fluctuation of the duration of the current video frame, and calculate the rendering time of the audio and video frames according to the selected mode.

2. The audio and video dynamic synchronization system based on play sequence and resource index according to claim 1, characterized in that, The 64-bit counter in the serial number generating module loops back after reaching the maximum value, and a loop comparison algorithm is used when comparing serial numbers. When the absolute value of the difference between two serial numbers exceeds 2^63, it is determined that the serial numbers have looped back and the serial number with the smaller difference is processed first.

3. The audio and video dynamic synchronization system based on play sequence and resource index according to claim 1, characterized in that, In the audio dominant mode, the rendering time of a video frame = current audio time + audio frame average duration x (video frame sequence number audio frame sequence) In the video-dominated mode, the video frames are rendered according to the dynamically calculated average frame interval, and the video playback rate is adjusted through frame skipping or delay strategies to synchronize with the audio clock.

4. The audio and video dynamic synchronization system based on play sequence and resource index according to claim 1, characterized in that, The clock switching module calculates the frame rate fluctuation threshold based on the duration of the last N video frames, and calculates the average frame duration using an exponential moving average algorithm. When the change rate of the current video frame duration and the average frame duration exceeds a preset threshold, the switching between the audio-dominated mode and the video-dominated mode is triggered.

5. The audio and video dynamic synchronization system based on play sequence and resource index according to claim 1, characterized in that, The resource index setting module assigns different resource indexes to audio and video data streams from different sources to achieve the distribution of multiple audio and video data streams. The buffer module stores the data packets in corresponding queues according to the different resource indexes, and when the serial numbers are the same, the data packet with the smaller resource index value is processed first under the three-level decision matrix.

6. A method for dynamic synchronization of audio and video based on play sequence and resource index, characterized in that, The method comprises the following steps: Step 1: receiving audio and video packets from multiple audio and video data streams; Step 2: assigning a globally unique and continuously increasing 64-bit serial number to each audio and video packet, and setting a resource index according to the source information of the data packet, with the resource index starting from 0 and increasing, and a smaller value indicating a higher priority; Step 3: sorting the audio and video packets according to the serial number to build a logical playback timeline at the receiving end, and placing the audio and video packets in corresponding buffer queues according to the resource index, with the packets in each buffer queue sorted in ascending order of the serial number; Step 4: determining the audio and video packets to be rendered according to a three-level decision matrix at each rendering, the decision matrix sequentially compares the sequence numbers, resource indexes and timestamps in priority, i.e. first selecting the packet with the smallest sequence number, then the packet with the smaller resource index when the sequence numbers are the same, and finally the packet with the earliest timestamp when the first two levels are the same; Step 5: dynamically switching the clock mode according to the current frame rate fluctuation, the clock mode including an audio dominant mode and a video dominant mode; in the audio dominant mode, the rendering time of a video frame is obtained by adding the current audio time and the product of the average duration of audio frames and the difference between the sequence numbers of the video frame and the audio frame; in the video dominant mode, the video frame is rendered according to the average interval dynamically calculated, and the video playback rate is adjusted by frame skipping or delay to follow the audio clock; when the change rate of the duration of the current video frame and the average duration of the last N frames exceeds a preset threshold, the switching of the clock mode is triggered.

7. The audio and video dynamic synchronization method based on play sequence and resource index according to claim 6, characterized in that, The globally unique and continuously increasing 64-bit sequence number is generated by a 64-bit counter, which loops back after reaching the maximum value; when comparing the sequence numbers, a loop comparison algorithm is used, and when the absolute value of the difference between two sequence numbers exceeds 2^63, it is judged that the loop back occurs, and the sequence number with the smaller difference is processed preferentially.

8. The audio and video dynamic synchronization method based on play sequence and resource index according to claim 6, characterized in that, The resource index is used to split the multi-channel audio and video data stream, and each channel of audio and video stream is assigned a different resource index; a plurality of cache queues are set according to the resource index, each queue corresponding to a resource index, and the cache queue stores the data packets of each audio and video stream in isolation to avoid time sequence conflicts caused by resolution switching or Seek operation.

9. The audio and video dynamic synchronization method based on play sequence and resource index according to claim 6, characterized in that, The dynamic switching of the audio dominant mode and the video dominant mode comprises the following processes: in the audio dominant mode, the rendering time of a video frame = current audio time + audio frame average duration × (video frame sequence number audio frame sequence); in the video dominant mode, the average frame duration of the last N frames of video is taken as a rendering interval target, and the video rendering is dynamically adjusted through a skip frame or delay strategy, so that the video playback rate follows the audio clock to keep synchronization.

10. The audio and video dynamic synchronization method based on play sequence and resource index according to claim 6, characterized in that, The frame rate fluctuation threshold is calculated based on the average frame duration of the last N frames of video, and when the change rate of the duration of the current video frame and the average frame duration exceeds a preset threshold, the switching between the audio dominant mode and the video dominant mode is triggered; the average frame duration is calculated by the exponential moving average method to smooth the duration fluctuation.