Audio and video delay processing method and system

Abstract

The invention relates to the technical field of audio and video processing, in particular to an audio and video delay processing method and system. According to the technical scheme, the method comprises the following steps: acquiring audio and video original data through an audio and video input module, and encoding the audio and video original data; according to the method, a storage module stores encoded audio and video data in a fragmentation mode according to preset delay playing time delta (t), a synchronization head is added into each fragmentation, and the synchronization head is obtained through calculation according to current equipment time t and the delay playing time delta (t). According to the invention, accurate quantitative control of time delay is realized by introducing fragmented storage and a synchronous head mechanism, persistent synchronization of audio and video is ensured by adopting a dynamic closed-loop compensation strategy, and smoothness and stability of a playing process are ensured by combining an intelligent flexible adjustment algorithm; finally, a complete scheme capable of providing fixed time delay, synchronous sound and picture and stable and smooth output is constructed by a modular architecture.

Description

Technical Field

[0001] This invention relates to the field of audio and video delay processing technology, and in particular to an audio and video delay processing method and system. Background Technology

[0002] In situations requiring strict oversight of broadcast content, such as important video conferences, received audio and video signals are typically delayed to prevent unauthorized information leaks or to respond to emergencies. This delayed playback mechanism provides a time window for content review and emergency response, with delay times ranging from several seconds to tens of minutes. Currently, technical solutions for implementing audio and video delays can be mainly divided into hardware and software categories. Hardware solutions typically rely on dedicated delay devices, which can provide fixed delays, but their delay durations are often not flexibly configurable, and the equipment is expensive and difficult to adapt to the input requirements of multi-source, heterogeneous audio and video signals. On the other hand, software solutions process data through servers and software algorithms. While offering some flexibility, they still have significant drawbacks in practical applications.

[0003] First, the precision of delay control is poor. Due to the complexity and diversity of audio and video input sources, their encoding formats, resolutions, frame rates, sampling rates, and other parameters vary. Existing processing methods struggle to dynamically and accurately calculate and manage the amount of data to be buffered based on different input sources, resulting in significant fluctuations in the actual delay time around the set value, making it impossible to achieve stable output with a fixed delay.

[0004] Secondly, audio and video synchronization is poor. Audio and video are two independent data streams during processing and transmission, potentially via different hardware paths or processing threads. Existing technologies often exhibit audio-video desynchronization when handling delayed playback, a problem known as "lip-syncing." This is primarily due to the lack of an effective and continuous synchronization maintenance mechanism during long-duration delayed playback. Initial alignment can gradually accumulate due to minor differences in decoders, system load fluctuations, or data packet processing timing issues, ultimately leading to noticeable audio-visual asynchrony.

[0005] Furthermore, existing processing methods often employ rather crude strategies to compensate for time-lapse, such as directly discarding or repeating a frame, which can easily lead to choppy playback. This abrupt compensation method can cause video jumps, stutters, or audio interruptions, severely impacting the viewing experience. Therefore, this application proposes an audio and video delay processing method and system. Summary of the Invention

[0006] The purpose of this invention is to address the problem in the existing audio and video delay processing methods in the background art of achieving stable output with fixed delay, and to propose an audio and video delay processing method and system.

[0007] In a first aspect, the present invention provides an audio and video delay processing method, comprising:

[0008] The audio and video input module collects raw audio and video data and encodes the raw audio and video data.

[0009] The encoded audio and video data is segmented and stored using the storage module according to the preset delay playback time δ(t), and a synchronization header is added to each segment. The synchronization header is calculated based on the current device time t and the delay playback time δ(t).

[0010] The decoding module reads and decodes the stored audio and video segment data during playback.

[0011] The synchronization module aligns the decoded audio and video data according to the synchronization head and compensates for delay deviation during playback for synchronized audio and video playback.

[0012] The playback module outputs synchronized audio and video data to achieve audio and video playback with a fixed delay.

[0013] Optionally, the step of segmenting and storing the encoded audio and video data includes:

[0014] Based on video resolution Video frame rate Delayed playback time and video size parameters Calculate the size of video segments The calculation formula is: ;

[0015] Based on audio sampling rate Delayed playback time and audio size parameters Calculate the size of audio segments The calculation formula is: .

[0016] Optionally, the synchronization header is calculated as follows: ,in The current time of the device. Delay the playback time for audio and video.

[0017] Optionally, the alignment process includes:

[0018] Based on the delayed playback time Get the synchronization header of the audio and video segments to be played, along with the device's current timestamp;

[0019] Obtain the display timestamp (pts) of the first frame of the audio and video segments respectively, and calculate the difference between the two.

[0020] If the difference exceeds a preset synchronization threshold, the audio or video segment alignment is completed by adjusting the reading of the next frame until the difference is less than the synchronization threshold.

[0021] Optionally, the delay deviation compensation includes:

[0022] During playback, the difference between the display timestamps of the audio and video is periodically calculated (diff).

[0023] When the diff is greater than a preset synchronization threshold, the playback order of the audio and video is determined.

[0024] If the video is faster than the audio, then the video frames are smoothly delayed.

[0025] If the audio is faster than the video, blank frames are inserted into the audio frames, and the video frames are dropped for audio-video synchronization.

[0026] In a second aspect, an audio / video delay processing system is provided for implementing the method described in the first aspect, the system comprising:

[0027] The audio / video input module is used to acquire and encode raw audio and video data.

[0028] The storage module is used to segment and store the encoded audio and video data, and add synchronization headers to the segments;

[0029] The decoding module is used to read and decode the stored audio and video segment data;

[0030] The synchronization module is used to align and compensate for deviations in the decoded audio and video data.

[0031] The playback module is used to output synchronized audio and video data.

[0032] Optionally, the storage module calculates the size of the audio and video segments based on the video resolution, frame rate, audio sampling rate, and playback delay time, and then stores the segments.

[0033] Optionally, the synchronization module includes:

[0034] Align audio and video segments according to the synchronization header;

[0035] Dynamic deviation detection and compensation are performed during playback to ensure synchronized audio and video output.

[0036] Compared with the prior art, this application includes at least one of the following beneficial technical effects:

[0037] By introducing a segmented storage calculation formula based on resolution, frame rate, sampling rate, and preset delay time, the abstract delay time is quantified into a specific data storage block size, achieving precise control from the data source. Combined with a synchronization header mechanism based on system timestamps, the system can accurately read data at specific time points during playback, effectively overcoming the delay fluctuation problem caused by complex input sources and ensuring the stability and accuracy of delayed output.

[0038] This invention not only focuses on initial alignment, but also innovatively introduces a dynamic, closed-loop synchronization maintenance mechanism. The synchronization module continuously monitors audio and video deviations during playback and performs real-time compensation through intelligent algorithms (such as smooth delay, frame interpolation, and frame dropping). This effectively eliminates minor errors accumulated due to factors such as system load and decoding differences, ensuring that audio and video remain perfectly synchronized during playback delays of up to tens of minutes.

[0039] The deviation compensation logic of this invention is more flexible and intelligent. It adopts different optimization strategies (such as 3x smooth delay or gradual adjustment) according to the direction and magnitude of the deviation, avoiding the discomfort caused by direct frame skipping or stuttering. This makes the entire delayed playback process smooth and fluid, greatly improving the user experience in serious scenarios such as confidential meetings.

[0040] The system employs a modular architecture design, clearly dividing it into five functional modules: input, storage, decoding, synchronization, and playback. This highly cohesive and loosely coupled design allows each module to be independently optimized and upgraded, facilitating system adaptation to new audio and video encoding formats or the integration of more advanced synchronization algorithms.

[0041] In summary, this invention achieves precise quantitative control of latency by introducing a segmented storage and synchronization header mechanism, ensures persistent audio and video synchronization by adopting a dynamic closed-loop compensation strategy, and guarantees smooth and stable playback by combining intelligent and flexible adjustment algorithms. Finally, a complete solution capable of providing fixed latency, audio-visual synchronization, and stable and smooth output is constructed with a modular architecture. Attached Figure Description

[0042] Figure 1 This is a block diagram illustrating the principle of an audio / video delay processing system.

[0043] Figure 2 This is a synchronization flowchart for an audio / video delay processing method. Detailed Implementation

[0044] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0045] Example: This example provides an audio and video delay processing system, which is described below in conjunction with the attached document. Figure 1 The audio and video delay processing system of the present invention will be described in detail.

[0046] like Figure 1 As shown, the audio and video delay processing system of the present invention includes the following five core modules:

[0047] The audio / video input module is used to receive and process audio and video input signals from various sources. For video input, the module first performs initialization operations, acquiring parameters such as video resolution (dbi) and frame rate (F), and then collects the raw video data. For audio input, the module preprocesses the audio data, sampling it at a fixed sampling rate R to convert the analog audio signal into a digital signal. Finally, the module encodes the raw audio and video data according to a preset encoding format (such as H.264 / H.265 for video and AAC for audio), generating a unified encoded data stream.

[0048] The storage module is responsible for managing the segmented storage of the encoded audio and video data. This module calculates the segment size based on the user-defined playback delay δ(t) and the audio and video parameters.

[0049] Video segment size:

[0050] Audio segment size:

[0051] Where β is the video size parameter (usually a value of 2-5), and α is the audio size parameter (usually a value of 1-3).

[0052] When storing each shard, the module adds a specific synchronization header. The synchronization header is calculated as follows: ,in This is the device's current system time. This segmented storage mechanism facilitates subsequent precise reading and synchronization control.

[0053] The decoding module reads the corresponding audio and video segment data from the storage module according to the playback delay δ(t) when audio and video need to be played. This module decodes the data according to the format used during encoding (such as H.264 / H.265, AAC) to restore the original audio and video data that can be played.

[0054] Synchronization module, such as Figure 2 As shown, the synchronization module is the core of this system, responsible for accurate audio and video synchronization and delay deviation compensation. This module first aligns the corresponding audio and video segment data according to the synchronization header, specifically including:

[0055] Based on the delayed playback time δ(t) and the device's current timestamp, calculate the segment header to be played: timestamp-δ(t); obtain the display timestamp pts of the first frame of the audio and video segments respectively, and calculate the difference between the two; adjust the reading of the next frame of the audio or video until the difference in pts is less than the preset synchronization threshold (usually 40-100ms).

[0056] During playback, the synchronization module also performs real-time delay deviation compensation: using audio PTS as a benchmark, it calculates the audio-video diff value; when the diff is greater than the synchronization threshold, it determines the order of audio and video; if the video is faster than the audio, the video frames are displayed with a delay of 3 times for smoothing; if the audio is faster than the video, the delay value is reduced, and blank frames are inserted between audio frames, while the video frames are dropped.

[0057] The playback module receives synchronized audio and video data and sends it to the corresponding audio and video output devices to achieve synchronized playback of audio and video with precise delay. This module ensures the smoothness and synchronization of the output audio and video, meeting the needs of special scenarios such as confidential meetings.

[0058] The audio / video delay processing method described in this invention achieves precise delay control and intelligent synchronization. Through a segmented storage strategy and a synchronization header mechanism, the abstract delay time δ(t) is transformed into specific data storage and retrieval instructions. This is achieved through the given specific calculation formula (such as...). This allows the system to accurately calculate the amount of data to be stored based on parameters such as video resolution and frame rate, providing a quantitative basis for precise latency from the source and avoiding latency fluctuations caused by improper data block partitioning in traditional methods. Secondly, the method's dynamic deviation compensation logic demonstrates significant intelligent advantages. It is not a one-time synchronization, but a closed-loop control that continuously monitors, calculates, and compensates throughout the entire playback cycle. By calculating the difference N and comparing it with the synchronization threshold, the system can diagnose the audio-visual offset status in real time and adopt differentiated compensation strategies such as "smooth latency" or "frame insertion / dropping." This flexible compensation mechanism effectively avoids video stuttering or sound pops, ensuring smooth playback while maintaining synchronization, ultimately achieving stable latency and high-quality audio-visual synchronization, making it particularly suitable for confidential meeting scenarios with extremely high security and stability requirements.

[0059] This embodiment also provides an audio and video delay processing method, which will be described in detail below with reference to the accompanying drawings.

[0060] like Figure 2 As shown, the audio and video delay processing method of the present invention includes the following steps:

[0061] Step S101: Audio and video acquisition and encoding. Different audio and video input signals are analyzed to obtain the video resolution (dbi) and frame rate (F), and raw video data is acquired. Raw audio data is acquired according to the set audio sampling rate (R). Then, the raw audio and video data are encoded according to a preset encoding format to generate a unified encoded data stream.

[0062] Step S102: Segmented storage processing. Calculate the audio / video segment size based on the preset delay playback time δ(t):

[0063] Video segment size: ;

[0064] Audio segment size: ;

[0065] In this case, β is 3 and α is 2. The encoded data is stored in fragments, and a synchronization header is added to each fragment. The synchronization header is calculated as: Sync = t / δ(t), where t is the current system time of the device.

[0066] Step S103: Data reading and decoding. At playback time, the corresponding audio and video segment data is read from the storage system according to the delayed playback time δ(t). The data is decoded according to the encoding format to restore the original audio and video data that can be played back.

[0067] Step S104: Audio and video synchronization processing, including two sub-steps: segment alignment and deviation compensation.

[0068] Segment Alignment: Based on the delayed playback time δ(t) and the device's current timestamp, obtain the segment header to be played: timestamp-δ(t); read the stored audio and video segment headers to obtain the raw data to be played; obtain the PTS of the first frame of the audio and video segments respectively, and calculate the difference between them; iterate through the next frame until the PTS difference is less than the synchronization threshold (recommended value is 50ms).

[0069] Deviation compensation: During playback, deviation is checked periodically. The calculation formula is: N=t / δ(t)-Sync. Based on the audio pts, the audio and video diff values ​​are calculated. When the diff is greater than the threshold: if the video is faster than the audio, the video frame is smoothly delayed by delay×3. If the audio is faster than the video, the delay is adjusted to FFMAX(0,delay+diff), and audio blank frames and video dropped frames are inserted.

[0070] Step S105: Audio and video playback output. The synchronized audio and video data is sent to the playback device to achieve synchronized audio and video output after precise delay, ensuring the security of the meeting content and the stability of the playback.

[0071] By implementing the above system and method, the present invention can effectively solve the problems of inaccurate audio and video delay control and poor synchronization effect in the prior art, and is particularly suitable for audio and video conferencing scenarios that require high security.

[0072] The audio and video delay processing system of this invention decomposes complex functions into five distinct professional modules (input, storage, decoding, synchronization, and playback). This highly cohesive and loosely coupled design results in extremely high reliability and maintainability. Each module can be independently optimized and upgraded. For example, the storage module can be optimized for read and write operations on fragmented storage, and the synchronization module can run complex compensation algorithms on a dedicated high-performance processor without affecting the stability of other modules. When a part requires maintenance or malfunctions, it can be quickly located and repaired, greatly improving the overall robustness of the system. Furthermore, the system architecture provides excellent engineering feasibility and scalability. The audio and video input module can encapsulate adaptation logic for different input sources, enabling the system to flexibly connect to various audio and video devices; while the synchronization module, as an independent computing unit, ensures the computing power required for real-time processing of large amounts of audio and video data, guaranteeing processing efficiency and real-time performance. This modular design also facilitates future expansion of new functions. For example, simply upgrading the algorithm of the synchronization module can support more complex synchronization strategies without changing the entire system architecture, thus giving the product a long lifespan and market competitiveness.

[0073] The above specific embodiments are merely several optional embodiments of the present invention. Based on the technical solutions of the present invention and the relevant teachings of the above embodiments, those skilled in the art can make various alternative improvements and combinations to the above specific embodiments.

Claims

1. An audio-video delay processing method, characterized in that, The system comprises: An audio and video input module is configured to collect audio and video raw data and encode the audio and video raw data; A storage module is configured to store the encoded audio and video data in slices according to a preset delay playback time δ(t) and add a sync header in each slice, wherein the sync header is calculated according to a current device time t and the delay playback time δ(t); A decoding module is configured to read the stored audio and video slice data and decode the audio and video slice data during playback; A synchronization module is configured to align the decoded audio and video data according to the sync header and compensate for a delay deviation during playback, so as to synchronize the audio and video data; A playback module is configured to output the synchronized audio and video data.

2. The audio-video delay processing method of claim 1, wherein, The storage module is configured to: According to the video resolution , video frame rate , time delay play time and video size parameters , the video slice size is calculated, and the calculation formula is: ; According to the audio sampling rate , the delay play time and the audio size parameter , the audio fragment size is calculated, and the calculation formula is: .

3. The audio-video delay processing method of claim 2, wherein, The calculation of the synchronization head is as follows: Wherein is the current time of the device, is the audio-video delay play time.

4. The audio-video delay processing method of claim 1, wherein, The alignment process comprises: According to the time delay playing time and the current time stamp of the device, the synchronization header of the audio and video slice to be played is acquired. The display time stamps pts of the first frames of the audio and video slices are obtained respectively, and a difference between the two is calculated; If the difference exceeds a preset synchronization threshold, the next frame of the audio or video is adjusted until the difference is less than the synchronization threshold, so as to complete the alignment of the audio and video slices.

5. The audio-video delay processing method of claim 1, wherein, The delay deviation compensation comprises: During playback, the display time stamp difference diff between the audio and the video is periodically calculated; When the diff is greater than a preset synchronization threshold, the playback sequence of the audio and the video is determined; If the video is faster than the audio, the video frame is subjected to a smooth delay processing; If the audio is faster than the video, a blank frame is inserted into the audio frame, and the video frame is subjected to a frame dropping processing, so as to synchronize the audio and the video.

6. An audio-video delay processing system for implementing the method according to any one of claims 1 to 5, characterized in that, The system comprises: An audio and video input module is configured to collect audio and video raw data and encode the audio and video raw data; A storage module is configured to store the encoded audio and video data in slices according to a preset delay playback time δ(t) and add a sync header in each slice, wherein the sync header is calculated according to a current device time t and the delay playback time δ(t); A decoding module is configured to read the stored audio and video slice data and decode the audio and video slice data during playback; A synchronization module is configured to align the decoded audio and video data according to the sync header and compensate for a delay deviation during playback, so as to synchronize the audio and video data; A playback module is configured to output the synchronized audio and video data.

7. The audio / video delay processing system of claim 6, wherein, The storage module is configured to:

8. The audio / video delay processing system of claim 6, wherein, The storage module is configured to: The alignment process comprises: The display time stamps pts of the first frames of the audio and video slices are obtained respectively, and a difference between the two is calculated; If the difference exceeds a preset synchronization threshold, the next frame of the audio or video is adjusted until the difference is less than the synchronization threshold, so as to complete the alignment of the audio and video slices. The delay deviation compensation comprises: During playback, the display time stamp difference diff between the audio and the video is periodically calculated; When the diff is greater than a preset synchronization threshold, the playback sequence of the audio and the video is determined; If the video is faster than the audio, the video frame is subjected to a smooth delay processing; If the audio is faster than the video, a blank frame is inserted into the audio frame, and the video frame is subjected to a frame dropping processing, so as to synchronize the audio and the video.

Images