Audio and video acquisition synchronization method based on kernel cooperation

Abstract

The application provides an audio and video acquisition synchronization method based on kernel cooperation, aiming at improving the synchronization quality of traditional audio and video acquisition through the cooperation technology of the bottom layer. The system realizes the unified association of the audio and video acquisition events by integrating the synchronization descriptor information in the video acquisition and the audio acquisition. The system comprises a video event module and an audio event module, the video event module is responsible for recording the video frame completion information, and the audio event module calculates the audio sample interval by using the period completion information. The system adopts a unified clock calibration module and a descriptor generation module to realize the synchronization processing. In addition, the system comprises a bypass output module for providing the synchronization information to the audio and video framework to adapt to the low delay requirement.

Description

Technical Field

[0001] This invention relates to the field of Linux multimedia acquisition technology, and in particular to a method and system for low-level audio and video collaborative synchronization that maintains compatibility between V4L2 and ALSA interfaces. Background Technology

[0002] Existing audio and video acquisition systems typically output video frames from the V4L2 subsystem and audio sample blocks from the ALSA subsystem. These two are independent data paths within the kernel.

[0003] Although V4L2 supports buffering mechanisms such as mmap and DMABUF, and ALSA also supports PCM mmap access, these mechanisms mainly solve the problem of data transfer within a single path and do not provide a unified correspondence between video frames and audio sample intervals.

[0004] In low-latency live streaming, conference capture, and edge analytics scenarios, the upper layer needs to wait for video and audio events separately and then maintain a synchronization queue based on timestamps, which can easily introduce additional waiting, scheduling jitter, and synchronization errors. Summary of the Invention

[0005] In view of this, the present invention proposes an audio and video acquisition synchronization method and system based on kernel collaboration.

[0006] Specifically, the technical solution of this application is: without changing the existing V4L2 and ALSA user-space ABI, to construct a kernel-level synchronization descriptor mechanism to uniformly mark and associate video frame completion events and audio cycle completion events. This includes:

[0007] Step 1: Listen for the V4L2 buffer completion event in the video capture path and record the video frame sequence number, buffer identifier, and completion timestamp.

[0008] Step 2: Listen for the ALSA cycle completion event in the audio acquisition path and record the audio hardware pointer, sampling rate, cycle frame count, and audio timestamp.

[0009] Step 3: Convert the video timestamp into audio sample position based on a unified clock, and generate the audio sample interval corresponding to the video frame.

[0010] Step 4: Encapsulate the video buffer reference, audio ring buffer offset, audio frame count, and synchronization error into a synchronization descriptor and provide it to the audio / video framework through a bypass interface. Attached Figure Description

[0011] Figure 1 is a schematic flowchart of an audio and video acquisition synchronization method according to an exemplary embodiment of this application.

[0012] Figure 2 is a detailed structural diagram of an audio and video acquisition synchronization system shown in an exemplary embodiment of this application. Detailed Implementation

[0013] The following embodiments illustrate the present invention, but the present invention is not limited to these embodiments. Modifications to the specific embodiments of the present invention or equivalent substitutions for some technical features, without departing from the spirit of the present invention, should all be covered within the scope of the technical solutions claimed in the present invention.

[0014] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this application refers to and includes any and all possible combinations of one or more associated listed items.

[0015] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this application, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0016] Referring to Figure 1, Figure 1 shows a flowchart of an audio and video acquisition synchronization method based on kernel collaboration according to an embodiment of this application.

[0017] During the video event acquisition phase, when the camera driver sets the buffer to the completed state, the kernel synchronization layer obtains the buffer sequence number, index, completion timestamp, and status flag.

[0018] During the audio event acquisition phase, when the sound card driver triggers the cycle completion notification, the kernel synchronization layer obtains the hardware pointer, cycle size, sampling rate, and timestamp.

[0019] In this application example, the clock calibration module converts the video frame completion time and audio sampling time to a unified time base and calculates the corresponding audio start sample and audio frame number.

[0020] The synchronization descriptor generation module does not copy video frame data and audio sample data, but instead saves video buffer references, audio offsets, audio lengths, and synchronization errors.

[0021] The bypass output module can be implemented using character devices, shared metadata queues, or event notification mechanisms, enabling the audio and video framework to obtain synchronization information while maintaining the original acquisition interface.

[0022] Referring to Figure 2, in one embodiment of this application, the system includes a video event module, an audio event module, a clock calibration module, a descriptor generation module, a drift estimation module, and a bypass output module.

[0023] For any video frame during continuous acquisition, the system uses the timestamp of that video frame to find the audio sample interval that has been completed but not yet covered, and generates the corresponding synchronization descriptor.

[0024] In this embodiment, ordinary applications can still complete the acquisition task through existing video and audio devices; when the audio and video framework is adapted to the bypass interface, the application-side synchronization queue waiting time can be reduced.

[0025] After the synchronization descriptor is input into the acquisition backend of the audio and video framework, the framework can directly obtain the correspondence between video frames and audio sample intervals, and perform joint scheduling based on the synchronization error.

Claims

1. A kernel-coordinated audio and video acquisition synchronization method, characterized in that, Includes the following steps: While keeping the video capture interface and audio capture interface unchanged, listen for the video frame completion event and the audio cycle completion event; Calculate the audio sample interval corresponding to the video frame based on a unified clock; Generate an audio-video synchronization descriptor that includes video buffer identifier, audio sample range, and synchronization error; The descriptor is provided to the audio / video framework via a bypass interface.

2. The audio and video acquisition synchronization method according to claim 1, characterized in that: The unified clock uses a kernel monotonic clock, a hardware timestamp, or a calibrated device clock to ensure that video frames and audio samples are on the same time base.

3. The audio and video acquisition synchronization method according to claim 1, characterized in that: The video buffer completion event includes the buffer sequence number, buffer index, completion timestamp, and frame status information.

4. The audio and video acquisition synchronization method according to claim 1, characterized in that: The audio cycle completion event includes a hardware pointer, cycle frame count, sampling rate, and timestamp, which are used to calculate the position of the audio sample.

5. The audio and video acquisition synchronization method according to claim 1, characterized in that: The audio / video synchronization descriptor only stores the video buffer reference, audio ring buffer offset, audio length, and synchronization metadata; it does not copy audio / video data.

6. A kernel-coordinated audio and video acquisition synchronization system, characterized in that, include: The video event module is used to obtain the video buffer completion event; The audio event module is used to obtain audio cycle completion events; Clock calibration module for unifying video and audio timestamps; The descriptor generation module is used to generate the correspondence between video frames and audio sample intervals; A bypass output module is used to output the synchronization descriptor.

7. The audio and video acquisition synchronization system according to claim 6, characterized in that: The bypass output module outputs descriptors through character devices, shared metadata queues, or kernel notification mechanisms, enabling the audio and video framework to obtain synchronization information without modifying ordinary applications.

8. The audio and video acquisition synchronization system according to claim 6, characterized in that: The system also includes a drift estimation module, which estimates clock drift based on the time difference between consecutive video frames and audio sample intervals, and provides compensation parameters.

Images