A cross-domain audio-video synchronization method, system, device and medium

Abstract

The application discloses a cross-domain audio and video synchronization method, system, device and medium, and the method comprises the following steps: acquiring an audio original timestamp of an audio frame rendering pipeline key event node based on a global hardware counter on a chip in an audio domain, calibrating the audio original timestamp based on a current acoustic delay error, determining an audio physical output time and writing into a cross-domain shared storage area; acquiring a video timestamp of a video frame rendering pipeline key event node based on a global hardware counter on a chip in a video domain; determining a video frame video rendering trigger time according to the difference between a target display time and the inherent delay of the video rendering pipeline, and triggering the video frame rendering based on the video rendering trigger time. Except for one-time trimming of the counter offset of each domain in the booting stage, no calibration between the cross-domain counters is required in the running period, the deviation between the software timestamp and the arrival of the sound wave at the reference measuring point in the vehicle is eliminated, the delay fluctuation is suppressed, the functional safety requirements of the vehicle are met, and audio and video synchronization is realized.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and in particular to a cross-domain audio and video synchronization method, system, device, and medium. Background Technology

[0002] With the rapid development of automotive intelligence and cockpit multimedia systems, automotive-grade System-on-Chip (SOC) generally adopts a multi-domain isolation architecture, isolating functional domains such as audio, video, and security domains to meet stringent functional and information security requirements. Under this architecture, video may be decoded and rendered in the entertainment domain before being output to the rear screen, while audio needs to be transmitted across domains to the cockpit domain for processing by a DSP (Digital Signal Processor) before being output to the speakers. Because the two domains are isolated, issues arise regarding memory isolation, CPU isolation, clock isolation, and scheduling isolation, posing a significant challenge to cross-domain audio-visual synchronization.

[0003] Existing cross-domain time synchronization solutions include: a daisy-chain-based hardware counter cross-domain calibration method, which connects counters from multiple hardware domains via a daisy chain, calculates the clock difference after measuring the transmission delay, and achieves subsystem timestamp synchronization; and a method that uses the generalized Precision Time Protocol (gPTP) to synchronize the network clocks and PTP (Precision Time Protocol) hardware clocks of each domain.

[0004] However, such protocols for establishing cross-domain time bases are highly complex, costly, and introduce additional errors, making them unsuitable for the low-overhead requirements of automotive-grade multi-domain isolation architectures. Daisy-chaining requires hardware link support, and gPTP requires a complete network protocol stack, both of which increase system complexity and cost. A systematic deviation exists between the software timestamp and the physical output time; this systematic deviation cannot be effectively detected or eliminated, resulting in limited audio-visual synchronization accuracy. Summary of the Invention

[0005] This invention provides a cross-domain audio and video synchronization method, system, device, and medium to meet automotive-grade functional safety requirements and achieve high-precision audio-visual synchronization.

[0006] According to a first aspect of the present invention, a cross-domain audio-video synchronization method is provided, applied to an automotive-grade system-on-a-chip (SoC), the SoC including a video domain and an audio domain, the method comprising:

[0007] The audio domain obtains the original audio timestamps of key event nodes in the audio frame rendering pipeline based on the on-chip global hardware counter of the SOC, and calibrates the original audio timestamps based on the current acoustic delay error to determine the physical audio output time and write it to the cross-domain shared storage area.

[0008] The video timestamps of key event nodes in the video frame rendering pipeline are obtained through the video domain based on the on-chip global hardware counter of the SOC.

[0009] The physical output time of the audio is used as the target display time of the video frame, and the rendering advance after the video domain reads the physical output time is evaluated based on the difference between the physical output time of the audio and the local reception time.

[0010] If sufficient, the video rendering trigger time of the video frame is determined based on the difference between the target display time and the inherent delay of the video rendering pipeline. The video frame rendering is triggered based on the video rendering trigger time, and the display time of the video frame after the inherent delay of the video rendering pipeline is aligned with the audio physical output time.

[0011] According to a second aspect of the present invention, an automotive-grade system-on-a-chip is provided, the automotive-grade system-on-a-chip including a video domain and an audio domain;

[0012] The audio domain is used to obtain the original audio timestamps of key event nodes of the audio frame rendering pipeline based on the global hardware counter on the SOC, and to calibrate the original audio timestamps based on the current acoustic delay error, determine the physical output time of the audio, and write it into the cross-domain shared storage area.

[0013] The video domain is used to obtain the video timestamps of key event nodes in the video frame rendering pipeline based on the on-chip global hardware counter of the SOC.

[0014] The physical output time of the audio is used as the target display time of the video frame, and the rendering advance after the video domain reads the physical output time is evaluated based on the difference between the physical output time of the audio and the local reception time.

[0015] If sufficient, the video rendering trigger time of the video frame is determined based on the difference between the target display time and the inherent delay of the video rendering pipeline. The video frame rendering is triggered based on the video rendering trigger time, and the display time of the video frame after the inherent delay of the video rendering pipeline is aligned with the audio physical output time.

[0016] According to a third aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the cross-domain audio and video synchronization method according to any embodiment of the present invention.

[0018] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the cross-domain audio and video synchronization method according to any embodiment of the present invention.

[0019] According to a fifth aspect of the present invention, embodiments of the present invention also provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the cross-domain audio and video synchronization method of any embodiment of the present invention.

[0020] The technical solution of this invention obtains the original audio timestamps of key event nodes in the audio frame rendering pipeline based on the on-chip global hardware counter in the audio domain, calibrates the original audio timestamps based on the current acoustic delay error, determines the physical audio output time, and writes it to the cross-domain shared storage area; it obtains the video timestamps of key event nodes in the video frame rendering pipeline based on the on-chip global hardware counter in the video domain; it determines the video rendering trigger time of the video frame based on the difference between the target display time and the inherent delay of the video rendering pipeline, and triggers video frame rendering based on the video rendering trigger time. Except for a one-time balancing of the counter offsets of each domain during the power-on phase, no cross-domain counter calibration is required during operation, eliminating the deviation between the software timestamp and the arrival of the sound wave at the in-vehicle reference measurement point, suppressing delay fluctuations, meeting automotive-grade functional safety requirements, and achieving audio-visual synchronization.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a cross-domain audio and video synchronization method provided according to Embodiment 1 of the present invention;

[0024] Figure 2 This is a flowchart of a cross-domain audio and video synchronization method provided according to Embodiment 2 of the present invention;

[0025] Figure 3 This is a system architecture diagram of a cross-domain audio and video synchronization method provided according to Embodiment 2 of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of an automotive-grade system-on-a-chip according to Embodiment 3 of the present invention;

[0027] Figure 5 This is a schematic diagram of the structure of an electronic device that implements an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] Figure 1 This is a flowchart illustrating a cross-domain audio and video synchronization method provided in Embodiment 1 of the present invention. This embodiment is applicable to audio and video synchronization in a multi-domain isolation architecture. The method is applied to an automotive-grade System-on-a-Chip (SoC), which includes a video domain and an audio domain. This automotive-grade SoC can be configured in electronic devices. Figure 1 As shown, the method includes:

[0032] S110: Obtain the original audio timestamps of key event nodes in the audio frame rendering pipeline through the audio domain based on the global hardware counter on the SOC chip, calibrate the original audio timestamps based on the current acoustic delay error, determine the physical output time of the audio, and write it to the cross-domain shared storage area.

[0033] In this embodiment, a System-on-a-Chip (SOC) refers to an integrated circuit that integrates multiple functional modules such as a central processing unit, graphics processing unit, digital signal processor, memory controller, and hardware counter onto a single chip. Automotive-grade SOCs comply with the AEC-Q100 standard and meet automotive functional safety requirements. The audio domain refers to a functional area within the SOC, defined using Hypervisor virtualization technology, specifically responsible for audio data acquisition, processing, decoding, rendering, and output. It includes independent CPU cores, DSPs, and audio peripheral resources, operating in isolation from the video domain. A global hardware counter refers to a hardware counter module integrated within the SOC chip (such as the CNTVCT_EL0 register of the ARM GenericTimer). It continuously increments after the SOC is powered on. Under the defined normal operating conditions of the automotive-grade SOC (without entering a deep sleep state that requires counter reset) and provided that the virtual counter offset CNTVOFF is uniformly configured and balanced by the Hypervisor during the power-on phase, all domains can be directly read, and the values ​​read from each domain are completely consistent and unaffected by domain scheduling switching. Except for the one-time balance of CNTVOFF of each domain by the Hypervisor during the power-on phase, a unified time base can be provided without cross-domain counter calibration during operation. The counting accuracy of the counter itself is determined by the SOC counting frequency (e.g., 100MHz corresponds to the order of 10ns). Finally, the audio-visual synchronization accuracy is limited by the edge quantization of the vertical synchronization VSync of the display controller. It should be noted that the timestamp acquisition operation involving reading the global hardware counter in the audio domain is executed on the CPU core side of the audio domain. The DSP in the audio domain is responsible for audio data processing. The timestamp information required by the DSP is read from the global hardware counter by the CPU core of the audio domain and then transmitted to the DSP through the domain's shared channel to ensure that heterogeneous processors operate based on the same global hardware counter time base. The audio rendering pipeline refers to the complete processing chain of audio data from application generation to final playback by the speaker, including audio frame submission to the DMA buffer, DAC conversion, power amplifier processing, and speaker playback. Critical event nodes refer to specific time points in the audio and video rendering pipeline used to mark rendering progress and record timestamps. The raw audio timestamp refers to the count value obtained by the audio domain by directly reading the global hardware counter on the SOC chip when a critical event node in the audio frame rendering pipeline occurs. Acoustic delay error is used to characterize the end-to-end time deviation between the software recording of the audio timestamp and the actual sound wave output by the speaker, including systematic deviations such as audio drive delay, DAC conversion delay, power amplifier processing delay, speaker acoustic propagation delay, and software scheduling jitter. The audio physical output moment refers to the moment when an audio frame, after acoustic delay error calibration, is actually played by a speaker, and is represented by a global hardware counter value.Cross-domain shared storage refers to the physical storage space within a System-on-a-Chip (SoC) that is shared and accessed by the audio and video domains through a Hypervisor configuration. It is used to enable cross-domain data transfer, including a dedicated timestamp transfer channel (lockless circular queue) and a normal data transfer channel.

[0034] Specifically, when a critical event occurs in the audio frame rendering pipeline, the audio domain directly reads the on-chip global hardware counter of the SOC to obtain the original audio timestamp (i.e., the planned playback time at the software level). Due to systematic deviations such as driver latency, DAC conversion latency, power amplifier latency, and speaker propagation latency, there is a fixed time deviation between this original audio timestamp and the actual physical moment when the sound wave is output by the speaker, i.e., acoustic delay error. The audio domain uses a pre-built acoustic feedback closed-loop calibration system to obtain the current acoustic delay error and correct the original audio timestamp to obtain the true physical audio output time. After calibration, the audio domain writes this time to the cross-domain shared storage area for subsequent use by the video domain.

[0035] S120: Obtain the video timestamps of key event nodes in the video frame rendering pipeline through the video domain based on the SOC on-chip global hardware counter.

[0036] In this embodiment, the video domain refers to a functional partition within the SOC, defined using Hypervisor virtualization technology, specifically responsible for video decoding, graphics rendering, and display output. The video timestamp refers to the count value obtained by the video domain from the on-chip global hardware counter when a critical event occurs in the video frame rendering pipeline. The video frame rendering pipeline refers to the complete processing chain that a video frame undergoes from its raw encoded data to its final presentation on the display screen.

[0037] The key event nodes of the video frame rendering pipeline include at least one of the following: video frame decoding completion time, video frame rendering completion time, video frame submission to the display queue time, and video frame display time. For example, it can be represented as: T_vdec_done: Video frame decoding completion time, which refers to the time when the video frame completes the decoding operation. T_render_done: Video frame rendering completion time, which refers to the time when the video frame completes all rendering and compositing operations. T_submit: Video frame submission to the display queue time, which indicates the time when the video frame is sent to the display controller's display queue. T_display: Video frame display time (obtained via the VSync signal), which refers to the time when the video frame is actually displayed on the screen.

[0038] Specifically, when critical event nodes occur in the video frame rendering pipeline, the video domain directly reads the on-chip global hardware counter of the SOC to obtain the video timestamp. Since the video and audio domains read the same physical hardware counter, their time bases are naturally unified, eliminating the need for any cross-domain counter calibration operations. This step is executed in parallel with step one, with the video domain continuously recording the actual progress timestamps of the video rendering pipeline, providing a rendering progress reference on the video side for subsequent synchronous calculations. A time base consistent with the audio domain can be obtained with only a single register read operation.

[0039] S130. Use the physical output time of the audio as the target display time of the video frame, and evaluate whether the rendering lead after the video domain reads the physical output time of the audio is sufficient based on the difference between the physical output time of the audio and the local reception time.

[0040] In this embodiment, the predicted current cross-domain transmission delay refers to the estimated current cross-domain transmission delay value predicted based on historical measured delay data. This value is used to predict the rendering lead and determine whether to trigger degradation processing prematurely before the latest timestamp entry is read. It also serves as a reference factor in the comprehensive evaluation of the rendering lead and is not directly included in the calculation of the video rendering trigger time T_render_trigger. The local reception time refers to the moment when the video domain reads the audio physical output time from the dedicated timestamp transmission channel (lockless circular queue) of the cross-domain shared storage area, which is recorded by the video domain based on the SOC's on-chip global hardware counter.

[0041] Specifically, the video domain reads the physical output time of the audio data written by the audio domain from the cross-domain shared storage area. This time is the reference time at which the audio frame is actually played by the speaker. Since the transmission of the physical output time from the audio domain to the video domain requires writing and reading from the cross-domain shared storage area, there is a cross-domain transmission delay. Directly using this time will cause the video frame display to lag behind the audio output. The video domain can use the physical output time as a reference to assess whether the rendering lead is sufficient by predicting the current cross-domain transmission delay.

[0042] S140. If sufficient, determine the video rendering trigger time of the video frame based on the difference between the target display time and the inherent delay of the video rendering pipeline, and trigger the video frame rendering based on the video rendering trigger time.

[0043] In this embodiment, the inherent latency of the video rendering pipeline refers to the fixed time delay between triggering video frame rendering and the actual display of that frame on the screen. It consists of the physical time consumed by video decoding, rendering compositing, and display refresh, and can be obtained through pre-measurement and calibration. The video rendering trigger moment refers to the specific point in time when the video domain initiates the video rendering action for a particular frame. The display moment of the video frame is aligned with the physical audio output moment on the timeline to achieve audio-visual synchronization. Video frame rendering refers to the process by which the video domain writes the decoded video frame data, after graphics rendering processing, into the frame buffer of the display controller.

[0044] Specifically, the video domain can determine the video rendering trigger time by combining the inherent latency of the video rendering pipeline. This allows it to calculate when a video frame needs to start rendering and when it needs to be displayed on the screen. The video domain writes the video rendering trigger time into the renderer's startup scheduling register, causing the renderer to start rendering the frame when the global hardware counter reaches that time. Simultaneously, the video domain writes the target display time into the display controller's frame toggle register. When the global hardware counter reaches the target display time, the display controller switches the rendered video frame to the display output at the nearest vertical synchronization (VSync) edge, aligning the display time of the video frame with the physical audio output time on the timeline, thus achieving cross-domain audio-visual synchronization. This invention is applicable to display controllers that support frame switching capabilities based on hardware counter-based scheduling.

[0045] The technical solution of this invention obtains the original audio timestamps of key event nodes in the audio frame rendering pipeline based on the on-chip global hardware counter in the audio domain, calibrates the original audio timestamps based on the current acoustic delay error, determines the physical audio output time, and writes it to the cross-domain shared storage area; it obtains the video timestamps of key event nodes in the video frame rendering pipeline based on the on-chip global hardware counter in the video domain; it determines the video rendering trigger time of the video frame based on the difference between the target display time and the inherent delay of the video rendering pipeline, and triggers video frame rendering based on the video rendering trigger time. Except for a one-time balancing of the counter offsets of each domain during the power-on phase, no cross-domain counter calibration is required during operation, eliminating the deviation between the software timestamp and the arrival of the sound wave at the in-vehicle reference measurement point, suppressing delay fluctuations, meeting automotive-grade functional safety requirements, and achieving audio-visual synchronization.

[0046] As a first optional embodiment of this embodiment, based on the above embodiment, the steps for determining the current acoustic delay error can be refined as follows:

[0047] A synchronization marker signal is periodically generated in the audio domain. The synchronization marker signal includes at least one of the following injection methods: acoustic injection and digital injection. The acoustic injection method involves injecting a short-time pulse signal of a specific frequency into the audio signal through the audio output link. The short-time pulse signal is played by a speaker and then collected by a microphone. The digital injection method involves inserting a special frame header marker or a low-energy audible feature marker into the digital audio frame buffer. The special frame header marker or low-energy audible feature marker is converted by a digital-to-analog converter (DAC) and amplified by a power amplifier along with the normal audio data. It is then played by a speaker and collected by a microphone. Its arrival time is detected based on frame header recognition or sequence matching. The planned playback time of the synchronization marker signal is recorded. The audio signal of the audio output link is collected by the microphone as an acoustic feedback signal. The synchronization marker signal is identified from the acoustic feedback signal and the actual playback time is recorded. The current acoustic delay error is determined based on the actual playback time and the planned playback time.

[0048] In this embodiment, the audio output link refers to the complete signal path that audio data takes from its generation in the audio domain to its final playback by the speaker, including DMA transmission, DAC conversion, power amplifier amplification, speaker sound generation, and other stages.

[0049] In this embodiment, the synchronization marker signal includes at least one of the following injection methods:

[0050] (i) Acoustic injection method: The synchronization marker signal is a short-time pulse signal of a specific frequency, which is injected into the audio output link as a synchronization marker. After being played by the speaker, it is collected and identified by the microphone to measure the acoustic delay. For example, a 22kHz sine pulse is preferred. This frequency band is beyond the upper limit of human hearing perception (about 20kHz) and does not affect the user experience. At the same time, the energy of the main noise sources in the vehicle (engine, tires, wind noise) is concentrated below 4kHz, and the signal-to-noise ratio of the 21-24kHz frequency band is high. Moreover, the upper limit of the frequency response range of the automotive-grade microphone covers 24kHz, the pulse duration is 1-10ms, preferably 5ms, and the injection period is 100ms-1000ms, preferably 500ms. It should be noted that when using acoustic injection, the frequency of the synchronization marker signal must be optimally selected under the hardware frequency response constraints of the chosen vehicle model, and should avoid the frequency bands sensitive to human hearing. In vehicles equipped with ultra-wideband speakers (effective frequency response coverage up to 24kHz), a 22kHz sine pulse can be used. In vehicles equipped with conventional speakers (-3dB point at 16-18kHz), since it is impossible to simultaneously satisfy "exceeding the upper limit of human hearing" and "not exceeding the upper limit of the effective frequency response of the speaker", the following digital injection method should be preferred as an alternative.

[0051] (II) Digital Injection Method: The synchronization marker signal is a special frame header marker or low-energy audible feature marker inserted into the digital audio frame buffer. This special frame header marker or low-energy audible feature marker is converted by the DAC and amplified by the power amplifier along with the normal audio data before being played by the speaker (its frequency is within the audible range but its energy is extremely low). The microphone calculates the acoustic delay by detecting the arrival time of the special frame header marker or low-energy audible feature marker. This method is suitable for most conventional vehicle models, is not limited by speaker frequency response, and has minimal impact on user experience. The pulse duration and injection period are the same as the acoustic injection method: the pulse duration is 1-10ms, preferably 5ms; the injection period is 100ms-1000ms, preferably 500ms.

[0052] Specifically, the audio domain generates a synchronization marker signal according to a preset period (e.g., 500ms), and the injection method of the synchronization marker signal includes at least one of the following:

[0053] (1) Acoustic injection method: The audio domain generates a short-time pulse signal of a specific frequency (e.g., a 22kHz sine pulse, suitable for vehicles equipped with ultra-wideband speakers; or, for example, a 16-18kHz sine pulse, suitable for vehicles equipped with conventional speakers) and injects it into the audio output link. This pulse signal will be converted by the DAC and amplified by the power amplifier along with the normal audio content and then played by the speaker. At the same time as injection, the audio domain reads the global hardware counter on the SOC chip and records the planned playback time of the synchronization marker signal. The in-vehicle microphone collects the audio signal played by the speaker in real time as the acoustic feedback signal. The audio domain filters the collected signal through a bandpass filter (e.g., a fourth-order Butterworth filter with a center frequency of the injection frequency and a bandwidth of 1kHz) to suppress low-frequency in-vehicle noise (the main energy of engine, tire and wind noise) and human-audible frequency signals, and extracts characteristic frequency components; when using dual-frequency parallel detection, the difference between the two selected frequency points should be greater than the passband width of a single-channel bandpass filter to ensure that the two channels can be separated. Subsequently, a detection algorithm (such as zero-crossing detection, energy detection, template matching, and autocorrelation detection) is used to identify the synchronization marker signal. After confirming the arrival of the pulse, the on-chip global hardware counter of the SOC is read to record the actual playback time of the signal.

[0054] (2) Digital Injection Method: The audio domain inserts a special frame header marker or low-energy audible feature marker (e.g., a specific frame header synchronization word or low-energy feature sampling sequence) into the digital audio frame buffer. This marker is then converted by the DAC and amplified by the power amplifier along with the normal audio data before being output by the speaker. Simultaneously with insertion, the audio domain reads the global hardware counter on the SOC chip to record the planned playback time of the synchronization marker signal. The in-vehicle microphone collects the audio signal of the audio output link in real time as the acoustic feedback signal. The audio domain detects the actual arrival time of the special frame header marker or low-energy audible feature marker using a frame header recognition algorithm or a sequence matching algorithm, reads the global hardware counter on the SOC chip, and records the actual playback time of the signal. The audio domain calculates the difference between the actual playback time and the planned playback time, which is the acoustic delay error. To suppress occasional noise interference, the audio domain can also continuously collect multiple (e.g., 5) acoustic delay values, remove the maximum and minimum values, and take the median of the remaining values ​​as the final calibration value.

[0055] An example scenario is a smart cockpit equipped with a multi-channel speaker system (such as a 7.1-channel in-vehicle audio system). Specific implementation steps may include:

[0056] Scenario 1 (Vehicle equipped with ultra-wideband speakers): A dual-frequency parallel detection scheme is adopted to enhance the robustness of acoustic feedback. Two characteristic frequency pulse signals with a frequency difference sufficient to be distinguished by a bandpass filter are simultaneously injected (e.g., 22kHz and 19kHz, with a frequency difference of 3kHz greater than the single-channel bandwidth of 1kHz). Simultaneously, the audio domain reads the global hardware counter on the SOC chip and records the planned playback time T_schedule of the synchronization marker signal. These signals are then detected by two independent bandpass filters. The arithmetic mean of the two detection results is taken as the final actual playback time T_detect. Furthermore, multiple consecutive acquisitions (e.g., 5 times) can be performed on each channel, removing the maximum and minimum values, and then taking the median. The final average of the two results is then taken, effectively suppressing the interference of complex in-vehicle acoustic environment (such as engine noise and wind noise) on the accuracy of single-frequency detection. Acoustic feedback calibration suppresses occasional noise interference by taking the median after multiple acquisitions of each channel and removing extreme values, obtaining a stable current acoustic delay error: current acoustic delay error T_acoustic = T_detect - T_schedule. For example, if T_schedule=0.500000s and T_detect=0.503000s, then T_acoustic=3ms; if the original audio timestamp T_dac_out read when any subsequent audio frame is output to the DAC is 0.997000s, then the physical audio output time T_audio_calibrated after acoustic delay error calibration is T_dac_out+T_acoustic=1.000000s.

[0057] Scenario 2 (Vehicles equipped with conventional speakers): A special frame header marker is inserted into the digital audio frame buffer as a synchronization marker. Simultaneously, the audio domain reads the on-chip global hardware counter of the SOC and records the planned playback time T_schedule of this synchronization marker signal. The microphone detects the actual arrival time T_detect of the special frame header marker using a frame header recognition algorithm. The acoustic delay error T_acoustic = T_detect - T_schedule.

[0058] In the first optional embodiment of this example, acoustic feedback closed-loop calibration is introduced under an automotive-grade multi-domain isolation architecture. This involves measuring the end-to-end error from software timestamp recording to actual sound wave output using either acoustic injection or digital injection. The acoustic delay error is then used to calibrate the original audio timestamp of the audio frame, eliminating systematic biases and converting the planned playback time at the software level into the actual physical audio output time. This achieves precise calibration of the error between the software timestamp and the physical acoustic output. The two injection methods can be used independently or simultaneously for mutual verification to improve calibration accuracy.

[0059] Example 2

[0060] Figure 2 This is a flowchart of a cross-domain audio and video synchronization method provided in Embodiment 2 of the present invention. This embodiment is a further refinement of the above embodiment. Figure 2 As shown, the method includes:

[0061] S201. Based on the current acoustic delay error, correct the original audio timestamp at the moment when the audio frame is output to the digital-to-analog converter (DAC) to obtain the physical audio output time.

[0062] The key event nodes of the audio frame rendering pipeline include at least one of the following: the moment the audio frame is submitted to the Direct Memory Access (DMA) buffer, the moment the audio frame is output to the Digital-to-Analog Converter (DAC), and the moment the audio frame is actually played. For example, it can be represented as: T_dma_submit: the moment the audio frame is submitted to the DMA buffer; T_dac_out: the moment the audio frame is output to the DAC; T_speaker: the moment the audio frame is actually played. Here, only the measured value is used on calibration frames where a synchronization marker signal is injected; on other unmarked frames, the estimated value is obtained by calibrating the corresponding audio frame output to the DAC using the currently effective global acoustic delay error, and is not associated with a specific frame sequence number.

[0063] In this embodiment, the digital-to-analog converter (DAC) is used to convert digital audio signals into analog audio signals to drive the speaker for playback. The time when an audio frame is output to the DAC refers to the moment the audio frame is output from the digital domain to the DAC, represented by a global hardware counter value. The time when an audio frame is submitted to the direct memory access (DMA) buffer refers to the moment the audio frame is sent into the DMA buffer, also represented by a global hardware counter value. The actual playback time of the audio frame refers to the estimated time when the audio frame is played by the speaker, obtained after calibrating the corresponding original audio timestamp based on the currently effective global acoustic delay error; wherein, the global acoustic delay error is obtained through periodic acoustic feedback closed-loop calibration and is not associated with a specific frame sequence number.

[0064] Specifically, the audio domain, based on the audio playback scheduling timetable, determines the planned DAC output time T_dac_out_scheduled for future audio frames that have not yet been output to the digital-to-analog converter (DAC) according to the sampling rate and DMA buffer scheduling. This timetable is then added to the current acoustic delay error to obtain the physical audio output time of that future audio frame.

[0065] T_audio_calibrated = T_dac_out_scheduled + T_acoustic, and is written to the cross-domain shared memory area at least "T_video_pipeline + cross-domain transmission delay + safety margin" before the scheduled DAC output time of the future audio frame. This ensures that the video domain has a rendering lead of no less than the inherent delay of the video rendering pipeline when it reads this time. When the audio frame is actually output to the DAC, the audio domain reads the on-chip global hardware counter of the SOC to obtain the measured original audio timestamp, which is used to verify and correct the planned DAC output time and acoustic delay error, forming a closed loop. This time is the reference time for subsequent cross-domain audio-visual synchronization. It should be noted that the acoustic delay error obtained by the acoustic feedback closed-loop calibration is a global calibration parameter and is not associated with a specific frame sequence number; this value is updated after each calibration cycle and used for the timestamp calibration of all subsequent audio frames. The acoustic calibration deviation field in the timestamp entry is filled with the currently effective global acoustic delay error value.

[0066] S202. Pack the original audio timestamp at the moment the audio frame is submitted to the direct memory access DMA buffer with the corresponding event type and frame sequence number into an original timestamp entry.

[0067] The actual playback time of the audio frame is the measured value only on the calibration frame with the injected synchronization marker signal. On the non-marked frame, it is the estimated value obtained by calibrating the original audio timestamp of the corresponding audio frame output to the digital-to-analog converter (DAC) based on the currently effective global acoustic delay error. It is packaged frame by frame without being associated with a specific frame sequence number.

[0068] In this embodiment, the event type refers to a field used to identify the event category to which a timestamp entry belongs, distinguishing between audio and video events. The frame sequence number is an incrementing number used to uniquely identify an audio or video frame, ensuring the association between timestamp entries and corresponding frames, facilitating frame-by-frame synchronization control and tracking. A timestamp entry refers to a data structure that packages timestamp data and its auxiliary information.

[0069] Specifically, the audio domain packages the raw audio timestamp of the moment the audio frame is submitted to the Direct Memory Access (DMA) buffer, along with the corresponding event type (identifying that the event is an audio frame submission to the DMA buffer) and frame sequence number (uniquely identifying the audio frame), into a raw timestamp entry according to a preset byte entry format. The actual playback time of the audio frame is only the measured value on calibration frames and appended to the entry; it is not packaged frame by frame on unmarked frames. This entry fully records the time information of each key event node of the audio frame in the rendering pipeline, facilitating subsequent traceability and synchronization control.

[0070] S203. Pack the audio physical output time with the corresponding event type and frame sequence number into a calibration timestamp entry.

[0071] Specifically, the audio domain will package the corrected audio physical output time, along with the corresponding event type and frame sequence number, into a calibration timestamp entry according to a preset byte entry format.

[0072] For example, each timestamp entry is 30 bytes long, with the following field definitions: Hardware counter timestamp value (timestamp), 8 bytes, filled with the physical audio output time; Send timestamp value (send_timestamp), 8 bytes, filled with the send time T_send recorded based on the global hardware counter when the audio field was written to this entry; Domain identifier (domain_id), 1 byte; Event type (event_type), 1 byte; Frame sequence number (sequence_num), 4 bytes; Acoustic calibration offset (acoustic_offset), 4 bytes, filled with the currently effective global acoustic delay error value; CRC-16 checksum (crc16), 2 bytes; Reserved field (reserved), 2 bytes. Calibration timestamp entries use the same entry format as original timestamp entries, the difference being that the timestamp field is filled with the corrected physical audio output time instead of the original audio timestamp.

[0073] S204. Write the original timestamp entry and the calibration timestamp entry to the dedicated timestamp transfer channel of the cross-domain shared storage area through the write pointer of the lockless circular queue.

[0074] In this embodiment, the lock-free circular queue refers to a concurrent data structure used for cross-domain data transfer. The lock-free circular queue is set up in a dedicated timestamp transfer channel within the cross-domain shared storage area, with a preferred queue depth of 64 entries. The dedicated timestamp transfer channel is a lock-free circular queue partitioned within the cross-domain shared storage area and physically isolated from ordinary data transmission channels. It is dedicated to the cross-domain transfer of timestamp entries, avoiding contention for bandwidth and lock resources with ordinary data communication, and ensuring low latency and determinism in timestamp transfer. The write pointer refers to a pointer in the lock-free circular queue that points to the next position to be written. After each data entry is written, the writing end moves this pointer forward by one position through an atomic operation, ensuring the atomicity and visibility of the write operation, allowing for safe updates without locking.

[0075] Specifically, the audio domain writes the packaged timestamp entries (including original and calibration timestamp entries) to the cross-domain shared memory area via the write pointer of a dedicated timestamp transmission channel (i.e., a lock-free circular queue). The write end maintains the write pointer atomically, writing the entry to the queue position pointed to by the write pointer and then atomically updating the write pointer. The lock-free circular queue adopts a single producer (audio domain) and single consumer (video domain) structure, completing the write operation without locking. This avoids the risks of priority inversion and deadlock caused by mutex locks, ensuring low latency and determinism in timestamp transmission, meeting the timing determinism requirements of automotive-grade functional safety. After the write is complete, the video domain can detect the arrival of new data by polling the difference between the read and write pointers and read the timestamp entries from the queue for subsequent synchronization calculations.

[0076] The dedicated timestamp transmission channel is a lock-free circular queue that is physically isolated from the ordinary data transmission channel in the cross-domain shared storage area. The lock-free circular queue has a single producer and single consumer structure, with the writing end being the audio domain and the reading end being the video domain. The writing end maintains the write pointer through atomic operations, and the reading end maintains the read pointer through atomic operations. The reading end determines whether new data has arrived by polling the difference between the read and write pointers.

[0077] In this embodiment, the ordinary data transmission channel refers to the channel used in the cross-domain shared storage area for transmitting regular cross-domain data (such as non-timestamped audio and video metadata, control commands, etc.). It is physically isolated from the dedicated timestamp transmission channel. This physical isolation avoids bandwidth contention and lock contention caused by ordinary data transmission, ensuring that the real-time performance and determinism of the dedicated timestamp transmission channel are not affected. The single-producer, single-consumer structure refers to a working mode of a lock-free circular queue, where only one producer (writer) writes data to the queue, and only one consumer (reader) reads data from the queue. This structure eliminates the need for complex multi-producer or multi-consumer synchronization mechanisms, simplifying concurrency control and ensuring orderly data transmission. In this embodiment, the producer is the audio domain, and the consumer is the video domain. The writer is the producer responsible for writing data to the queue in the lock-free circular queue. The reader is the consumer responsible for reading data from the queue in the lock-free circular queue. An atomic operation is a single, uninterruptible operation that either succeeds completely or fails entirely, preventing interruption by other operations. The read pointer is a pointer in the lock-free circular queue that points to the next position to be read.

[0078] Specifically, the lock-free circular queue adopts a single-producer, single-consumer model, where the producer is uniquely identified as the audio domain and the consumer as the video domain. The audio domain writes the calibrated physical output time of the audio as a timestamp entry to the queue, and the video domain reads this entry from the queue for cross-domain audio-visual synchronization alignment calculations. The unidirectional transmission mechanism ensures a clear data flow direction, and the single-producer, single-consumer structure simplifies concurrency control, eliminating the need for complex multi-producer or multi-consumer synchronization logic and ensuring that timestamp entries are read in the order they are written. After writing each timestamp entry, the audio domain (writer) moves the write pointer forward by one position through an atomic operation; after reading each timestamp entry, the video domain (reader) moves the read pointer forward by one position through an atomic operation. Atomic operations guarantee the integrity and visibility of pointer updates, allowing the writer and reader to safely share and update pointers without locking, avoiding data races and inconsistencies. The atomic operations, combined with the lock-free design, ensure predictable read / write operation latency, suitable for automotive-grade real-time requirements. The video domain (read end) continuously checks the difference between the write pointer and the read pointer. When the difference is greater than 0, it indicates that a new timestamp entry has been written to the audio domain and has not yet been read. The video domain then reads the entry from the position pointed to by the read pointer and atomically updates the read pointer after reading. The polling method does not rely on interrupts or semaphores, avoiding additional inter-domain communication overhead and priority inversion risks. Combined with a lock-free design, read and write latency is controlled at the microsecond level, meeting the real-time requirements of high-precision audio-visual synchronization. At the same time, the number of backlogged entries can be obtained by comparing the difference, which helps the system determine the load status of the transmission channel.

[0079] S205. Obtain the video timestamps of key event nodes in the video frame rendering pipeline through the video domain based on the SOC on-chip global hardware counter.

[0080] Specifically, the video domain can determine if new data has arrived by polling the difference between the read and write pointers of the lock-free circular queue. When the difference is greater than 0, the video domain reads the timestamp entry from the position pointed to by the read pointer and performs a CRC-16 check. After the check passes, the audio physical output time T_audio_calibrated is parsed from the entry. After reading, the video domain updates the read pointer through atomic operations. This step completes the transfer of the audio physical output time from the audio domain to the video domain through a cross-domain shared memory area.

[0081] S206. Based on the cross-domain delay prediction model, determine the predicted current cross-domain transmission delay.

[0082] In this embodiment, the cross-domain delay prediction model refers to a mathematical model that predicts future delays based on historical cross-domain transmission delay data. Preferably, algorithms such as Kalman filters, adaptive filters, particle filters, or exponential smoothing prediction are used. The measured cross-domain delay is input as an observation value into the model for state updates, and the predicted delay is output. This predicted delay is used to evaluate rendering lead and predict degradation processing. The model input is the difference between the cross-domain timestamp sending and receiving times, and the output is the predicted current cross-domain transmission delay D_predict, used to evaluate whether the rendering lead after the video domain reads the audio physical output time is sufficient and to predict whether degradation processing should be triggered.

[0083] Specifically, the video domain can obtain the predicted cross-domain transmission delay D_predict at the current moment from the cross-domain delay prediction model, which is used to evaluate the rendering lead and predict whether to trigger degradation processing.

[0084] Furthermore, based on the above embodiments, the steps for determining the predicted current cross-domain transmission delay based on the cross-domain delay prediction model can be refined as follows:

[0085] Obtain the historical transmission time when the audio field in the historical timestamp entry was written to the historical audio physical output time, and obtain the historical local reception time of the historical audio physical output time. Use the difference between the historical local reception time and the historical transmission time as the cross-domain delay observation value. Adjust the model parameters in the cross-domain delay prediction model according to the current operating condition signal to obtain the current cross-domain delay prediction model. Input the cross-domain delay observation value into the current cross-domain delay prediction model to obtain the predicted current cross-domain transmission delay.

[0086] In this embodiment, the historical transmission time refers to the moment when the audio domain historically wrote the audio physical output time into the dedicated timestamp transmission channel (lockless circular queue) of the cross-domain shared storage area, and the time when the audio domain read and recorded the write operation based on the SOC on-chip global hardware counter, represented by the hardware counter value. The historical local reception time refers to the moment when the video domain historically read the audio physical output time from the dedicated timestamp transmission channel (lockless circular queue) of the cross-domain shared storage area, and the time when the video domain read and recorded the read operation based on the SOC on-chip global hardware counter. The cross-domain delay observation value refers to the measured cross-domain transmission delay value obtained by subtracting the historical local reception time from the historical transmission time, reflecting the actual time delay experienced by the audio physical output time from the audio domain to the video domain. The current operating condition signal refers to information reflecting the current operating status of the vehicle and the SOC load status, including the vehicle speed signal output by the vehicle speed sensor and the load signal output by the SOC load monitoring module. The model parameters refer to the parameters controlling the response characteristics of the filter in the cross-domain delay prediction model. The current cross-domain delay prediction model refers to the cross-domain delay prediction model that is adapted to the current operating state after adjusting the model parameters according to the current operating condition signal.

[0087] Specifically, when the audio domain writes the calibrated audio physical output time to the cross-domain shared storage area, it records the time of this write operation as the transmission time T_send based on the SOC's on-chip global hardware counter, and writes it into the queue as part of a timestamp entry. When the video domain reads this entry from the queue, it parses the transmission time T_send and reads the current count value based on the SOC's on-chip global hardware counter as the local reception time T_recv. The video domain calculates the difference between T_recv and T_send to obtain the cross-domain delay observation D_observed = T_recv - T_send. The video domain acquires the current operating condition signal (such as the vehicle speed signal output by the vehicle speed sensor and the load signal output by the SOC load monitoring module) and dynamically adjusts the model parameters in the cross-domain delay prediction model according to changes in operating conditions. The video domain inputs the calculated cross-domain delay observation value D_observed as the observation value into the current cross-domain delay prediction model (preferably using a Kalman filter) after the operating conditions are adjusted. After the model performs the state update step, it outputs the predicted current cross-domain transmission delay D_predict.

[0088] For example, a cross-domain delay prediction model can employ a Kalman filter, where the Kalman filter state variables are defined as follows:

[0089] X = [D, ΔD]ᵀ

[0090] Where: D is the estimated cross-domain transmission delay, and ΔD is the delay change rate.

[0091] The state equation is:

[0092] X(k+1) = A·X(k) + w(k)

[0093] Where: A = [1, Δt; 0, 1], Δt is the sampling interval (in seconds), and w(k) is the process noise.

[0094] The observation equation is:

[0095] Z(k) = H·X(k) + v(k)

[0096] Where: H = [1, 0], v(k) is the observation noise.

[0097] Initialization of the process noise covariance matrix Q:

[0098] Q = [σ_D², 0; 0, σ_ΔD²]

[0099] Where: σ_D is the standard deviation of cross-domain transmission delay (in μs), and σ_ΔD is the standard deviation of the rate of change of delay (in μs / s). Since the state components D and ΔD have different dimensions, the two diagonal elements of the Q matrix have different dimensions, μs² and (μs / s)², respectively. σ_D and σ_ΔD can be taken as empirical values, for example...

[0100] For example, σ_D takes 50μs and σ_ΔD takes 10μs / s.

[0101] Initialization of observation noise covariance R:

[0102] R = [σ_measure²]

[0103] Where: σ_measure is the standard deviation of the timestamp measurement error (in μs), which can be taken as an empirical value, such as 20μs.

[0104] The prediction step is as follows: D_hat = X1(k|k-1), which is the first component of the prior estimate of the state vector; after the update step (Kalman gain correction), the first component of the posterior estimate X1(k|k) is taken as the current cross-domain transmission delay D_predict.

[0105] When a shift in SOC load from low to high is detected, the σ_ΔD value in the process noise covariance matrix Q is increased to make the filter more sensitive to the rate of change in delay. When a shift in operating condition from idling to high-speed driving is detected, the σ_D value is increased (e.g., from 50μs to 100μs) to make the filter more sensitive to the absolute value of delay, adapting to changes in inter-domain communication delay in high-speed scenarios. Once the operating condition stabilizes, the value returns to normal. By adjusting the model parameters, the cross-domain delay prediction model is adapted to the current operating conditions, enabling rapid tracking when delay fluctuations are large and smooth filtering when delays are stable.

[0106] S207. When the video domain has not yet read the audio physical output time corresponding to the current frame from the cross-domain shared storage area, calculate the predicted available margin based on the current cross-domain transmission delay. If the predicted available margin is less than the inherent delay of the video rendering pipeline, trigger the degradation process in advance.

[0107] In this embodiment, the predicted available margin can be understood as a value used to pre-evaluate whether the video domain has enough time to complete rendering before the timestamp is actually read.

[0108] Specifically, when the video domain has not yet read the audio physical output time corresponding to the current frame from the cross-domain shared storage area, since the measured available margin cannot be obtained, the video domain can predict the rendering advance based on the current cross-domain transmission delay prediction value D_predict output by the cross-domain delay prediction model of the previous cycle. Specifically, it derives the estimated value of the audio physical output time and the estimated value of the local reception time based on D_predict and calculates the predicted available margin. The video domain evaluates whether the rendering advance is sufficient using the available margin T_audio_calibrated-T_recv: if the predicted available margin is greater than the inherent delay of the video rendering pipeline, i.e., T_audio_calibrated-T_recv>T_video_pipeline, rendering is triggered according to the target display time; otherwise, degradation processing is performed, which includes at least one of the following: delaying the video frame to the next alignable vertical synchronization cycle, maintaining the display output of the previous frame, or discarding the frame and re-evaluating the synchronization status in the next calibration cycle. The predicted current cross-domain transmission delay D_predict is used to predict the rendering lead and decide whether to trigger degradation in advance before the latest timestamp entry is read. It also serves as a reference factor for the comprehensive evaluation of the rendering lead and is not directly included in the calculation formula of the video rendering trigger time T_render_trigger.

[0109] S208. If the predicted available margin is greater than or equal to the inherent latency of the video rendering pipeline, then read the audio physical output time from the dedicated timestamp transfer channel of the cross-domain shared storage area.

[0110] S209. Use the physical output time of the audio as the target display time of the video frame, and evaluate whether the rendering lead after the video domain reads the physical output time of the audio is sufficient based on the difference between the physical output time of the audio and the local reception time.

[0111] In this embodiment, the target display time refers to the time when the video frame is expected to be displayed on the screen, which is equal to the audio physical output time; the rendering lead refers to the time margin available for video rendering between the time the video domain reads the target display time and the time the target display time arrives; the video rendering trigger time is obtained by subtracting the inherent delay of the video rendering pipeline from the target display time, that is, the video domain starts rendering at the trigger time, so that the display time of the video frame after the inherent delay of the pipeline is aligned with the audio physical output time.

[0112] Specifically, the video domain uses the physical audio output time T_audio_calibrated as the target display time T_video_display of the video frame (i.e., T_video_display = T_audio_calibrated). Since T_audio_calibrated is determined based on the original audio timestamp at the time the audio frame is output to the digital-to-analog converter (DAC) and the current acoustic delay error, representing the absolute time when the sound wave arrives at the reference measurement point inside the vehicle, this time is a fixed reference on the time axis and does not change with its transmission between the audio and video domains. Therefore, the target display time is directly taken as the arrival time itself, without needing to add or subtract cross-domain transmission delay. When the video domain reads this time, it is already lagging behind the audio domain's write by one cross-domain transmission delay. This lag is already reflected in the local reception time T_recv, so there is no need to add or subtract the cross-domain transmission delay again to the target display time.

[0113] S210. If sufficient, determine the video rendering trigger time of the video frame based on the difference between the target display time and the inherent delay of the video rendering pipeline, and trigger the video frame rendering based on the video rendering trigger time.

[0114] Specifically, the video domain calculates the difference between the target display time T_video_display and the inherent delay of the video rendering pipeline T_video_pipeline to obtain the video frame's video rendering trigger time (e.g., determined by the formula: T_render_trigger = T_video_display - T_video_pipeline, where T_video_display = T_audio_calibrated). This calculation, after deducting the inherent delay of the video rendering pipeline, ensures that the video frame starts rendering at the video rendering trigger time, and the display time after the pipeline's inherent delay is aligned with the audio physical output time. The video domain writes the video rendering trigger time into the renderer's startup scheduling register, causing the renderer to start rendering the frame when the on-chip global hardware counter reaches that time. Simultaneously, it writes the target display time into the display controller's frame toggle register. When the on-chip global hardware counter reaches the target display time, the display controller switches the rendered video frame to the display output at the nearest vertical synchronization (VSync) edge, aligning the video frame's display time with the audio physical output time on the timeline, achieving cross-domain audio-visual synchronization.

[0115] For example, the video rendering trigger time T_render_trigger can be calculated using the following formula:

[0116] T_render_trigger=T_video_display-T_video_pipeline

[0117] For example, taking a vehicle equipped with an ultra-wideband speaker as an example, the main end-to-end synchronization process is as follows:

[0118] (1) Acoustic calibration: Two characteristic frequency pulse signals of 22kHz and 19kHz are periodically injected simultaneously in the audio domain. The planned playback time T_schedule=0.500000s is recorded by the global hardware counter. The two bandpass filters are detected and the arithmetic mean is taken to obtain the actual playback time T_detect=0.503000s. Then the current acoustic delay error T_acoustic=T_detect-T_schedule=3ms.

[0119] (2) Determining the physical output time of audio (degradation special case): For audio frames that cannot be written in advance before the planned DAC output time, such as frames that have not been pre-arranged at the beginning of system startup or frames with insufficient advance due to scheduling abnormalities, the audio domain acquires and writes according to the current frame in real time. Its rendering advance is usually insufficient and needs to be degraded. Taking frame A as an example, the degradation path is explained: When the frame is output to the DAC, the global hardware counter is read to obtain the original audio timestamp T_dac_out=0.997000s. The audio domain adds the current acoustic delay error to it for calibration, and obtains the physical output time of audio T_audio_calibrated=T_dac_out+T_acoustic=0.997000+0.003=1.000000s. It is packaged into a timestamp entry and written to the dedicated timestamp transmission channel of the cross-domain shared storage area. At the same time, the sending time T_send=0.998000s is recorded.

[0120] (3) Cross-domain transmission and delay prediction: When the video domain reads the entry from the queue, it reads the local reception time T_recv=0.999000s, and the cross-domain delay observation D_observed=T_recv-T_send=1ms; after filtering by the current cross-domain delay prediction model (Kalman filter), the output D_predict=1.05ms. It should be noted that during the period when the video domain has not yet read the latest timestamp entry, the rendering advance of the next frame can be predicted based on D_predict obtained in the previous cycle: if the available prediction margin T_audio_calibrated_est-T_recv_est is less than T_video_pipeline, then the degradation is triggered in advance, without waiting for the entry to be read, thus avoiding the judgment lag.

[0121] (4) Rendering advance assessment: The inherent delay of the video rendering pipeline is T_video_pipeline = 16ms. The available margin is T_audio_calibrated - T_recv = 1.000000 - 0.999000 = 1ms, which is less than T_video_pipeline (16ms). The rendering advance is insufficient, triggering degradation processing and delaying the frame to the next alignable vertical synchronization cycle. In this example, the judgment is based on the measured margin after the latest timestamp entry has been read. When the latest timestamp entry has not yet been read, the predicted current cross-domain transmission delay D_predict is used to predict the rendering advance and decide whether to trigger degradation in advance. D_predict is also used as a reference factor for the comprehensive evaluation of rendering advance and does not directly enter the calculation formula of T_render_trigger at the video rendering trigger time.

[0122] (5) Rendering trigger and alignment: The audio frame is a pre-arranged future audio frame. The audio domain has written T_audio_calibrated=1.020000s to the cross-domain shared storage area before its planned DAC output time (1.017s). The video domain reads this time at T_recv=1.000000s, and the available margin is 20ms (1.020000-1.000000), which is greater than T_video_pipeline (16ms). Therefore, the video rendering trigger time T_render_trigger=T_audio_calibrated-T_video_pipeline=1.020000-0.016=1.004000s. The video domain starts rendering at this time and displays it at 1.020000s after the inherent delay of 16ms pipeline, which is aligned with the audio physical output time.

[0123] The technical solution of this invention directly reads the same SOC global hardware counter as the time base from each domain, eliminating the need for inter-counter calibration and significantly reducing system complexity and protocol overhead. Time base acquisition requires only one register read operation, resulting in extremely low latency. The hardware counter is bound to key event nodes of the audio / video rendering pipeline for audio-visual synchronization, rather than a general clock synchronization. The timestamp directly reflects the actual progress of the rendering pipeline and can be directly used for synchronization control, eliminating the semantic gap between the timestamp and actual playback. A Kalman filter latency prediction model is established to predict cross-domain latency in advance to assess rendering lead and anticipate degradation processing, transforming passive synchronization into proactive predictive synchronization. Compared to the passive detection and adjustment mode of existing technologies, this effectively suppresses the accumulation of synchronization errors caused by dynamic fluctuations in cross-domain transmission latency. A lock-free circular queue design avoids lock contention, meeting the automotive-grade functional safety requirements for timing determinism and memory safety. Read / write latency is low and predictable, avoiding packet loss. Compared to existing solutions that rely on complex calibration mechanisms or passive compensation, this method significantly improves cross-domain audio-visual synchronization accuracy from the perspective of end-to-end error calibration and proactive prediction of rendering lead. The core computations include hardware counter reading, addition, subtraction, multiplication, and 2×2 matrix operations. A single Kalman filter state update is a constant-level operation. The overall computational complexity increases linearly with the number of events N. No GPU / DSP acceleration is required. It can be processed in real time on a regular CPU of the vehicle SoC, resulting in low computational overhead.

[0124] Furthermore, after completing the continuous monitoring of the acoustic feedback closed-loop calibration results, the current acoustic delay error fluctuation is calculated. Specifically, a synchronization marker signal is periodically injected (either acoustically or digitally), and the actual acoustic output is collected using a microphone. The measured acoustic delay error of the current calibration cycle is compared with the acoustic delay error of the previous calibration cycle to obtain the current acoustic delay error fluctuation value. This monitoring process uses the same hardware and detection algorithm as the acoustic feedback calibration in the above steps, requiring no additional hardware overhead. The current acoustic delay error fluctuation data is obtained after each calibration cycle (preferably 500ms). When the acoustic delay error fluctuation continuously exceeds a preset threshold, it is determined that the current operating conditions have changed drastically (such as a sudden large fluctuation in SOC load, a change in inter-domain scheduling strategy, etc.), causing the existing prediction model and calibration parameters to be unable to effectively track delay changes. At this time, a recalibration process is triggered. The preset threshold should be no less than a single frame display cycle, which can be approximately 16ms for a 60Hz display.

[0125] The recalibration process includes resetting the Kalman filter state, remeasuring the inherent latency of the video rendering pipeline, and forcing a complete acoustic feedback calibration to update the acoustic latency error. After recalibration, the system resumes normal operation. Furthermore, if D_predict predicts a degradation, but subsequent readings of timestamp entries reveal sufficient margin T_audio_calibrated-T_recv, the degradation decision is canceled, and rendering is triggered at the target display time based on the measured value, avoiding frame dropping due to prediction errors.

[0126] For example, the overall approach can be explained as a system. Figure 3This is a system architecture diagram of a cross-domain audio and video synchronization method provided in Embodiment 2 of the present invention. The automotive-grade SOC multi-domain isolation architecture system includes a global hardware counter inside the SOC chip, a video source domain (entertainment domain), an audio output domain (cockpit domain), a shared memory area, a synchronization controller, speakers, and a reference microphone. The global hardware counter is a hardware module at the SOC chip level. Both the video source domain (same as the video domain mentioned above) and the audio output domain (same as the audio domain mentioned above) directly read the same global hardware counter to obtain timestamps, ensuring a naturally unified time base and eliminating the need for cross-domain calibration. The video source domain contains a video decoder, a video renderer, and a display controller. When a key event node of its rendering pipeline occurs (including at least one of the following: video frame decoding completion time, video frame rendering completion time, video frame submission to the display queue time, and video frame display time), the video domain directly reads the global hardware counter to obtain the video timestamp, which is used to record the actual progress of the video rendering pipeline. The audio output domain contains an audio decoder and a DSP processor. When a key event node of its rendering pipeline occurs, the audio domain directly reads the global hardware counter to obtain the original audio timestamp. The audio domain simultaneously generates a synchronization marker signal (the synchronization marker signal includes at least one of the following injection methods: injecting a short-time pulse signal of a specific frequency through the audio output link, or inserting a special frame header marker or a low-energy audible feature marker into the digital audio frame buffer). The acoustic delay error is determined by the acoustic feedback signal collected by the reference microphone. Based on this acoustic delay error, the original audio timestamp is calibrated to determine the physical audio output time, and then packaged into timestamp entries and written to the timestamp transfer channel in the shared memory area. A dedicated timestamp transfer channel (lockless circular queue) is partitioned within the shared memory area, physically isolated from the ordinary data transmission channel. A single producer-single consumer structure is adopted, with the writing end being the audio output domain and the reading end being the video source domain. Low-latency transfer of timestamp entries is achieved by maintaining read and write pointers through atomic operations. The synchronization controller is used to implement cross-domain audio-visual synchronization decisions and control. The synchronization controller uses the physical audio output time as the target display time of the video frame. It assesses whether the rendering lead after the video domain reads the physical audio output time is sufficient based on the difference between the physical audio output time and the local reception time. Combining this with the inherent latency of the video rendering pipeline, it determines whether to trigger rendering at the target display time, ensuring that the target display time of the video frame is aligned with the physical audio output time on the timeline. Simultaneously, the synchronization controller continuously monitors acoustic feedback results and dynamically updates the cross-domain delay prediction model parameters based on synchronization errors. When the synchronization error continuously exceeds a preset threshold, a recalibration process is triggered. The Kalman filter predictor, located within the synchronization controller, takes the cross-domain delay observation (the difference between the local reception time and the transmission time) as input and outputs the predicted current cross-domain transmission delay after state updates. The cross-domain delay observation can optionally be superimposed with operating condition signals such as SOC load. The synchronization controller also adjusts the model parameters of the Kalman filter predictor based on the current operating condition signals, adapting the predictor to the current operating conditions.A reference microphone is installed inside the vehicle to collect acoustic feedback signals from the audio output link. The synchronization controller identifies the synchronization marker signal from the acoustic feedback signal and records the actual playback time. It determines the acoustic delay error based on the difference between the actual and planned playback times, thus achieving closed-loop calibration of the physical audio output time. When the synchronization marker signal is digitally injected, the reference microphone is used to collect the acoustic signal generated after passing through the audio output link by special frame header markers or low-energy audible feature markers. The synchronization marker signal is detected and identified using a frame header recognition algorithm or a sequence matching algorithm.

[0127] Example 3

[0128] Figure 4 This is a schematic diagram of the structure of an automotive-grade system-on-a-chip provided in Embodiment 3 of the present invention. Figure 4 As shown, the system includes an audio domain 31 and a video domain 32.

[0129] The audio domain 31 is used to obtain the original audio timestamp of the key event node of the audio frame rendering pipeline based on the global hardware counter on the SOC, and to calibrate the original audio timestamp based on the current acoustic delay error, determine the physical output time of the audio, and write it into the cross-domain shared storage area.

[0130] The video field 32 is used to obtain the video timestamps of key event nodes in the video frame rendering pipeline based on the on-chip global hardware counter of the SOC.

[0131] The physical output time of the audio is used as the target display time of the video frame, and the rendering advance after the video domain reads the physical output time is evaluated based on the difference between the physical output time of the audio and the local reception time.

[0132] If sufficient, the video rendering trigger time of the video frame is determined based on the difference between the target display time and the inherent delay of the video rendering pipeline. The video frame rendering is triggered based on the video rendering trigger time, and the display time of the video frame after the inherent delay of the video rendering pipeline is aligned with the audio physical output time.

[0133] The technical solution of this invention obtains the original audio timestamps of key event nodes in the audio frame rendering pipeline based on the on-chip global hardware counter in the audio domain, calibrates the original audio timestamps based on the current acoustic delay error, determines the physical audio output time, and writes it to the cross-domain shared storage area; it obtains the video timestamps of key event nodes in the video frame rendering pipeline based on the on-chip global hardware counter in the video domain; it determines the video rendering trigger time of the video frame based on the difference between the target display time and the inherent delay of the video rendering pipeline, and triggers video frame rendering based on the video rendering trigger time. Except for a one-time balancing of the counter offsets of each domain during the power-on phase, no cross-domain counter calibration is required during operation, eliminating the deviation between the software timestamp and the arrival of the sound wave at the in-vehicle reference measurement point, suppressing delay fluctuations, meeting automotive-grade functional safety requirements, and achieving audio-visual synchronization.

[0134] Furthermore, the key event nodes of the audio frame rendering pipeline include at least one of the following: the moment when the audio frame is submitted to the direct memory access (DMA) buffer, the moment when the audio frame is output to the digital-to-analog converter (DAC), and the moment when the audio frame is actually played. Accordingly, the audio domain 31 is specifically used for:

[0135] Based on the current acoustic delay error, the original audio timestamp at the moment when the audio frame is output to the digital-to-analog converter (DAC) is corrected to obtain the physical audio output time.

[0136] The original audio timestamp at the moment the audio frame is submitted to the Direct Memory Access (DMA) buffer is packaged with the corresponding event type and frame sequence number into an original timestamp entry. The actual playback time of the audio frame is a measured value only on the calibration frame with the injected synchronization marker signal, and an estimated value obtained by calibrating the original audio timestamp at the moment the corresponding audio frame is output to the digital-to-analog converter (DAC) based on the currently effective global acoustic delay error. The timestamps are packaged frame by frame without being associated with a specific frame sequence number.

[0137] The audio physical output time, along with the corresponding event type and frame sequence number, is packaged into a calibration timestamp entry;

[0138] The original timestamp entry and the calibration timestamp entry are written to the dedicated timestamp transfer channel of the cross-domain shared storage area using the write pointer of the lockless circular queue.

[0139] The dedicated timestamp transmission channel is a lock-free circular queue physically isolated from the ordinary data transmission channel within the cross-domain shared storage area; the lock-free circular queue has a single producer-single consumer structure, the writing end is the audio domain, and the reading end is the video domain; the writing end maintains the write pointer through atomic operations, the reading end maintains the read pointer through atomic operations, and the reading end determines whether new data has arrived by polling the difference between the read and write pointers.

[0140] The key event nodes of the video frame rendering pipeline include at least one of the following: video frame decoding completion time, video frame rendering completion time, video frame submission to the display queue time, and video frame display time.

[0141] Furthermore, the video domain 32 is specifically used for:

[0142] Based on the cross-domain delay prediction model, the predicted current cross-domain transmission delay is determined;

[0143] When the video domain has not yet read the audio physical output time corresponding to the current frame from the cross-domain shared storage area, the predicted available margin is calculated based on the current cross-domain transmission delay. If the predicted available margin is less than the inherent delay of the video rendering pipeline, the degradation processing is triggered in advance.

[0144] If the predicted available margin is greater than or equal to the inherent latency of the video rendering pipeline, then the audio physical output time is read from the dedicated timestamp delivery channel of the cross-domain shared storage area;

[0145] The physical output time of the audio is used as the target display time of the video frame, and the rendering advance after the video domain reads the physical output time is evaluated based on the difference between the physical output time of the audio and the local reception time.

[0146] Furthermore, the video domain 32 is also specifically used for:

[0147] Obtain the historical transmission time when the audio field in the historical timestamp entry was written to the historical audio physical output time, and obtain the historical local reception time of the historical audio physical output time. Use the difference between the historical local reception time and the historical transmission time as the cross-domain delay observation value.

[0148] Adjust the model parameters in the cross-domain delay prediction model according to the current operating condition signal to obtain the current cross-domain delay prediction model;

[0149] The cross-domain delay observation is input into the current cross-domain delay prediction model to obtain the predicted current cross-domain transmission delay.

[0150] Optionally, the audio domain 31 is further used for:

[0151] A synchronization marker signal is periodically generated through the audio domain, and the synchronization marker signal includes at least one of the following injection methods: acoustic injection method and digital injection method;

[0152] The acoustic injection method involves injecting a short-time pulse signal of a specific frequency into an audio signal through an audio output link. The short-time pulse signal is played by a speaker and then collected by a microphone.

[0153] The digital injection method involves inserting a special frame header marker or a low-energy audible feature marker into the digital audio frame buffer. The special frame header marker or low-energy audible feature marker is converted by a digital-to-analog converter (DAC) along with the normal audio data, amplified by a power amplifier, and then played by a speaker. The arrival time is detected by a microphone based on frame header recognition or sequence matching.

[0154] Record the planned playback time of the synchronization marker signal;

[0155] The audio signal from the audio output link is collected by the microphone as an acoustic feedback signal, and the synchronization marker signal is identified from the acoustic feedback signal and the actual playback time is recorded.

[0156] The current acoustic delay error is determined based on the actual playback time and the planned playback time.

[0157] The automotive-grade on-chip system provided in this embodiment of the invention can execute the cross-domain audio and video synchronization method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0158] Example 4

[0159] Figure 5 A schematic diagram of an electronic device 40 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0160] like Figure 5 As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded from storage unit 48 into the RAM 43. The RAM 43 may also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0161] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0162] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as cross-domain audio and video synchronization methods.

[0163] In some embodiments, the cross-domain audio / video synchronization method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the cross-domain audio / video synchronization method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to perform the cross-domain audio / video synchronization method by any other suitable means (e.g., by means of firmware).

[0164] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0165] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0166] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0167] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0168] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0169] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0170] In one embodiment, the present invention further includes a computer program product, which includes a computer program that, when executed by a processor, implements the cross-domain audio and video synchronization method of any embodiment of the present invention.

[0171] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0172] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0173] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A cross-domain audio and video synchronization method, characterized in that, Applied to automotive-grade system-on-a-chip (SoC), the SoC includes a video domain and an audio domain, the method includes: The audio domain obtains the original audio timestamps of key event nodes in the audio frame rendering pipeline based on the on-chip global hardware counter of the SOC, and calibrates the original audio timestamps based on the current acoustic delay error to determine the physical audio output time and write it to the cross-domain shared storage area. The video timestamps of key event nodes in the video frame rendering pipeline are obtained through the video domain based on the on-chip global hardware counter of the SOC. The physical output time of the audio is used as the target display time of the video frame, and the rendering advance after the video domain reads the physical output time is evaluated based on the difference between the physical output time of the audio and the local reception time. If sufficient, the video rendering trigger time of the video frame is determined based on the difference between the target display time and the inherent delay of the video rendering pipeline. The video frame rendering is triggered based on the video rendering trigger time, and the display time of the video frame after the inherent delay of the video rendering pipeline is aligned with the audio physical output time.

2. The method according to claim 1, characterized in that, The key event nodes of the audio frame rendering pipeline include at least one of the following: the time when the audio frame is submitted to the direct memory access (DMA) buffer, the time when the audio frame is output to the digital-to-analog converter (DAC), and the time when the audio frame is actually played. Correspondingly, the step of obtaining the original audio timestamp of the key event node of the audio frame rendering pipeline through the audio domain based on the on-chip global hardware counter of the SOC, calibrating the original audio timestamp based on the current acoustic delay error, determining the physical audio output time, and writing it to the cross-domain shared memory area includes: Based on the current acoustic delay error, the original audio timestamp at the moment when the audio frame is output to the digital-to-analog converter (DAC) is corrected to obtain the physical audio output time. The original audio timestamp at the moment the audio frame is submitted to the Direct Memory Access (DMA) buffer is packaged with the corresponding event type and frame sequence number into an original timestamp entry. The actual playback time of the audio frame is a measured value only on the calibration frame with the injected synchronization marker signal, and an estimated value obtained by calibrating the original audio timestamp at the moment the corresponding audio frame is output to the digital-to-analog converter (DAC) based on the currently effective global acoustic delay error. The timestamps are packaged frame by frame without being associated with a specific frame sequence number. The audio physical output time, along with the corresponding event type and frame sequence number, is packaged into a calibration timestamp entry; The original timestamp entry and the calibration timestamp entry are written to the dedicated timestamp transfer channel of the cross-domain shared storage area using the write pointer of the lockless circular queue.

3. The method according to claim 2, characterized in that, The dedicated timestamp transmission channel is a lock-free circular queue that is physically isolated from the ordinary data transmission channel within the cross-domain shared storage area; the lock-free circular queue has a single producer and single consumer structure, the writing end is the audio domain, and the reading end is the video domain; The write end maintains the write pointer through atomic operations, the read end maintains the read pointer through atomic operations, and the read end determines whether new data has arrived by polling the difference between the read and write pointers.

4. The method according to claim 1, characterized in that, The key event nodes of the video frame rendering pipeline include at least one of the following: the time when video frame decoding is completed, the time when video frame rendering is completed, the time when video frame is submitted to the display queue, and the time when video frame is displayed.

5. The method according to claim 1, characterized in that, The step of using the physical audio output time as the target display time of the video frame, and evaluating whether the rendering lead after the video domain reads the physical audio output time is sufficient based on the difference between the physical audio output time and the local reception time, includes: Based on the cross-domain delay prediction model, the predicted current cross-domain transmission delay is determined; When the video domain has not yet read the audio physical output time corresponding to the current frame from the cross-domain shared storage area, the predicted available margin is calculated based on the current cross-domain transmission delay. If the predicted available margin is less than the inherent delay of the video rendering pipeline, the degradation processing is triggered in advance. If the predicted available margin is greater than or equal to the inherent latency of the video rendering pipeline, then the audio physical output time is read from the dedicated timestamp delivery channel of the cross-domain shared storage area; The physical output time of the audio is used as the target display time of the video frame, and the rendering advance after the video domain reads the physical output time is evaluated based on the difference between the physical output time of the audio and the local reception time.

6. The method according to claim 5, characterized in that, The method for determining the predicted current cross-domain transmission delay based on the cross-domain delay prediction model includes: Obtain the historical transmission time when the audio field in the historical timestamp entry was written to the historical audio physical output time, and obtain the historical local reception time of the historical audio physical output time. Use the difference between the historical local reception time and the historical transmission time as the cross-domain delay observation value. Adjust the model parameters in the cross-domain delay prediction model according to the current operating condition signal to obtain the current cross-domain delay prediction model; The cross-domain delay observation is input into the current cross-domain delay prediction model to obtain the predicted current cross-domain transmission delay.

7. The method according to claim 1, characterized in that, The steps for determining the current acoustic delay error include: A synchronization marker signal is periodically generated through the audio domain, and the synchronization marker signal includes at least one of the following injection methods: acoustic injection method and digital injection method; The acoustic injection method involves injecting a short-time pulse signal of a specific frequency into an audio signal through an audio output link. The short-time pulse signal is played by a speaker and then collected by a microphone. The digital injection method involves inserting a special frame header marker or a low-energy audible feature marker into the digital audio frame buffer. The special frame header marker or low-energy audible feature marker is converted by a digital-to-analog converter (DAC) along with the normal audio data, amplified by a power amplifier, and then played by a speaker. The arrival time is detected by a microphone based on frame header recognition or sequence matching. Record the planned playback time of the synchronization marker signal; The audio signal from the audio output link is collected by the microphone as an acoustic feedback signal, and the synchronization marker signal is identified from the acoustic feedback signal and the actual playback time is recorded. The current acoustic delay error is determined based on the actual playback time and the planned playback time.

8. An automotive-grade system-on-a-chip, characterized in that, The automotive-grade system-on-a-chip includes a video domain and an audio domain; The audio domain is used to obtain the original audio timestamps of key event nodes of the audio frame rendering pipeline based on the global hardware counter on the SOC, and to calibrate the original audio timestamps based on the current acoustic delay error, determine the physical output time of the audio, and write it into the cross-domain shared storage area. The video domain is used to obtain the video timestamps of key event nodes in the video frame rendering pipeline based on the on-chip global hardware counter of the SOC. The physical output time of the audio is used as the target display time of the video frame, and the rendering advance after the video domain reads the physical output time is evaluated based on the difference between the physical output time of the audio and the local reception time. If sufficient, the video rendering trigger time of the video frame is determined based on the difference between the target display time and the inherent delay of the video rendering pipeline. The video frame rendering is triggered based on the video rendering trigger time, and the display time of the video frame after the inherent delay of the video rendering pipeline is aligned with the audio physical output time.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the cross-domain audio-video synchronization method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the cross-domain audio and video synchronization method according to any one of claims 1-7.

Images