A low-latency multi-source data stream synchronization and control method for virtual production

Abstract

This invention relates to the field of virtual production technology, specifically to a low-latency multi-source data stream synchronization and control method for virtual production. In this method, after each data node receives the main frame phase reference signal, it retrieves the corresponding phase mapping function based on the data type identifier pre-configured in the node register, maps the original sampled data to the main frame phase coordinate system, and outputs synchronized data that is phase-aligned with the sampled phase points of the rendering frame. This invention uses the vertical synchronization signal output by the rendering engine as a single main frame phase reference, and retrieves different corresponding phase mapping functions based on the data type identifiers (continuous pose, discrete event control, and display refresh) pre-configured in the node register. This enables multi-source data with different sampling clocks to achieve precise phase alignment within the rendering frame cycle, effectively eliminating relative time ambiguity and inter-frame cumulative drift caused by multi-clock domain cross-sampling to a certain extent.

Description

Technical Field

[0001] This invention relates to the field of virtual production technology, specifically to a low-latency multi-source data stream synchronization and control method for virtual production. Background Technology

[0002] In the field of virtual production, with the deep coupling of diverse data sources such as LED giant screens, multi-camera real-time rendering, motion capture, and lighting control within the same production workflow, achieving high-precision frame-level synchronization between multi-source data streams and the rendering engine has become a core bottleneck restricting production quality and efficiency. Current mainstream synchronization solutions mostly rely on a single external synchronization source, or use Genlock hardware synchronization signals to lock the camera and LED screen refresh phase, or rely on the PTP network protocol to distribute time bases at the Ethernet link layer, while others completely rely on the rendering engine's internal clock to control the data processing rhythm.

[0003] However, virtual production workflows contain continuous high-frequency sampling data (such as camera six-DOF pose and optical motion capture skeleton point clouds), discrete event-based control commands (such as lighting switching, prop triggering, and special effects activation), and display refresh data (such as LED screen drive signals). These data types differ significantly in their native sampling rate, real-time requirements, and jitter tolerance. A single external synchronization reference is insufficient to meet the alignment requirements of all data types within a unified phase coordinate system. Furthermore, in complex production environments, external synchronization signals may temporarily fail due to loose cables, network congestion, or signal source drift. Existing systems generally lack smooth degradation switching mechanisms. Once the synchronization reference is interrupted, it will directly cause phase disorder in the entire data link, ranging from introducing a few frames of delay fluctuations to serious production accidents such as a disconnect between the captured action and the rendering. Furthermore, the processing latency introduced by the differences in algorithm complexity of each data node has dynamic time-varying characteristics. Traditional open-loop synchronization schemes lack the ability to perceive and compensate for the end-to-end latency in real time, and cannot stably constrain the total end-to-end latency within the target threshold, making it difficult to meet the stringent requirements of high-end virtual production. Summary of the Invention

[0004] The purpose of this invention is to provide a low-latency multi-source data stream synchronization and control method for virtual filmmaking, so as to solve the problems mentioned in the background art.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a low-latency multi-source data stream synchronization and control method for virtual filmmaking, comprising: S1. Using the vertical synchronization signal output by the rendering engine as the trigger source, the start time of each rendering frame is recorded as phase zero and the end time of the frame is recorded as phase 1. A normalized main frame phase coordinate system is established, a main frame phase reference signal carrying the frame number and phase value is generated, and the obtained main frame phase reference signal is sent down to each data node. Specifically, the phase value ranges from [0,1); the main frame phase reference signal generated in this invention is sent to all data nodes via the SDI synchronization bus or the Ethernet PTP bus; S2. After receiving the main frame phase reference signal, each data node retrieves the corresponding phase mapping function according to the data type identifier pre-configured in the node register, maps the original sampled data to the main frame phase coordinate system, and outputs synchronization data that is phase-aligned with the sampled phase points of the rendering frame; the data types include continuous pose data, discrete event control data, and display refresh data; the continuous pose data uses phase linear interpolation alignment, the discrete event control data uses phase point locking alignment, and the display refresh data uses multi-sub-phase mapping alignment; S3. Delay probes are deployed at each data node to measure and process the latency in real time, and the data is reported to the central synchronization controller on a frame cycle. The central synchronization controller summarizes the total latency of the entire link and, in combination with the preset target latency threshold, dynamically adjusts the output buffer queue depth of each data node to compensate for the latency fluctuation of the entire link, so that the total latency of the entire link is stabilized within the allowable deviation range of the target threshold. Specifically, timestamp markers are set at the input and output ends of each data node. The delay probe calculates the processing delay ΔT of this node and reports it to the central synchronization controller on a frame cycle. The central synchronization controller summarizes the processing delay of each node and the link transmission delay, calculates the total link delay Tcur, and obtains the delay deviation of the entire link by subtracting it from the preset target delay threshold Ttar. By issuing buffer depth adjustment instructions to each data node, the depth of the node output buffer queue is dynamically adjusted to compensate for delay fluctuations, so that the total link delay is stabilized within the allowable deviation range of the target delay threshold. The output buffer queue is set after the phase mapping process. S4 maintains a three-level synchronization reference system, consisting of the hardware Genlock signal, the PTP network time signal, and the local temperature-controlled crystal oscillator signal. When the upper-level synchronization reference fails, it automatically switches to the lower-level reference, maintaining the continuity of the main frame phase coordinate system during the switching process. The main frame phase reference after the switch is generated using the currently effective synchronization reference as the clock source. When switching to the local crystal oscillator, the relative phase difference between each node is latched to maintain relative synchronization. When the upper-level synchronization reference is restored, a multi-frame phase gradient method is used for smooth alignment.

[0006] Specifically, this invention automatically switches to a lower-level reference when the upper-level synchronization reference fails. During the switching process, the main frame phase coordinate system remains continuous and is not reset, with only the clock source switching. The relative phase difference between each node is latched at the instant of switching to the local crystal oscillator reference to maintain the relative synchronization relationship of multi-source data. After the upper-level synchronization reference is restored, the total deviation between the current phase and the target phase is calculated, and the deviation is evenly distributed to a preset number of transition frames for frame-by-frame adjustment to achieve smooth phase alignment without jumps.

[0007] Furthermore, the phase mapping function in S2 is constructed as follows: For a data source with a sampling frequency of fs, the main frame frequency is denoted as fr, and the corresponding frequency ratio is denoted as k, where k = fs / fr; When k is an integer value, k uniformly distributed sub-phase points are constructed, and each sub-phase point corresponds to a native sample value, which is directly mapped to the main frame phase. When k is a non-integer value, linear interpolation is used to calculate the data value of the target phase point. The interpolation formulas involved are as follows: ; in, Indicates phase The interpolation result at the location; This represents the nth native sampled value adjacent to the target phase; This represents the (n+1)th native sampled value adjacent to the target phase; express The corresponding main frame phase value; express The corresponding main frame phase value.

[0008] Furthermore, the continuous pose data includes camera six-DOF pose data and motion capture skeleton data, and both the camera six-DOF pose data and motion capture skeleton data are aligned using the phase linear interpolation method. When the corresponding frequency ratio k is greater than or equal to the first preset frequency ratio, the sampling mode is automatically switched, and the native sampled value with the smallest phase distance from the target phase point is directly selected as the output. The purpose is to effectively reduce the computational overhead.

[0009] Furthermore, the discrete event-type control data includes lighting switching signals, prop triggering signals, and special effects activation signals; The specific process of using phase point locking alignment for the discrete event-type control data includes: recording the main frame phase value corresponding to the trigger time of the corresponding discrete event, denoted as... The phase of the rendering frame is accumulated through a hardware timer. The event response is executed instantly, rather than being postponed to the start of the next full frame.

[0010] Furthermore, the specific method for measuring and processing latency in real time at each data node by deploying delay probes in S3 includes: The input data is stamped with a first timestamp, denoted as Tin, at the input end of the data node, and the output data is stamped with a second timestamp, denoted as Tout, at the output end of the data node. The processing delay of the real-time measurement is denoted as ΔT. Then, ΔT = Tout - Tin. Both the first and second timestamps are generated based on the main frame phase reference signal, and the time references of each node are consistent. The measurement period of the delay probe is equal to one main frame period, and the processing delay of the corresponding data node is reported to the central synchronization controller once per frame. The total delay across the entire link is equal to the sum of the processing delays of each node and the transmission delay of the link, which are aggregated by the central synchronization controller.

[0011] Furthermore, in S3, an incremental PID control algorithm is used to dynamically adjust the output buffer queue depth of each data node. The calculation method involved is as follows: The difference between the obtained total end-to-end latency and the preset target latency threshold is taken as the end-to-end latency deviation. Let e(i) be the full-link delay deviation of the i-th measurement, then e(i) = Tcur(i) - Ttar; Calculate the buffer depth adjustment amount for the i-th time, denoted as . , ; Where Tcur(i) represents the total end-to-end latency of the i-th measurement; Ttar represents the preset target latency threshold; This indicates the preset scaling factor; Indicates the preset integral coefficient; Indicates the preset differential coefficient; Let represent the end-to-end delay deviation of the j-th measurement; e(i-1) represents the end-to-end delay deviation of the (i-1)-th measurement. When the total latency across the entire link is within the allowable deviation range of the preset target threshold, it is determined that no buffer depth adjustment will be performed. This mechanism can effectively avoid data jitter caused by frequent fine-tuning.

[0012] Furthermore, S3 also includes a delay overload reduction mechanism, specifically: when the processing delay of a certain data node exceeds a preset delay upper limit threshold for N consecutive frames, the central synchronization controller issues a reduction command to the data node, where N is a preset constant; The generation of the load reduction command issued in this invention is affected by the data node type. When the data node is a tracking node, the load reduction method for the data node is to reduce the number of marker reconstructions or lower the sampling frame rate; When a data node is a rendering node, the unloading method for that data node is to reduce the rendering resolution or turn off unnecessary post-processing effects. Once the processing latency of the corresponding data node is detected to have returned to the normal range, the corresponding load reduction command is removed, and the system automatically returns to its original working state.

[0013] Furthermore, the switching determination conditions for the three-level synchronization reference of hardware Genlock signal, PTP network time signal, and local isothermal crystal oscillator signal in S4 include: For the primary reference Genlock signal, if the synchronization pulse is missing for more than g consecutive frame periods, it is determined to be faulty and the signal is switched to the secondary PTP reference, where g represents a preset constant. For the secondary reference PTP signal, if the network jitter exceeds the preset jitter threshold and continues for more than a preset number of frames, it is determined to be ineffective and switched to the tertiary local crystal oscillator reference; the tertiary local crystal oscillator reference uses a temperature-compensated crystal oscillator.

[0014] Furthermore, the specific execution steps of S4, which employs a multi-frame phase gradient method for smooth alignment during the recovery of the upper-level synchronization reference, include: S41. After detecting the recovery of the upper-level synchronization reference, calculate the original difference between the current phase and the target phase of a single data node. Combine this with the circular characteristics of the phase interval [0,1) to solve for the shortest phase deviation. This shortest phase difference is denoted as the total deviation, and is recorded as follows: The calculation formulas involved in solving for the shortest phase deviation are as follows: ; in, Indicates the target phase of the corresponding data node; Indicates the current phase of the corresponding data node; This represents a range-finding function used to solve for... Perform an increment or decrement operation, and the processed value belongs to the interval [0,1); S42. Calculate the phase adjustment amount for each frame based on a preset number of transition frames, wherein the phase adjustment amount for each frame is equal to... The quotient of the preset number of transition frames; S43. Perform a phase adjustment once for each transition frame, and the adjustment amount for a single frame shall not exceed the preset maximum allowable phase offset for a single frame. The purpose is to ensure that the phase change is below the human eye perception threshold. S44. After a preset number of transition frames, complete the alignment and switch back to the original synchronization mode.

[0015] Furthermore, the main frame phase reference signal serves as the time reference for phase mapping, the time reference for delay measurement, and the phase-locked reference for synchronous degradation. In this mechanism, the three share the same phase reference, which can effectively avoid the cumulative error caused by introducing multiple clock references.

[0016] Compared with the prior art, the beneficial effects achieved by the present invention are: (1) This invention uses the vertical synchronization signal output by the rendering engine as the single main frame phase reference, and calls the corresponding different phase mapping functions based on the data type identifier (continuous pose, discrete event control and display refresh) pre-configured in the node register, so that the multi-source data with different sampling clocks can achieve precise phase alignment within the rendering frame period, which effectively eliminates the relative time ambiguity and inter-frame cumulative drift caused by cross-sampling of multiple clock domains to a certain extent. (2) This invention deploys delay probes at the input and output ends of each data node, measures the processing delay of each node in real time according to the frame period, and summarizes the total delay of the entire link by the central synchronous controller. Combined with the incremental PID control algorithm, the output buffer queue depth of each data node is dynamically adjusted, thereby realizing the tracking and compensation of link delay fluctuations and stabilizing the total delay of the entire link within the allowable deviation range of the target threshold. (3) This invention maintains three-level synchronization references in parallel: hardware Genlock signal, PTP network time signal and local isothermal crystal oscillator signal. When the upper reference fails, it automatically switches to the lower reference. During the switching process, the main frame phase coordinate system remains continuous and does not reset, only the clock source is switched. When switching to the local crystal oscillator, the relative phase difference between each node is latched to maintain relative synchronization. After the upper reference is restored, a multi-frame phase gradient method is used to smoothly align. This mechanism avoids the visual jump caused by the phase change at the moment of synchronization switching. The three-level redundancy design significantly improves the anti-interference capability and operational reliability of the synchronization system under complex working conditions on site, ensuring that the virtual production process can continue to work stably when the synchronization source is abnormal. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating a low-latency multi-source data stream synchronization and control method for virtual film production according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1 This embodiment provides a low-latency multi-source data stream synchronization and control method for virtual filmmaking, including: S1. Using the vertical synchronization signal output by the rendering engine as the trigger source, the start time of each rendering frame is recorded as phase zero and the end time of the frame is recorded as phase 1. A normalized main frame phase coordinate system is established, a main frame phase reference signal carrying the frame number and phase value is generated, and the obtained main frame phase reference signal is sent down to each data node. Specifically, the phase value ranges from [0,1); the main frame phase reference signal generated in this invention is sent to all data nodes via the SDI synchronization bus or the Ethernet PTP bus; S2. After receiving the main frame phase reference signal, each data node retrieves the corresponding phase mapping function according to the data type identifier pre-configured in the node register, maps the original sampled data to the main frame phase coordinate system, and outputs synchronization data that is phase-aligned with the sampled phase points of the rendering frame; the data types include continuous pose data, discrete event control data, and display refresh data; the continuous pose data uses phase linear interpolation alignment, the discrete event control data uses phase point locking alignment, and the display refresh data uses multi-sub-phase mapping alignment; The phase mapping function in S2 is constructed as follows: For a data source with a sampling frequency of fs, the main frame frequency is denoted as fr, and the corresponding frequency ratio is denoted as k, where k = fs / fr; When k is an integer value, k uniformly distributed sub-phase points are constructed, and each sub-phase point corresponds to a native sample value, which is directly mapped to the main frame phase. When k is a non-integer value, linear interpolation is used to calculate the data value of the target phase point. The interpolation formulas involved are as follows: ; in, Indicates phase The interpolation result at the location; This represents the nth native sampled value adjacent to the target phase; This represents the (n+1)th native sampled value adjacent to the target phase; express The corresponding main frame phase value; express The corresponding main frame phase value.

[0020] The continuous pose data includes camera six-DOF pose data and motion capture skeleton data, and both the camera six-DOF pose data and motion capture skeleton data are aligned using the phase linear interpolation method. When the corresponding frequency ratio k is greater than or equal to the first preset frequency ratio (in this embodiment, the first preset frequency ratio is 8), the sampling mode is automatically switched, and the original sampled value with the smallest phase distance from the target phase point is directly selected as the output. The purpose is to effectively reduce the computational overhead.

[0021] The discrete event-type control data includes lighting switching signals, prop triggering signals, and special effects activation signals; The specific process of using phase point locking alignment for the discrete event-type control data includes: recording the main frame phase value corresponding to the trigger time of the corresponding discrete event, denoted as... The phase of the rendering frame is accumulated through a hardware timer. Instead of delaying the response to the next full frame, the phase-locked alignment method described above enables the response accuracy of discrete events to reach the sub-millisecond level.

[0022] In this embodiment, the refresh data includes LED screen drive signals, and the LED screen refresh rate is usually much higher than the rendering frame rate. In this embodiment, the LED screen refresh rate is 7680Hz, and the main frame frequency is 24Hz, so k = 7680 ÷ 24 = 320. The LED refresh within a single main frame cycle is divided into 320 sub-phase points to ensure that the refresh phase is accurately matched with the camera exposure phase, which can effectively avoid the appearance of scan lines or dark bands in the captured image to a certain extent.

[0023] S3. Delay probes are deployed at each data node to measure and process the latency in real time, and the data is reported to the central synchronization controller on a frame cycle. The central synchronization controller summarizes the total latency of the entire link and, in combination with the preset target latency threshold, dynamically adjusts the output buffer queue depth of each data node to compensate for the latency fluctuation of the entire link, so that the total latency of the entire link is stabilized within the allowable deviation range of the target threshold. Specifically, timestamp markers are set at the input and output ends of each data node. The delay probe calculates the processing delay ΔT of this node and reports it to the central synchronization controller on a frame cycle. The central synchronization controller summarizes the processing delay of each node and the link transmission delay, calculates the total link delay Tcur, and obtains the delay deviation of the entire link by subtracting it from the preset target delay threshold Ttar. By issuing buffer depth adjustment instructions to each data node, the depth of the node output buffer queue is dynamically adjusted to compensate for delay fluctuations, so that the total link delay is stabilized within the allowable deviation range of ±5% to ±10% of the target delay threshold. The output buffer queue is set after the phase mapping process. The specific method for measuring and processing latency in real time at each data node by deploying latency probes in S3 includes: The input data is stamped with a first timestamp, denoted as Tin, at the input end of the data node, and the output data is stamped with a second timestamp, denoted as Tout, at the output end of the data node. The processing delay of the real-time measurement is denoted as ΔT. Then, ΔT = Tout - Tin. Both the first and second timestamps are generated based on the main frame phase reference signal, and the time references of each node are consistent. The measurement period of the delay probe is equal to one main frame period, and the processing delay of the corresponding data node is reported to the central synchronization controller once per frame. The total delay across the entire link is equal to the sum of the processing delays of each node and the transmission delay of the link, which are aggregated by the central synchronization controller.

[0024] The incremental PID control algorithm used in S3 dynamically adjusts the output buffer queue depth of each data node, and the calculation method involved is as follows: The difference between the obtained total end-to-end latency and the preset target latency threshold is taken as the end-to-end latency deviation. Let e(i) be the full-link delay deviation of the i-th measurement, then e(i) = Tcur(i) - Ttar; Calculate the buffer depth adjustment amount for the i-th time, denoted as . , ; Where Tcur(i) represents the total end-to-end latency of the i-th measurement; Ttar represents the preset target latency threshold; This indicates the preset scaling factor; Indicates the preset integral coefficient; Indicates the preset differential coefficient; Let represent the end-to-end delay deviation of the j-th measurement; e(i-1) represents the end-to-end delay deviation of the (i-1)-th measurement. When the total latency across the entire link is within the allowable deviation range of the preset target threshold, it is determined that the buffer depth adjustment will not be performed to avoid data jitter caused by frequent fine-tuning.

[0025] The S3 also includes a delay overload reduction mechanism, which is manifested as follows: when the processing delay of a certain data node exceeds the preset delay upper limit threshold for N consecutive frames, the central synchronization controller issues a reduction command to the data node, where N is a preset constant, and in this embodiment, N is a positive integer between 3 and 10. In this embodiment, the generation of the load reduction command is affected by the data node type. When the data node is a tracking node, the load reduction method for the data node is to reduce the number of marker reconstructions or lower the sampling frame rate; When a data node is a rendering node, the unloading method for that data node is to reduce the rendering resolution or turn off unnecessary post-processing effects. Once the processing latency of the corresponding data node is detected to have returned to the normal range, the corresponding load reduction command is removed, and the system automatically returns to its original working state.

[0026] S4 maintains a three-level synchronization reference system, consisting of the hardware Genlock signal, the PTP network time signal, and the local temperature-controlled crystal oscillator signal. When the upper-level synchronization reference fails, it automatically switches to the lower-level reference, maintaining the continuity of the main frame phase coordinate system during the switching process. The main frame phase reference after the switch is generated using the currently effective synchronization reference as the clock source. When switching to the local crystal oscillator, the relative phase difference between each node is latched to maintain relative synchronization. When the upper-level synchronization reference is restored, a multi-frame phase gradient method is used for smooth alignment.

[0027] Specifically, the system maintains three levels of synchronization references in parallel: hardware Genlock signal, PTP network time signal, and local isothermal crystal oscillator signal, and monitors their effectiveness in real time. When the upper-level synchronization reference fails, it automatically switches to the lower-level reference. During the switching process, the main frame phase coordinate system remains continuous and does not reset, with only the clock source changing. The relative phase difference between each node is latched at the instant of switching to the local crystal oscillator reference, thereby maintaining the relative synchronization relationship of multi-source data. After the upper-level synchronization reference is restored, the total deviation between the current phase and the target phase is calculated, and the deviation is evenly distributed to a preset number of transition frames for frame-by-frame adjustment to achieve smooth phase alignment without jumps.

[0028] The switching criteria for the three-level synchronization references—hardware Genlock signal, PTP network time signal, and local isothermal crystal oscillator signal—in S4 include: For the primary reference Genlock signal, if the synchronization pulse is missing for more than g consecutive frame periods, it is determined to be faulty and the signal is switched to the secondary PTP reference. Here, g represents a preset constant, and in this embodiment, the value of g is 3. For the secondary reference PTP signal, if the network jitter exceeds the preset jitter threshold (in this embodiment, the preset jitter threshold is 1ms) and continues for more than a preset number of frames (5 frames), it is determined to be ineffective and switches to the tertiary local crystal oscillator reference; the tertiary local crystal oscillator reference uses a temperature-compensated crystal oscillator.

[0029] The specific execution steps of S4, which employs a multi-frame phase gradient method for smooth alignment during the recovery of the upper-level synchronization reference, include: S41. After detecting the recovery of the upper-level synchronization reference, calculate the original difference between the current phase and the target phase of a single data node. Combine this with the circular characteristics of the phase interval [0,1) to solve for the shortest phase deviation. This shortest phase difference is denoted as the total deviation, and is recorded as follows: The calculation formulas involved in solving for the shortest phase deviation are as follows: ; in, Indicates the target phase of the corresponding data node; Indicates the current phase of the corresponding data node; This represents a range-finding function used to solve for... Perform an increment or decrement operation, and the processed value belongs to the interval [0,1); S42. Calculate the phase adjustment amount for each frame based on a preset number of transition frames, wherein the phase adjustment amount for each frame is equal to... The quotient of the preset number of transition frames; in this embodiment, the preset number of transition frames is a positive integer between 30 and 120. S43. Perform a phase adjustment once for each transition frame, and the adjustment amount per frame shall not exceed the preset maximum allowable phase offset per frame (in this embodiment, the preset maximum allowable phase offset per frame is one-thirtieth of the main frame period). The purpose is to ensure that the phase change is lower than the human eye perception threshold. S44. After a preset number of transition frames, complete the alignment and switch back to the original synchronization mode.

[0030] In this embodiment, the main frame phase reference signal is used as the time reference for phase mapping, the time reference for delay measurement, and the phase locking reference for synchronization degradation. All three share the same phase reference, which can effectively avoid the cumulative error caused by introducing multiple clock references.

[0031] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0032] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A low-latency multi-source data stream synchronization and control method for virtual filmmaking, characterized in that, include: S1. Using the vertical synchronization signal output by the rendering engine as the trigger source, the start time of each rendering frame is recorded as phase zero and the end time of the frame is recorded as phase 1. A normalized main frame phase coordinate system is established, a main frame phase reference signal carrying the frame number and phase value is generated, and the obtained main frame phase reference signal is sent down to each data node. S2. After receiving the main frame phase reference signal, each data node retrieves the corresponding phase mapping function according to the data type identifier pre-configured in the node register, maps the original sampled data to the main frame phase coordinate system, and outputs synchronization data that is phase-aligned with the sampled phase points of the rendering frame; the data types include continuous pose data, discrete event control data, and display refresh data; the continuous pose data uses phase linear interpolation alignment, the discrete event control data uses phase point locking alignment, and the display refresh data uses multi-sub-phase mapping alignment; S3. Delay probes are deployed at each data node to measure and process the latency in real time, and the data is reported to the central synchronization controller on a frame cycle. The central synchronization controller summarizes the total latency of the entire link and, in combination with the preset target latency threshold, dynamically adjusts the output buffer queue depth of each data node to compensate for the latency fluctuation of the entire link, so that the total latency of the entire link is stabilized within the allowable deviation range of the target threshold. S4. Parallel maintenance of three-level synchronization references: hardware Genlock signal, PTP network time signal, and local isothermal crystal oscillator signal. When the upper-level synchronization reference fails, it automatically switches to the lower-level reference and maintains the continuity of the main frame phase coordinate system during the switching process. The main frame phase reference after the switch is generated with the currently effective synchronization reference as the clock source. When switching to the local crystal oscillator, the relative phase difference between each node is latched to maintain relative synchronization. During the recovery of the upper-level synchronization reference, a multi-frame phase gradient method is used for smooth alignment.

2. The low-latency multi-source data stream synchronization and control method for virtual filmmaking according to claim 1, characterized in that, The phase mapping function in S2 is constructed as follows: For a data source with a sampling frequency of fs, the main frame frequency is denoted as fr, and the corresponding frequency ratio is denoted as k, where k = fs / fr; When k is an integer value, k uniformly distributed sub-phase points are constructed, and each sub-phase point corresponds to a native sample value, which is directly mapped to the main frame phase. When k is a non-integer value, linear interpolation is used to calculate the data value of the target phase point. The interpolation formulas involved are as follows: ； in, Indicates phase The interpolation result at the location; This represents the nth native sampled value adjacent to the target phase; This represents the (n+1)th native sampled value adjacent to the target phase; express The corresponding main frame phase value; express The corresponding main frame phase value.

3. The low-latency multi-source data stream synchronization and control method for virtual filmmaking according to claim 2, characterized in that, The continuous pose data includes camera six-DOF pose data and motion capture skeleton data, and both the camera six-DOF pose data and motion capture skeleton data are aligned using the phase linear interpolation method. When the corresponding frequency ratio k is greater than or equal to the first preset frequency ratio, the sampling mode is automatically switched, and the native sampled value with the smallest phase distance from the target phase point is directly selected as the output.

4. The low-latency multi-source data stream synchronization and control method for virtual filmmaking according to claim 1, characterized in that, The discrete event-type control data includes lighting switching signals, prop triggering signals, and special effects activation signals; The specific process of using phase point locking alignment for the discrete event-type control data includes: recording the main frame phase value corresponding to the trigger time of the corresponding discrete event, denoted as... The phase of the rendering frame is accumulated through a hardware timer. The event response is executed instantly, rather than being postponed to the start of the next full frame.

5. A low-latency multi-source data stream synchronization and control method for virtual filmmaking according to claim 1, characterized in that, The specific method for measuring and processing latency in real time at each data node by deploying latency probes in S3 includes: At the input end of the data node, the input data is stamped with the first timestamp, denoted as Tin. At the output end of the data node, the output data is stamped with the second timestamp, denoted as Tout. The processing delay of the real-time measurement is denoted as ΔT. Then, ΔT = Tout - Tin. Both the first and second timestamps are generated based on the main frame phase reference signal, and the time references of each node are consistent. The measurement period of the delay probe is equal to one main frame period, and the processing delay of the corresponding data node is reported to the central synchronization controller once per frame. The total delay across the entire link is equal to the sum of the processing delays of each node and the transmission delay of the link, which are aggregated by the central synchronization controller.

6. The low-latency multi-source data stream synchronization and control method for virtual filmmaking according to claim 1, characterized in that, The incremental PID control algorithm used in S3 dynamically adjusts the output buffer queue depth of each data node, and the calculation method involved is as follows: The difference between the obtained total end-to-end latency and the preset target latency threshold is taken as the end-to-end latency deviation. Let e(i) be the full-link delay deviation of the i-th measurement, then e(i) = Tcur(i) - Ttar; Calculate the buffer depth adjustment amount for the i-th time, denoted as . , ； Where Tcur(i) represents the total end-to-end latency of the i-th measurement; Ttar represents the preset target latency threshold; This indicates the preset scaling factor; Indicates the preset integral coefficient; Indicates the preset differential coefficient; Let represent the end-to-end delay deviation of the j-th measurement; e(i-1) represents the end-to-end delay deviation of the (i-1)-th measurement. If the total latency across the entire link is within the allowable deviation range of the preset target threshold, then it is determined that buffer depth adjustment will not be performed.

7. A low-latency multi-source data stream synchronization and control method for virtual filmmaking according to claim 1, characterized in that, The S3 also includes a delay overload reduction mechanism, which is manifested as follows: when the processing delay of a certain data node exceeds the preset delay upper limit threshold for N consecutive frames, the central synchronization controller issues a reduction command to the data node, where N is a preset constant.

8. A low-latency multi-source data stream synchronization and control method for virtual filmmaking according to claim 1, characterized in that, The switching criteria for the three-level synchronization references—hardware Genlock signal, PTP network time signal, and local isothermal crystal oscillator signal—in S4 include: For the primary reference Genlock signal, if the synchronization pulse is missing for more than g consecutive frame periods, it is determined to be faulty and the signal is switched to the secondary PTP reference, where g represents a preset constant. For the secondary reference PTP signal, if the network jitter exceeds the preset jitter threshold and continues for more than a preset number of frames, it is determined to be ineffective and switched to the tertiary local crystal oscillator reference; the tertiary local crystal oscillator reference uses a temperature-compensated crystal oscillator.

9. A low-latency multi-source data stream synchronization and control method for virtual filmmaking according to claim 1, characterized in that, The specific execution steps of S4, which employs a multi-frame phase gradient method for smooth alignment during the recovery of the upper-level synchronization reference, include: S41. After detecting the recovery of the upper-level synchronization reference, calculate the original difference between the current phase and the target phase of a single data node. Combine this with the circular characteristics of the phase interval [0,1) to solve for the shortest phase deviation. This shortest phase difference is denoted as the total deviation, and is recorded as follows: The calculation formulas involved in solving for the shortest phase deviation are as follows: ； in, Indicates the target phase of the corresponding data node; Indicates the current phase of the corresponding data node; This represents a range-finding function used to solve for... Perform an increment or decrement operation, and the processed value belongs to the interval [0,1); S42. Calculate the phase adjustment amount for each frame based on a preset number of transition frames, wherein the phase adjustment amount for each frame is equal to... The quotient of the preset number of transition frames; S43. Perform a phase adjustment once for each transition frame, and the adjustment amount for a single frame shall not exceed the preset maximum allowable phase offset for a single frame. S44. After a preset number of transition frames, complete the alignment and switch back to the original synchronization mode.

10. A low-latency multi-source data stream synchronization and control method for virtual filmmaking according to claim 1, characterized in that, The main frame phase reference signal serves as the time reference for phase mapping, the time reference for delay measurement, and the phase locking reference for synchronous degradation.

Images