VR system multi-mode feedback synchronous driving method based on time sequence calibration

Abstract

The invention discloses a VR system multi-modal feedback synchronous driving method based on time sequence calibration, and relates to the technical field of multi-modal feedback synchroniss.The method comprises the steps that when a user interaction event is detected, the event is captured, and an initial timestamp is recorded; based on the timestamp, task scheduling and resource pre-allocation are carried out through a quantum heuristic scheduling algorithm; predicting a dynamic delay compensation value of each mode by using a neural symbol AI model; calculating the accurate triggering time fed back by each mode according to the compensation value; executing high-priority parallel instruction scheduling at the calculated time point; the driving hardware outputs feedback and collects actual execution time to optimize subsequent scheduling, synchronous driving is achieved in an operating system kernel layer, dynamic delay prediction and compensation are combined, the problem of timing sequence misalignment caused by hardware heterogeneous delay and software scheduling uncertainty of multi-mode feedback is solved, synchronous control of microsecond-level precision is achieved, and the service life of the system is prolonged. And the immersion and robustness of the VR system are improved.

Description

Technical Field

[0001] This invention relates to the field of multimodal feedback synchronization technology, and in particular to a multimodal feedback synchronization driving method for VR systems based on timing calibration. Background Technology

[0002] The immersive experience of virtual reality systems relies on the precise coordination of multimodal feedback, including visual, auditory, and tactile feedback. Currently, the industry primarily attempts to unify the output timing of each modality by coordinating upper-layer application software or using display vertical synchronization signals as a reference. Such methods can achieve coarse synchronization at the millisecond level under ideal loads, constituting the main technical path for achieving multimodal synchronization at this stage.

[0003] Existing methods have inherent limitations. Their synchronization logic operates in a non-real-time user space, making it impossible to avoid random latency jitter introduced by operating system scheduling, data buffering, and inter-process communication. More critically, there are inherent and heterogeneous microsecond-level processing delays between different feedback hardware (such as display controllers, audio codecs, and haptic motor drivers), and existing solutions lack mechanisms for precise measurement and compensation at the driver layer. This results in significant dispersion of modal feedback at the user's perception level in complex scenarios or under high loads, severely impairing the realism and immersion of the interaction and potentially causing user discomfort. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a VR system multimodal feedback synchronization driving method based on timing calibration to solve the timing inaccuracy problem caused by software scheduling uncertainty and uncompensated inherent hardware delay in existing synchronization methods, and to solve the problem of how to achieve microsecond-level deterministic multimodal feedback synchronization.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a multimodal feedback synchronization driving method for a VR system based on timing calibration, characterized by comprising the following steps: When the VR application layer detects a user interaction event, it captures the event and records the initial timestamp; Based on the initial timestamp, a quantum heuristic scheduling algorithm is used for task scheduling and resource pre-allocation; Based on the scheduling results, the dynamic delay compensation value for each modality is predicted using a neural symbolic AI model; The precise trigger time of each modal feedback is calculated based on the dynamic delay compensation value; Execute high-priority parallel instruction scheduling at the calculated time point; The driver outputs feedback and collects the actual execution time to optimize subsequent scheduling strategies.

[0007] As a preferred embodiment of the VR system multimodal feedback synchronization driving method based on time-series calibration described in this invention, the step of capturing the event and recording the initial timestamp when the VR application layer detects a user interaction event includes the following specific steps: The VR application layer captures user interaction events through a system-level high-precision clock source and assigns a unique event identifier to each event; The event identifier, event type, scene context information when the event occurred, and the current time recorded by a high-precision clock source are encapsulated into an event data packet; Write the event data packet into a thread-safe circular buffer and immediately trigger a high-priority interrupt to notify the subsequent processing module; After the event data packet is successfully written to the buffer, the initial timestamp is extracted and recorded as the base time of the interaction event.

[0008] As a preferred embodiment of the time-series calibration-based multimodal feedback synchronization driving method for VR systems described in this invention, the specific steps of task scheduling and resource pre-allocation based on an initial timestamp using a quantum heuristic scheduling algorithm are as follows: Read event data packets from the circular buffer and parse out the list of tasks that require multimodal feedback; Each feedback task is encoded into a quantum state representation that can be processed by a quantum heuristic algorithm, based on its type, priority, and estimated resource consumption. The quantum state is evolved using quantum rotation gate operations, and the expected execution cost of different task scheduling sequences is evaluated by simulating the measurement operations. Based on the evaluation results, the scheduling sequence with the lowest cost is selected, and detailed resource allocation instructions are generated accordingly, including CPU core allocation, memory bandwidth, and GPU rendering channels.

[0009] As a preferred embodiment of the VR system multimodal feedback synchronization driving method based on timing calibration described in this invention, the specific steps of predicting the dynamic delay compensation value of each modality using a neural symbolic AI model based on the scheduling results are as follows: Collect current VR system operating status data, including processor temperature, memory usage, GPU load, and network latency; This real-time data is then matched and preliminarily inferred against symbolic rules loaded from a neural symbolic AI model for the current type of interaction event; The matching results and raw data are input into a trained neural network module, which outputs a correction factor for the inherent delay of each modality. By combining the inference results of symbolic rules and the correction factor of the neural network, dynamic delay correction amounts for each modality are generated. Among them, the total intrinsic delay of each mode Provided by the hardware-inherent latency database, and this total inherent latency Delayed by the corresponding software path With inherent hardware latency The sum of them constitutes, that is, satisfies ,in, This includes hardware processing latency and software path latency.

[0010] As a preferred embodiment of the VR system multimodal feedback synchronization driving method based on timing calibration described in this invention, the specific steps for calculating the precise trigger time of each modal feedback based on the dynamic delay compensation value are as follows: Obtain ideal perception time from VR runtime system This time is determined based on the V-Sync signal or the predicted trajectory of head posture changes; For each mode that requires feedback, obtain its total intrinsic delay. and the dynamic delay correction predicted by the neural symbolic AI model ; Introducing formulas Calculate the precise trigger time of the modal feedback command. ,in, The instruction must be sent to the mode. Triggering time of the underlying driver; The theoretical trigger times for all modes are sorted, and the system scheduling overhead is taken into account. The trigger times are then fine-tuned to avoid instruction conflicts and ensure parallel execution.

[0011] As a preferred embodiment of the VR system multimodal feedback synchronization driving method based on timing calibration described in this invention, the step of executing high-priority parallel instruction scheduling at the calculated time point specifically includes the following steps: Create a high-priority timer task for each calculated precise trigger time point and register it in the timer queue of the real-time operating system; When the system timer reaches the preset trigger time, it immediately wakes up the corresponding timer task, which is responsible for sending the modal feedback instruction to the hardware driver layer. Non-blocking I / O operations are used to send instructions in parallel to the visual rendering module, audio playback device, and haptic feedback device; The scheduling module continuously monitors the command sending status and handles retrying or error reporting for possible sending failures or timeouts.

[0012] As a preferred embodiment of the VR system multimodal feedback synchronization driving method based on timing calibration described in this invention, the driving hardware outputs feedback and collects the actual execution time to optimize subsequent scheduling strategies. The specific steps are as follows: After receiving the feedback command, the VR system hardware module executes the corresponding output operation and internally records the time of command reception and actual execution completion. The hardware module feeds back the actual execution completion time to the VR system's time calibration module via a dedicated high-speed bus; The timing calibration module compares the theoretical trigger time of the instruction with the actual execution completion time to calculate the feedback timing error for each mode. These timing error data and the corresponding system operating status data are used as inputs to adjust and optimize the parameters of the quantum heuristic scheduling algorithm and the prediction model of the neural symbolic AI model online.

[0013] Secondly, the present invention provides a VR system multimodal feedback synchronization drive system based on time-series calibration, comprising, The event capture module captures user interaction events and records the initial timestamp when the VR application layer detects such events. The scheduling pre-allocation module performs task scheduling and resource pre-allocation based on the initial timestamp using a quantum heuristic scheduling algorithm; The delay prediction module uses a neural symbolic AI model to predict the dynamic delay compensation value for each modality based on the scheduling results. The trigger calculation module calculates the precise trigger time for each modal feedback based on the dynamic delay compensation value; The instruction scheduling module executes high-priority parallel instructions at the calculated time points; The feedback optimization module drives the hardware to output feedback and collects the actual execution time, which is used to optimize subsequent scheduling strategies.

[0014] Thirdly, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the timing-calibrated VR system multimodal feedback synchronization driving method described in the first aspect of the present invention.

[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the timing-calibrated VR system multimodal feedback synchronization driving method as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: By sinking the synchronization mechanism of multimodal feedback to the operating system kernel layer and constructing a compensation model based on the hardware inherent latency database and neural symbol AI dynamic prediction, the timing inaccuracy problem caused by software scheduling uncertainty and uncompensated heterogeneous hardware latency in the prior art is effectively solved. This method uses quantum heuristic scheduling to optimize task sequences and resource allocation, combined with a high-priority timed triggering mechanism, to achieve microsecond-level deterministic synchronization control of visual, auditory and tactile feedback. Its closed-loop optimization architecture further improves the timing stability and adaptability of the system under dynamic load. While ensuring synchronization accuracy, this scheme enhances the adaptability and robustness of the system, providing a reliable technical foundation for the realism and immersion of VR interactive experience. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a block diagram of the system state machine nodes of the VR system multimodal feedback synchronization driving method based on timing calibration in Example 1.

[0019] Figure 2 This is the system sequence UML diagram of the VR system multimodal feedback synchronization driving method based on timing calibration in Example 1. Detailed Implementation

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0023] Example 1, referring to Figure 1 and Figure 2This is the first embodiment of the present invention, which provides a multimodal feedback synchronization driving method for a VR system based on timing calibration, characterized by including the following steps: When the VR application layer detects a user interaction event, it captures the event and records the initial timestamp; Based on the initial timestamp, a quantum heuristic scheduling algorithm is used for task scheduling and resource pre-allocation; Based on the scheduling results, the dynamic delay compensation value for each modality is predicted using a neural symbolic AI model; The precise trigger time of each modal feedback is calculated based on the dynamic delay compensation value; Execute high-priority parallel instruction scheduling at the calculated time point; The driver outputs feedback and collects the actual execution time to optimize subsequent scheduling strategies.

[0024] It should be noted that interactive events are captured in real-time by calling a system-level high-precision clock source at the VR application layer, and a unique identifier is assigned to each event. The event identifier, type, context, and precise timestamp are then encapsulated into a data packet, written to a thread-safe circular buffer, and a high-priority interrupt is triggered to notify subsequent modules. This step establishes a unique, reliable, and low-latency event starting point and data path for the entire synchronization process, ensuring the accuracy and consistency of the time base upon which all subsequent calculations and scheduling rely, and avoiding initial errors caused by event capture delays or data transmission irregularities.

[0025] The synchronization-driven module reads event packets from the buffer, parses out the list of multimodal feedback tasks to be triggered, and then uses a quantum heuristic scheduling algorithm to encode each task as a quantum state according to its attributes. By simulating quantum rotation gate evolution and measurement, the resource consumption and time cost of different task execution sequences are evaluated. Finally, the optimal sequence is selected, and detailed pre-allocation instructions for resources such as processor cores, memory, and graphics processor rendering channels are generated. This step optimizes global resource planning before task execution, effectively reducing resource contention and scheduling conflicts between tasks, laying the foundation for reducing overall execution latency and jitter.

[0026] The system continuously collects real-time operational status data, including processor load, temperature, and memory usage. The neural symbolic AI model first loads predefined symbolic rules related to the current event type for logical reasoning. Then, the reasoning results, along with the original state data, are input into its internal neural network component. This network, after training, can output real-time correction factors for the inherent latency of each modality, ultimately synthesizing the dynamic latency correction amount ΔDM. This step achieves real-time dynamic compensation for the inherent hardware latency DM, enabling the system to adapt to changes in the operating environment such as chip temperature and load fluctuations, significantly improving the accuracy of latency prediction and environmental robustness.

[0027] The ideal perception time Ttarget, determined based on vertical synchronization signals or head posture prediction during VR runtime, is obtained. For each feedback modality, its inherent latency database value DM is queried, and the dynamic correction amount ΔDM predicted in the previous step is obtained. The precise trigger time of each modal command is calculated using the formula Ttrigger,M = Ttarget - (DM + ΔDM). The trigger times of all modalities are then sorted and fine-tuned to avoid conflicts. This step transforms the dynamically compensated total latency into independent, precise command issuance times for each modality. It is the core computational step for achieving multimodal synchronization with the user's senses at the target perception time, ensuring the logical correctness of the synchronization.

[0028] Based on each calculated Trigger, M, a corresponding high-priority timer task is created in the real-time operating system and registered in the timer queue. When the system clock reaches the preset time, the corresponding task is immediately triggered. This task sends control commands in parallel to the display driver, audio device driver, and haptic feedback driver using non-blocking input / output. This step, leveraging the high-priority timing and scheduling mechanism of the operating system kernel, ensures that the calculated trigger times are executed strictly and precisely, achieving deterministic and parallel triggering of instructions for various heterogeneous hardware drivers. This is a crucial execution guarantee for translating theoretical time synchronization into actual hardware synchronization.

[0029] Each hardware driver module executes instructions and drives physical hardware (such as display pixel refresh, speaker sound emission, and motor vibration) to generate feedback. Simultaneously, it internally records the timestamps of instruction reception and actual signal output completion, feeding back the actual completion time to the system's time calibration module via a dedicated high-speed bus. The calibration module compares the theoretical trigger time with the actual completion time, calculates the timing error, and uses this error, along with system state data, to iteratively optimize the parameters of the quantum scheduling algorithm and the weights of the neural symbolic AI model. This process constitutes a complete closed-loop optimization system, continuously calibrating and optimizing the prediction and scheduling model using actual execution feedback, enabling the system to self-learn and continuously improve synchronization accuracy during long-term operation.

[0030] Specifically, when the VR application layer detects a user interaction event, it captures the event and records the initial timestamp. The specific steps are as follows: The VR application layer captures user interaction events through a system-level high-precision clock source and assigns a unique event identifier to each event; The event identifier, event type, scene context information when the event occurred, and the current time recorded by a high-precision clock source are encapsulated into an event data packet; Write the event data packet into a thread-safe circular buffer and immediately trigger a high-priority interrupt to notify the subsequent processing module; After the event data packet is successfully written to the buffer, the initial timestamp is extracted and recorded as the base time of the interaction event.

[0031] It should be noted that the VR application layer captures user interaction events in real time by calling the system-level high-precision clock source (such as a crystal oscillator-based timer) provided by the embedded system chip, and then generates a globally unique sequence number for each event as an event identifier. This operation assigns a uniquely traceable tag to each interaction event and ensures the original accuracy of the event's occurrence time by relying on the hardware clock. This eliminates time reference errors caused by inconsistent software timing or event confusion from the source, laying a reliable foundation for all subsequent delay calculations and scheduling decisions that depend on this moment.

[0032] The system packages the generated event identifier, the event type determined by application logic (such as click, collision), the virtual scene context information at the time of the event (such as object ID, location), and the current time value read from a high-precision clock source into a fixed-structure event data packet. This operation binds and encapsulates the core semantic information of the event with its precise physical time of occurrence, forming a data unit containing a complete spatiotemporal context. This allows subsequent processing modules to not only know "when it happened" but also understand "what happened" and its context, providing the necessary information input for intelligent scheduling and delay prediction related to event type.

[0033] The encapsulated event data packet is written to a thread-safe circular buffer pre-allocated in memory and protected by mechanisms such as mutexes. Upon successful writing, the system immediately notifies the kernel-mode synchronous driver processing module by triggering a high-priority hardware interrupt signal managed by the operating system kernel. This operation utilizes the circular buffer to achieve efficient data transfer under the producer-consumer model and uses a hardware interrupt mechanism to achieve extremely low-latency, high-reliability asynchronous event notification between the user application and the kernel driver. This ensures that critical interactive events are delivered to the core processing stage in real time and without errors, avoiding notification delays or loss that might occur due to polling or ordinary message queues.

[0034] After confirming that the event data packet has been persistently stored in the buffer, the synchronization driver module extracts the original time value recorded by the high-precision clock source from a specific field of the data packet and officially records and confirms this time as the "initial timestamp" of this interaction event. This operation marks the completion of the event capture phase and establishes this timestamp as the unique absolute time reference followed by all subsequent steps in the entire synchronization pipeline. Through this step, the system establishes a clear and unified timing reference origin, enabling all time-related operations such as subsequent delay compensation and trigger time calculation to be performed based on the same origin, fundamentally ensuring the consistency and computability of the global timing logic.

[0035] Specifically, the steps for task scheduling and resource pre-allocation based on the initial timestamp using a quantum heuristic scheduling algorithm are as follows: Read event data packets from the circular buffer and parse out the list of tasks that require multimodal feedback; Each feedback task is encoded into a quantum state representation that can be processed by a quantum heuristic algorithm, based on its type, priority, and estimated resource consumption. The quantum state is evolved using quantum rotation gate operations, and the expected execution cost of different task scheduling sequences is evaluated by simulating the measurement operations. Based on the evaluation results, the scheduling sequence with the lowest cost is selected, and detailed resource allocation instructions are generated accordingly, including CPU core allocation, memory bandwidth, and GPU rendering channels.

[0036] It should be noted that the synchronization driver module reads the complete event data packet written by the event capture module from the shared circular buffer via kernel system calls. Subsequently, the driver module parses the event type and scene context within the data packet, and determines the specific combination of feedback modalities (such as visual highlighting, specific sound effects, and controller vibration) to be triggered for this interaction based on a preset rule mapping table, thereby generating a structured list of tasks to be executed. This step transforms the original interaction events into a set of explicit feedback instructions that the system can recognize and process, providing clear and unambiguous processing objects for subsequent optimization and scheduling, ensuring that scheduling decisions are based on complete and accurate business requirements.

[0037] For each feedback task in the task list, the scheduler extracts its predefined task type (e.g., real-time, buffered), the priority weight set by the user or system, and the computational and input / output resource consumption estimated based on historical data or models. Then, the scheduler employs an encoding strategy inspired by quantum computing concepts to map these multidimensional attributes of each task into a complex vector (i.e., a quantum state representation), where different dimensions and phases of the vector represent different states and attribute probability amplitudes of the task. This operation transforms the complex, multi-constrained scheduling problem into a mathematical model that can evolve and be evaluated in parallel in a high-dimensional probability space, laying the foundation for using heuristic optimization algorithms to find a globally approximate optimal solution.

[0038] In a simulated quantum computing environment, the scheduling algorithm applies a series of simulated "quantum rotation gate" operations to all encoded task quantum states. These operations are essentially a set of parameterized matrix transformations, whose parameters are dynamically adjusted based on the current system resource status and optimization objectives to change the probability amplitude distribution of each quantum state. Subsequently, the algorithm collapses multiple possible classical task execution sequences from the evolved quantum states through multiple simulated "measurement" operations, and quickly evaluates the expected execution cost of each sequence using a cost function (comprehensively considering completion time, resource utilization balance, etc.). This process efficiently selects high-performance candidate sequences by exploring a massive number of potential scheduling schemes in parallel.

[0039] After evaluating all candidate scheduling sequences, the algorithm selects the sequence with the lowest expected execution cost as the final scheduling scheme. Based on the task execution order and dependencies specified in this scheme, the scheduler generates a detailed set of resource allocation instructions that can be directly executed by the operating system kernel resource manager. These instructions precisely specify when each feedback task will be executed, by which one or more specific CPU cores, how much memory bandwidth needs to be reserved, and which rendering channel and time slice of the graphics processor will be used. This step transforms the optimized scheduling scheme into low-level instructions that can directly control hardware resources, achieving precise pre-allocation and reservation of computing, storage, and rendering resources. This minimizes resource contention uncertainty before actual task execution, providing crucial resource guarantees for achieving low-jitter, high-determinism instruction execution.

[0040] Specifically, based on the scheduling results, the dynamic delay compensation value for each modality is predicted using a neural symbolic AI model. The specific steps are as follows: Collect current VR system operating status data, including processor temperature, memory usage, GPU load, and network latency; This real-time data is then matched and preliminarily inferred against symbolic rules loaded from a neural symbolic AI model for the current type of interaction event; The matching results and raw data are input into a trained neural network module, which outputs a correction factor for the inherent delay of each modality. By combining the inference results of symbolic rules and the correction factor of the neural network, dynamic delay correction amounts for each modality are generated. Among them, the total intrinsic delay of each mode Provided by the hardware-inherent latency database, and this total inherent latency Delayed by the corresponding software path With inherent hardware latency The sum of them constitutes, that is, satisfies ,in, This includes hardware processing latency and software path latency.

[0041] It should be noted that the system periodically samples and acquires processor core temperature, current utilization rates of cache levels and system memory, load percentages of each GPU computing unit, and network interface round-trip latency using built-in sensors (such as temperature sensors) and performance monitoring counters provided by the operating system kernel. This data is aggregated and formatted into a unified state vector. This step provides the latency prediction model with crucial real-time input describing the system's instantaneous operating environment and resource pressure, enabling the model to perceive external variables that may affect hardware processing speed and providing a data foundation for subsequent environment-related dynamic latency compensation.

[0042] The neurosymbolic AI model loads a set of predefined symbolic rules from its own knowledge base based on the type of the current interaction event (e.g., "shooting" or "collision"). These rules are expressed in formal logical statements such as "IF-THEN," for example, "If the event type is 'shooting,' then audio latency is sensitive to CPU load." The model matches the collected real-time system state data with the preconditions of these rules, performs symbolic reasoning, and draws preliminary qualitative conclusions based on rules about how latency in each modality might be affected. This step encodes human domain knowledge (about the sensitivity of different events to latency) into computable logic, providing prior constraints and interpretable reasoning paths for purely data-driven neural networks, enhancing the rationality of the prediction process and its generalization ability under sparse data.

[0043] The qualitative conclusions generated by the previous symbolic reasoning step (such as "highly sensitive") are concatenated with the original, quantified system state data vector to form a comprehensive input that integrates symbolic knowledge and numerical features. This input is fed into a deep neural network component within the neuro-symbolic AI model. This network component has been trained offline using historical system state data and corresponding actual measured delay data, enabling it to learn the nonlinear mapping relationship between complex, high-dimensional system states and subtle changes in the inherent delays of each modality. After processing the fused input, the network outputs a quantified correction factor for each modality. This step leverages the powerful pattern recognition and function approximation capabilities of neural networks to extract deep, complex patterns affecting delays from the mixed real-time data, achieving a quantified refinement and supplement to rule-based reasoning.

[0044] Defined explicitly at the driver layer Due to software path delay and inherent hardware latency The two parts are linearly superimposed. This definition is based on an in-depth analysis of the entire path from the issuance of the drive command to the output of the hardware signal, laying the theoretical foundation for subsequent accurate measurement (calibration) and separate compensation. A calibrable "hardware inherent delay database" is established and maintained to store the latency of different hardware in different modes. The model's final decision module receives qualitative guidance from symbolic reasoning and quantitative correction factors from the neural network output. It combines these two factors according to a pre-defined fusion strategy (such as weighted averaging or confidence-based selection) to generate a final dynamic latency correction ΔDM for each of the visual, auditory, and tactile modalities. This correction ΔDM is then combined with a baseline latency DM, retrieved from a hardware-inherent latency database and characterizing the inherent characteristics of the hardware and driver software, to calculate the precise instruction triggering time. This step achieves a complementary advantage between the interpretability of symbolic logic and the perceptual accuracy of neural networks, generating a latency compensation value that conforms to domain common sense and adapts to the complex dynamics of real-time systems, thus upgrading static inherent latency compensation to dynamic adaptive intelligent compensation.

[0045] Specifically, the steps for calculating the precise trigger time of each modal feedback based on the dynamic delay compensation value are as follows: Obtain ideal perception time from VR runtime system This time is determined based on the V-Sync signal or the predicted trajectory of head posture changes; For each mode that requires feedback, obtain its total intrinsic delay. and the dynamic delay correction predicted by the neural symbolic AI model ; Introducing formulas Calculate the precise trigger time of the modal feedback command. ,in, The instruction must be sent to the mode. Triggering time of the underlying driver; The theoretical trigger times for all modes are sorted, and the system scheduling overhead is taken into account. The trigger times are then fine-tuned to avoid instruction conflicts and ensure parallel execution.

[0046] It should be noted that the synchronization driver module calls a dedicated interface provided by the VR runtime to obtain the ideal perception time determined by the rendering engine based on the precise timestamp of the next frame's vertical synchronization (V-Sync) signal. Alternatively, it can obtain a future moment calculated by the head pose prediction module based on inertial sensor data as... This operation provides a unified physical time target point determined by the upper layer of the system and aligned with all feedback modes, establishing a precise reference system for subsequent calculation of the independent command issuance time of each mode.

[0047] For each mode to be fed back from the event parsing (e.g., visual, auditory) The synchronization drive module first queries the preset hardware inherent delay database based on the hardware identifier to obtain the total inherent delay of the mode under standard conditions. Simultaneously, the dynamic delay correction amount, predicted for the current real-time state of the system and used to compensate for environmental fluctuations, is obtained from the running neural symbolic AI model. This operation combines the static baseline and dynamic adjustment values ​​required for delay compensation, providing complete delay input parameters for accurate trigger time calculation.

[0048] The core computing unit of the driver module is designed for each mode. The ideal perception time will be obtained As the minuend, the total inherent delay With dynamic delay correction The sum is used as the subtrahend, and arithmetic subtraction is performed according to the formula. Calculations were performed to determine the precise trigger time of the modal feedback command. This calculation step is a crucial transformation from the goal of "when it is expected to be perceived" to an executable plan of "when instructions need to be issued in advance." Its output is the direct basis for the driver layer to control when each piece of hardware should begin to act.

[0049] After calculating all modes The scheduler then sorts these time points in chronological order. Subsequently, considering the minor scheduling overhead of the real-time operating system kernel itself, such as task switching and interrupt responses, it performs nanosecond-level fine-tuning of the sorted time sequence to ensure that instruction sending tasks of different modalities are completely staggered in time during their critical execution phases, avoiding resource contention at the driver layer. This operation is an engineering correction to the ideal calculation results, ensuring that the theoretically parallel triggering plan can be deterministically and conflict-free executed sequentially in a real, non-ideal scheduling environment, thus reliably translating into parallel hardware output.

[0050] Specifically, the step of executing high-priority parallel instruction scheduling at the calculated time point involves the following steps: Create a high-priority timer task for each calculated precise trigger time point and register it in the timer queue of the real-time operating system; When the system timer reaches the preset trigger time, it immediately wakes up the corresponding timer task, which is responsible for sending the modal feedback instruction to the hardware driver layer. Non-blocking I / O operations are used to send instructions in parallel to the visual rendering module, audio playback device, and haptic feedback device; The scheduling module continuously monitors the command sending status and handles retrying or error reporting for possible sending failures or timeouts.

[0051] It should be noted that the synchronization driver module, based on each calculated precise trigger time point Ttrigger,M, calls the high-precision timer application programming interface provided by the real-time operating system (RTOS) kernel to create an independent timed task for each modality. When creating the task, the task attributes are explicitly set to the highest scheduling priority available to the system, and Ttrigger,M is used as the absolute wake-up time parameter for that task. Subsequently, the driver module, through system calls, inserts these created high-priority timed task objects into the kernel-managed timer event queue according to their respective wake-up time sequences. This operation transforms the abstract time calculation result into a concrete execution entity that the operating system kernel can recognize and strictly schedule according to plan, providing a kernel-guaranteed scheduling carrier for subsequent timely triggering.

[0052] When a hardware timer interrupt in the real-time operating system kernel reaches a preset Trigger,M time, the kernel immediately responds to the interrupt. Within its timer interrupt service routine, it precisely retrieves the task to be awakened at that time from the timer queue. The kernel then forces a context switch, preempting any currently running low-priority task and immediately allocating CPU execution rights to the awakened high-priority timer task. Once the task gains execution rights, its function entry logic is activated, and it begins executing code that sends feedback instructions to the hardware driver layer. This process, leveraging kernel interrupts and preemptive scheduling, ensures that regardless of the overall system load, the calculated trigger time can be responded to and executed by the operating system with the highest priority and in the most timely manner, achieving a deterministic transformation from time to action.

[0053] When a high-priority timer task is awakened, it employs non-blocking input / output operations to send instructions to the underlying hardware drivers. Through system calls, it simultaneously initiates parallel instruction write requests to the graphics processing unit (GPU) display rendering driver, the audio digital signal processor (DSP) or codec driver, and the haptic motor microcontroller (MCU) driver. The operating system kernel or driver framework ensures that these write operations do not wait for device responses and thus block task execution, allowing the instruction sending processes to different physical hardware to fully overlap in time. This operation minimizes the sequential latency that may be introduced by serial instruction sending, achieving near-simultaneous triggering of instructions from various heterogeneous hardware drivers, and is a key technical means to achieve parallel output at the hardware level.

[0054] After the instruction sending process begins, an independent monitoring thread or callback mechanism within the scheduling module starts working. It monitors the transmission status of each issued instruction. If it detects that an instruction has not received an acknowledgment from the hardware driver within a preset timeout window, or if it receives a clear transmission error flag, the monitoring module will initiate instruction retransmission according to a predefined strategy (such as immediate retry or delayed retry). If the retry failure exceeds a threshold, a detailed error log is generated and reported to the system management module. This operation establishes a fault-tolerance mechanism that can effectively handle abnormal situations such as momentary hardware busyness and bus interference, preventing the loss of the entire multimodal feedback event due to a single instruction sending failure, thus enhancing the system's robustness and reliability in non-ideal hardware environments.

[0055] Specifically, the driver hardware outputs feedback and collects the actual execution time to optimize subsequent scheduling strategies. The specific steps are as follows: After receiving the feedback command, the VR system hardware module executes the corresponding output operation and internally records the time of command reception and actual execution completion. The hardware module feeds back the actual execution completion time to the VR system's time calibration module via a dedicated high-speed bus; The timing calibration module compares the theoretical trigger time of the instruction with the actual execution completion time to calculate the feedback timing error for each mode. These timing error data and the corresponding system operating status data are used as inputs to adjust and optimize the parameters of the quantum heuristic scheduling algorithm and the prediction model of the neural symbolic AI model online.

[0056] It should be noted that after receiving feedback instructions from the synchronization driver module through their respective core drivers, each hardware module in the VR system (such as the display controller, audio DAC, and haptic motor driver chip) immediately begins executing the specific output operations defined by the instructions. Simultaneously, these hardware modules utilize their internal or tightly coupled high-precision timing circuits to record the precise time points at which the instructions are received by their driver interfaces, and the precise time points at which their physical signals (such as pixel array refresh completion, analog sound wave output establishment, and motor reaching the target amplitude) are actually completed. This operation completes the final execution stage from instruction issuance to physical effect generation, accurately capturing the actual processing time consumed by each hardware component, providing the most direct and final measured data for evaluating and calibrating the synchronization accuracy of the entire system.

[0057] After each hardware module completes its signal output and records the actual completion time, it immediately packages and sends the time data, along with the corresponding instruction identifier, back to the system's main control chip via a dedicated, low-latency on-chip system bus or peripheral interconnect bus. This bus is typically configured with high priority to ensure the real-time nature of the feedback data. The time calibration module on the system's main control chip continuously listens for and receives these time feedback data packets from different hardware modules through the controller interface of this bus. This operation establishes a reverse, high-speed data feedback channel from the distributed hardware to the central processing unit, enabling the system to collect the actual completion time of each modal feedback in near real-time.

[0058] The time calibration module receives a certain mode After obtaining the actual execution completion time data, immediately retrieve the theoretical trigger time corresponding to the instruction from the internal records. Subsequently, the calibration module performs a subtraction operation: subtracting the theoretical trigger time from the actual completion time. The difference obtained is the timing error of this feedback. This error value quantifies the overall deviation between the instruction plan and the actual hardware output, including the effects of all unmodeled dynamic factors and random disturbances. This operation transforms the gap between theory and practice into a measurable and analyzable scalar error value, providing a quantitative basis for the system to understand the precise state of its current synchronization performance.

[0059] The calibration module combines the calculated timing error data for all modes with previously collected system operating status data (such as CPU load and temperature) corresponding to the instruction trigger times, into a training sample dataset. This dataset is fed into an online learning engine, which uses these samples to iteratively adjust the weight parameters used to evaluate the cost of the task sequence in the quantum heuristic scheduling algorithm through gradient descent or other optimization algorithms, while simultaneously fine-tuning the connection weights of the neural network in the neural symbolic AI model. This operation constitutes a complete closed-loop feedback control system that continuously and automatically optimizes the front-end prediction and scheduling model using the actual results after each interaction event, enabling the system to adaptively correct errors and gradually improve the accuracy and robustness of synchronization over long-term operation.

[0060] This embodiment also provides a VR system multimodal feedback synchronization drive system based on timing calibration, including: The event capture module captures user interaction events and records the initial timestamp when the VR application layer detects such events. The scheduling pre-allocation module performs task scheduling and resource pre-allocation based on the initial timestamp using a quantum heuristic scheduling algorithm; The delay prediction module uses a neural symbolic AI model to predict the dynamic delay compensation value for each modality based on the scheduling results. The trigger calculation module calculates the precise trigger time for each modal feedback based on the dynamic delay compensation value; The instruction scheduling module executes high-priority parallel instructions at the calculated time points; The feedback optimization module drives the hardware to output feedback and collects the actual execution time, which is used to optimize subsequent scheduling strategies.

[0061] This embodiment also provides a computer device applicable to the timing-calibrated VR system multimodal feedback synchronization driving method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the timing-calibrated VR system multimodal feedback synchronization driving method proposed in the above embodiment.

[0062] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0063] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the multimodal feedback synchronization driving method for a VR system based on timing calibration as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0064] In summary, this invention effectively solves the timing inaccuracies caused by software scheduling uncertainties and uncompensated heterogeneous hardware latency in existing technologies by sinking the multimodal feedback synchronization mechanism to the operating system kernel layer and constructing a compensation model based on a hardware-inherent latency database and dynamic prediction by neural symbolic AI. This method utilizes quantum heuristic scheduling to optimize task sequences and resource allocation, combined with a high-priority timed triggering mechanism, to achieve microsecond-level deterministic synchronization control of visual, auditory, and tactile feedback. Its closed-loop optimization architecture further enhances the system's timing stability and adaptability under dynamic loads. This solution, while ensuring synchronization accuracy, enhances the system's adaptability and robustness, providing a reliable technical foundation for the realism and immersion of VR interactive experiences.

[0065] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for multi-modal feedback synchronization driving of a VR system based on timing calibration, characterized in that, Includes the following steps: When the VR application layer detects a user interaction event, it captures the event and records the initial timestamp; Based on the initial timestamp, a quantum heuristic scheduling algorithm is used for task scheduling and resource pre-allocation; Based on the scheduling results, the dynamic delay compensation value for each modality is predicted using a neural symbolic AI model; The precise trigger time of each modal feedback is calculated based on the dynamic delay compensation value; Execute high-priority parallel instruction scheduling at the calculated time point; The driver outputs feedback and collects the actual execution time to optimize subsequent scheduling strategies.

2. The timing calibration based VR system multi-modal feedback synchronization driving method of claim 1, wherein: When the VR application layer detects a user interaction event, it captures the event and records the initial timestamp. The specific steps are as follows: The VR application layer captures user interaction events through a system-level high-precision clock source and assigns a unique event identifier to each event; The event identifier, event type, scene context information when the event occurred, and the current time recorded by a high-precision clock source are encapsulated into an event data packet; Write the event data packet into a thread-safe circular buffer and immediately trigger a high-priority interrupt to notify the subsequent processing module; After the event data packet is successfully written to the buffer, the initial timestamp is extracted and recorded as the base time of the interaction event.

3. The VR system multimodal feedback synchronization driving method based on timing calibration as described in claim 2, characterized in that: The specific steps for task scheduling and resource pre-allocation based on the initial timestamp and using a quantum heuristic scheduling algorithm are as follows: Read event data packets from the circular buffer and parse out the list of tasks that require multimodal feedback; Each feedback task is encoded into a quantum state representation that can be processed by a quantum heuristic algorithm, based on its type, priority, and estimated resource consumption. The quantum state is evolved using quantum rotation gate operations, and the expected execution cost of different task scheduling sequences is evaluated by simulating the measurement operations. Based on the evaluation results, the scheduling sequence with the lowest cost is selected, and detailed resource allocation instructions are generated accordingly, including CPU core allocation, memory bandwidth, and GPU rendering channels.

4. The VR system multimodal feedback synchronization driving method based on timing calibration as described in claim 3, characterized in that: The specific steps for predicting the dynamic delay compensation value for each modality based on the scheduling results using a neural symbolic AI model are as follows: Collect current VR system operating status data, including processor temperature, memory usage, GPU load, and network latency; This real-time data is then matched and preliminarily inferred against symbolic rules loaded from a neural symbolic AI model for the current type of interaction event; The matching results and raw data are input into a trained neural network module, which outputs a correction factor for the inherent delay of each modality. By combining the inference results of symbolic rules and the correction factor of the neural network, dynamic delay correction amounts for each modality are generated. Among them, the total intrinsic delay of each mode Provided by the hardware-inherent latency database, and this total inherent latency Delayed by the corresponding software path With inherent hardware latency The sum of them constitutes, that is, satisfies ,in, This includes hardware processing latency and software path latency.

5. The VR system multimodal feedback synchronization driving method based on timing calibration as described in claim 4, characterized in that: The specific steps for calculating the precise trigger time of each modal feedback based on the dynamic delay compensation value are as follows: Obtain ideal perception time from VR runtime system This time is determined based on the V-Sync signal or the predicted trajectory of head posture changes; For each mode that requires feedback, obtain its total intrinsic delay. and the dynamic delay correction predicted by the neural symbolic AI model ; Introducing formulas Calculate the precise trigger time of the modal feedback command. ,in, The instruction must be sent to the mode. Triggering time of the underlying driver; The theoretical trigger times for all modes are sorted, and the system scheduling overhead is taken into account. The trigger times are then fine-tuned to avoid instruction conflicts and ensure parallel execution.

6. The VR system multimodal feedback synchronization driving method based on timing calibration as described in claim 5, characterized in that: The specific steps for executing high-priority parallel instruction scheduling at the calculated time point are as follows: Create a high-priority timer task for each calculated precise trigger time point and register it in the timer queue of the real-time operating system; When the system timer reaches the preset trigger time, it immediately wakes up the corresponding timer task, which is responsible for sending the modal feedback instruction to the hardware driver layer. Non-blocking I / O operations are used to send instructions in parallel to the visual rendering module, audio playback device, and haptic feedback device; The scheduling module continuously monitors the command sending status and handles retrying or error reporting for possible sending failures or timeouts.

7. The VR system multimodal feedback synchronization driving method based on timing calibration as described in claim 6, characterized in that: The driver hardware outputs feedback and collects the actual execution time to optimize subsequent scheduling strategies. The specific steps are as follows: After receiving the feedback command, the VR system hardware module executes the corresponding output operation and internally records the time of command reception and actual execution completion. The hardware module feeds back the actual execution completion time to the VR system's time calibration module via a dedicated high-speed bus; The timing calibration module compares the theoretical trigger time of the instruction with the actual execution completion time to calculate the feedback timing error for each mode. These timing error data and the corresponding system operating status data are used as inputs to adjust and optimize the parameters of the quantum heuristic scheduling algorithm and the prediction model of the neural symbolic AI model online.

8. A VR system multimodal feedback synchronization driving system based on timing calibration, based on the VR system multimodal feedback synchronization driving method based on timing calibration according to any one of claims 1 to 7, characterized in that: include, The event capture module captures user interaction events and records the initial timestamp when the VR application layer detects such events. The scheduling pre-allocation module performs task scheduling and resource pre-allocation based on the initial timestamp using a quantum heuristic scheduling algorithm; The delay prediction module uses a neural symbolic AI model to predict the dynamic delay compensation value for each modality based on the scheduling results. The trigger calculation module calculates the precise trigger time for each modal feedback based on the dynamic delay compensation value; The instruction scheduling module executes high-priority parallel instructions at the calculated time points; The feedback optimization module drives the hardware to output feedback and collects the actual execution time, which is used to optimize subsequent scheduling strategies.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the VR system multimodal feedback synchronization driving method based on timing calibration as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the VR system multimodal feedback synchronization driving method based on timing calibration as described in any one of claims 1 to 7.

Images