An ar interaction closed-loop control method and system for improving user immersion

Abstract

The application provides an AR interaction closed-loop control method and system for improving user immersion, and relates to the technical field of artificial intelligence. The method comprises the following steps: establishing a globally unified closed-loop control timing reference; in response to the user's limb movement, acquiring the spatial positions of each control node of the limb at fixed time intervals through a tracking device; inputting the spatial positions of each control node acquired at the current time as observation values into a state estimator for state estimation to obtain the optimal state vector of each control node at the current time; using the speed estimated by the state estimator at the previous time as a feedback signal to adjust the parameters of the state estimator; and using the speed estimated by the state estimator at the previous time as a precursor signal to drive the spatial positions of each control node after the prediction compensation of the end-to-end delay based on the optimal state vector; and finally controlling the virtual limb to be output on a display output device according to the spatial positions of each control node after the compensation of the end-to-end delay, thereby improving the immersion of AR interaction.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to an AR interactive closed-loop control method and system for enhancing user immersion. Background Technology

[0002] Augmented Reality (AR) technology provides users with an immersive interactive experience by overlaying virtual image information onto the real environment. During AR interaction, the response speed and tracking accuracy of virtual limbs to the user's real-world movements directly determine the level of immersion. When the virtual limbs lag behind the real limbs, users experience a noticeable sense of disconnect and unnaturalness, severely diminishing the realism and usability of AR interaction.

[0003] Traditional AR systems mainly employ the following two technical solutions for interactive responses: The first type is the pure software solution. This type of solution relies entirely on the default task scheduler of general operating systems (such as Android, iOS, and Windows) to complete the acquisition, processing, and rendering of interactive data. Since the operating system scheduler prioritizes fairness and throughput over the real-time response of the interaction, AR interaction tasks need to compete with background system tasks for CPU / GPU resources. This results in significant fluctuations in the latency from the issuance of the interaction command to the virtual limb's response (usually 15–30ms). As a result, users frequently experience unpleasant sensations such as the virtual limbs not responsive, movements being choppy, and responses being inconsistently fast or slow during the interaction process, severely damaging the sense of immersion.

[0004] The second type is an integrated screen compensation technology solution that combines asynchronous time warp technology, motion prediction algorithms, and dedicated processing hardware. This type of solution is based on motion prediction to calculate the expected pose between the current moment and the next display refresh moment. This expected pose serves as the input parameter for the rendering thread. The rendering thread and the display update control are asynchronous and independent. The rendering process has an unstable time consumption (usually 8–20ms). If the rendering cannot be completed before the display refresh, the system will discard the data of the current moment and directly use the data of the previous moment. When the user's body movements undergo substantial changes (such as the hand changing from an extended hand to a clenched fist), the asynchronous time warp technology cannot update the new data used for display in real time, resulting in a significant asynchrony between virtual body movements and actual body movements, and a break in the user's immersion. Summary of the Invention

[0005] The purpose of this application is to provide an AR interaction closed-loop control method and system for improving user immersion, so as to achieve close following of virtual limbs to real limbs and achieve imperceptible delay at the human sensory level, thereby enhancing the immersion of AR interaction.

[0006] In a first aspect, this application provides an AR interaction closed-loop control method for enhancing user immersion, comprising: S1. Synchronously trigger the tracking device, state estimator, and display output device through the same hardware clock source to establish a globally unified closed-loop control timing reference. S2. In response to the user's limb movements in the augmented reality space, under the drive of the closed-loop control timing reference, the tracking device acquires the spatial position of each control node of the limb at fixed time intervals; wherein, the time interval between two adjacent acquisitions is a preset value and the timing jitter is less than a preset threshold. S3. The spatial positions of each control node of the limb acquired at the current moment are used as observation values ​​and input to the state estimator corresponding to each control node for state estimation to obtain the optimal state vector of each control node at the current moment; the state estimator is set for each control node of the limb, and the state vector of each state estimator includes three-dimensional position, three-dimensional velocity components and three-dimensional acceleration components. S4. The three-dimensional velocity components estimated by the state estimator corresponding to each control node at the previous moment are used as feedback signals to dynamically adjust the filtering parameters of the state estimator. At the same time, they are used as precursor signals to drive the prediction and compensation of the spatial position of each control node after end-to-end delay based on the optimal state vector of each control node at the current moment. S5. Based on the spatial position of each control node after the end-to-end delay is compensated, drive control commands for the virtual limbs are generated, and the virtual limbs are controlled to output on the display output device to realize the spatial position synchronization between the virtual limbs and the real limbs, thus completing the closed-loop control of AR interaction.

[0007] According to the technical solution provided in this application, the tracking device includes an image sensor and a supplementary lighting device. The image sensor and the supplementary lighting device are triggered by a hardware clock source through a fan-out buffer. The transmission clock of the USB controller used to transmit the data collected by the image sensor to the state estimator is synchronized with the hardware clock source. The preset value is 8.33ms, corresponding to a 120fps sampling rate. The preset threshold is 0.05ms.

[0008] According to the technical solution provided in this application, S4 specifically includes the following steps: S401. Extract the three-dimensional velocity components of the previous moment from the optimal state vector output by the state estimator corresponding to each control node at the previous moment and calculate the resultant velocity of the three-dimensional velocity components at the previous moment. Dynamically determine the process noise figure and observation noise figure at the current moment based on the resultant velocity of the three-dimensional velocity components at the previous moment, and update the filtering parameters of the state estimator corresponding to each control node accordingly. S402. Filter the observations of each control node at the current time using the updated filtering parameters to obtain the optimal state vector of each control node at the current time. S403. Extract the three-dimensional velocity component and three-dimensional acceleration component from the optimal state vector at the current moment as the prediction driving signal; estimate the end-to-end delay from state acquisition to virtual limb display output in real time; dynamically determine the compensation time window based on the end-to-end delay and the three-dimensional velocity component and three-dimensional acceleration component at the current moment. S404. Based on the optimal state vector at the current moment, extrapolate the spatial positions of each control node after the compensation time window using a motion model.

[0009] According to the technical solution provided in this application, the process noise figure and observation noise figure at the current moment are dynamically determined based on the resultant velocity of the three-dimensional velocity components at the previous moment, including: A preset low-speed resultant velocity threshold and the low-speed process noise figure and low-speed observation noise figure corresponding to the low-speed resultant velocity threshold are defined. A preset high-speed resultant velocity threshold and a high-speed process noise figure and a high-speed observation noise figure corresponding to the low-speed resultant velocity threshold are defined; the high-speed resultant velocity threshold is greater than the low-speed resultant velocity threshold, the high-speed process noise figure is greater than the low-speed process noise figure, and the high-speed observation noise figure is less than the low-speed observation noise figure. The difference coefficient is calculated based on the difference between the resultant velocity and the low-speed resultant velocity threshold, and the difference between the high-speed resultant velocity threshold and the low-speed resultant velocity threshold at the previous moment. When the resultant velocity at the previous moment is less than or equal to the low-speed resultant velocity threshold, the process noise figure at the current moment is set to equal the low-speed process noise figure and the observation noise figure is set to equal the low-speed observation noise figure. When the resultant velocity at the previous moment is greater than or equal to the high-speed resultant velocity threshold, the process noise figure at the current moment is set to be equal to the high-speed process noise figure and the observation noise figure is set to be equal to the high-speed observation noise figure. When the resultant velocity at the previous moment is greater than the low-speed resultant velocity threshold and less than the high-speed resultant velocity threshold, the process noise coefficient at the current moment is calculated by linear interpolation based on the low-speed process noise coefficient, the high-speed process noise coefficient and the difference coefficient; the observation noise coefficient at the current moment is calculated by linear interpolation based on the low-speed observation noise coefficient, the high-speed observation noise coefficient and the difference coefficient.

[0010] According to the technical solution provided in this application, the state estimator adopts a Kalman filter, the Kalman filter is configured as a constant acceleration model, and a state transition matrix corresponding to the constant acceleration model is configured, wherein the time step in the state transition matrix is ​​equal to the preset value.

[0011] According to the technical solution provided in this application, the end-to-end delay includes at least one of the following: tracking device exposure readout delay, data transmission bus delay, state estimation and motion prediction calculation delay, graphics rendering pipeline delay, and display output device scan output delay.

[0012] According to the technical solution provided in this application, the compensation time window is dynamically determined based on the end-to-end delay and the three-dimensional velocity component and three-dimensional acceleration component at the current moment, including: Calculate the resultant velocity and resultant acceleration at the current moment based on the three-dimensional velocity components and three-dimensional acceleration components at the current moment; The motion phase of each control node at the current moment is identified based on the resultant velocity and resultant acceleration at the current moment. The motion phase includes an acceleration phase, a constant velocity phase, and a deceleration phase. When the motion phase is the acceleration phase, the length of the compensation time window is set to the sum of the end-to-end delay and the preset offset duration; When the motion phase is a constant velocity phase, the length of the compensation time window is set to the end-to-end delay; When the motion phase is a deceleration phase, the length of the compensation time window is set to the difference between the end-to-end delay and the preset offset duration.

[0013] According to the technical solution provided in this application, the motion phase of each control node at the current moment is identified based on the resultant velocity and resultant acceleration at the current moment, including: When the absolute value of the resultant acceleration at the current moment is less than a preset static threshold, it is determined to be in a uniform velocity phase; When the absolute value of the resultant acceleration at the current moment is greater than or equal to a preset rest threshold, and the direction of the resultant acceleration at the current moment is the same as the direction of the resultant velocity, it is determined to be an acceleration phase. When the absolute value of the resultant acceleration at the current moment is greater than or equal to a preset rest threshold, and the direction of the resultant acceleration at the current moment is opposite to the direction of the resultant velocity, it is determined to be a deceleration phase.

[0014] According to the technical solution provided in this application, prior to S5, the method further includes: The first-order low-pass smoothing coefficient is dynamically determined based on the resultant velocity of the three-dimensional velocity components output by the state estimator corresponding to each control node at the previous moment. The spatial positions of each control node after compensation for end-to-end delay are low-pass filtered using the first-order low-pass smoothing coefficient. The spatial positions of each control node after filtering are used to generate the drive control command.

[0015] Secondly, this application provides an AR interactive closed-loop control system for enhancing user immersion, comprising: Hardware timing synchronization module: configured to synchronously trigger the tracking device, state estimator and display output device through the same hardware clock source, and establish a globally unified closed-loop control timing reference; Tracking device: configured to respond to the user's limb movements in the augmented reality space, and under the drive of the closed-loop control timing reference, acquire the spatial position of each control node of the limb at fixed time intervals; wherein the time interval between two adjacent acquisitions is a preset value and the timing jitter is less than a preset threshold; State estimation module: Configured to set state estimators for each control node of the limb. The state vector of each state estimator includes three-dimensional position, three-dimensional velocity components and three-dimensional acceleration components. The spatial position of each control node of the limb acquired at the current moment is used as the observation value to perform state estimation and obtain the optimal state vector of each control node at the current moment. The control module is used to take the three-dimensional velocity components estimated by the state estimator corresponding to each control node at the previous moment. On the one hand, it serves as a feedback signal to dynamically adjust the filtering parameters of the state estimator. On the other hand, it also serves as a precursor signal to drive the prediction and compensation of the spatial position of each control node after end-to-end delay based on the optimal state vector of each control node at the current moment. Based on the spatial position of each control node after end-to-end delay compensation, it generates driving control commands for the virtual limbs and controls the output of the virtual limbs on the display output device to realize the synchronization of the spatial position of the virtual limbs and the real limbs, thus completing the closed-loop control of AR interaction.

[0016] In summary, this application provides an AR interaction closed-loop control method to enhance user immersion, comprising: synchronously triggering a tracking device, a state estimator, and a display output device through the same hardware clock source to establish a globally unified closed-loop control timing reference; responding to the user's limb movement in the augmented reality space, under the drive of the closed-loop control timing reference, acquiring the spatial positions of each control node of the limb at fixed time intervals through the tracking device; wherein the time interval between two adjacent acquisitions is a preset value and the timing jitter is less than a preset threshold; using the spatial positions of each control node of the limb acquired at the current moment as observation values, inputting them to the state estimator corresponding to each control node for state estimation, and obtaining the optimal state vector of each control node at the current moment; for Each control node of the limb is equipped with a state estimator. The state vector of each state estimator includes three-dimensional position, three-dimensional velocity components, and three-dimensional acceleration components. The three-dimensional velocity components estimated by the state estimator corresponding to each control node at the previous moment are used as feedback signals to dynamically adjust the filtering parameters of the state estimator. At the same time, they also serve as precursor signals to drive the prediction and compensation of the spatial position of each control node after end-to-end delay based on the optimal state vector of each control node at the current moment. The virtual limb is driven by the spatial position of each control node after end-to-end delay compensation. The virtual limb is controlled to output on the display output device, thereby synchronizing the spatial position of the virtual limb with the real limb and completing the closed-loop control of AR interaction.

[0017] This application establishes a globally unified closed-loop control timing reference through hardware-level timing synchronization, enabling the state estimator to obtain a jitter-free observation sequence with equal time intervals, thus eliminating jitter at its source. This lays a deterministic data foundation for the real-time forward response of the filter parameters and the prediction compensation of end-to-end delay. Based on this, the speed output of the state estimator at the previous moment is used as a feedback signal to dynamically adjust the state estimator's own filter parameters. This allows the filter parameters to respond in real-time to the user's limb movement speed, achieving speed-based control. Strong filtering at low speeds ensures extremely smooth output under static drift, while weak filtering at high speeds ensures no lag in dynamic response. Furthermore, it serves as a precursor signal to drive delay prediction compensation, ensuring that the prediction compensation is based on the optimal state vector at the current moment rather than lag. The data may be distorted or inaccurate. By using the same speed information twice in feedback adjustment and forward drive, the two contradictory links of filtering smoothing and delay prediction, which are independent in the traditional solution, are unified into the same state vector. This makes the optimization of filtering parameters and the driving of prediction compensation work together: the improvement of filtering accuracy provides more reliable state information for prediction, and the improvement of prediction accuracy verifies the filtering effect in turn. The two form a mutually reinforcing closed-loop control link. Finally, the virtual limb output is directly driven by the spatial position of the control node after prediction. When the user sees the virtual limb, its display position corresponds exactly to the current position of the real limb. This eliminates the floating feeling of the virtual limb being half a beat slower and achieves imperceptible delay at the human sensory level, thus improving the immersiveness of AR interaction.

[0018] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this embodiment, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1A flowchart illustrating an AR interaction closed-loop control method for enhancing user immersion, provided in an embodiment of this application; Figure 2 A schematic diagram illustrating the process of step S4 provided in this application embodiment; Figure 3 This is a schematic diagram of an AR interactive closed-loop control system for enhancing user immersion, provided as an embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The descriptions in this section are merely illustrative and explanatory, and should not be construed as limiting the scope of protection of this application. Specifically, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort should fall within the scope of protection of this application.

[0022] It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0023] To facilitate understanding of this application, a brief introduction to the relevant technologies is given first: Augmented Reality (AR) overlays virtual images onto the real environment to achieve seamless interaction between the virtual and real worlds. It is now widely used in industrial training, motion-sensing games, remote assistance, and medical aids. End-to-end interaction latency and temporal stability are crucial for evaluating the AR limb interaction experience, virtual-real tracking effect, and wearing comfort. Current mainstream optimization solutions have significant shortcomings: pure software solutions rely on system scheduling, resulting in temporal interval fluctuations of ±10ms and latency fluctuations between 15-30ms. Poor virtual limb tracking leads to frequent user experiences of unresponsive virtual limbs, stuttering movements, and inconsistent response times, severely disrupting the user's immersion. Furthermore, traditional hardware-software hybrid solutions using ATW (Asynchronous Time Warp), basic motion prediction, and dedicated hardware can only make simple corrections to the image position, failing to guarantee synchronization between virtual and real movements. Coupled with unstable rendering times, issues such as image drift, bounce, and ghosting easily occur during high-speed limb movements and transitions, disrupting the user's immersion. Therefore, existing technologies result in persistent synchronization errors between virtual and real limbs, and cannot eliminate issues such as interaction stuttering and image distortion, significantly reducing the user's immersive experience.

[0024] In view of this, embodiments of this application provide an AR interaction closed-loop control method and system for improving user immersion, so as to achieve close following of virtual limbs to real limbs and achieve imperceptible delay at the human sensory level, thereby enhancing the immersion of AR interaction.

[0025] To make the technical solution of this application clearer and easier to understand, the AR interactive closed-loop control method for improving user immersion provided in the embodiments of this application is described below.

[0026] It should be noted that the subject executing this method can be an AR terminal device.

[0027] This embodiment provides an AR interaction closed-loop control method for improving user immersion, including: S1. Synchronously trigger the tracking device, state estimator, and display output device through the same hardware clock source to establish a globally unified closed-loop control timing reference. Specifically, in this embodiment, a 48MHz TCXO temperature-compensated crystal oscillator is used as the hardware clock source. The tracking device is used to acquire the spatial position of each control node of the limb at fixed time intervals. The state estimator is used to estimate the optimal state vector of each control node of the limb based on the spatial position acquired at the current moment. The display output device is used to present to the user the spatial position of each control node after end-to-end delay compensation based on the optimal state vector, as displayed through the virtual limb. The tracking device, state estimator and display output device are synchronously triggered by the same hardware clock source. The purpose is to ensure that the data objects processed by the tracking device, state estimator and display output device when they perform their respective steps in a single time interval (the spatial position of each control node of the limb, the optimal state vector, and the spatial position of each control node after end-to-end delay compensation) originate from the same point in time, avoiding the introduction of additional time jitter errors and laying the foundation for subsequent precise calculations.

[0028] S2. In response to the user's limb movements in the augmented reality space, under the drive of the closed-loop control timing reference, the spatial position of each control node of the limb is acquired at fixed time intervals through the tracking device; wherein, the time interval between two adjacent acquisitions is a preset value and the timing jitter is less than a preset threshold. Specifically, this step relies on a unified hardware timing benchmark built by S1 to conduct data acquisition. A tracking device periodically collects the spatial positions of limb control nodes such as the wrist, elbow, and fingertips at fixed time intervals. These spatial positions are the three-dimensional positions corresponding to each control node. The interval between two consecutive data acquisitions by the tracking device is a pre-set fixed duration (i.e., a preset value). Simultaneously, the difference between the actual acquisition time interval and the fixed time interval (i.e., timing jitter) is strictly controlled within a preset threshold. This ensures that the tracking device outputs raw observation data with uniform timing and minimal fluctuations, avoiding computational bias caused by uneven sampling rhythms. This provides an accurate and reliable input data source for subsequent steps such as state estimation and motion prediction.

[0029] S3. The spatial positions of each control node of the limb acquired at the current moment are used as observation values ​​and input into the state estimator corresponding to each control node to perform state estimation, so as to obtain the optimal state vector of each control node at the current moment; a state estimator is set for each control node of the limb, and the state vector of each state estimator includes three-dimensional position, three-dimensional velocity components and three-dimensional acceleration components. Specifically, in this embodiment, the state estimator is a Kalman filter. Since random noise and observation bias are inevitably introduced during image acquisition and data transmission, directly using the raw acquired spatial positions of the control nodes will cause the virtual limb image to jitter and the trajectory to distort. The core function of the Kalman filter is to filter out noise and optimally estimate the motion state of the raw observation data of the control nodes. For each control node, based on its optimal state vector at the previous moment and the observed spatial position (i.e., three-dimensional position) at the current moment, the Kalman filter calculates the three-dimensional position, three-dimensional velocity components, and three-dimensional acceleration components of each control node at the current moment. These three components together constitute the optimal state vector of each control node at the current moment, providing stable and high-precision data support for subsequent motion trend prediction and virtual limb posture rendering. Therefore, in this embodiment, a Kalman filter is independently configured for each control node, such as the wrist joint and elbow joint, synchronously incorporating the three-dimensional position, three-dimensional velocity components, and three-dimensional acceleration components of the limb control node to comprehensively describe the real-time motion state of the control node and obtain the optimal state vector of each control node at the current moment. The specific steps are as follows: Taking one of the control nodes as an example, firstly, the optimal state vector output by the Kalman filter at the previous time step is used. The prior state vector of the control node at the current time is predicted according to the following formula (1), thus obtaining the prior state vector of the control node at the current time. : Formula (1) in, This is the prior state vector of the control node at the current moment; This is the optimal state vector output by the Kalman filter at the previous moment; Let be the state transition matrix of the Kalman filter.

[0030] Then, based on the state transition matrix The process noise covariance matrix Q of the Kalman filter at the current time step, and the posterior state covariance matrix of the Kalman filter at the previous time step. The prior state covariance matrix of the Kalman filter at the current time is calculated according to the following formula (2). .

[0031] Formula (2) in, Let be the prior state covariance matrix of the Kalman filter at the current moment; Let be the posterior state covariance matrix of the Kalman filter at the previous time step; is the state transition matrix of the Kalman filter; Q is the process noise covariance matrix of the Kalman filter at the current time.

[0032] It should be noted that the covariance matrix of the posterior state at the previous time step... The Kalman filter of the control node has historical operation parameters that are retained after each iteration. After each round of filtering iteration, the posterior state covariance matrix at the current time is updated and cached. When the prior covariance is recursively calculated in the next frame, the cached posterior covariance matrix of the previous time is directly retrieved as the data to be used in the calculation, without the need for additional data collection or external data import.

[0033] Complete the prior state vector Prior state covariance matrix After calculation, combined with the observation matrix H and the prior state covariance moment of the Kalman filter at the current time... The Kalman gain is calculated using formula (3) based on the observation noise covariance matrix R of the Kalman filter at the current time. : Formula (3) in, Kalman gain; H is the prior state covariance matrix of the Kalman filter at the current time; H is the observation matrix; and R is the observation noise covariance matrix of the Kalman filter at the current time.

[0034] It should be noted that the expression for the observation matrix H is shown in the following formula (4): Formula (4) Where H is the observation matrix; It is a 3-order identity matrix.

[0035] The spatial position (i.e., three-dimensional position) of the control node at the current moment. For the observed values, use Kalman gain The prior state vector of the control node at the current moment The corrections are made to eliminate the discrepancy between model predictions and actual observations, resulting in the optimal state vector at the current moment, which includes three-dimensional position, three-dimensional velocity, and three-dimensional acceleration. The calculation process is shown in formula (5): Formula (5) in, This represents the optimal state vector of the control node at the current moment. H is the Kalman gain; H is the observation matrix; To determine the spatial position of the control node at the current moment; This is the prior state vector of the control node at the current moment.

[0036] Therefore, when When the value approaches 0, it indicates that the optimal state vector output by the Kalman filter is almost entirely determined by the prior state vector, and the correction effect of the observation value is extremely weak; when When the value approaches the theoretical upper limit (a very large value), it indicates that the optimal state vector output by the Kalman filter will significantly adopt the correction amount of the observations, weakening the weight of the prior state vector; in summary, the Kalman gain... Used to assign confidence weights between prior predictions and observed data; a larger gain value indicates that the filter places more weight on the observed values ​​collected by the tracking device, amplifying the observed residuals. The degree to which the prior state vector is corrected; conversely, the degree to which the prior state vector is relied upon is greater. ).

[0037] It should be noted that the observation matrix H is used to preserve the prior state vector. The three-dimensional position is determined by removing the three-dimensional velocity and acceleration components to match the spatial position (i.e., three-dimensional position) obtained by the tracking device at the current moment.

[0038] The optimal state vector of the control node at the current moment The expression for is shown in the following formula (6): Formula (6) in, This represents the optimal state vector of the control node at the current moment. The X-axis coordinate of the control node at the current moment; The Y-axis coordinate of the control node at the current moment; The Z-axis coordinate of the control node at the current moment; The velocity component of the control node along the X-axis at the current moment; The velocity component of the control node along the Y-axis at the current moment; The velocity component of the control node along the Z-axis at the current moment; The acceleration component along the X-axis of the control node at the current moment; The Y-axis acceleration component of the control node at the current moment; This represents the acceleration component of the control node along the Z-axis at the current moment.

[0039] S4. The three-dimensional velocity components estimated by the state estimator corresponding to each control node at the previous moment are used as feedback signals to dynamically adjust the filtering parameters of the state estimator. At the same time, they are used as precursor signals to drive the prediction and compensation of the spatial position of each control node after end-to-end delay based on the optimal state vector of each control node at the current moment. Specifically, this step simultaneously updates the filtering parameters of the state estimator and predicts the spatial positions of each control node after compensating for the end-to-end delay. First, the three-dimensional velocity components of the previous time step are extracted from the optimal state vector output by the state estimator corresponding to each control node at the previous time step, and the resultant velocity of the three-dimensional velocity components at the previous time step is calculated. Based on the resultant velocity of the three-dimensional velocity components at the previous time step, the process noise figure and observation noise figure at the current time step are dynamically determined, and the filtering parameters of the state estimator corresponding to each control node are updated accordingly, so that the filtering parameters of the state estimator respond to the user's limb movement speed in real time. Then, the updated filtering parameters are used to filter the observation values ​​of each control node at the current time step to obtain the optimal state vector of each control node at the current time step. Next, the three-dimensional velocity components and three-dimensional acceleration components of the current time step are extracted from the optimal state vector at the current time step as prediction driving signals, and the end-to-end delay from state acquisition to virtual limb display output is estimated in real time. The compensation time window is dynamically determined based on the end-to-end delay and the three-dimensional velocity components and three-dimensional acceleration components at the current time step. Finally, based on the optimal state vector at the current time step, the spatial positions of each control node after compensating for the time window are extrapolated using a motion model.

[0040] S5. Generate drive control commands for the virtual limbs based on the spatial positions of each control node after compensating for end-to-end delay, control the virtual limbs to output on the display output device, realize the spatial position synchronization between the virtual limbs and the real limbs, and complete the closed-loop control of AR interaction.

[0041] Specifically, the spatial positions of each control node after the calculated compensation time window are packaged and encapsulated into standardized drive control commands, which are then sent to the display output device. The display output device has a built-in graphics rendering unit that can directly parse the received drive control commands, thereby rendering the spatial position of the virtual limb. The rendered virtual limb image is then synchronously output to the display screen for the user to observe, completing the AR interaction closed loop from limb acquisition, state estimation, motion prediction to image display.

[0042] In a preferred embodiment, the tracking device includes an image sensor and a supplementary lighting device. The image sensor and the supplementary lighting device are triggered by a hardware clock source distributed via a fan-out buffer. Furthermore, the transmission clock of the USB controller used to transmit the data collected by the image sensor to the state estimator is synchronized with the hardware clock source. The preset value is 8.33ms, corresponding to a sampling rate of 120fps. The preset threshold is 0.05ms.

[0043] Specifically, the tracking device includes an image sensor and a supplementary lighting device. The image sensor is responsible for acquiring the spatial position of each control node of the limb. The supplementary lighting device, in this embodiment, provides synchronous illumination to ensure clear and stable imaging; this is an LED. The clock signal from the hardware clock source is evenly distributed through a fan-out buffer and synchronously triggers the image sensor and supplementary lighting device. Acquisition is performed cyclically at a fixed time interval of 8.33ms (preset value) corresponding to a sampling rate of 120fps, with timing jitter controlled within 0.05ms (preset threshold) throughout the process. Furthermore, the USB controller is responsible for transmitting the spatial position data; its transmission clock is also synchronized with the hardware clock source to prevent transmission delay fluctuations, data misalignment, and time loss caused by clock asynchrony during data transmission.

[0044] In a preferred embodiment, such as Figure 2 As shown, S4 specifically includes the following steps: S401. Extract the three-dimensional velocity components of the previous moment from the optimal state vector output by the state estimator corresponding to each control node at the previous moment and calculate the resultant velocity of the three-dimensional velocity components at the previous moment. Dynamically determine the process noise figure and observation noise figure at the current moment based on the resultant velocity of the three-dimensional velocity components at the previous moment, and update the filtering parameters of the state estimator corresponding to each control node accordingly. Specifically, in this embodiment, the filtering parameters of the state estimator include the process noise covariance matrix and the observation noise covariance matrix of the state estimator, and the specific update process is as follows: First, the optimal state vector output by the Kalman filter at the previous time step. In the process, extract the three-dimensional velocity components from the previous moment. , , The optimal state vector output by the Kalman filter at the previous time step. The expression for is shown in the following formula (7): Formula (7) in, This is the optimal state vector output by the Kalman filter at the previous moment; The X-axis coordinate of the Kalman filter output at the previous moment; The Y-axis coordinate of the Kalman filter output at the previous moment; The Z-axis coordinate is the output of the Kalman filter at the previous moment; The velocity component of the X-axis output by the Kalman filter at the previous moment; The velocity component of the Y-axis output by the Kalman filter at the previous moment; The Z-axis velocity component is the output of the Kalman filter at the previous moment; This represents the X-axis acceleration component output by the Kalman filter at the previous moment; This represents the Y-axis acceleration component output by the Kalman filter at the previous moment; This represents the Z-axis acceleration component output by the Kalman filter at the previous moment.

[0045] Then, based on the three-dimensional velocity components of the previous moment... , , The resultant velocity of the three-dimensional velocity components at the previous moment is calculated according to the following formula (8). : Formula (8) in, The resultant velocity of the three-dimensional velocity components at the previous moment; The velocity component of the X-axis output by the Kalman filter at the previous moment; The velocity component of the Y-axis output by the Kalman filter at the previous moment; This represents the Z-axis velocity component output by the Kalman filter at the previous moment.

[0046] Then, a low-speed resultant velocity threshold and corresponding low-speed process noise figure and low-speed observation noise figure are preset; a high-speed resultant velocity threshold and corresponding high-speed process noise figure and high-speed observation noise figure are also preset; the high-speed resultant velocity threshold is greater than the low-speed resultant velocity threshold, the high-speed process noise figure is greater than the low-speed process noise figure, and the high-speed observation noise figure is less than the low-speed observation noise figure. Based on the resultant velocity at the previous moment... The difference coefficient is calculated by taking the difference between the low-speed combined velocity threshold and the difference between the high-speed combined velocity threshold and the low-speed combined velocity threshold. When the combined velocity at the previous moment is less than or equal to the low-speed combined velocity threshold, the process noise coefficient at the current moment is set to be equal to the low-speed process noise coefficient and the observation noise coefficient is set to be equal to the low-speed observation noise coefficient. When the combined velocity at the previous moment is greater than or equal to the high-speed combined velocity threshold, the process noise coefficient at the current moment is set to be equal to the high-speed process noise coefficient and the observation noise coefficient is set to be equal to the high-speed observation noise coefficient. When the combined velocity at the previous moment is greater than the low-speed combined velocity threshold and less than the high-speed combined velocity threshold, the process noise coefficient at the current moment is calculated by linear interpolation of the low-speed process noise coefficient, the high-speed process noise coefficient and the difference coefficient. The observation noise coefficient at the current moment is calculated by linear interpolation of the low-speed observation noise coefficient, the high-speed observation noise coefficient and the difference coefficient.

[0047] Finally, by updating the process noise covariance matrix of the Kalman filter using formula (9) and the observation noise covariance matrix of the Kalman filter using formula (10), the filter parameters of the state estimator can be updated.

[0048] Formula (9) Formula (10) Where Q is the process noise covariance matrix of the Kalman filter at the current time; R is the observation noise covariance matrix of the Kalman filter at the current time; The process noise figure at the current moment; The current observation noise figure; The basic process noise covariance matrix; Based on the observation noise covariance matrix.

[0049] It should be noted that, , All were obtained through offline calibration: the basic observation noise covariance matrix The calibration is performed on the image sensor to characterize the inherent sampling noise of the hardware; the basic process noise covariance matrix is ​​obtained. The constant acceleration model used in this embodiment is calibrated to obtain the reference error that characterizes the constant acceleration model itself.

[0050] S402. Filter the observations at the current time using the updated filtering parameters to obtain the optimal state vector of each control node at the current time. Specifically, a filtering iteration operation is performed independently for each limb control node. The process noise covariance matrix and observation noise covariance matrix updated by S401 are used as input parameters for this filtering operation, combined with the spatial position of the control node at the current moment and the optimal state vector output by the state estimator at the previous moment. The optimal state vector of the control node at the current moment can then be obtained. After all control nodes complete the filtering operation synchronously and in parallel, the optimal state vectors of all control nodes at the current moment can be obtained by summing them up. The specific process has been explained in detail in S3, and will not be repeated here.

[0051] S403. Extract the three-dimensional velocity component and three-dimensional acceleration component of the current moment from the optimal state vector at the current moment as the prediction driving signal; estimate the end-to-end delay from state acquisition to virtual limb display output in real time; dynamically determine the compensation time window based on the end-to-end delay and the three-dimensional velocity component and three-dimensional acceleration component of the current moment. Specifically, end-to-end latency refers to the total time difference between the moment a real limb begins to move and the moment the virtual limb image is finally refreshed and displayed on the display output device. Since the entire process—visual acquisition, data transmission, algorithm computation, graphics rendering, and screen output—involves inherent time consumption, these multi-stage delays can accumulate, causing the system's output virtual limb posture to consistently lag behind the real limb's real-time posture. This easily leads to problems such as misalignment between virtual and real limbs, image ghosting, and poor interactive responsiveness. Therefore, it is necessary to estimate the total end-to-end latency in real time and dynamically and adaptively adjust the compensation time window length based on the real-time motion status of the control nodes. This ensures that the final output virtual limb node coordinates can preemptively offset the motion lag caused by end-to-end latency, match the spatial position of the real limb, and effectively reduce the synchronization error between virtual and real limbs. The specific steps for determining the compensation time window are as follows: Calculate the resultant velocity and resultant acceleration at the current moment based on the three-dimensional velocity components and three-dimensional acceleration components at the current moment; identify the motion phase of each control node at the current moment based on the resultant velocity and resultant acceleration, where the motion phase includes acceleration, constant velocity, and deceleration phases; when the motion phase is acceleration, set the length of the compensation time window to the sum of the end-to-end delay and the preset offset time; when the motion phase is constant velocity, set the length of the compensation time window to the end-to-end delay; when the motion phase is deceleration, set the length of the compensation time window to the difference between the end-to-end delay and the preset offset time.

[0052] S404. Based on the optimal state vector at the current moment, the spatial positions of each control node after the compensation time window are extrapolated using a motion model.

[0053] Specifically, in this embodiment, the motion model adopts constant acceleration. Therefore, based on the following formulas (11), (12), and (13), the spatial position of the control node after the compensation time window length can be extrapolated to compensate for the lag caused by the end-to-end delay in advance, thereby offsetting the display deviation caused by the end-to-end delay and ensuring that the virtual limb moves in sync with the real limb: Formula (11) Formula (12) Formula (13) in, The X-axis coordinate after the control node compensation time window length; The Y-axis coordinate after the control node compensation time window length; The Z-axis coordinate after the control node compensation time window length; The X-axis coordinate of the control node at the current moment; The Y-axis coordinate of the control node at the current moment; The Z-axis coordinate of the control node at the current moment; The velocity component of the control node along the X-axis at the current moment; The velocity component of the control node along the Y-axis at the current moment; The velocity component of the control node along the Z-axis at the current moment; The acceleration component along the X-axis of the control node at the current moment; The Y-axis acceleration component of the control node at the current moment; The acceleration component along the Z-axis of the control node at the current moment; To compensate for the length of the time window.

[0054] In a preferred embodiment, the process noise figure and observation noise figure at the current moment are dynamically determined based on the resultant velocity of the three-dimensional velocity components at the previous moment, including: Preset low-speed resultant velocity threshold and corresponding low-speed process noise figure and low-speed observation noise figure; Preset a high-speed resultant velocity threshold and the corresponding high-speed process noise figure and high-speed observation noise figure; the high-speed resultant velocity threshold is greater than the low-speed resultant velocity threshold, the high-speed process noise figure is greater than the low-speed process noise figure, and the high-speed observation noise figure is less than the low-speed observation noise figure. Specifically, a low-speed combined velocity threshold is preset in advance. Low-speed observation noise figure Noise figure during low-speed processes Simultaneously, a high-speed resultant velocity threshold is preset. High-speed observation noise figure High-speed process noise figure All the thresholds and coefficients mentioned above were determined based on the device hardware parameters, limb movement characteristics, offline calibration experiments, and multiple field tests and adjustments. Specifically, they were determined by considering the inherent noise levels of the image sensor and data transmission link, and by differentiating the speed ranges of micro-movements, normal movements, and extreme movements of the human limbs to define low-speed composite speed thresholds. and high-speed combined speed threshold Then, using the absence of virtual limb jitter and lag as the evaluation criteria, the noise coefficient was repeatedly adjusted to match different motion states. In this embodiment, the high-speed resultant velocity threshold... Greater than the low-speed combined velocity threshold High-speed process noise figure Greater than the noise figure of the low-speed process High-speed observation noise figure Less than the low-speed observation noise figure .

[0055] The difference coefficient is calculated based on the difference between the resultant velocity at the previous moment and the low-speed resultant velocity threshold, and the difference between the high-speed resultant velocity threshold and the low-speed resultant velocity threshold. Specifically, based on the resultant velocity of the three-dimensional velocity components at the previous moment... Combined speed threshold with low speed The difference, the high-speed combined velocity threshold Combined speed threshold with low speed The difference is calculated using the following formula (14) to obtain the difference coefficient. : Formula (14) in, The difference coefficient; The resultant velocity of the three-dimensional velocity components at the previous moment; The high-speed combined velocity threshold; This is the low-speed combined velocity threshold.

[0056] When the resultant velocity at the previous moment is less than or equal to the low-speed resultant velocity threshold, set the process noise figure at the current moment to be equal to the low-speed process noise figure and the observation noise figure to be equal to the low-speed observation noise figure. When the resultant velocity at the previous moment is greater than or equal to the high-speed resultant velocity threshold, set the process noise figure at the current moment to be equal to the high-speed process noise figure and the observation noise figure to be equal to the high-speed observation noise figure. When the resultant velocity at the previous moment is greater than the low-speed resultant velocity threshold and less than the high-speed resultant velocity threshold, the process noise coefficient at the current moment is calculated by linear interpolation based on the low-speed process noise coefficient, the high-speed process noise coefficient, and the difference coefficient; the observation noise coefficient at the current moment is calculated by linear interpolation based on the low-speed observation noise coefficient, the high-speed observation noise coefficient, and the difference coefficient.

[0057] Specifically, the resultant velocity combining the three-dimensional velocity components from the previous moment. Combined speed threshold with low speed High-speed combined speed threshold The magnitude relationship is used to determine the process noise figure at the current moment by differentiating between three motion conditions. and observation noise figure : Operating Condition 1: When At this time, the limbs are in a static or slow micro-movement state. In order to suppress sampling noise and smooth the motion trajectory, directly cause... , .

[0058] Operating Condition 2: When At that time, the limbs are in a state of rapid movement. To avoid filtering lag and ensure data tracking accuracy, the command is directly... , .

[0059] Operating Condition 3: When At that time, the process noise figure at the current moment is calculated using the following formula (15). The observation noise figure is calculated using the following formula (16). : Formula (15) Formula (16) in, The process noise figure at the current moment; The current observation noise figure; The difference coefficient; This represents the noise figure for low-speed observations. The noise figure for low-speed processes; The noise figure for high-speed observation; The noise figure represents the high-speed process noise figure.

[0060] In a preferred embodiment, the state estimator employs a Kalman filter, which is configured as a constant acceleration model and configured with a corresponding state transition matrix for the constant acceleration model. The time step in the state transition matrix is ​​equal to a preset value.

[0061] Specifically, since the state estimator uses a Kalman filter, and the Kalman filter is configured as a constant acceleration model, and a corresponding constant acceleration model state transition matrix is ​​configured, the time step in the state transition matrix is ​​equal to a preset value. Therefore, the state transition matrix of the Kalman filter... The expression for is shown in the following formula (17): Formula (17) in, Here is the state transition matrix of the Kalman filter; It is a 3-order identity matrix; This is the default value.

[0062] It should be noted that, since the Kalman filter in this embodiment is configured as a constant acceleration model, the state transition matrix is ​​derived based on the laws of classical uniform acceleration kinematics. State transition matrix Internally, by combining the system with a fixed time step, a mathematical mapping relationship is established between the position, velocity, and acceleration of the control node at two adjacent moments. Without introducing the visual observation data at the current moment, only using the complete motion state at the previous moment, the theoretical motion state of the control node after a certain time can be calculated through matrix multiplication operations according to the laws of physical motion, thus obtaining the prior state vector at the current moment. The specific process has been explained in S3 and will not be repeated here.

[0063] In a preferred embodiment, the end-to-end latency includes at least one of the following: tracking device exposure readout latency, data transmission bus latency, state estimation and motion prediction calculation latency, graphics rendering pipeline latency, and display output device scan output latency.

[0064] Specifically, in this implementation, the end-to-end latency includes at least one of the following: tracking device exposure readout latency, data transmission bus latency, state estimation and motion prediction calculation latency, graphics rendering pipeline latency, and display output device scan output latency; it may also be a combination of several of these. The tracking device exposure readout latency is the time consumed by the image sensor to complete exposure and read the raw observation data. The data transmission bus latency is the transmission time for data generated in each step to be transferred to the main processor via USB. The state estimation and motion prediction calculation latency is the time consumed by algorithms such as Kalman filtering, motion phase determination, and position extrapolation. The graphics rendering pipeline latency is the rendering time consumed by loading the virtual limb model, calculating the pose, and drawing the image. The display output device scan output latency is the output time consumed by sending the rendered frame data to the display controller and completing the screen pixel scan refresh.

[0065] In a preferred embodiment, the compensation time window is dynamically determined based on the end-to-end delay and the three-dimensional velocity and three-dimensional acceleration components at the current moment, including: Calculate the resultant velocity and resultant acceleration at the current moment based on the three-dimensional velocity components and three-dimensional acceleration components at the current moment; The motion phase of each control node at the current moment is identified based on the resultant velocity and resultant acceleration at the current moment. The motion phase includes the acceleration phase, the constant velocity phase, and the deceleration phase. When the motion phase is the acceleration phase, the length of the compensation time window is set to the sum of the end-to-end delay and the preset offset duration; When the motion phase is a constant velocity phase, the length of the compensation time window is set to an end-to-end delay; When the motion phase is the deceleration phase, the length of the compensation time window is set to the difference between the end-to-end delay and the preset offset duration.

[0066] Specifically, based on formulas (11), (12), and (13), it can be seen that predicting the future spatial position of the control node after the compensation window duration is essentially based on extrapolating the current velocity and acceleration to the future spatial position of the control node. However, human limbs are non-rigid and dynamically variable. Within the compensation window duration, the acceleration of the actual limb movement may continuously change, that is, it may remain unchanged, increase, or decrease. If the position after the fixed compensation time window is extrapolated using the constant acceleration at the current moment, it will lead to insufficient extrapolation displacement when the acceleration increases, causing the virtual control node to lag behind the real limb, and excessive extrapolation displacement when the acceleration decreases, causing the virtual control node to lead the real limb, ultimately resulting in virtual-real alignment deviation and image jitter. Therefore, different compensation time window lengths should be determined for different limb movement phases to compensate for the deficiency that fixed-duration extrapolation cannot adapt to acceleration changes. Theoretically, jerk is needed to determine whether acceleration is increasing or decreasing. However, jerk amplifies noise significantly, and introducing jerk would cause frequent phase jumps, which is counterproductive. Therefore, this embodiment uses the acceleration at the current moment for judgment. That is, when the acceleration at the current moment is close to 0, it is assumed that the acceleration within the compensation time window will not change, and the length of the compensation time window is directly taken as the measured end-to-end delay of the system. When the acceleration at the current moment is positive, it is predicted that the acceleration within the compensation time window will increase, so the length of the compensation time window needs to be increased. When the acceleration at the current moment is negative, it is predicted that the acceleration within the compensation time window will decrease, so the length of the compensation time window needs to be shortened.

[0067] Specifically, it is based on the optimal state vectors of each control node at the current moment. Extract the three-dimensional velocity components at the current moment. , , and three-dimensional acceleration components , , Then calculate the resultant velocity at the current moment according to the following formulas (18) and (19). and the resultant acceleration at the current moment : Formula (18) Formula (19) in, The resultant velocity at the current moment; The resultant acceleration at the current moment; The velocity component of the control node along the X-axis at the current moment; The velocity component of the control node along the Y-axis at the current moment; The velocity component of the control node along the Z-axis at the current moment; The acceleration component along the X-axis of the control node at the current moment; The Y-axis acceleration component of the control node at the current moment; This represents the acceleration component of the control node along the Z-axis at the current moment.

[0068] Next, based on the resultant velocity at the current moment... and the resultant acceleration at the current moment The system determines whether each control node is currently in an acceleration, constant speed, or deceleration phase. If the motion phase is determined to be acceleration, the compensation time window length is set to the sum of the end-to-end system delay and the preset offset duration to compensate for insufficient displacement extrapolated from the fixed delay. If the motion phase is determined to be constant speed, the compensation time window length is set to the end-to-end system delay. Under constant speed conditions, the limb movement rate is stable with no significant speed changes, and accurate position compensation can be achieved by matching only the inherent system delay, ensuring that the virtual limb smoothly follows the movement of the real limb. If the motion phase is determined to be deceleration, the compensation time window length is set to the difference between the end-to-end system delay and the preset offset duration to reduce overshoot errors caused by the model overestimating displacement during deceleration, preventing the virtual limb from leading the real limb, and effectively suppressing screen drift and misalignment during the deceleration start-stop phase. The preset offset duration is obtained through experimental calibration by combining the sampling time interval (preset duration) and the acceleration and deceleration motion variation amplitude of normal human limb activities.

[0069] In a preferred embodiment, identifying the motion phase of each control node at the current moment based on the resultant velocity and resultant acceleration at the current moment includes: When the absolute value of the resultant acceleration at the current moment is less than the preset static threshold, it is determined to be a uniform velocity phase; When the absolute value of the resultant acceleration at the current moment is greater than or equal to the preset rest threshold, and the direction of the resultant acceleration at the current moment is the same as the direction of the resultant velocity, it is determined to be the acceleration phase. When the absolute value of the resultant acceleration at the current moment is greater than or equal to the preset rest threshold, and the direction of the resultant acceleration at the current moment is opposite to the direction of the resultant velocity, it is determined to be in the deceleration phase.

[0070] Specifically, in this embodiment, the preset static threshold is a value close to 0. When the absolute value of the resultant acceleration is less than the preset static threshold, it indicates that the speed of the control node has almost no dynamic change, the overall limb movement is stable, and there is no obvious acceleration or deceleration trend, which is determined to be the uniform speed stage. When the absolute value of the resultant acceleration is greater than or equal to the preset static threshold, and the direction of the resultant acceleration is the same as that of the resultant velocity, the acceleration continues to have a boosting effect on the limb movement, and the movement rate of the control node continues to increase, which is consistent with the characteristics of limb acceleration movement, and is therefore determined to be the acceleration stage. When the absolute value of the resultant acceleration is greater than or equal to the preset static threshold, and the direction of the resultant acceleration is opposite to that of the resultant velocity, the acceleration cancels out the original movement speed of the limb, and the movement rate of the control node continues to decrease, which is consistent with the characteristics of limb deceleration and braking, and is therefore determined to be the deceleration stage.

[0071] It should be noted that this embodiment only ensures directional accuracy, i.e., the direction is extended during acceleration, shortened during deceleration, and remains unchanged during constant speed. Correct direction is a necessary condition for effective compensation; amplitude errors in the closed-loop system can be corrected through sampling and state estimation at the next time step. The core strategy of this embodiment is to first ensure correct direction, and then absorb amplitude deviations through closed-loop iteration, rather than calculating the optimal offset precisely in one go.

[0072] In a preferred embodiment, the method further includes the following steps before S5: The first-order low-pass smoothing coefficient is dynamically determined based on the resultant velocity of the three-dimensional velocity components output by the state estimator corresponding to each control node at the previous moment. The spatial positions of each control node after compensation for end-to-end delay are low-pass filtered using a first-order low-pass smoothing coefficient. The spatial positions of each control node after filtering are used to generate drive control commands.

[0073] Specifically, although the spatial positions of each control node obtained through motion model extrapolation compensate for the end-to-end delay, they still contain a small amount of high-frequency jitter noise. Directly using these positions for rendering might cause slight flickering or jittering of the virtual limbs. Therefore, it is necessary to perform smoothing filtering on the spatial positions of each control node to remove high-frequency noise and improve image stability. Specifically, for each control node, a first-order low-pass smoothing coefficient is determined based on the resultant velocity of the three-dimensional velocity components output by the state estimator at the previous moment. This first-order low-pass smoothing coefficient is then multiplied by the predicted spatial position of the control node after compensating for the end-to-end delay, outputting the smoothed spatial position. Virtual limb drive control commands are then generated based on these smoothed spatial positions, thus balancing the smoothness of the virtual limb image with the responsiveness of high-speed interactions. The dynamic determination of the first-order low-pass smoothing coefficient includes: Preset low-speed smoothing coefficient and high-speed smoothing coefficient; When the resultant velocity of the three-dimensional velocity components at the previous moment is less than or equal to the low-velocity resultant velocity threshold, the low-velocity smoothing coefficient is set to the first-order low-pass smoothing coefficient. When the resultant velocity of the three-dimensional velocity components at the previous moment is greater than or equal to the high-speed resultant velocity threshold, the high-speed smoothing coefficient is set to the first-order low-pass smoothing coefficient. When the resultant velocity of the three-dimensional velocity components at the previous moment is greater than the low-speed resultant velocity threshold and less than the high-speed resultant velocity threshold, the first-order low-pass smoothing coefficient is calculated based on the low-speed smoothing coefficient, the high-speed smoothing coefficient, and the difference coefficient.

[0074] Specifically, the resultant velocity of the three-dimensional velocity components passing through the previous moment. Combined speed threshold with low speed High-speed combined speed threshold The magnitude relationship is used to determine the process noise figure at the current moment by differentiating between three motion conditions. and observation noise figure Similarly, this step also relies on the resultant velocity of the three-dimensional velocity components from the previous moment. Combined speed threshold with low speed High-speed combined speed threshold The first-order low-pass smoothing coefficient at the current moment is determined by classifying the three motion conditions based on their magnitude relationship. : First, preset the low-speed smoothing coefficient. and high-speed smoothing coefficient Low-speed smoothing coefficient Greater than the high-speed smoothing coefficient ; Then, in condition one: when When the limbs are stationary or in a slow, micro-movement state, sampling noise can easily cause subtle jitters in the virtual limbs. To fully suppress high-frequency positional disturbances, the system directly... .

[0075] Operating Condition 2: When At this time, the limbs are in a state of rapid movement, requiring higher sensitivity to follow the posture. To avoid excessive smoothing that causes sluggishness in the movement, it is directly caused... .

[0076] Operating Condition 3: When At that time, the first-order low-pass smoothing coefficient is calculated using the following formula (20). : Formula (20) in, The first-order low-pass smoothing coefficient at the current moment; This is a low-speed smoothing coefficient; This is the high-speed smoothing coefficient; This is the difference coefficient.

[0077] It should be noted that, since stronger smoothing is needed to suppress jitter at low speeds and lower smoothing weights are needed at high speeds to ensure sensitive movements without trailing, the smoothing coefficient for low speeds in this embodiment is... Greater than the high-speed smoothing coefficient .

[0078] This application relies on a hardware clock source to achieve end-to-end timing uniformity, outputting the spatial positions of each control node at equal intervals without jitter, thereby eliminating calculation deviations caused by timing fluctuations and providing stable data support for adaptive filtering and delay prediction. By reusing the motion velocity of the previous moment to adaptively adjust the filter parameters, the filtering is enhanced to suppress jitter and drift in the output data when the limb movement is at low speed, and the filtering is relaxed to ensure no lag in the action response when the movement is at high speed. At the same time, the motion velocity of the previous moment is used as the basis for prediction, improving the accuracy of end-to-end delay compensation. Filter optimization and delay prediction share the same motion state data, which improves the accuracy of the filter output and provides more reliable state information for prediction. The prediction results inversely verify the adaptability of the filter, forming a two-way gain closed loop. Furthermore, the spatial positions of each control node after prediction compensation are directly used for virtual limb rendering, achieving synchronous alignment of virtual and real actions, eliminating the phenomenon of virtual limb lag and floating, realizing imperceptible human-human interaction delay, and effectively improving the smoothness and immersive experience of AR interaction.

[0079] The above text combined Figure 1 and Figure 2 This application provides a detailed description of the AR interaction closed-loop control method for enhancing user immersion, as illustrated in the embodiments below. Figure 3 This application provides an AR interactive closed-loop control system for enhancing user immersion, as illustrated in its embodiments.

[0080] like Figure 3 As shown in the figure, this is a schematic diagram of an AR interactive closed-loop control system 700 for enhancing user immersion provided in this application. The system includes: Hardware timing synchronization module 701: configured to synchronously trigger the tracking device 702, the state estimator and the display output device through the same hardware clock source, and establish a globally unified closed-loop control timing reference; Tracking device 702: configured to respond to the user's limb movements in the augmented reality space, and under the drive of a closed-loop control timing reference, acquire the spatial position of each control node of the limb at fixed time intervals; wherein the time interval between two adjacent acquisitions is a preset value and the timing jitter is less than a preset threshold. State estimation module 703: configured to set state estimators for each control node of the limb, and the state vector of each state estimator includes three-dimensional position, three-dimensional velocity components and three-dimensional acceleration components; using the spatial position of each control node of the limb acquired at the current time as the observation value to perform state estimation, and obtain the optimal state vector of each control node at the current time; Control module 704: It is used to take the three-dimensional velocity components estimated by the state estimator corresponding to each control node in the previous moment as a feedback signal to dynamically adjust the filtering parameters of the state estimator. On the other hand, it also serves as a precursor signal to drive the prediction and compensation of the spatial position of each control node after end-to-end delay based on the optimal state vector of each control node in the current moment. Based on the spatial position of each control node after end-to-end delay compensation, it generates the driving control command of the virtual limb, controls the virtual limb to output on the display output device, realizes the synchronization of the spatial position of the virtual limb and the real limb, and completes the closed-loop control of AR interaction.

[0081] Furthermore, the control module 704 is specifically used for: Extract the three-dimensional velocity components of the previous moment from the optimal state vector output by the state estimator of each control node at the previous moment. Dynamically determine the process noise figure and observation noise figure at the current moment based on the resultant velocity of the three-dimensional velocity components of the previous moment, and update the filtering parameters of the state estimator of each control node accordingly. The updated filtering parameters are used to filter the observations of each control node at the current time to obtain the optimal state vector of each control node at the current time. Extract the three-dimensional velocity and three-dimensional acceleration components of the current moment from the optimal state vector at the current moment as the prediction driving signal; estimate the end-to-end delay from state acquisition to virtual limb display output in real time; dynamically determine the compensation time window based on the end-to-end delay and the three-dimensional velocity and three-dimensional acceleration components of the current moment; Based on the optimal state vector at the current moment, the spatial positions of each control node after the compensation time window are extrapolated using a motion model.

[0082] Furthermore, the control module 704 is specifically used for: Preset low-speed resultant velocity threshold and corresponding low-speed process noise figure and low-speed observation noise figure; Preset a high-speed resultant velocity threshold and the corresponding high-speed process noise figure and high-speed observation noise figure; the high-speed resultant velocity threshold is greater than the low-speed resultant velocity threshold, the high-speed process noise figure is greater than the low-speed process noise figure, and the high-speed observation noise figure is less than the low-speed observation noise figure. The difference coefficient is calculated based on the difference between the resultant velocity at the previous moment and the low-speed resultant velocity threshold, and the difference between the high-speed resultant velocity threshold and the low-speed resultant velocity threshold. When the resultant velocity at the previous moment is less than or equal to the low-speed resultant velocity threshold, set the process noise figure at the current moment to be equal to the low-speed process noise figure and the observation noise figure to be equal to the low-speed observation noise figure. When the resultant velocity at the previous moment is greater than or equal to the high-speed resultant velocity threshold, set the process noise figure at the current moment to be equal to the high-speed process noise figure and the observation noise figure to be equal to the high-speed observation noise figure. When the resultant velocity at the previous moment is greater than the low-speed resultant velocity threshold and less than the high-speed resultant velocity threshold, the process noise coefficient at the current moment is calculated by linear interpolation based on the low-speed process noise coefficient, the high-speed process noise coefficient, and the difference coefficient; the observation noise coefficient at the current moment is calculated by linear interpolation based on the low-speed observation noise coefficient, the high-speed observation noise coefficient, and the difference coefficient.

[0083] Furthermore, the control module 704 is specifically used for: Calculate the resultant velocity and resultant acceleration at the current moment based on the three-dimensional velocity components and three-dimensional acceleration components at the current moment; The motion phase of each control node at the current moment is identified based on the resultant velocity and resultant acceleration at the current moment. The motion phase includes the acceleration phase, the constant velocity phase, and the deceleration phase. When the motion phase is the acceleration phase, the length of the compensation time window is set to the sum of the end-to-end delay and the preset offset duration; When the motion phase is a constant velocity phase, the length of the compensation time window is set to an end-to-end delay; When the motion phase is the deceleration phase, the length of the compensation time window is set to the difference between the end-to-end delay and the preset offset duration.

[0084] Furthermore, the control module 704 is specifically used for: When the absolute value of the resultant acceleration at the current moment is less than the preset static threshold, it is determined to be a uniform velocity phase; When the absolute value of the resultant acceleration at the current moment is greater than or equal to the preset rest threshold, and the direction of the resultant acceleration at the current moment is the same as the direction of the resultant velocity, it is determined to be the acceleration phase. When the absolute value of the resultant acceleration at the current moment is greater than or equal to the preset rest threshold, and the direction of the resultant acceleration at the current moment is opposite to the direction of the resultant velocity, it is determined to be in the deceleration phase.

[0085] Optionally, the control module 704 is also used for: The first-order low-pass smoothing coefficient is dynamically determined based on the resultant velocity of the three-dimensional velocity components output by the state estimator corresponding to each control node at the previous moment. The spatial positions of each control node after compensation for end-to-end delay are low-pass filtered using a first-order low-pass smoothing coefficient. The spatial positions of each control node after filtering are used to generate drive control commands.

[0086] The AR interactive closed-loop control system for enhancing user immersion provided in this application embodiment can correspond to executing the AR interactive closed-loop control method for enhancing user immersion described in this application embodiment, and the above-mentioned functions of each module and device of the system correspond to realizing... Figure 1 The corresponding process of the method shown will not be elaborated here for the sake of brevity.

[0087] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. The above are only preferred embodiments of this application. It should be noted that due to the limitations of textual expression, while there are objectively infinite specific structures, those skilled in the art can make several improvements, modifications, or changes without departing from the principles of this application, and can also combine the above technical features in an appropriate manner. These improvements, modifications, changes, or combinations, or the direct application of the inventive concept and technical solution to other situations without modification, should all be considered within the scope of protection of this application.

Claims

1. A closed-loop control method for AR interaction to enhance user immersion, characterized in that, include: S1. Synchronously trigger the tracking device, state estimator, and display output device through the same hardware clock source to establish a globally unified closed-loop control timing reference. S2. In response to the user's limb movements in the augmented reality space, under the drive of the closed-loop control timing reference, the tracking device acquires the spatial position of each control node of the limb at fixed time intervals; wherein, the time interval between two adjacent acquisitions is a preset value and the timing jitter is less than a preset threshold. S3. The spatial positions of each control node of the limb acquired at the current moment are used as observation values ​​and input to the state estimator corresponding to each control node for state estimation to obtain the optimal state vector of each control node at the current moment; the state estimator is set for each control node of the limb, and the state vector of each state estimator includes three-dimensional position, three-dimensional velocity components and three-dimensional acceleration components. S4. The three-dimensional velocity components estimated by the state estimator corresponding to each control node at the previous moment are used as feedback signals to dynamically adjust the filtering parameters of the state estimator. At the same time, they are used as precursor signals to drive the prediction and compensation of the spatial position of each control node after end-to-end delay based on the optimal state vector of each control node at the current moment. S4 specifically includes the following steps: S401. Extract the three-dimensional velocity components of the previous moment from the optimal state vector output by the state estimator corresponding to each control node at the previous moment and calculate the resultant velocity of the three-dimensional velocity components at the previous moment. Dynamically determine the process noise figure and observation noise figure at the current moment based on the resultant velocity of the three-dimensional velocity components at the previous moment, and update the filtering parameters of the state estimator corresponding to each control node accordingly. The process noise figure and observation noise figure at the current moment are dynamically determined based on the resultant velocity of the three-dimensional velocity components at the previous moment, including: A preset low-speed resultant velocity threshold and the low-speed process noise figure and low-speed observation noise figure corresponding to the low-speed resultant velocity threshold are defined. A preset high-speed resultant velocity threshold and a high-speed process noise figure and a high-speed observation noise figure corresponding to the high-speed resultant velocity threshold are defined; the high-speed resultant velocity threshold is greater than the low-speed resultant velocity threshold, the high-speed process noise figure is greater than the low-speed process noise figure, and the high-speed observation noise figure is less than the low-speed observation noise figure. The difference coefficient is calculated based on the difference between the resultant velocity and the low-speed resultant velocity threshold, and the difference between the high-speed resultant velocity threshold and the low-speed resultant velocity threshold at the previous moment. When the resultant velocity at the previous moment is less than or equal to the low-speed resultant velocity threshold, the process noise figure at the current moment is set to equal the low-speed process noise figure and the observation noise figure is set to equal the low-speed observation noise figure. When the resultant velocity at the previous moment is greater than or equal to the high-speed resultant velocity threshold, the process noise figure at the current moment is set to be equal to the high-speed process noise figure and the observation noise figure is set to be equal to the high-speed observation noise figure. When the resultant velocity at the previous moment is greater than the low-speed resultant velocity threshold and less than the high-speed resultant velocity threshold, the process noise coefficient at the current moment is calculated by linear interpolation based on the low-speed process noise coefficient, the high-speed process noise coefficient, and the difference coefficient; the observation noise coefficient at the current moment is calculated by linear interpolation based on the low-speed observation noise coefficient, the high-speed observation noise coefficient, and the difference coefficient. S402. Filter the observations of each control node at the current time using the updated filtering parameters to obtain the optimal state vector of each control node at the current time. S403. Extract the three-dimensional velocity component and three-dimensional acceleration component from the optimal state vector at the current moment as the prediction driving signal; estimate the end-to-end delay from state acquisition to virtual limb display output in real time; dynamically determine the compensation time window based on the end-to-end delay and the three-dimensional velocity component and three-dimensional acceleration component at the current moment. S404. Based on the optimal state vector at the current moment, extrapolate the spatial positions of each control node after the compensation time window using a motion model; S5. Based on the spatial position of each control node after the end-to-end delay is compensated, drive control commands for the virtual limbs are generated, and the virtual limbs are controlled to output on the display output device to realize the spatial position synchronization between the virtual limbs and the real limbs, thus completing the closed-loop control of AR interaction.

2. The method according to claim 1, characterized in that, The tracking device includes an image sensor and a supplementary lighting device, wherein the image sensor and the supplementary lighting device are triggered by a hardware clock source distributed via a fan-out buffer. Furthermore, the transmission clock of the USB controller used to transmit the data acquired by the image sensor to the state estimator is synchronized with the hardware clock source; The preset value is 8.33ms, corresponding to a sampling rate of 120fps; the preset threshold is 0.05ms.

3. The method according to any one of claims 1-2, characterized in that, The state estimator employs a Kalman filter, which is configured as a constant acceleration model and is configured with a state transition matrix corresponding to the constant acceleration model. The time step in the state transition matrix is ​​equal to the preset value.

4. The method according to any one of claims 1-2, characterized in that, The end-to-end latency includes at least one of the following: tracking device exposure readout latency, data transmission bus latency, state estimation and motion prediction calculation latency, graphics rendering pipeline latency, and display output device scan output latency.

5. The method according to claim 1, characterized in that, The compensation time window is dynamically determined based on the end-to-end delay and the current three-dimensional velocity and acceleration components, including: Calculate the resultant velocity and resultant acceleration at the current moment based on the three-dimensional velocity components and three-dimensional acceleration components at the current moment; The motion phase of each control node at the current moment is identified based on the resultant velocity and resultant acceleration at the current moment. The motion phase includes an acceleration phase, a constant velocity phase, and a deceleration phase. When the motion phase is the acceleration phase, the length of the compensation time window is set to the sum of the end-to-end delay and the preset offset duration; When the motion phase is a constant velocity phase, the length of the compensation time window is set to the end-to-end delay; When the motion phase is a deceleration phase, the length of the compensation time window is set to the difference between the end-to-end delay and the preset offset duration.

6. The method according to claim 5, characterized in that, Identifying the motion phase of each control node at the current moment based on the resultant velocity and resultant acceleration includes: When the absolute value of the resultant acceleration at the current moment is less than a preset static threshold, it is determined to be in a uniform velocity phase; When the absolute value of the resultant acceleration at the current moment is greater than or equal to a preset rest threshold, and the direction of the resultant acceleration at the current moment is the same as the direction of the resultant velocity, it is determined to be an acceleration phase. When the absolute value of the resultant acceleration at the current moment is greater than or equal to a preset rest threshold, and the direction of the resultant acceleration at the current moment is opposite to the direction of the resultant velocity, it is determined to be a deceleration phase.

7. The method according to any one of claims 1-2, characterized in that, Prior to S5, the method further includes: The first-order low-pass smoothing coefficient is dynamically determined based on the resultant velocity of the three-dimensional velocity components output by the state estimator corresponding to each control node at the previous moment. The spatial positions of each control node after compensation for end-to-end delay are low-pass filtered using the first-order low-pass smoothing coefficient. The spatial positions of each control node after filtering are used to generate the drive control command.

8. An AR interactive closed-loop control system for enhancing user immersion, characterized in that, For implementing the control method as described in any one of claims 1-2, comprising: Hardware timing synchronization module: configured to synchronously trigger the tracking device, state estimator and display output device through the same hardware clock source, and establish a globally unified closed-loop control timing reference; Tracking device: configured to respond to the user's limb movements in the augmented reality space, and under the drive of the closed-loop control timing reference, acquire the spatial position of each control node of the limb at fixed time intervals; wherein the time interval between two adjacent acquisitions is a preset value and the timing jitter is less than a preset threshold; State estimation module: Configured to set state estimators for each control node of the limb. The state vector of each state estimator includes three-dimensional position, three-dimensional velocity components and three-dimensional acceleration components. The spatial position of each control node of the limb acquired at the current moment is used as the observation value to perform state estimation and obtain the optimal state vector of each control node at the current moment. The control module is used to take the three-dimensional velocity components estimated by the state estimator corresponding to each control node at the previous moment. On the one hand, it serves as a feedback signal to dynamically adjust the filtering parameters of the state estimator. On the other hand, it also serves as a precursor signal to drive the prediction and compensation of the spatial position of each control node after end-to-end delay based on the optimal state vector of each control node at the current moment. Based on the spatial position of each control node after end-to-end delay compensation, it generates driving control commands for the virtual limbs and controls the output of the virtual limbs on the display output device to realize the synchronization of the spatial position of the virtual limbs and the real limbs, thus completing the closed-loop control of AR interaction.

Images