VR equipment observability enhancement method based on relative-absolute state estimation, medium and application

Abstract

The invention relates to a VR equipment observability enhancement method based on relative-absolute state estimation, a medium and application, and the method comprises the steps: defining a coupled head display state and a controller state, and building a continuous time dynamic model based on an IMU; constructing an overall measurement model based on an absolute measurement model of the head-mounted display and a relative measurement model between the head-mounted display and the controller; a relative-absolute state estimation problem is constructed and solved, and state estimation of the head-mounted display and the controller is obtained through joint optimization; based on relative-absolute state estimation, VR equipment observability enhancement is realized by utilizing kinematics constraints among equipment provided by a relative measurement model; when the method is applied to the condition that the head-mounted display is lack of visual absolute measurement, the accelerometer deviation is also observed through the state correlation with the controller; media are implemented based on the method. According to the method, the stability and availability of head-mounted display absolute pose estimation are maintained in a time period lacking visual absolute reference, and the long-term precision and stability of inertial navigation are fundamentally improved.

Description

Technical Field

[0001] This invention relates to the field of electrical digital data processing technology, and in particular to a method, medium, and application for enhancing the observability of VR devices based on relative-absolute state estimation. Background Technology

[0002] As virtual reality (VR) systems increasingly demand higher levels of immersion and interaction precision, head-mounted display (HUD) positioning technology based on visual inertial odometry (VIO) has become the mainstream solution for achieving high-precision spatial tracking. However, in practical applications, especially in complex, dynamic, or feature-poor environments, traditional single-device VIO systems have revealed several fundamental technical defects, severely limiting their robustness and reliability. These technical defects include:

[0003] (1) The absolute pose estimation of the system is highly dependent on the continuous observation of static visual features in the environment. In scenes with rapid movement, sudden changes in illumination, sparse texture or no texture at all (such as white walls), visual information will be severely degraded or lost, resulting in the lack of absolute reference and the estimation results becoming unstable or even invalid.

[0004] (2) The accelerometer and gyroscope bias of the IMU will change slowly over time. When the device is stationary or in uniform motion, these biases are coupled with gravity or motion state. The traditional VIO framework is difficult to separate them effectively, resulting in insufficient observability and thus affecting long-term accuracy.

[0005] (3) During the short-term failure of visual information, due to the integral effect of IMU noise and uncorrected bias, its position and attitude estimation will generate cumulative errors that increase over time, i.e., unbounded drift, which will disrupt the user's spatial perception consistency.

[0006] (4) Existing solutions usually treat the state estimation of the head-mounted display and the handheld controller as two independent or loosely coupled problems, failing to fully explore the inherent, rigid spatial kinematic constraints between them, and missing the opportunity for mutual constraints and error correction;

[0007] (5) Without external absolute reference information, the position and yaw angle of the system in the global coordinate system are unobservable. Summary of the Invention

[0008] This invention addresses the problems existing in the prior art and provides a method, medium, and application for enhancing the observability of VR devices based on relative-absolute state estimation. It effectively enhances the objectivity and system stability of absolute estimation of head-mounted displays and improves the positioning accuracy and robustness of VR systems in complex usage scenarios.

[0009] The technical solution adopted in this invention is a VR device observability enhancement method based on relative-absolute state estimation. The method defines coupled head-mounted display state and controller state, and constructs a continuous-time dynamic model based on IMU.

[0010] An overall measurement model is constructed based on the absolute measurement model of the head-mounted display and the relative measurement model between the head-mounted display and the controller.

[0011] Construct and solve the relative-absolute state estimation problem, and jointly optimize to obtain the state estimates of the head-mounted display and controller;

[0012] Based on the relative-absolute state estimation, the visual appeal of VR devices is enhanced by utilizing the inter-device kinematic constraints provided by the relative measurement model.

[0013] Preferably, the head-mounted display state and controller state include position, speed, Lie algebra for attitude representation, accelerometer deviation, and gyroscope deviation.

[0014] Preferably, the output of the absolute measurement model includes the position, velocity, and attitude of the head-mounted display in the world coordinate system;

[0015] The output of the relative measurement model includes the relative position of the controller in the head-mounted display coordinate system and the relative posture from the head-mounted display to the controller.

[0016] Preferably, a cost function is constructed and solved based on geometric consistency, statistical optimality, dynamic consistency, and measurement consistency to jointly optimize the relative-absolute state estimation problem. The cost function is associated with the absolute measurement residual term, the relative measurement residual term, the head-mounted display state transition constraint term, and the controller state transition constraint term.

[0017] Preferably, based on the relative-absolute state estimation, and utilizing the inter-device kinematic constraints provided by the relative measurement model, the accelerometer deviation of the head-mounted display is measured through its correlation with the state of the controller, even when the head-mounted display lacks visual absolute measurement.

[0018] The conditions to be met include the existence of effective relative measurement between the head-mounted display and the controller, the controller experiencing sufficient motion excitation, and the system not being in a purely static equilibrium state.

[0019] Preferably, a bidirectional error correction framework is defined, which uses accurate absolute state estimation of the head-mounted display to constrain and optimize the state estimation of the controller relative to the head-mounted display.

[0020] Preferably, during the degradation of absolute visual measurements, constraint propagation is performed between the head-mounted display and the controller state using relative measurement constraints.

[0021] Preferably, the relative-absolute state estimate remains unchanged under three-dimensional global translation and global rotation transformation about the gravity axis.

[0022] The application of the VR device observability enhancement method based on relative-absolute state estimation allows the accelerometer deviation to be observed through its correlation with the controller's state, even when the headset lacks absolute visual measurements.

[0023] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the VR device observability enhancement method based on relative-absolute state estimation.

[0024] This invention relates to a method, medium, and application for enhancing the observability of VR devices based on relative-absolute state estimation. It defines coupled head-mounted display (HMD) and controller states, and constructs a continuous-time dynamics model based on an IMU. Based on the absolute measurement model of the HMD and the relative measurement model between the HMD and the controller, an overall measurement model is constructed. A relative-absolute state estimation problem is constructed and solved, and joint optimization is performed to obtain state estimates for the HMD and controller. Based on the relative-absolute state estimation, the observability of the VR device is enhanced using the kinematic constraints between the devices provided by the relative measurement model. The method is applied even when the HMD lacks absolute visual measurements, allowing its accelerometer deviation to be observed through its correlation with the controller's state. The method is implemented using a medium.

[0025] The beneficial effects of this invention are as follows:

[0026] (1) By introducing stable relative measurement between the head display and the controller, even if the visual information of the head display is completely lost, the system can still indirectly transmit the motion information of the controller to the head display state through relative constraints, or use prior relative geometric relationships to constrain it, so as to maintain the stability and availability of the absolute pose estimation of the head display during the period when there is no absolute visual reference, and effectively overcome the absolute dependence on environmental features.

[0027] (2) Based on theoretical analysis and Lie derivative proof, when the controller has appropriate motion, the deviation of the accelerometer and gyroscope of the head display can become observable, thereby achieving more accurate and faster online deviation calibration and compensation, fundamentally improving the long-term accuracy of inertial navigation;

[0028] (3) Construct a two-way error suppression mechanism. This closed-loop mechanism of mutual correction couples and manages and suppresses the traditionally independent error sources, significantly reducing the overall drift rate of the entire system and improving the stability of long-term operation.

[0029] (4) Unlike the simple fusion of two independent estimation results, the error spaces of the two are associated by relative constraints during the estimation process, so that the reliable information of any device can directly optimize the state of the other device, thereby obtaining a statistically consistent and better collaborative estimation result. Attached Figure Description

[0030] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to embodiments, but the scope of protection of the present invention is not limited thereto.

[0032] This invention relates to a method for enhancing the observability of VR devices based on relative-absolute state estimation, the method comprising the following steps:

[0033] (1) Define the coupled head-mounted display state and controller state, and construct a continuous-time dynamic model based on the IMU;

[0034] (2) Based on the absolute measurement model of the head-mounted display and the relative measurement model between the head-mounted display and the controller, an overall measurement model is constructed;

[0035] (3) Construct and solve the relative-absolute state estimation problem, and jointly optimize to obtain the state estimates of the head-mounted display and the controller;

[0036] (4) Based on the relative-absolute state estimation, the device kinematic constraints provided by the relative measurement model are used to enhance the visual appeal of VR devices.

[0037] The method will be described below with reference to specific implementation methods.

[0038] (1) Define the coupled head-mounted display state and controller state, and construct a continuous-time dynamic model based on the IMU;

[0039] The head-mounted display status and controller status include position, velocity, Lie algebra for attitude representation, accelerometer bias, and gyroscope bias.

[0040] In this invention, the coupled head-mounted display-controller state space is defined as follows:

[0041]

[0042] Among them, the head-mounted display status is and controller status They are respectively

[0043]

[0044]

[0045] Where p is position and v is velocity, using Lie algebra. To represent the attitude and avoid over-parameterization issues, For accelerometer deviation, This refers to gyroscope bias.

[0046] Modeling a continuous-time dynamic system, the dynamic equations of the head-mounted display IMU are as follows:

[0047]

[0048]

[0049]

[0050]

[0051]

[0052] in, For IMU accelerometer data, This data comes from the IMU gyroscope, where g is the acceleration due to gravity. , These are the driving noises corresponding to the deviations.

[0053] The controller IMU dynamic equations are the same, where, Let be the left Jacobian matrix of the SO(3) group.

[0054] (2) Based on the absolute measurement model of the head-mounted display and the relative measurement model between the head-mounted display and the controller, an overall measurement model is constructed;

[0055] The output of the absolute measurement model includes the position, velocity, and attitude of the head-mounted display in the world coordinate system, satisfying the following:

[0056]

[0057]

[0058] Specifically, this includes absolute position, absolute velocity, and absolute attitude measurements:

[0059]

[0060] in, For the absolute measurement of the head-mounted display, For absolute measurement functions, For absolute measurement of noise, This represents the actual position of the headset in the world coordinate system. This represents the actual speed of the headset in the world coordinate system. This is the actual posture of the headset.

[0061] The output of the relative measurement model includes the relative position of the controller in the head-mounted display coordinate system and the relative posture from the head-mounted display to the controller.

[0062]

[0063]

[0064] Including relative position and attitude:

[0065]

[0066] in, For the relative observables between the head-mounted display and the controller, For relative measurement functions, For relative measurement noise, For head-mounted display rotation matrix, Let be the displacement vector in the world coordinate system. This refers to the relative rotation from the head-mounted display posture to the controller posture.

[0067] (3) Construct and solve the relative-absolute state estimation problem, and jointly optimize to obtain the state estimates of the head-mounted display and the controller;

[0068] Based on geometric consistency, statistical optimality, dynamic consistency, and measurement consistency, a cost function is constructed and solved to jointly optimize the relative-absolute state estimation problem. The cost function is associated with the absolute measurement residual term, the relative measurement residual term, the head-mounted display state transition constraint term, and the controller state transition constraint term.

[0069] Specifically, the geometric consistency condition requires that for all time instances... (Within the noise range), specifically in terms of position and attitude,

[0070]

[0071] in, This is an estimate of the state. The theoretical relative position is calculated based on the position estimates of the head-mounted display and controller. The theoretical relative attitude is calculated based on the attitude estimates of both.

[0072] Statistical optimality is achieved by constructing the complete cost function.

[0073]

[0074] in, For absolute measurement residuals, For relative measurement residuals, For dynamic constraints, and Let be the discrete-time state transition function, u be the IMU input, and Q be the covariance of the process noise;

[0075] Dynamic consistency is guaranteed by state transition constraints;

[0076] Measurement consistency is achieved by minimizing residuals;

[0077] These four factors together constitute the necessary and sufficient conditions for the optimal estimate.

[0078] (4) Based on the relative-absolute state estimation, the inter-device kinematic constraints provided by the relative measurement model are used to enhance the visual appeal of VR devices;

[0079] Unlike traditional VIO systems, which suffer from observability limitations due to coupling with the gravity vector, accelerometer bias (ba) and gyroscope bias are not observable under static conditions. Unobservable without rotational motion excitation, and unobservable when global position and yaw angle are not absolutely measured in the world coordinate system; this invention introduces a new observable mechanism through a relative-absolute framework—

[0080] Relative measurement constraints propagate, while relative observations create kinematic constraints between devices;

[0081] Cross-device status correlation: deviations become correlated through shared relative measurements;

[0082] Virtual absolute measurement, with relative constraints serving as pseudo-absolute references.

[0083] Based on the relative-absolute state estimation, and utilizing the inter-device kinematic constraints provided by the relative measurement model, the accelerometer deviation of the head-mounted display is determined under the condition that it lacks absolute visual measurements. The measured object is obtained by relating it to the state of the controller;

[0084] The conditions to be met include the existence of effective relative measurement between the head-mounted display and the controller, the controller experiencing sufficient motion excitation, and the system not being in a purely static equilibrium state.

[0085] By considering the second time derivative of the relative position measurement, d 2 d t 2 [ R h ⊤ ( p c - p h )] = d dt -[ oh m h - b g,h ] × R h ⊤ ( p c - p h )+ R h ⊤ ( v c - v h ) = R h ⊤ [ R c ( a m c - b a,c )- R h ( a m h - b a,h )]+ P

[0086] Rearranging and focusing on the deviation terms can yield the following results.

[0087] b a,h = d 2 d t 2 [ R h ⊤ ( p c - p h )]+ R h ⊤ R c b a,c - F

[0088] This indicates Through relative constraints and Coupling, if If observable (the controller has motion excitation), then Observable through coupling;

[0089] Furthermore, this invention uses nonlinear observability theory to analyze the system through Lie derivatives, resulting in the system's dynamic vector field being...

[0090]

[0091] The relative measurement function is,

[0092]

[0093] Calculate the second-order Lie derivative Revealing depends and The terms are used to confirm observability under appropriate incentive conditions.

[0094] A bidirectional error correction framework is defined, which uses accurate absolute state estimation of the head-mounted display to constrain and optimize the state estimation of the controller relative to the head-mounted display, satisfying the following conditions:

[0095]

[0096] This optimization anchors the relative attitude to an absolute reference, preventing cumulative drift.

[0097] During periods of visual absolute measurement degradation, constraint propagation is performed between the head-mounted display and controller states using relative measurement constraints.

[0098]

[0099] Maintain global consistency even when absolute measurements are unavailable.

[0100] Based on this, a joint optimization framework is obtained by fully optimizing the combination of absolute, relative, and dynamic terms.

[0101]

[0102] The relative-absolute state estimate remains invariant under three-dimensional global translation and global rotation transformation about the gravity axis.

[0103] The unobservable subspace of a relative-absolute constrained system is generated by the following symmetric transformation:

[0104]

[0105] The action applied to the state space is,

[0106]

[0107] in, This refers to rotation about the axis of gravity.

[0108] Verify that the observation function is in the symmetric group Remains unchanged under the influence of the action, for relative position observation

[0109] p c|h (g⋅ x )=[ R z (ψ) R h ] ⊤ (( p c + t )-( p h + t ))= R h ⊤ R z (ψ ) ⊤ R z (ψ)( p c - p h )= R h ⊤ ( p c - p h )= p c|h ( x )

[0110] For relative attitude observation

[0111] i c|h (g⋅ x ) = log ( [ R z (ψ) R h ] ⊤ R z (ψ) R c ) = log ( R h ⊤ R z (ψ ) ⊤ R z (ψ) R c ) = log ( R h ⊤ R c )= i c|h ( x )

[0112] IMU measurements remain unchanged under symmetric transformations;

[0113] therefore, It is indeed the symmetric group of the system.

[0114] Velocity-state observability theorem: In a relative-absolute constraint system, even visual observation... Head-mounted display speed and controller speed It is still locally observable; the proof steps are as follows:

[0115] Construct observable Lie algebras, let Calculate the first-order Lie derivative:

[0116] L f v ϕ 1 = ∂ ϕ 1 ∂ x f v ( x ) = R h ⊤ ( v c - v h )- R h ⊤ [ oh h ] × ( p c - p h )

[0117] Define a new observation function:

[0118] ϕ 3 ( x )= L f v ϕ 1 + R h ⊤ [ oh h ] × ( p c - p h )= R h ⊤ ( v c - v h )

[0119] Jacobian matrix relating to velocity state:

[0120]

[0121] Both Jacobian matrices are full rank. The matrix proves the observability of the velocity state.

[0122] The present invention also relates to an application of the aforementioned VR device observability enhancement method based on relative-absolute state estimation, in which the accelerometer deviation of the head-mounted display can also be observed through its correlation with the state of the controller, even when the head-mounted display lacks absolute visual measurements.

[0123] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned method for enhancing the observability of VR devices based on relative-absolute state estimation.

[0124] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure one One or more processes and / or boxes Figure one A device that provides the functions specified in one or more boxes.

[0126] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure one One or more processes and / or boxes Figure oneThe function specified in one or more boxes.

[0127] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure one One or more processes and / or boxes Figure one The steps of the function specified in one or more boxes.

[0128] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0129] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for enhancing the observability of VR devices based on relative-absolute state estimation, characterized in that: The method defines the coupled head-mounted display state and controller state, and constructs a continuous-time dynamic model based on the IMU; An overall measurement model is constructed based on the absolute measurement model of the head-mounted display and the relative measurement model between the head-mounted display and the controller. Construct and solve the relative-absolute state estimation problem, and jointly optimize to obtain the state estimates of the head-mounted display and controller; Based on the relative-absolute state estimation, the visual appeal of VR devices is enhanced by utilizing the inter-device kinematic constraints provided by the relative measurement model.

2. The VR device observability enhancement method based on relative-absolute state estimation according to claim 1, characterized in that: The head-mounted display status and controller status include position, velocity, Lie algebra for attitude representation, accelerometer bias, and gyroscope bias.

3. The VR device observability enhancement method based on relative-absolute state estimation according to claim 1, characterized in that: The output of the absolute measurement model includes the position, velocity, and attitude of the head-mounted display in the world coordinate system; The output of the relative measurement model includes the relative position of the controller in the head-mounted display coordinate system and the relative posture from the head-mounted display to the controller.

4. The VR device observability enhancement method based on relative-absolute state estimation according to claim 1, characterized in that: Based on geometric consistency, statistical optimality, dynamic consistency, and measurement consistency, a cost function is constructed and solved to jointly optimize the relative-absolute state estimation problem. The cost function is associated with the absolute measurement residual term, the relative measurement residual term, the head-mounted display state transition constraint term, and the controller state transition constraint term.

5. The VR device observability enhancement method based on relative-absolute state estimation according to claim 1, characterized in that: Based on the relative-absolute state estimation, and utilizing the inter-device kinematic constraints provided by the relative measurement model, the accelerometer deviation of the head-mounted display is measured by its correlation with the state of the controller, even when the head-mounted display lacks visual absolute measurement. The conditions to be met include the existence of effective relative measurement between the head-mounted display and the controller, the controller experiencing sufficient motion excitation, and the system not being in a purely static equilibrium state.

6. The VR device observability enhancement method based on relative-absolute state estimation according to claim 1, characterized in that: A bidirectional error correction framework is defined, which uses accurate absolute state estimation of the head-mounted display to constrain and optimize the state estimation of the controller relative to the head-mounted display.

7. The VR device observability enhancement method based on relative-absolute state estimation according to claim 6, characterized in that: During the degradation of absolute visual measurements, constraint propagation is performed between the head-mounted display and controller states using relative measurement constraints.

8. The VR device observability enhancement method based on relative-absolute state estimation according to claim 1, characterized in that: The relative-absolute state estimate remains invariant under three-dimensional global translation and global rotation transformation about the gravity axis.

9. An application of the VR device observability enhancement method based on relative-absolute state estimation as described in any one of claims 1 to 8, characterized in that: Even when the head-mounted display lacks absolute visual measurements, its accelerometer bias can still be observed through its correlation with the controller's state.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the VR device observability enhancement method based on relative-absolute state estimation as described in any one of claims 1 to 8.

Images