A bidirectional coupled visual-inertial joint pose estimation method, device and medium

Abstract

The application discloses a bidirectional coupled visual-inertial combined pose estimation method, device and medium, and relates to the positioning and tracking technology field of virtual reality and augmented reality. The method comprises the following steps: collecting inertial measurement data through the inertial measurement units of a head-mounted device and a handle device, and collecting images through the head-mounted device; collecting first state parameters of the head-mounted device, second state parameters of the handle device relative to the head-mounted device, and gravity vector parameters; constructing a double pre-integration factor for constraining the relative motion state between the head-mounted device and the handle device based on the inertial measurement data; constructing a visual re-projection factor and a gravity constraint factor according to visual features; and performing joint optimization on a joint state vector by using the double pre-integration factor, the visual re-projection factor and the gravity constraint factor, so as to obtain the pose information of the head-mounted device and the handle device. The application realizes bidirectional transmission of the state between the head-mounted device and the handle device, joint optimization and dynamic interference suppression through the above method.

Description

Technical Field

[0001] This application relates to the field of positioning and tracking technology in virtual reality and augmented reality, and in particular to a bidirectional coupled visual-inertial joint pose estimation method, device and medium. Background Technology

[0002] In virtual reality and augmented reality systems, high-precision, low-latency six-DOF pose tracking is crucial for ensuring an immersive experience. Currently, consumer-grade VR / AR devices generally employ inside-out tracking solutions, which rely on the headset's integrated cameras and inertial measurement units to achieve self-localization. The headset's cameras then optically track feature points or emitters on the controller's surface to estimate the controller's relative pose. Although commercially available, this approach typically treats the pose estimation of the headset and controllers as a unidirectional or independent process.

[0003] Specifically, existing solutions mostly adopt a head-mounted display (HUD)-based architecture with controllers as secondary devices. The HUD estimates its absolute pose in the world coordinate system using visual inertial odometry or SLAM (Simultaneous Localization and Mapping) technology, while the controllers rely on the HUD's pose and their own sensor data to calculate their relative pose. This architecture results in the state estimation of the HUD and controllers being decoupled or only unidirectionally constrained. When one side experiences estimation errors or drift, it cannot be effectively corrected by the other. At the same time, the moving controllers, as dynamic objects appearing in the HUD's field of view, interfere with the HUD's visual SLAM in extracting and tracking static environmental features, further reducing the overall accuracy and robustness of the system in complex motion and interaction scenarios.

[0004] Based on the above analysis, the problems and shortcomings of the existing technology are as follows: In existing technologies, the head-mounted display and controller pose estimation are decoupled, errors cannot be corrected bidirectionally, and the motion controller interferes with the head-mounted display's visual SLAM, resulting in decreased tracking accuracy and insufficient system robustness. Summary of the Invention

[0005] This application provides a bidirectional coupled visual-inertial joint pose estimation method, device, and medium, which can solve the problems of decoupling between head-mounted display and controller pose estimation, inability to bidirectionally correct errors, and reduced tracking accuracy and insufficient system robustness caused by motion controller interference with head-mounted display visual SLAM.

[0006] In a first aspect, embodiments of this application provide a bidirectional coupled visual-inertial joint pose estimation method, characterized in that the method includes: acquiring inertial measurement data through the inertial measurement units of a head-mounted display (HMD) device and a controller device, and acquiring images through the HMD device; acquiring first state parameters of the HMD device, second state parameters of the controller device relative to the HMD device, and gravity vector parameters to construct a joint state vector; constructing a dual pre-integration factor based on the inertial measurement data to constrain the relative motion state between the HMD device and the controller device; extracting visual features from the images, constructing a visual reprojection factor based on the visual features, and constructing a gravity constraint factor based on the gravity vector parameters; constructing a joint optimization model using the dual pre-integration factor, the visual reprojection factor, and the gravity constraint factor to jointly optimize the joint state vector and obtain an optimized state estimate; and obtaining the pose information of the HMD device and the controller device based on the optimized state estimate.

[0007] In one implementation of this application, a dual pre-integration factor is constructed based on inertial measurement data to constrain the relative motion state between the head-mounted display device and the controller device. Specifically, this includes: pre-integrating the inertial measurement data of the head-mounted display device and the controller device respectively to obtain the head-mounted display motion increment and the controller motion increment; and constructing the dual pre-integration factor based on the head-mounted display motion increment and the controller motion increment.

[0008] In one implementation of this application, a joint state vector is constructed by collecting first state parameters of the head-mounted display device, second state parameters of the handle device relative to the head-mounted display device, and gravity vector parameters. Specifically, the first state parameters include the position and velocity of the head-mounted display device in the world coordinate system and the offset of the head-mounted display inertial measurement unit; the second state parameters include the rotation angle, position, and velocity of the handle device relative to the coordinate system of the head-mounted display device, and the offset of the handle inertial measurement unit; the representation of the gravity vector parameters in the world coordinate system in the coordinate system of the head-mounted display device is defined; and the first state parameters, the second state parameters, and the representation are combined to obtain the joint state vector.

[0009] In one implementation of this application, visual features are extracted from the image, and a visual reprojection factor is constructed based on the visual features. Specifically, this includes: predicting the projection area of ​​the controller device in the image based on the currently estimated pose of the controller device relative to the head-mounted display device, and generating a dynamic object mask; extracting static environment feature points outside the area covered by the dynamic object mask; and constructing the visual reprojection factor based on the static environment feature points.

[0010] In one implementation of this application, based on the currently estimated pose of the handle relative to the head-mounted display, the projection area of ​​the handle in the head-mounted display camera image is predicted and a dynamic object mask is generated. Specifically, this includes: obtaining the coordinates of multiple preset 3D key points in the 3D model of the handle device; projecting the 3D key points onto the image based on the pose estimation value, the intrinsic parameters of the head-mounted display device, and the extrinsic parameters of the head-mounted display device to obtain multiple 2D projection points; and connecting the multiple 2D projection points to obtain a dynamic object mask surrounding the projection area of ​​the handle device.

[0011] In one implementation of this application, a joint optimization model is constructed using a double pre-integration factor, a visual reprojection factor, and a gravity constraint factor. Specifically, this includes: explicitly introducing a gravity vector as a state variable to be optimized into the joint optimization model; obtaining explicit linear constraint equations for the gravity vector parameters based on inertial measurement data from the head-mounted display and the handheld device; and adding the linear constraint equations to the joint optimization model in the form of gravity constraint factors.

[0012] In one implementation of this application, a joint optimization model is constructed using a double pre-integration factor, a visual reprojection factor, and a gravity constraint factor to jointly optimize the joint state vector and obtain the optimized state estimate. Specifically, this includes: using a sliding window to manage the joint state vector, where the sliding window includes the joint state vectors corresponding to consecutive keyframes; when a new keyframe is added, using the joint optimization model and a nonlinear optimization algorithm to jointly optimize all states within the window; after optimization, old states outside the window are transformed into prior information through marginalization, and observability is monitored during marginalization, with covariance inflation or retention performed on old states in weakly observable directions.

[0013] In one implementation of this application, the method further includes: feeding back the bias and gravity vector parameters of the head-mounted display inertial measurement unit obtained after joint optimization; using the fed-back bias of the inertial measurement unit to perform online correction of the pre-integration process of the inertial measurement data; and using the fed-back gravity vector parameters to correct the extraction of visual features and the gravity compensation term.

[0014] Secondly, embodiments of this application also provide a bidirectional coupled visual-inertial joint pose estimation device, the device including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: perform any step of a bidirectional coupled visual-inertial joint pose estimation method.

[0015] Thirdly, embodiments of this application also provide a bidirectional coupled visual-inertial joint pose estimation non-volatile computer storage medium storing computer-executable instructions, which are configured to execute any one of the steps of a bidirectional coupled visual-inertial joint pose estimation method.

[0016] This application provides a bidirectional coupled visual-inertial joint pose estimation method, device, and medium. By constructing a unified joint state vector and dual pre-integration factors, the states of the head-mounted display and the controller can mutually constrain and correct each other, effectively suppressing error accumulation and improving overall tracking accuracy. By actively removing the projection area of ​​the controller in the image through a dynamic sensing visual front end, the interference of dynamic objects on the visual SLAM of the head-mounted display is eliminated, enhancing the robustness of the system in occlusion, weak texture, and fast-moving scenarios. Joint optimization of gravity direction avoids virtual world tilting and enhances immersion. Sliding window optimization and edge-mapping strategies ensure real-time operation of the system under high-frequency IMU and image data. Through a gravity joint estimation and feedback correction mechanism that integrates dual-device information, the stability of the gravity direction in the virtual world and the consistency of the system state are ensured, thereby significantly improving the user's immersive experience. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a bidirectional coupled visual-inertial joint pose estimation method provided in this application embodiment; Figure 2 A schematic diagram of the overall architecture of a bidirectional coupled visual-inertial joint pose estimation method provided in an embodiment of this application; Figure 3 A schematic diagram of the joint factors for a bidirectional coupled visual-inertial joint pose estimation method provided in an embodiment of this application; Figure 4 A flowchart of the dynamic perception visual front-end processing of a bidirectional coupled visual-inertial joint pose estimation method provided in an embodiment of this application; Figure 5 This is a schematic diagram of the internal structure of a bidirectional coupled visual-inertial joint pose estimation device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] This application provides a bidirectional coupled visual-inertial joint pose estimation method, device, and medium, which solves the problems of decoupling between head-mounted display and controller pose estimation, inability to bidirectionally correct errors, and decreased tracking accuracy and insufficient system robustness caused by interference from the motion controller with head-mounted display visual SLAM.

[0020] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0021] Figure 1 This is a flowchart illustrating a bidirectional coupled visual-inertial joint pose estimation method provided in an embodiment of this application. Figure 1 As shown in the figure, the bidirectional coupled visual-inertial joint pose estimation method provided in this application embodiment specifically includes the following steps: Step 10: Acquire inertial measurement data and images through the inertial measurement units of the head-mounted display and the handheld device.

[0022] First, it is understandable that, such as Figure 2 As shown, this application mainly includes four core modules: a bidirectional coupled state estimation module, a dynamic perception visual front-end module, a joint factor graph optimization module, and a high-precision gravity estimation module.

[0023] Specifically, the inputs are the raw data of angular velocity and linear acceleration of the inertial measurement units of the head-mounted display and the controller, as well as the image sequence captured by the head-mounted display camera; the outputs are the optimized pose, velocity, and IMU (Inertial Measurement Unit) offset of the head-mounted display in the world coordinate system, the pose and velocity of the controller relative to the head-mounted display, and the accurate estimate of the global gravity vector in the head-mounted display coordinate system.

[0024] In this step, the inertial measurement unit built into the head-mounted display continuously collects its own angular velocity and linear acceleration data to form head-mounted display inertial measurement data; the IMU built into the controller also continuously collects its own angular velocity and linear acceleration data to form controller inertial measurement data; one or more cameras mounted on the head-mounted display simultaneously collect image sequences of the external environment.

[0025] Step 20: Collect the first state parameters of the head-mounted display device, the second state parameters of the controller device relative to the head-mounted display device, and the gravity vector parameters to construct a joint state vector; As an optional embodiment, the first state parameters of the head-mounted display device, the second state parameters of the handle device relative to the head-mounted display device, and the gravity vector parameters are collected to construct a joint state vector. Specifically, this may include: Step 201: The first state parameters include the position and velocity of the head-mounted display device in the world coordinate system and the offset of the head-mounted display inertial measurement unit.

[0026] In this step, the position refers to the transformation of the head-mounted display device from a fixed world coordinate system to its own body coordinate system. This transformation includes position translation and orientation rotation in three-dimensional space. The velocity refers to the three-dimensional motion velocity of the head-mounted display device in the world coordinate system. The IMU bias refers to the inherent measurement errors of the gyroscope and accelerometer in the inertial measurement unit of the head-mounted display device. These biases change slowly over time and need to be estimated and compensated in real time.

[0027] Step 202: The second state parameters include the rotation angle, position, and speed of the handle device relative to the head-mounted display device coordinate system, as well as the offset of the handle inertial measurement unit.

[0028] In this step, relative rotation and relative position refer to the rotation matrix and translation vector of the coordinate system of the controller device relative to the coordinate system of the head-mounted display device, which directly express the orientation and distance of the controller relative to the head-mounted display device; relative velocity refers to the velocity of the controller device relative to the head-mounted display device, which is defined in the coordinate system of the head-mounted display device and has eliminated the influence of gravitational acceleration, purely reflecting relative motion; controller IMU bias refers to the measurement bias of the gyroscope and accelerometer in the inertial measurement unit of the controller device, which also needs to be estimated and corrected.

[0029] Step 203: Define the representation of the gravity vector parameters in the world coordinate system in the head-mounted display device coordinate system.

[0030] In this step, the known gravitational acceleration vector in the world coordinate system is transformed and represented in the head-mounted display's own coordinate system at the current moment, reflecting the pitch and roll attitude of the head-mounted display, which is a common benchmark connecting inertial measurement and visual observation.

[0031] Step 204: Combine the first state parameter, the second state parameter, and the representation to obtain the joint state vector.

[0032] In this step, the first state parameters of the head-mounted display, the second state parameters of the controllers, and the gravity vector representation defined in the previous steps are combined in a predetermined order into a comprehensive, high-dimensional joint state vector. This vector corresponds to an instance at each processing moment or keyframe and will be estimated and optimized as a whole in subsequent steps. Through this unified mathematical expression, the absolute motion of the head-mounted display, the relative motion of the controllers, and the global gravity direction are closely linked within the same optimization framework.

[0033] Step 30: Based on inertial measurement data, construct a double pre-integral factor to constrain the relative motion state between the head-mounted display and the handheld device.

[0034] It is understood that traditional methods typically perform independent pre-integration on the IMU data of the head-mounted display and the controller, and constrain their own motions during optimization, lacking direct kinematic constraints based on the raw measurement data between the two.

[0035] In this step, by co-processing the IMU data of both devices, a mathematical model is constructed that directly and tightly constrains the relative motion changes between the headset and the controller. A unified joint state vector is built, and bidirectional constraints on the states of the headset and the controller are realized within the optimization framework.

[0036] As an optional embodiment, based on inertial measurement data, a dual pre-integration factor is constructed to constrain the relative motion state between the head-mounted display device and the handheld device. Specifically, it may include: Step 301: pre-integrating the inertial measurement data of the head-mounted display device and the handheld device respectively to obtain the motion increment of the head-mounted display device and the motion increment of the handheld device.

[0037] In this step, to achieve efficient optimization and coupling of the motions of both, this application adopts a two-layer pre-integration strategy: the first layer is the head-mounted display layer pre-integration, which involves optimizing the head-mounted display IMU in adjacent keyframes. and Pre-integrate the data between them to obtain incremental observations independent of the initial state of the head-mounted display. The specific formula is as follows:

[0038] These represent the changes in rotation, velocity, and position, respectively. The values ​​are the raw IMU measurements, where i represents the time of the i-th keyframe. Let k be the sampling interval (from time i to j), and k be the index of the sampling time of the IMU within the time interval from i to j. This represents the exponential mapping from Lie algebras to Lie groups. This is for the IMU's gyroscope bias and accelerometer bias.

[0039] In other words, for a head-mounted display device, all IMU measurements between two adjacent keyframe moments, such as angular velocity and acceleration, are integrated. By subtracting the currently estimated bias from the measured values ​​and then integrating, a set of relative motion increments that are independent of the absolute state of the head-mounted display at the initial moment are obtained. These increments include rotation increments, velocity increments, and position increments. They depend only on the IMU measurement sequence and IMU bias between the two keyframes, thereby compressing the high-frequency IMU data into constraints between several keyframes.

[0040] The second-layer pre-integration and double pre-integration construction involves pre-integrating the handheld IMU data. A dual pre-integral factor is constructed, which depends on the pre-integral values ​​of both the head-mounted display and the controller, directly constraining the relative state changes of both; this factor is constructed from time [time value missing]. arrive The relative motion model:

[0041] Similarly, for handheld devices, the same pre-integration operation is performed on the IMU measurements within the same time interval to obtain independent handheld motion increments, including the handheld's own rotation increment, velocity increment, and position increment.

[0042] Step 302: Construct a dual pre-integration factor based on the motion increments of the head-mounted display and the motion increments of the controller.

[0043] In this step, the double pre-integration residual can be defined. This factor is used to penalize the deviation between the predicted relative state and the estimated value during optimization; it is connected to the head-mounted display state. relative to the handle This achieves bidirectional coupling of motion constraints.

[0044] In other words, instead of simply treating the two pre-integral increments as two separate constraints, we combine them to construct a completely new double pre-integral factor.

[0045] Specifically, a mathematical relationship was established regarding how the state, rotation, position, and speed of the controller relative to the head-mounted display should change from the previous keyframe to the current keyframe. The rotation of the controller relative to the head-mounted display at the current moment should be determined by the rotation increment of the head-mounted display itself, the relative rotation of the controller at the previous moment, and the rotation increment of the controller itself. The position and speed of the controller relative to the head-mounted display at the current moment are derived from the motion increment of the head-mounted display, the motion increment of the controller, and the relative state of the controller at the previous moment through specific kinematic formulas.

[0046] Then, a predicted relative state of the handle at the current moment can be calculated. During the joint optimization process, this predicted value is compared with the corresponding estimated value in the state vector to be optimized. The difference constitutes the double pre-integration residual. This residual is added to the overall optimization objective function as an independent constraint term (double pre-integration factor). Its function is to force the optimized head-mounted display state and the relative state of the handle to simultaneously satisfy the relative motion law derived from the original IMU data of both.

[0047] In this way, by introducing a dual pre-integration factor, the IMU data from the headset and controllers are no longer isolated, but are tightly linked by a unified kinematic model. During optimization, adjustments to the headset state directly affect the constraints on the controller's relative state through this factor, and vice versa, thus mathematically achieving bidirectional coupling and mutual correction of the motion estimates of the two. This improves the accuracy and consistency of the system's relative pose estimation under rapid movement or sensor noise interference.

[0048] Step 40: Extract visual features from the image, construct a visual reprojection factor based on the visual features, and construct a gravity constraint factor based on the gravity vector parameters.

[0049] In this step, the head-mounted display image first passes through the dynamic perception visual front-end module, such as... Figure 3 As shown, after removing the dynamic features caused by the handles, static environmental features are extracted, aiming to eliminate the interference of moving handles on head-mounted display visual SLAM. The input is the raw image captured by the head-mounted display and the current relative pose estimate of the handles from the state estimation module. In other words, by actively eliminating dynamic interference through a dynamic sensing visual front end and constructing gravity constraint factors by fusing information from both devices, the accuracy of gravity estimation is improved through joint optimization. Furthermore, during periods when the user is stationary or moving slowly, the system, through numerous such constraints, can accurately observe and optimize the direction of gravity, resulting in an optimized gravity estimate. It will provide real-time feedback to the inertial navigation calculation module of the head-mounted display to ensure the stability of the virtual horizon and to correct the gravity compensation term in the subsequent pre-integration, forming an enhanced loop from relative motion information to absolute gravity perception.

[0050] As an optional embodiment, visual features are extracted from the image, and a visual reprojection factor is constructed based on the visual features. Specifically, it may include: Step 401: Based on the currently estimated pose of the handle device relative to the head-mounted display device, predict the projection area of ​​the handle device in the image and generate a dynamic object mask. As an optional embodiment, based on the currently estimated pose of the handle relative to the head-mounted display, the projection area of ​​the handle in the head-mounted display camera image is predicted and a dynamic object mask is generated. Specifically, this may include: Step 4011: Obtaining the coordinates of multiple preset 3D key points in the 3D model of the handle device; Step 4012: Projecting the 3D key points onto the image based on the pose estimation value, the intrinsic parameters of the head-mounted display device, and the extrinsic parameters of the head-mounted display device to obtain multiple 2D projection points; Step 4013: Connecting the multiple 2D projection points to obtain a dynamic object mask surrounding the projection area of ​​the handle device.

[0051] In this step, the known 3D model of the handle can be simplified to a set of 3D key points in the body coordinate system. Combining the estimated relative pose of the handles with the known intrinsic parameters of the head-mounted display camera and camera-headset IMU external parameters The projection area of ​​the handle on the image can be predicted:

[0052] in For perspective projection functions, , For the reason The transformation matrix is ​​constructed by connecting all the predicted two-dimensional points. This forms one or more convex hull regions, which are the dynamic object masks.

[0053] Step 402: Extract static environment feature points outside the area covered by the dynamic object mask.

[0054] In this step, after obtaining the dynamic object mask, no image feature points such as FAST corner points or ORB features are extracted or actively discarded within the image area covered by the mask; feature point detection and descriptor calculation are only performed in the image area outside the mask, that is, the static background area; these static features are matched with existing static map points for subsequent head-mounted display pose estimation. This method effectively cuts off the contamination path of hand controller movement on head-mounted display visual observation, significantly improving the robustness and accuracy of head-mounted display visual SLAM in intense interactive scenarios.

[0055] Step 403: Visual reprojection factor is constructed based on static environmental feature points.

[0056] In this step, for each successfully tracked static environmental feature point, given its 3D map coordinates in the world coordinate system and its observed 2D pixel coordinates in the current image, the 3D map point can be reprojected onto the current image plane based on the currently estimated head-mounted display pose and camera parameters, resulting in a theoretical 2D projected coordinate. The visual reprojection factor is the difference between this theoretical projected coordinate and the actual observed coordinate. In subsequent joint optimization, minimizing all such visual reprojection errors directly optimizes the head-mounted display pose estimation and is more robust due to its pure static features.

[0057] Step 50: Construct a joint optimization model using the double pre-integration factor, visual reprojection factor, and gravity constraint factor to jointly optimize the joint state vector and obtain the optimized state estimate. In this step, the joint state estimation module initializes a joint state vector that integrates the absolute state of the headset, the relative state of the controller, and the sensor bias, such as... Figure 4 As shown, the joint factor graph optimization module receives pre-integration factors from the dual pre-integrators, static visual reprojection factors from the visual front end, and gravity constraint factors, and performs nonlinear optimization within a sliding window. The optimization results output the final state estimate and, on the other hand, feed back the updated head-mounted display IMU bias and gravity vector to the head-mounted display pre-integrator and the visual front end.

[0058] As an optional embodiment, a joint optimization model is constructed using a double pre-integration factor, a visual reprojection factor, and a gravity constraint factor. Specifically, this may include: Step 501: Explicitly introducing a gravity vector as a state variable to be optimized in the joint optimization model.

[0059] In this step, the gravity vector Constant in the navigation coordinate system Let g be the local gravitational acceleration scalar, in the head-mounted display coordinate system. The observed values ​​are as follows By treating the direction of gravity as the optimization variable, the state can be parameterized as a two-dimensional perturbation, since the magnitude of gravity is known and only its direction is unknown. For example, a perturbation applied to a tangential plane can be used. .

[0060] In other words, the representation of the gravity vector in the head-mounted display coordinate system is explicitly included as one of the state variables to be optimized in the joint state vector, which breaks the traditional limitation that gravity is only an implicit parameter or estimated by a single sensor.

[0061] Step 502: Based on the inertial measurement data of the head-mounted display and the controller, obtain the explicit linear constraint equations for the gravity vector parameters.

[0062] In this step, one aspect comes from the head-mounted display IMU accelerometer model. The acceleration measured by the head-mounted display IMU includes gravity, motion acceleration, and bias. In optimization, the accelerometer measurement residual implicitly contains constraints on gravity. The other aspect comes from the relative motion constraints of the handles. According to relative kinematics, the acceleration of the handles relative to the head-mounted display is related to the IMU measurements of both and gravity.

[0063] In the double pre-integral model, by introducing a gravitational state, gravity can be explicitly separated from the relative motion constraints, constructing a direct gravity constraint factor. Specifically, by deriving the relative motion equations of the handle, a formula for... The linear constraint term is in the form of:

[0064] in, and The residual, determined by the pre-integral values ​​of the head-mounted display and the controllers and their relative pose, is incorporated into the joint optimization.

[0065] In other words, instead of directly using the original inertial measurement data, it is based on the relative kinematic model between the head-mounted display and the controller implied by the dual pre-integral factor constructed in step 30. By analyzing the physical relationship between the acceleration of the controller relative to the head-mounted display and the IMU measurements and gravity vector of the two, an explicit linear constraint equation for the gravity vector at the current moment can be derived.

[0066] Step 503: Add the linear constraint equations to the joint optimization model in the form of gravity constraint factors.

[0067] In this step, the linear physical constraint equation is defined as a new optimization error term, namely the gravity constraint factor. As an independent constraint, it is added to the overall objective function of the joint optimization model along with the double pre-integration factor and the visual reprojection factor.

[0068] As an optional embodiment, a joint optimization model is constructed using a dual pre-integration factor, a visual reprojection factor, and a gravity constraint factor to jointly optimize the joint state vector and obtain the optimized state estimate. Specifically, it may include: Step 504: Using a sliding window to manage the joint state vector, the sliding window includes the joint state vectors corresponding to consecutive keyframes. In this step, to balance computational complexity and estimation accuracy, a sliding window strategy is adopted. The window maintains a fixed-size set in memory, arranged in chronological order, which contains the joint state vectors corresponding to the most recent keyframe moments. When a new keyframe is generated, the state is added to the window. At the same time, if the window is full, the oldest keyframe state will be removed, thus keeping the window size constant.

[0069] The first in the sliding window At each keyframe, the system state is defined as follows:

[0070] in, This indicates that the head-mounted display is in the world coordinate system. To its body coordinate system The transformation matrix, which includes rotation and translation, This indicates the speed of the headset in the world coordinate system. This indicates the bias of the gyroscope and accelerometer of the head-mounted display's IMU. Indicates the handle coordinate system Relative to the head-mounted display coordinate system The rotation matrix, This represents the position and velocity of the controller relative to the headset, defined in the headset coordinate system. In this context, this speed is defined as... To eliminate the influence of gravity. This indicates the bias of the gyroscope and accelerometer of the handle IMU. Represents the gravity vector in the world coordinate system In the head-mounted display coordinate system The following is a representation, namely The state is modeled as a slowly changing or constant parameter within the window.

[0071] Step 505: When a new keyframe is added, use a nonlinear optimization algorithm to jointly optimize all states within the window through a joint optimization model.

[0072] In this step, once a new keyframe is added to the sliding window, a batch optimization is immediately triggered for all states within the window. The goal of this optimization is to minimize a comprehensive cost function, which is the sum of the squared Mahalanobis distance weighted sums of the residuals corresponding to all constraint factors within the window, mathematically represented as minimizing Σ ||r||². These residuals r include: dual pre-integration residuals: measuring the consistency between state evolution and dual IMU pre-integration predictions; visual reprojection residuals: measuring the consistency between static map point projections and observations; gravity constraint residuals: measuring the consistency between gravity vector estimation and physical derivation constraints; and prior residuals: derived from the marginalization operation, representing information left behind by the old states that were removed from the window.

[0073] Furthermore, a nonlinear least squares optimization algorithm, such as the Levenberg-Marquardt method, is used to iteratively solve the above cost function. By adjusting the joint state vector of all keyframes within the sliding window, the total residual is continuously reduced, eventually converging to a set of state estimates that maximize the global probability, i.e., the optimized state estimate.

[0074] Specifically, within a sliding window containing M keyframes, the objective function to be optimized is:

[0075] in, It is the set of all states within the window, and the meanings of each residual are as follows: prior residual This indicates that the old state information from the sliding-out window is preserved in a priori form, indicating that the operation originates from the edge. (Double pre-integration residual) Constraining the relative motion between adjacent frames of the head-up display and the controller; visual reprojection residuals Indicates for the first A static map point, its position in the world coordinate system is... In the Observations on the frame image are The residual is: Observations here Static features extracted solely from the dynamic perception visual front end; gravity-constrained residuals are expressed as Used to optimize gravity vector .

[0076] Step 506: After optimization, the old states that exceed the window are transformed into prior information through marginalization, and observability is monitored during marginalization. Covariance expansion or retention is performed on the old states in the weak observable direction.

[0077] In this step, the old keyframe states that are removed from the window are removed from the variables to be optimized. Through first-order linearization approximation and Shur complement operation, these old states and their associated constraint factors are transformed into a Gaussian prior distribution acting on the remaining window states, which encodes the information carried by the removed states and is incorporated into subsequent optimization as a summary of historical knowledge.

[0078] Furthermore, to prevent the linearization error during the marginalization process from accumulating in weakly observable directions and causing estimation divergence when the system's motion excitation is insufficient, the information matrix corresponding to the part to be marginalized is analyzed before marginalization. The information content in each direction is quantitatively assessed by examining the eigenvalues ​​of this matrix in the theoretically observable directions of the system. When the eigenvalues ​​in one or more directions are detected to be below a preset safety threshold, it indicates that the constraints in these directions are very weak due to motion degradation. Therefore, special processing is applied to these directions. The processing methods include: covariance inflation: artificially increasing the covariance of the corresponding state in the generated prior factors to weaken its constraint strength and prevent overfitting; and selective retention: temporarily deciding not to marginalize specific states strongly correlated with the weakly observable direction, temporarily retaining them within a sliding window until subsequent motion provides sufficient excitation and observability is restored before marginalization. This strategy significantly enhances the long-term robustness and consistency of the system under challenging motion scenarios.

[0079] Step 60: Based on the optimized state estimation, obtain the pose information of the head-mounted display and the controller.

[0080] As an optional embodiment, the method may further include: feeding back the bias and gravity vector parameters of the head-mounted display inertial measurement unit obtained after joint optimization; In this step, the latest pose of the head-mounted display (HMD) in the world coordinate system, namely its position, orientation, and velocity, is directly extracted from the optimized state vector. This is the optimal estimate after incorporating visual, dual IMU, and gravity constraints. The rotation, position, and velocity of the controller relative to the HMD's coordinate system are obtained from the optimized state vector. Then, combined with the already output HMD pose in the world coordinate system, the absolute pose of the controller in the world coordinate system is calculated and output through coordinate transformation. This pose also benefits from bidirectional coupling optimization, and its accuracy no longer depends solely on the HMD pose but is obtained by mutual correction with the HMD pose.

[0081] The pre-integration process of inertial measurement data is corrected online by using the bias of the inertial measurement unit with feedback; the extraction of visual features and the gravity compensation term are corrected by using the gravity vector parameters with feedback.

[0082] In this step, during IMU pre-integration between subsequent keyframes, the more accurate head-mounted display IMU bias estimate fed back will be used as the bias input in the pre-integration formula. This immediately reduces the pre-integration error introduced by inaccurate bias estimation, making the relative motion constraints constructed by the dual pre-integration factors more accurate. In the dynamic perception visual front end, if any inertial-based motion prediction or gravity influence assessment is involved, the fed-back gravity vector parameters will be used for more accurate compensation. More importantly, in the inertial navigation solution or motion prediction stages of the head-mounted display and controllers, especially during brief visual loss, accurate gravity direction is crucial for correctly decomposing the gravity component and motion acceleration component in the accelerometer measurement. Compensation using the accurate fed-back gravity vector can significantly improve the accuracy of pure inertial recursion, providing more reliable pose prediction for the system in challenging scenarios such as occlusion and rapid movement, thus enhancing robustness.

[0083] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a bidirectional coupled visual-inertial joint pose estimation device, the structure of which is as follows: Figure 5 As shown.

[0084] Figure 5 This is a schematic diagram of the internal structure of a bidirectional coupled visual-inertial joint pose estimation device provided in an embodiment of this application. Figure 5 As shown, the device includes: At least one processor 201; And a memory 502 that is communicatively connected to at least one processor; The memory 502 stores instructions that can be executed by at least one processor, which are executed by at least one processor 501 to enable at least one processor 501 to: perform the steps of any one of the bidirectional coupled visual-inertial joint pose estimation methods.

[0085] Some embodiments of this application provide corresponding to Figure 1 A non-volatile computer storage medium for bidirectional coupled visual-inertial joint pose estimation stores computer-executable instructions, which are configured to execute any one of the steps of a bidirectional coupled visual-inertial joint pose estimation method.

[0086] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for IoT devices and media are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0087] The systems, media, and methods provided in this application are one-to-one correspondences. Therefore, the systems and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be repeated here.

[0088] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0093] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0094] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0095] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0096] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A bidirectional coupled visual-inertial joint pose estimation method, characterized in that, The method includes: Inertial measurement data is acquired through the inertial measurement units of the head-mounted display and the handheld device, and images are acquired through the head-mounted display. The first state parameters of the head-mounted display device, the second state parameters of the handle device relative to the head-mounted display device, and the gravity vector parameters are collected to construct a joint state vector; Based on the inertial measurement data, a double pre-integral factor is constructed to constrain the relative motion state between the head-mounted display device and the handheld device; Visual features are extracted from the image, a visual reprojection factor is constructed based on the visual features, and a gravity constraint factor is constructed based on the gravity vector parameters. Using the aforementioned double pre-integration factor, visual reprojection factor, and gravity constraint factor, a joint optimization model is constructed to jointly optimize the joint state vector, thereby obtaining the optimized state estimate. Based on the optimized state estimation, the pose information of the head-mounted display device and the handheld device is obtained.

2. The bidirectional coupled visual-inertial joint pose estimation method according to claim 1, characterized in that, The construction of a dual pre-integral factor based on the inertial measurement data to constrain the relative motion state between the head-mounted display and the controller specifically includes: The inertial measurement data of the head-mounted display and the controller are pre-integrated to obtain the motion increment of the head-mounted display and the motion increment of the controller. Based on the motion increments of the head-mounted display and the motion increments of the controller, a dual pre-integration factor is constructed.

3. The bidirectional coupled visual-inertial joint pose estimation method according to claim 1, characterized in that, The process of collecting the first state parameters of the head-mounted display device, the second state parameters of the controller device relative to the head-mounted display device, and the gravity vector parameters to construct a joint state vector specifically includes: The first state parameters include the position and velocity of the head-mounted display in the world coordinate system and the offset of the head-mounted display inertial measurement unit; The second state parameters include the rotation angle, position, and speed of the handle device relative to the coordinate system of the head-mounted display device, as well as the offset of the handle inertial measurement unit; Define the representation of the gravity vector parameters in the world coordinate system in the head-mounted display device coordinate system; The joint state vector is obtained by combining the first state parameter, the second state parameter, and the representation.

4. The bidirectional coupled visual-inertial joint pose estimation method according to claim 1, characterized in that, The step of extracting visual features from the image and constructing a visual reprojection factor based on the visual features specifically includes: Based on the currently estimated pose of the handle device relative to the head-mounted display device, predict the projection area of ​​the handle device in the image and generate a dynamic object mask; Extract static environment feature points outside the area covered by the dynamic object mask; The visual reprojection factor is constructed based on the static environmental feature points.

5. The bidirectional coupled visual-inertial joint pose estimation method according to claim 4, characterized in that, The step of predicting the projection area of ​​the handle in the head-mounted display camera image and generating a dynamic object mask based on the currently estimated pose of the handle relative to the head-mounted display specifically includes: Obtain the coordinates of multiple preset 3D key points in the 3D model of the handle device; Based on the pose estimation value, the intrinsic parameters of the head-mounted display device, and the extrinsic parameters of the head-mounted display device, the three-dimensional key points are projected onto the image to obtain multiple two-dimensional projection points; By connecting the multiple two-dimensional projection points, a dynamic object mask surrounding the projection area of ​​the handle device is obtained.

6. The bidirectional coupled visual-inertial joint pose estimation method according to claim 1, characterized in that, The construction of a joint optimization model using the dual pre-integration factor, visual reprojection factor, and gravity constraint factor specifically includes: The gravity vector is explicitly introduced as a state variable to be optimized in the joint optimization model. Based on the inertial measurement data of the head-mounted display and the handheld device, the explicit linear constraint equations for the gravity vector parameters are obtained. The linear constraint equations are incorporated into the joint optimization model in the form of gravity constraint factors.

7. The bidirectional coupled visual-inertial joint pose estimation method according to claim 1, characterized in that, Using the aforementioned dual pre-integration factor, visual reprojection factor, and gravity constraint factor, a joint optimization model is constructed to jointly optimize the joint state vector, obtaining the optimized state estimate, specifically including: The joint state vector is managed using a sliding window, which includes the joint state vectors corresponding to consecutive keyframes. When a new keyframe is added, the joint optimization model is used to perform joint optimization on all states within the window using a nonlinear optimization algorithm. After optimization, old states that exceed the window are transformed into prior information through marginalization, and observability is monitored during marginalization. Covariance inflation or retention is performed on old states in weakly observable directions.

8. The bidirectional coupled visual-inertial joint pose estimation method according to claim 1, characterized in that, The method further includes: The bias and gravity vector parameters of the head-mounted display inertial measurement unit obtained after joint optimization are fed back. The pre-integration process of the inertial measurement data is corrected online by using the bias of the feedback inertial measurement unit; The extracted visual features and gravity compensation terms are corrected using the feedback gravity vector parameters.

9. A bidirectional coupled visual-inertial joint pose estimation device, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: Perform the steps of the bidirectional coupled visual-inertial joint pose estimation method as described in any one of claims 1-8.

10. A bidirectional coupled visual-inertial joint pose estimation non-volatile computer storage medium, storing computer-executable instructions, characterized in that, The computer-executable instructions are set as follows: Perform the steps of the bidirectional coupled visual-inertial joint pose estimation method as described in any one of claims 1-8.

Images