A dual-handle extrinsic parameter calibration and fusion method and system based on splittable merging
By employing a dual-handle extrinsic parameter calibration and fusion method, and utilizing dual IMU data and visual observations to estimate extrinsic parameters online, the problem of rotation accuracy and drift when merging dual handles in VR/AR systems is solved. This achieves high-precision rotation estimation and system consistency, and supports dynamic disassembly and assembly scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV
- Filing Date
- 2026-04-20
- Publication Date
- 2026-07-14
AI Technical Summary
In existing VR/AR systems, when two controllers are combined, the external parameters are unknown or require offline calibration, rotation accuracy is limited, drift accumulates, there is a lack of a unified fusion framework, and asynchronous time leads to errors, making it unsuitable for dynamic splitting and joining scenarios.
A dual-handle extrinsic parameter calibration and fusion method based on splittable merging is adopted. The merging mode is determined by cross-correlation calculation. The extrinsic parameters are estimated online using dual IMU data and visual observation. B-spline trajectory modeling and factor graph optimization are used to achieve high-precision rotation estimation and time offset calibration, and dynamic splitting and merging are supported.
It improves rotation accuracy, reduces drift, enhances system robustness and consistency, supports dynamic splitting and joining, eliminates asynchronous fusion errors, provides reliable rotation estimation, and avoids cumulative errors.
Smart Images
Figure CN122391376A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of virtual reality and augmented reality technologies, and in particular to a method and system for extrinsic parameter calibration and fusion based on a detachable and merging dual-handle controller. Background Technology
[0002] In VR / AR systems, the controllers are the core interaction devices, and their tracking accuracy directly impacts the user experience. Current mainstream solutions (such as the Meta Quest series and Pimax Crystal) employ inside-out tracking: the head-mounted display (HMD) observes infrared LEDs or feature points on the controllers through cameras, and combines this with the controllers' built-in IMUs to achieve six degrees of freedom (6-DoF) pose estimation. However, existing solutions typically treat each controller as an independent entity, failing to fully utilize the redundant information available when multiple controllers work together.
[0003] With the expansion of VR application scenarios, there is an increasing demand for integrated devices that can combine two controllers into one (such as motion-sensing guns, nunchucks, and modular controllers). When two controllers are rigidly connected, they constitute a multi-IMU system. If the extrinsic parameters (rotation and translation) between the two controllers can be estimated online and the measurement data from the two IMUs can be fused, rotational accuracy can be significantly improved, drift can be reduced, and system robustness can be enhanced.
[0004] However, existing controller technology still has the following shortcomings: 1. When using two handles together, the external parameters are unknown or require offline calibration, making it unsuitable for dynamic disassembly and assembly scenarios.
[0005] 2. The redundant information from the dual IMUs was not utilized to improve rotation accuracy and reduce drift.
[0006] 3. Lack of a unified fusion framework, making it impossible to jointly optimize the 6-DoF pose of two handles in merge mode.
[0007] 4. Asynchronous time (clocks not synchronized) leads to fusion errors and lacks online time offset calibration. Summary of the Invention
[0008] This application provides a method and system for calibrating and fusing extrinsic parameters of a dual-handle controller based on a detachable and merging mechanism, which solves the following technical problems: In existing controller tracking technologies, extrinsic parameters are unknown or require offline calibration, and rotational accuracy is easily limited and drift accumulates, resulting in error anomalies in controller merging mode.
[0009] The embodiments of this application adopt the following technical solutions: On one hand, this application provides a dual-handle extrinsic parameter calibration and fusion method based on a detachable and merging mechanism, comprising: performing normalized cross-correlation calculation between the first angular velocity measurement sequence of the first handle and the second angular velocity measurement sequence of the second handle according to the first angular velocity measurement sequence of the first handle and the second angular velocity measurement sequence of the second handle, and determining the merging mode or the separation mode of the handle based on the cross-correlation result; if the handle is determined to be in the merging mode, then performing rotational motion-based constraint control between the first handle and the second handle according to the extrinsic parameter state and time offset to obtain the rotational motion constraint result; when the handle is determined to be in the merging mode, according to the first 6-D of the first handle... Based on the first handle's 6-DoF pose and the second handle's second 6-DoF pose, visual observation-based constraint control is applied between the first handle and the second handle to obtain visual observation constraint results. If the handle is determined to be in the merging mode, the rotational motion constraint results are subjected to residual joint fusion processing based on measurements from two IMU gyroscopes to obtain a high-precision rotational trajectory of the merged handle. If the handle is determined to be in the merging mode, the visual observation constraint results are subjected to unified observation control processing based on shared feature points to obtain a high-precision 6-DoF pose of the merged handle. If the handle is determined to be in the separation mode, both the first handle and the second handle are independently tracked and controlled.
[0010] This application embodiment achieves seamless switching between separate and merge modes for the dual handles. In merge mode, it estimates extrinsic parameters online, calibrates time offset, and fuses dual IMU data with visual observations, improving rotation accuracy, reducing drift, and enhancing system robustness and consistency. Specifically, by fusing dual IMU gyroscope data, rotation accuracy is significantly improved compared to a single IMU gyroscope, and drift is significantly reduced. Furthermore, offline calibration is unnecessary, and dynamic separation and reassembly are supported, improving the user experience. It can estimate clock differences online, eliminating asynchronous fusion errors. It can also automatically detect merge / separate states and seamlessly switch tracking modes. Even in visually occluded or weakly textured scenes, dual IMU fusion still provides reliable rotation estimation. Simultaneously, joint optimization ensures the rigid consistency of the poses of the two handles and avoids accumulated errors during separation.
[0011] In one feasible implementation, based on the first angular velocity measurement sequence of the first handle and the second angular velocity measurement sequence of the second handle, a normalized cross-correlation calculation is performed between the first angular velocity measurement sequence and the second angular velocity measurement sequence, specifically including: based on... The cross-correlation value between the first angular velocity measurement sequence and the second angular velocity measurement sequence is obtained. ;in, The first angular velocity measurement sequence of the first handle; This is the first preset angular velocity measurement sequence; The second angular velocity measurement sequence for the second handle; This is the second preset angular velocity measurement sequence; k is a mathematical constant.
[0012] In one feasible implementation, based on the cross-correlation results, the merging or separating mode of the handle is determined, specifically including: if the cross-correlation value is greater than the cross-correlation reference value and the duration in the time window is greater than the time reference value, then the handle is determined to be in the merging mode; otherwise, the handle is determined to be in the separating mode; when the separating mode switches to the merging mode, the extrinsic parameter state and time offset are initialized, and the online calibration and fusion module in the dual-handle extrinsic parameter calibration and fusion system is activated; when the merging mode switches to the separating mode, the update of the extrinsic parameter state is stopped, independent tracking control of each individual handle is restored, the current extrinsic parameter state in the last merging mode is retained, and the current extrinsic parameter state is determined as the initial value of the extrinsic parameter state when the next merging mode is started.
[0013] In one feasible implementation, if the handle is determined to be in the merging mode, then based on the extrinsic parameter state and time offset, rotational motion-based constraint control is performed between the first handle and the second handle to obtain the rotational motion constraint result. Specifically, this includes: modeling the first rotational trajectory of the first handle using a continuous-time B-spline, and constraining the rotational trajectory of the second handle based on the control point set and the introduced extrinsic parameter state to obtain the second rotational trajectory of the second handle; according to The residual of the second angular velocity measurement sequence of the second handle is obtained. ;in, The second angular velocity measurement sequence for the second handle; The first angular velocity measurement sequence of the first handle; This is the time offset; The second gyroscope on the second handle has zero bias; The rotational extrinsic parameter in the aforementioned extrinsic parameter state; The term is obtained based on differential calculations using B-splines; according to The residual of the first angular velocity measurement sequence of the first handle is obtained. ;in, The first gyroscope of the first handle has zero bias; The angular velocity measurement sequence is obtained based on the B-spline differential; based on the residuals of the second angular velocity measurement sequence and the first angular velocity measurement sequence, the rotational motion is constrained and controlled on the second handle and the first handle respectively, and the rotational motion constraint result is obtained.
[0014] In one feasible implementation, when the handle is determined to be in the merged mode, constraint control based on visual observation is performed between the first handle and the second handle according to the first 6-DoF pose of the first handle and the second 6-DoF pose of the second handle, to obtain the visual observation constraint result. Specifically, this includes: when both the first handle and the second handle are observed by the head-mounted display camera, acquiring the first 6-DoF pose of the first handle and the second 6-DoF pose of the second handle in the world coordinate system; wherein, the second 6-DoF pose satisfies rigid constraints based on the first 6-DoF pose and the external parameter state; according to The time extrinsic residual between the first handle and the second handle is obtained. ;in, This is the first 6-DoF pose; This is the second 6-DoF pose; The external parameter state is defined; based on the time external parameter residual, constraint control based on visual observation is performed between the first handle and the second handle respectively to obtain the visual observation constraint result.
[0015] In one feasible implementation, when the handle is determined to be in the merging mode, the rotational motion constraint results are subjected to residual fusion processing related to the measurements of the two IMU gyroscopes to obtain a high-precision rotational trajectory of the merged handle. Specifically, this includes: extracting the residuals of the second angular velocity measurement sequence and the residuals of the first angular velocity measurement sequence; according to... The joint residuals based on measurements from the first handle IMU gyroscope and the second handle IMU gyroscope were obtained. ;in, The first angular velocity measurement sequence is the residual of the first angular velocity measurement sequence; The second angular velocity measurement sequence based on the time offset is the residual of the second angular velocity measurement sequence; This refers to the time offset; The rotational extrinsic parameter in the aforementioned extrinsic parameter state; The second gyroscope on the second handle has zero bias; The first gyroscope of the first handle has zero bias; The cumulative B-spline is used for the handle assembly. Angular velocity at time t; k is a mathematical constant; This is the first term to be summed; The second summation term is used; through the joint residual, the rotational motion constraint result is subjected to joint fusion processing of the residuals of the two IMU gyroscope measurements, and based on the optimized control of the control point, external parameter state, time offset, and gyroscope zero bias of the combined rotation trajectory of the handle, the high-precision rotation trajectory in the combined mode is obtained.
[0016] In one feasible implementation, before performing unified observation control processing based on shared feature points on the visual observation constraint results, the method further includes: when the handle is determined to be in the merging mode, determining the state vector of the handle merging body based on the pose, velocity, IMU zero bias, gravity vector, extrinsic parameter state, and time offset of the handle merging body in the world coordinate system; and according to the state vector, performing factor graph optimization processing based on a sliding window on the IMU pre-integration factor, visual reprojection factor, extrinsic parameter factor, time offset factor, and prior factor after extrinsic parameter motion association in each handle to obtain an optimized factor graph.
[0017] In one feasible implementation, when the handle is determined to be in the merging mode, the visual observation constraint results are subjected to unified observation control processing based on shared feature points to obtain the high-precision 6-DoF pose of the handle merging body. Specifically, this includes: extracting the shared feature points of the visual front end of each handle; based on the shared feature points and through the optimization factor map, fusing the first IMU pre-integration of the first handle and the IMU pre-integration of the second handle, the visual observation constraint results, and the extrinsic state constraints; performing unified observation control on the observation data in the fusion result under the same map, and outputting the high-precision 6-DoF pose of the handle merging body.
[0018] In one feasible implementation, if the handle is determined to be in the separation mode, then both the first handle and the second handle are subjected to independent tracking control. Specifically, this includes: if the handle is determined to be in the separation mode, then both the first handle and the second handle are subjected to 6-DoF pose estimation calculation based on independent VI-SLAM to obtain an independent 6-DoF pose for each handle; based on the independent 6-DoF pose, velocity, IMU zero bias, and gravity vector, an independent state vector for each handle is determined; and based on the independent state vector, independent tracking control of both the first handle and the second handle is completed.
[0019] On the other hand, embodiments of this application also provide a dual-handle extrinsic parameter calibration and fusion system based on a detachable and merging mechanism. The dual-handle extrinsic parameter calibration and fusion system is executable by instructions from at least one processor, enabling the at least one processor to execute a dual-handle extrinsic parameter calibration and fusion method based on a detachable and merging mechanism according to any of the above embodiments.
[0020] This application provides a method and system for extrinsic parameter calibration and fusion based on a splittable and mergeable dual-handle system. Compared with the prior art, the embodiments of this application have the following beneficial technical effects: 1. High-precision rotation estimation: By fusing data from dual IMU gyroscopes, the rotation accuracy is significantly improved compared to a single IMU gyroscope, and drift is significantly reduced.
[0021] 2. Online external parameter calibration: No offline calibration is required, and dynamic splitting and joining are supported, improving the user experience.
[0022] 3. Time offset compensation: Online estimation of clock difference to eliminate asynchronous fusion error.
[0023] 4. Modal Adaptive: Automatically detects merging / separation status and seamlessly switches tracking modes.
[0024] 5. Strong robustness: Even in scenes with visual occlusion or weak texture, dual IMU fusion can still provide reliable rotation estimation.
[0025] 6. High system consistency: Joint optimization ensures the rigid consistency of the poses of the two handles, avoiding the accumulation of errors when they are separated. Attached Figure Description
[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart of a dual-handle extrinsic parameter calibration and fusion method based on detachable and merging is provided for embodiments of this application; Figure 2 A flowchart of a dual-handle dual-modal detection process is provided for an embodiment of this application; Figure 3 This is a schematic diagram of a factor graph in a dual-handle merging mode provided in an embodiment of this application; Figure 4 A schematic diagram of a B-spline rotation trajectory provided in an embodiment of this application; Figure 5 This is a schematic diagram of a detachable and merging dual-handle extrinsic parameter calibration and fusion device provided in an embodiment of this application. Detailed Implementation
[0027] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.
[0028] It should be noted that this application aims to address the following problems existing in VR / AR controller tracking systems in scenarios where two controllers can be detached and merged: 1) Unknown extrinsic parameters or need for offline calibration: When the two controllers are rigidly connected, the rotation and translation relationship between them is unknown, making it impossible to directly fuse the dual IMU data. 2) Limited rotation accuracy and drift accumulation: A single IMU experiences drift during long-term operation or rapid rotation, and redundant IMUs are not utilized to improve accuracy. 3) Asynchronous time: The IMU clocks of the two controllers are not synchronized, introducing errors through direct fusion. 4) Inflexible mode switching: The system cannot automatically detect the merging / separation state and switch tracking modes, resulting in a fragmented user experience.
[0029] This application includes the following core modules: 1) a dual-modal detection module: capable of determining whether the two handles are currently in a separated or merged mode based on IMU motion consistency; 2) an online extrinsic parameter time offset estimation module: in merged mode, utilizing dual IMUs and visual observation to estimate rotational extrinsic parameters online. Translation of external parameters and time offset 3) Dual IMU fusion attitude estimation module: It can use continuous-time B-spline trajectory modeling, fuse the gyroscope data of two IMUs, optimize the rotation trajectory, and improve rotation accuracy; 4) Joint 6-DoF fusion module: In merging mode, it can combine the independent 6-DoF estimates of the two handles (from visual-inertial odometry) with the dual IMU fusion results to output a higher-precision merged 6-DoF pose; 5) Modal switching and state management module: Based on the modal detection results, it switches the optimization strategy, manages the state vector, and realizes a smooth transition between separation and merging.
[0030] Furthermore, the inputs to the dual-handle extrinsic parameter calibration and fusion system are: raw IMU data (angular velocity, acceleration) from both handpieces, head-mounted display camera images, and the head-mounted display's own pose (which can be provided by the head-mounted display's VIO). The outputs of the dual-handle extrinsic parameter calibration and fusion system are: independent 6-DoF poses of the two handpieces in separate mode, or combined 6-DoF poses (including high-precision rotation) in combined mode, extrinsic parameter estimates, and time offset.
[0031] This application provides a dual-handle extrinsic parameter calibration and fusion method based on spliable and merging extrinsic parameters, such as... Figure 1 As shown, the dual-handle extrinsic parameter calibration and fusion method based on splittable and mergeable methods specifically includes steps S101-S106: S101. Based on the first angular velocity measurement sequence of the first handle and the second angular velocity measurement sequence of the second handle, perform normalized cross-correlation calculation between the first angular velocity measurement sequence and the second angular velocity measurement sequence, and determine the merging mode or separation mode of the handle based on the cross-correlation result.
[0032] Specifically, first define the time window. Internal, angular velocity measurement sequence of handle 1 and handle 2 , Then, calculate the normalized cross-correlation (NCC) of their angular velocities, i.e.: according to The cross-correlation value between the first angular velocity measurement sequence and the second angular velocity measurement sequence was obtained. ;in, The first angular velocity measurement sequence for the first handle; This is the first preset angular velocity measurement sequence; The second angular velocity measurement sequence for the second handle; This is the second preset angular velocity measurement sequence; k is a mathematical constant.
[0033] Furthermore, if the cross-correlation value of the cross-correlation result is greater than the cross-correlation baseline value and the duration in the time window is greater than the time baseline value, then the handle is set to merge mode; otherwise, the handle is set to separate mode.
[0034] In one embodiment, Figure 2 A flowchart of a dual-handle dual-modal detection process is provided for an embodiment of this application, as follows: Figure 2 As shown, if And the duration exceeds If the result is positive, it is determined to be in merging mode; otherwise, it is in separation mode. Furthermore, it can assist in detecting the consistency of acceleration magnitude values to improve robustness.
[0035] Furthermore, when the separation mode is switched to the merging mode, the external parameter status and time offset are initialized, and the online calibration and fusion module in the dual-handle external parameter calibration and fusion system is activated.
[0036] Furthermore, when the merge mode switches to the separate mode, the update of the external parameter state is stopped, independent tracking control of each individual handle is restored, the current external parameter state in the last merge mode is retained, and the current external parameter state is determined as the initial value of the external parameter state when the next merge mode is started.
[0037] As a possible implementation method, such as Figure 2 As shown, when switching from separation mode to merging mode: after merging is detected, the extrinsic parameters are initialized with a time offset (which can be roughly estimated visually or set to zero), and the online calibration and fusion module is activated. When switching from merging mode to separation mode: after separation is detected, extrinsic parameter updates are stopped, independent tracking is resumed, and the last estimated extrinsic parameters are retained as the initial values for the next merging.
[0038] S102. If the handle is determined to be in merge mode, then based on the external parameter state and time offset, the first handle and the second handle are subjected to rotational motion-based constraint control to obtain the rotational motion constraint result.
[0039] It should be noted that in the merge mode, the external parameter state also needs to be introduced. and time offset (The clock difference between handle 2 and handle 1, i.e., the measurement time of handle 2) The extrinsic parameters are modeled as slowly changing quantities (or constants) and optimized within a sliding window.
[0040] Specifically, the first rotation trajectory of the first handle is first modeled using a continuous-time B-spline, and the rotation trajectory of the second handle is constrained based on the control point set and the introduced extrinsic state, thus obtaining the second rotation trajectory of the second handle.
[0041] In one embodiment, Figure 3 This application provides a schematic diagram of a factor graph in a dual-handle merging mode, as shown in the embodiment of the present application. Figure 3 As shown, the rotation trajectory of handle 1 is obtained using a continuous-time B-spline. Modeling is performed, and the control point set is... The second rotation trajectory of handle 2 should satisfy: .
[0042] Furthermore, using the gyroscope measurement of handle 2, the residual of the second angular velocity measurement sequence is constructed, that is: according to The residual of the second angular velocity measurement sequence of the second handle is obtained. ;in, The second angular velocity measurement sequence for the second handle; The first angular velocity measurement sequence for the first handle; This is the time offset; The second gyroscope on the second handle has zero bias; The extrinsic parameter is the rotating extrinsic parameter in the extrinsic parameter state; The term is obtained by differential calculation based on B-splines.
[0043] Furthermore, the gyroscope of handle 1 is used to measure the constrained B-spline trajectory: according to The residual of the first angular velocity measurement sequence of the first handle is obtained. ;in, The first gyroscope of the first handle has zero bias; This is the angular velocity measurement sequence obtained based on B-spline differentiation.
[0044] Furthermore, based on the residuals from the second angular velocity measurement sequence and the residuals from the first angular velocity measurement sequence, the rotational motion of the second handle and the first handle are constrained and controlled respectively to obtain the rotational motion constraint results.
[0045] S103. When the handle is determined to be in the merge mode, according to the first 6-DoF pose of the first handle and the second 6-DoF pose of the second handle, the constraint control between the first handle and the second handle based on visual observation is performed to obtain the visual observation constraint result.
[0046] Specifically, when both the first and second handles are observed by the head-mounted display camera, the first 6-DoF pose of the first handle and the second 6-DoF pose of the second handle are obtained in the world coordinate system; wherein, the second 6-DoF pose satisfies the rigid constraints based on the first 6-DoF pose and the external parameter state.
[0047] In one embodiment, when both handles are simultaneously observed by the head-mounted display camera, their respective 6-DoF poses in the world coordinate system can be obtained. (First 6-DoF pose) and (The second 6-DoF pose is provided by the VI-SLAM of each handle). In merged mode, the 6-DoF poses of the two handles should satisfy rigid constraints: .
[0048] Furthermore, according to The time extrinsic residual between the first handle and the second handle is obtained. ;in, This is the first 6-DoF pose; This is the second 6-DoF pose; This is the external parameter state. That is, the visual observation constraints between the first and second handles are completed using the time external parameter residual.
[0049] Furthermore, based on the time extrinsic residual, visual observation-based constraint control is applied to the first handle and the second handle respectively to obtain the visual observation constraint results.
[0050] As a feasible implementation method, in the time offset constraint of the two handles, the time offset τ is used as an optimization variable, and the time offset compensation is performed on the rotation trajectory in the residual (e.g., (Interpolation at the point of intersection) to align with gyroscope measurements. The Jacobian of the time offset can be obtained by differentiating the B-spline.
[0051] S104. When the handle is determined to be in merge mode, the rotational motion constraint results are subjected to residual fusion processing of the measurements from the two IMU gyroscopes to obtain the high-precision rotational trajectory of the handle merged body.
[0052] Specifically, Figure 4 A schematic diagram of a B-spline rotation trajectory provided in this application embodiment, such as... Figure 4As shown, the rotational trajectory of the merged body can be represented using a cumulative B-spline (or Cumulative B-spline). Furthermore, the control points are uniformly distributed, and the angular velocity of the trajectory can be obtained by differentiating the B-spline. Then, the residuals of the second angular velocity measurement sequence and the first angular velocity measurement sequence are extracted. That is, the residual of the first angular velocity measurement sequence... Residual with the second angular velocity measurement sequence .
[0053] Furthermore, according to The joint residuals based on measurements from the first handle IMU gyroscope and the second handle IMU gyroscope were obtained. ;in, The first angular velocity measurement sequence is the first angular velocity measurement sequence in the residual of the first angular velocity measurement sequence; The second angular velocity measurement sequence based on time offset is the residual of the second angular velocity measurement sequence. This is the time offset; The extrinsic parameter is the rotating extrinsic parameter in the extrinsic parameter state; The second gyroscope on the second handle has zero bias; The first gyroscope of the first handle has zero bias; The cumulative B-spline is used for the handle assembly. Angular velocity at time t; k is a mathematical constant; This is the first term to be summed; This is the second summation term.
[0054] Furthermore, by combining the residuals, the rotational motion constraint results are jointly fused from the residuals of the two IMU gyroscope measurements. Based on the optimized control points, extrinsic parameter states, time offset, and gyroscope zero bias of the combined handle rotation trajectory, a high-precision rotation trajectory in fusion mode is obtained. After fusing the two IMUs in the accuracy analysis, the noise variance of the rotation estimation is theoretically reduced to half that of a single IMU (if the noise level is the same), achieving a 2-fold improvement in accuracy and suppressing zero bias drift.
[0055] S105. When the handle is determined to be in merge mode, the visual observation constraint results are processed by unified observation control based on shared feature points to obtain the high-precision 6-DoF pose of the merged handle.
[0056] Specifically, when the controller is determined to be in merge mode, the state vector of the controller merge is determined based on the pose, velocity, IMU zero bias, gravity vector, external parameter state and time offset of the controller merge in the world coordinate system.
[0057] In one embodiment, under merge mode, the state vector of the merged entity is defined as follows: ;in, The pose of the merged body in the world coordinate system (the handle 1 coordinate system can be selected as the body coordinate system). For speed, For the IMU zero bias term in the first and second handles, The gravity vector In external parameter state, This is the time offset.
[0058] Furthermore, based on the state vector, the IMU pre-integration factor, visual reprojection factor, extrinsic factor, time offset factor, and prior factor after extrinsic motion association in each handle are all optimized using a sliding window-based factor graph to obtain an optimized factor graph.
[0059] In one embodiment, Figure 3 This application provides a schematic diagram of a factor graph in a dual-handle merging mode, as shown in the embodiment of the present application. Figure 3 As shown, in factor graph optimization, a factor graph is first constructed within a sliding window, which needs to include: 1) IMU pre-integration factor, where each handle pre-integrates independently but needs to be associated with extrinsic parameters; 2) visual reprojection factor, derived from the visual feature points of the two handles, if the feature points are fixed in the world frame; 3) extrinsic parameter factor, constraining the relative pose of the two handles; 4) time offset factor, derived from gyroscope alignment; 5) prior factor, marginalizing historical information. That is, for the IMU pre-integration of handles 1 and 2, extrinsic parameters are needed to associate their motions. A "dual IMU pre-integration factor" can be defined to complete the constraint between the two handles.
[0060] Furthermore, shared feature points are first extracted from the visual front end of each handle. Then, based on the shared feature points and by optimizing the factor map, the first IMU pre-integration of the first handle, the IMU pre-integration of the second handle, the visual observation constraint results, and the extrinsic state constraints are fused together.
[0061] In one embodiment, for the fusion of visual observations from the two handles, feature points can be extracted from the visual front end of each handle (if there are independent feature points in the merged object) or shared feature points can be used. Then, in fusion mode, the observations from the two handles are unified into one map, and joint bundle adjustment (BA) is achieved using extrinsic constraints. Subsequently, the IMU pre-integration, visual observations, and extrinsic state constraints of the two handles are fused in the unified factor map to output the high-precision pose of the merged object; that is, the observation data in the fusion result is uniformly controlled under the same map, and the high-precision 6-DoF pose of the handle merged object is output.
[0062] S106. If the handle is determined to be in split mode, then both the first handle and the second handle will be independently tracked and controlled.
[0063] Specifically, if the handle is determined to be in a separate mode, then both the first and second handles will be subjected to 6-DoF pose estimation calculations based on independent VI-SLAM to obtain the independent 6-DoF pose of each handle.
[0064] Furthermore, based on independent 6-DoF pose, velocity, IMU bias, and gravity vector, an independent state vector for each controller is determined. Then, based on the independent state vectors, independent tracking control of both the first and second controllers is achieved.
[0065] In one embodiment, in the separate mode, each handle uses independent VI-SLAM for 6-DoF estimation. The state vector no longer includes extrinsic parameters and time offset; that is, only the independent 6-DoF pose, velocity, IMU null bias, and gravity vector of each handle are needed to determine the independent state vector of each handle. Then, the extrinsic parameter state estimated during the last merging is retained as the initial value of the extrinsic parameter state for the next merging.
[0066] As a feasible implementation method, this application is based on a dual-handle extrinsic parameter calibration and fusion method that can be split and merged. It features a dual-modal detection and switching mechanism: automatically identifying the handle merging / separation state based on IMU motion consistency and dynamically switching the tracking mode. A joint estimation mechanism for online extrinsic parameters and time offset is implemented: in merging mode, dual IMU gyroscopes and visual observations are used to simultaneously optimize rotational extrinsic parameters, translational extrinsic parameters, and time offset. A dual IMU fusion rotation optimization mechanism is employed: continuous-time B-spline modeling of the rotation trajectory is used, fusing the angular velocity data of the two IMUs to achieve high-precision rotation estimation and drift suppression. A joint 6-DoF fusion framework mechanism is used: IMU pre-integration, visual observations, and extrinsic parameter constraints of the two handles are fused in a unified factor graph to output the high-precision pose of the merged object. A modal smoothing transition mechanism is implemented: during separation / merging switching, edge detection is used to retain historical information and avoid state jumps.
[0067] In addition, embodiments of this application also provide a dual-handle extrinsic parameter calibration and fusion system based on detachable and merging, such as... Figure 5 As shown, the dual-handle extrinsic parameter calibration and fusion system 500 specifically includes: The dual-modal detection module 510 is used to perform normalized cross-correlation calculation between the first angular velocity measurement sequence and the second angular velocity measurement sequence of the first handle and the second angular velocity measurement sequence of the second handle, and determine the merging mode or separation mode of the handle based on the cross-correlation result. The online extrinsic parameter and time offset estimation module 520, in merged mode, uses dual IMUs and visual observation to estimate rotational extrinsic parameters, translational extrinsic parameters and time offset online. The dual IMU fusion attitude estimation module 530 is used to perform rotational motion-based constraint control between the first and second handles based on the external parameter state and time offset if the handle is determined to be in merging mode, and to obtain rotational motion constraint results; and is used to perform visual observation-based constraint control between the first and second handles based on the first 6-DoF pose of the first handle and the second 6-DoF pose of the second handle when the handle is determined to be in merging mode, and to obtain visual observation constraint results. The 6-DoF fusion module 540 is used to perform residual fusion processing on the rotational motion constraint results related to the measurements of two IMU gyroscopes when the handle is determined to be in fusion mode, so as to obtain a high-precision rotational trajectory of the combined handle; and to perform unified observation control processing on the visual observation constraint results based on shared feature points when the handle is determined to be in fusion mode, so as to obtain a high-precision 6-DoF pose of the combined handle.
[0068] The modal switching and state management module 550 is used to independently track and control both the first and second handles if the handle is determined to be in a split mode; and to switch optimization strategies and manage state vectors based on modal detection results to achieve a smooth transition between splitting and merging.
[0069] This application embodiment achieves seamless switching between separate and merge modes for the dual handles. In merge mode, it estimates extrinsic parameters online, calibrates time offset, and fuses dual IMU data with visual observations, improving rotation accuracy, reducing drift, and enhancing system robustness and consistency. Specifically, by fusing dual IMU gyroscope data, rotation accuracy is significantly improved compared to a single IMU gyroscope, and drift is significantly reduced. Furthermore, offline calibration is unnecessary, and dynamic separation and reassembly are supported, improving the user experience. It can estimate clock differences online, eliminating asynchronous fusion errors. It can also automatically detect merge / separate states and seamlessly switch tracking modes. Even in visually occluded or weakly textured scenes, dual IMU fusion still provides reliable rotation estimation. Simultaneously, joint optimization ensures the rigid consistency of the poses of the two handles and avoids accumulated errors during separation.
[0070] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0071] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0072] The above description is merely an embodiment of this application and is not intended to limit this application. For those skilled in the art, various modifications and variations can be made to the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the embodiments of this application should be included within the scope of the claims of this application.
Claims
1. A method for calibrating and fusing extrinsic parameters based on a splittable and merging dual-handle system, characterized in that, The method includes: Based on the first angular velocity measurement sequence of the first handle and the second angular velocity measurement sequence of the second handle, a normalized cross-correlation calculation is performed between the first angular velocity measurement sequence and the second angular velocity measurement sequence, and based on the cross-correlation result, the merging mode or separation mode of the handle is determined. If the handle is determined to be in the merge mode, then based on the external parameter state and time offset, a rotational motion-based constraint control is performed between the first handle and the second handle to obtain the rotational motion constraint result. When the handle is determined to be in the merged mode, constraint control based on visual observation is performed between the first handle and the second handle according to the first 6-DoF pose of the first handle and the second 6-DoF pose of the second handle to obtain the visual observation constraint result. When the handle is determined to be in the merged mode, the rotational motion constraint results are subjected to residual fusion processing of measurements from two IMU gyroscopes to obtain a high-precision rotational trajectory of the handle merged body. When the handle is determined to be in the merging mode, the visual observation constraint results are processed by unified observation control based on shared feature points to obtain the high-precision 6-DoF pose of the merged handle. If the handle is determined to be in the separation mode, then both the first handle and the second handle will be independently tracked and controlled.
2. The method for calibrating and fusing extrinsic parameters based on a splittable and mergeable dual-handle according to claim 1, characterized in that, Based on the first angular velocity measurement sequence of the first handle and the second angular velocity measurement sequence of the second handle, a normalized cross-correlation calculation is performed between the first angular velocity measurement sequence and the second angular velocity measurement sequence, specifically including: according to The cross-correlation value between the first angular velocity measurement sequence and the second angular velocity measurement sequence is obtained. ;in, The first angular velocity measurement sequence of the first handle; This is the first preset angular velocity measurement sequence; The second angular velocity measurement sequence for the second handle; This is the second preset angular velocity measurement sequence; k is a mathematical constant.
3. The dual-handle extrinsic parameter calibration and fusion method based on detachable and merging as described in claim 2, characterized in that, Based on the cross-correlation results, the merging or separating mode of the handle is determined, specifically including: If the cross-correlation result shows a cross-correlation value greater than the cross-correlation baseline value and the duration within the time window is greater than the time baseline value, then the handle is set to the merging mode; otherwise, the handle is set to the separation mode. When the separation mode is switched to the merging mode, the external parameter status and time offset are initialized, and the online calibration and fusion module in the dual-handle external parameter calibration and fusion system is activated. When the merge mode is switched to the separation mode, the update of the external parameter state is stopped, independent tracking control of each individual handle is restored, the current external parameter state in the last merge mode is retained, and the current external parameter state is determined as the initial value of the external parameter state when the next merge mode is started.
4. The dual-handle extrinsic parameter calibration and fusion method based on detachable and merging as described in claim 1, characterized in that, If the handle is determined to be in the merged mode, then based on the external parameter state and time offset, rotational motion-based constraint control is performed between the first handle and the second handle to obtain the rotational motion constraint result, specifically including: The first rotation trajectory of the first handle is modeled using continuous-time B-splines, and the rotation trajectory of the second handle is constrained based on the control point set and the introduced extrinsic state, thus obtaining the second rotation trajectory of the second handle. according to The residual of the second angular velocity measurement sequence of the second handle is obtained. ;in, The second angular velocity measurement sequence for the second handle; The first angular velocity measurement sequence of the first handle; This is the time offset; The second gyroscope on the second handle has zero bias; The rotational extrinsic parameter in the aforementioned extrinsic parameter state; The term is obtained based on differential calculations using B-splines; according to The residual of the first angular velocity measurement sequence of the first handle is obtained. ;in, The first gyroscope of the first handle has zero bias; This is a sequence of angular velocity measurements obtained based on B-spline differentiation; Based on the residuals from the second and first angular velocity measurement sequences, rotational motion constraint control is applied to the second and first handles respectively, resulting in rotational motion constraint results.
5. The dual-handle extrinsic parameter calibration and fusion method based on detachable and merging according to claim 1, characterized in that, When the controller is determined to be in the merged mode, based on the first 6-DoF pose of the first controller and the second 6-DoF pose of the second controller, visual observation-based constraint control is performed between the first controller and the second controller to obtain the visual observation constraint result, specifically including: When both the first and second handles are observed by the head-mounted display camera, the first 6-DoF pose of the first handle and the second 6-DoF pose of the second handle are obtained in the world coordinate system; wherein, the second 6-DoF pose satisfies the rigid constraints based on the first 6-DoF pose and the external parameter state. according to The time extrinsic residual between the first handle and the second handle is obtained. ;in, This is the first 6-DoF pose; This is the second 6-DoF pose; The external parameter state; Based on the time extrinsic residual, visual observation-based constraint control is applied to the first handle and the second handle respectively to obtain the visual observation constraint result.
6. The dual-handle extrinsic parameter calibration and fusion method based on detachable and merging as described in claim 1, characterized in that, When the handle is determined to be in the merged mode, the rotational motion constraint results are subjected to residual fusion processing of measurements from the two IMU gyroscopes to obtain a high-precision rotational trajectory of the merged handle, specifically including: Extract the residuals of the second angular velocity measurement sequence and the first angular velocity measurement sequence; according to The joint residuals based on measurements from the first handle IMU gyroscope and the second handle IMU gyroscope were obtained. ;in, The first angular velocity measurement sequence is the residual of the first angular velocity measurement sequence; The second angular velocity measurement sequence based on the time offset is the residual of the second angular velocity measurement sequence; This refers to the time offset; The rotational extrinsic parameter in the aforementioned extrinsic parameter state; The second gyroscope on the second handle has zero bias; The first gyroscope of the first handle has zero bias; The cumulative B-spline is used for the handle assembly. Angular velocity at time t; k is a mathematical constant; This is the first term to be summed; This is the second summation term; The rotational motion constraint results are subjected to joint fusion processing of residuals from two IMU gyroscope measurements using the joint residuals. Based on the optimized control of the control points, external parameter states, time offset, and gyroscope zero bias of the combined handle rotation trajectory, the high-precision rotation trajectory in the combined mode is obtained.
7. The dual-handle extrinsic parameter calibration and fusion method based on detachable and merging as described in claim 1, characterized in that, Before performing unified observation control processing based on shared feature points on the visual observation constraint results, the method further includes: When the controller is determined to be in the merged mode, the state vector of the controller merged body is determined based on the pose, velocity, IMU zero bias, gravity vector, external parameter state and time offset of the controller merged body in the world coordinate system. Based on the state vector, the IMU pre-integration factor, visual reprojection factor, extrinsic factor, time offset factor, and prior factor after extrinsic motion association in each handle are all optimized using a sliding window-based factor graph to obtain an optimized factor graph.
8. The dual-handle extrinsic parameter calibration and fusion method based on detachable and merging as described in claim 7, characterized in that, When the handle is determined to be in the merged mode, the visual observation constraint results are subjected to unified observation control processing based on shared feature points to obtain the high-precision 6-DoF pose of the handle merged body, specifically including: Extract the shared feature points of the visual front end of each handle; Based on the shared feature points, and through the optimization factor map, the first IMU pre-integration of the first handle and the IMU pre-integration of the second handle, the visual observation constraint results, and the external parameter state constraints are fused. The observation data in the fusion result are uniformly controlled under the same map, and the high-precision 6-DoF pose of the handle merge is output.
9. The dual-handle extrinsic parameter calibration and fusion method based on detachable and merging as described in claim 1, characterized in that, If the controller is determined to be in the separated mode, then both the first controller and the second controller will be independently tracked and controlled, specifically including: If the handle is determined to be in the separation mode, then both the first handle and the second handle will be subjected to 6-DoF pose estimation calculation based on independent VI-SLAM to obtain the independent 6-DoF pose of each handle. Based on the independent 6-DoF pose, velocity, IMU zero bias, and gravity vector, the independent state vector of each handle is determined; Based on the independent state vector, independent tracking control is achieved for both the first handle and the second handle.
10. A dual-handle extrinsic parameter calibration and fusion system based on detachable and merging mechanisms, characterized in that, The dual-handle extrinsic calibration and fusion system is executable by at least one processor, which enables the at least one processor to execute a dual-handle extrinsic calibration and fusion method based on a splittable and mergeable method according to any one of claims 1-9.