Handle positioning method and device for sparse features

Abstract

The invention relates to a handle positioning method and device for sparse features, and the method comprises the steps: constructing a non-rigid multi-sensor joint state estimation model, and building a non-rigid connection through a camera and an IMU (Inertial Measurement Unit); performing joint optimization on the visual re-projection error and the IMU pre-integration error in the sliding window to obtain initial attitude estimation of the handle; based on the number of visible feature points, a geometric degradation state is analyzed through observability, a self-adaptive marginalization strategy of motion perception is established, and a sliding window is dynamically adjusted; based on a self-adaptive marginalization strategy, a prior quality monitoring and multi-hypothesis maintenance mechanism is implemented, initial attitude estimation is optimized, and an optimal handle attitude is output; equipment is realized based on the method. According to the method, high pose estimation precision can still be maintained in a complex interaction scene, long-term drift caused by inaccurate models is remarkably reduced, the survivability under the severe conditions of shielding, rapid rotation and the like is greatly improved, limited visual information in each frame is efficiently and accurately utilized, and the real-time operation capability of a system is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric digital data processing, and particularly relates to a handle positioning method and device for sparse features. BACKGROUND

[0002] With the continuous development of virtual reality technology towards high immersion and strong interaction, users have almost stringent requirements for the positioning accuracy, real-time performance and robustness of VR handles. Under this background, the tightly coupled positioning system based on visual-inertial odometry (VIO) has become a mainstream technical solution for realizing high-precision six-degree-of-freedom pose estimation, because it can fuse visual geometric constraints and dynamic information of an inertial measurement unit (IMU).

[0003] However, when the traditional VIO framework is directly applied to this specific scene of VR handles, its theoretical basis and engineering assumptions face severe challenges, including the following reasons:

[0004] (1) The core premise of the traditional VIO method is that the camera and the IMU are fixed on the same rigid body, and the relative pose (external parameter) between the two remains unchanged during system operation. However, in the VR system, the camera array is fixed on the head-mounted display device, while the IMU is built into the handle held by the user, i.e., the "camera-IMU" pair constitutes a dynamically changing non-rigid system, and thus the offline calibrated external parameter will quickly become invalid. If the rigid assumption is forcibly applied, it will introduce system errors that cannot be modeled, leading to significant drift in pose estimation;

[0005] (2) In order to meet the requirements of power consumption, volume and tracking, the VR handle usually only embeds several to tens of actively emitting infrared LEDs as visual feature points. Compared with the rich corner and texture features in natural scenes, the visual features provided by the handle are extremely sparse in quantity and lack uniqueness in quality, which makes it impossible to extract discriminative descriptors for cross-frame matching, and thus the feature point matching fails, the dense or semi-dense map cannot be constructed for repositioning, greatly weakening the observation ability and loop detection possibility of the system;

[0006] (3) The handle motion in the VR interactive scene has the characteristics of high speed, large acceleration and irregular pattern, which easily leads to image motion blur, making the already sparse feature point detection unstable or even lost;

[0007] (4) During the interaction, the handle is often obstructed by the user's hand, body or other objects, or due to rapid rotation, causing the camera to only observe one or two feature points. From the perspective of computer vision geometry, this is a typical P1P (single-point perspective) and P2P (two-point perspective) problem. Theoretical analysis shows that in these cases, the system state (especially the rotation and translation of the handle) has multiple unobservable degrees of freedom. Since visual constraints cannot provide effective information, the system will rely excessively on IMU integration, resulting in rapid accumulation of errors, scale drift, and a very high probability of positioning failure. Summary of the Invention

[0008] This invention addresses the problems existing in the prior art and provides a handle positioning method and device for sparse features. In environments where visual features are sparse, insignificant, and dynamically changing, it effectively solves the problem of positioning accuracy degradation under conditions of insufficient features by establishing a non-rigid multi-sensor fusion framework, sliding window optimization, and observability analysis mechanism.

[0009] The technical solution adopted in this invention is a handle positioning method for sparse features, the method comprising the following steps:

[0010] S1 constructs a non-rigid multi-sensor joint state estimation model, establishing a non-rigid connection between the camera and the handle's IMU;

[0011] S2 performs joint optimization of visual reprojection error and IMU pre-integration error within a sliding window to obtain the initial pose estimate of the handle;

[0012] S3 Based on the number of visible feature points, the geometric degradation state is analyzed through observability analysis, and a motion-aware adaptive edge-forming strategy is established accordingly to dynamically adjust the sliding window;

[0013] S4 Based on the adaptive edge-shifting strategy, a priori quality monitoring and multiple hypothesis maintenance mechanism are implemented to optimize the initial pose estimation and output the optimal handle pose.

[0014] Preferably, in S1, the non-rigid multi-sensor joint state estimation model satisfies,

[0015]

[0016] in, Let the rotation matrix of the handle be in the world coordinate system. It is a translation vector. For velocity vectors, and These are the deviation vectors of the accelerometer and gyroscope, respectively.

[0017] Preferably, in S2, the objective function for joint optimization is,

[0018]

[0019] in, For IMU pre-integration residuals, For IMU bias residuals, For visual reprojection residuals, and This is the covariance matrix corresponding to the IMU pre-integration residual and the visual reprojection residual. Let i and j be the set of keyframe indices within the sliding window, where i and j are keyframes respectively. Let i be the set of feature points in the i-th frame. For feature point indexes.

[0020] Preferably, the visual reprojection error is associated with a feature projection model, and the 2D pixel coordinates of the κ-th light point on the i-th frame camera image are obtained based on the feature projection model. ,

[0021]

[0022] in, For camera projection model, This refers to the calibration position of the feature point in the handle coordinate system. To measure noise in the image, and The camera's orientation.

[0023] Preferably, in S3, the adaptive edge-shifting strategy for motion perception is to define a motion irregularity metric. ,

[0024]

[0025] in, , and They are acceleration, jerk, and jerk-jerk respectively. , and They are respectively , and The corresponding weights For time windows.

[0026] Preferably, based on Update the sliding window size to,

[0027]

[0028] in, , These are the upper and lower limits of the window size. This indicates that the sensitivity parameter is being adjusted; and These are the minimum and maximum values ​​for measuring motion irregularity;

[0029] A state priority scoring function is established, and states to be marginalized are selected based on the scores; the state priority scoring function is associated with time scores, geometric scores, and motion sensitivity scores.

[0030] Preferably, in S4, a priori quality indicators are defined for priori quality monitoring. The priori quality indicators are:

[0031]

[0032] in, The number of residual terms corresponding to the prior constraints. Let i be the i-th prior residual vector. This is the prior information matrix generated after marginalization. It is the diagonal block corresponding to the i-th residual block in the inverse of the prior information matrix;

[0033] when or At that time, recalibrate, among which, The minimum prior quality threshold, It is an adjustable parameter. is the trace of the prior covariance matrix.

[0034] Preferably, in S4, the set of multiple motion hypotheses Each hypothesis Corresponding to different motion patterns and marginalized priors , and These are the prior information matrix and gradient vector for the m-th motion pattern, respectively.

[0035] Effective priors are synthesized through weighted synthesis.

[0036] Preferably, a priori weight decay strategy is set.

[0037]

[0038]

[0039]

[0040] in The decay rate parameter is adaptively adjusted according to motion irregularities. This is a moment of marginalization.

[0041] A handle positioning device for sparse features, the device comprising a processor and a memory:

[0042] The memory is used to store program code and transmit the program code to the processor;

[0043] The processor is used to execute the handle localization method for sparse features according to the instructions in the program code.

[0044] This invention relates to a method and device for handle localization based on sparse features. It constructs a non-rigid multi-sensor joint state estimation model, establishing a non-rigid connection between the camera and the handle's IMU. Within a sliding window, it jointly optimizes the visual reprojection error and the IMU pre-integration error to obtain an initial handle pose estimate. Based on the number of visible feature points, it analyzes the geometric degradation state through observability analysis and establishes an adaptive edge-shifting strategy for motion perception, dynamically adjusting the sliding window. Based on the adaptive edge-shifting strategy, it implements a priori quality monitoring and multiple hypothesis maintenance mechanisms to optimize the initial pose estimate and output the optimal handle pose. The device is implemented based on this method.

[0045] The beneficial effects of this invention are as follows:

[0046] (1) By constructing a non-rigid multi-sensor joint state estimation model, the relative motion between the handle and the head display is allowed and modeled, which fundamentally eliminates the systematic error caused by model mismatch, so that the positioning system can truly reflect and adapt to the non-rigid physical relationship of hand-head separation in VR use, thus maintaining high pose estimation accuracy in complex interactive scenarios and significantly reducing long-term drift caused by model inaccuracy.

[0047] (2) By bringing the observability analysis of P1P / P2P problems forward and integrating it into the state estimation process, the system can actively perceive its own geometric degradation state. When features are sparse, it can intelligently adjust the optimization strategy and rely more on the IMU to make up for the lack of visual information, thereby avoiding the problem that traditional methods will directly collapse or produce meaningless outputs when geometrically degraded, and greatly improving its survivability under harsh conditions such as occlusion and rapid rotation.

[0048] (3) Adopting a dynamic map point management strategy, feature points are treated as dynamic entities bound to the motion state of the handle for management, which greatly improves the utilization efficiency and consistency of feature constraints, avoids constraint conflicts caused by mistakenly treating dynamic points as static points, and enables the system to utilize the limited visual information in each frame more efficiently and accurately, adapting to the fast and irregular motion characteristics of the handle.

[0049] (4) Under the extremely demanding condition that each frame of visual observation contains only 1 to 2 light spots without texture or descriptor, it can still achieve sub-centimeter (millimeter) positioning accuracy, meet the stringent requirements of high-end VR applications for precise interaction, and solve the long-standing problem of sparse feature positioning in the industry.

[0050] (5) By introducing a motion-aware adaptive edge-off strategy, the sliding window optimization is dynamically associated with the irregularity measurement of real-time motion. The retention and edge-off strategies of historical states are intelligently determined according to the intensity of motion, ensuring high-precision output while effectively controlling computational complexity and ensuring the real-time operation capability of the system. Attached Figure Description

[0051] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0052] The present invention will be further described in detail below with reference to embodiments, but the scope of protection of the present invention is not limited thereto.

[0053] This invention relates to a handle localization method for sparse features, the method comprising the following steps:

[0054] (1) Construct a non-rigid multi-sensor joint state estimation model and establish a non-rigid connection between the camera and the handle's IMU;

[0055] (2) The visual reprojection error and IMU pre-integration error are jointly optimized within the sliding window to obtain the initial attitude estimate of the handle;

[0056] (3) Based on the number of visible feature points, the geometric degradation state is analyzed through observability analysis, and a motion-aware adaptive edge-forming strategy is established accordingly to dynamically adjust the sliding window;

[0057] (4) Based on the adaptive edge-out strategy, implement the prior quality monitoring and multiple hypothesis maintenance mechanism to optimize the initial attitude estimation and output the optimal handle attitude.

[0058] The steps are explained below with reference to specific implementation methods.

[0059] (1) Construct a non-rigid multi-sensor joint state estimation model and establish a non-rigid connection between the camera and the handle's IMU;

[0060] To be precise, the multiple cameras on the head-mounted device and the IMU on the handle are considered as a non-rigid connection system.

[0061] The non-rigid multi-sensor joint state estimation model satisfies...

[0062]

[0063] in, Let the rotation matrix of the handle be in the world coordinate system. It is a translation vector. For velocity vectors, and These are the deviation vectors of the accelerometer and gyroscope, respectively.

[0064] (2) The visual reprojection error and IMU pre-integration error are jointly optimized within the sliding window to obtain the initial attitude estimate of the handle;

[0065] The objective function for joint optimization is,

[0066]

[0067] in, For IMU pre-integration residuals, For IMU bias residuals, For visual reprojection residuals, and This is the covariance matrix corresponding to the IMU pre-integration residual and the visual reprojection residual. Let i and j be the set of keyframe indices within the sliding window, where i and j are keyframes respectively. Let i be the set of feature points in the i-th frame. For feature point indexes.

[0068] In this invention, a soft edge-out strategy is used to remove old states during edge-out, while retaining their constraint information to prevent the accumulation of linearization errors.

[0069] The visual reprojection error is associated with the feature projection model, and the 2D pixel coordinates of the κ-th light point on the i-th frame camera image are obtained based on the feature projection model. ,

[0070]

[0071] in, For camera projection model, This refers to the calibration position of the feature point in the handle coordinate system. To measure noise in the image, and The camera's orientation.

[0072] (3) Based on the number of visible feature points, the geometric degradation state is analyzed through observability analysis, and a motion-aware adaptive edge-forming strategy is established accordingly to dynamically adjust the sliding window;

[0073] To address the issue of insufficient points, this invention provides observability analysis and degenerate motion recognition for P1P / P2P localization, performing observability analysis for both single-point (P1P) and two-point (P2P) localization problems:

[0074] P1P problem: There is unobservability in 3 rotational degrees of freedom and 1 scale degree of freedom;

[0075] P2P problem: There is an unobservable subspace dimension with 2 degrees of freedom;

[0076] If a geometrically degenerate state exists, proceed with the next steps.

[0077] The motion-aware adaptive edge-shifting strategy is to define a motion irregularity metric. ,

[0078]

[0079] in, , and They are acceleration, jerk, and jerk-jerk respectively. , and They are respectively , and The corresponding weights For time windows.

[0080] based on Update the sliding window size to,

[0081]

[0082] in, , These are the upper and lower limits of the window size. This indicates that the sensitivity parameter is being adjusted; and These are the minimum and maximum values ​​for measuring motion irregularity;

[0083] A state priority scoring function is established, and states to be marginalized are selected based on the scores; the state priority scoring function is associated with time scores, geometric scores, and motion sensitivity scores.

[0084] Specifically, the state priority scoring function is:

[0085]

[0086] in, This is the i-th state;

[0087] Time rating , Let i be the time corresponding to the i-th state. For the current time, The time decay constant;

[0088] Geometric scoring ,

[0089] Motion sensitivity score Let r be the residual function, a be the motion variable, and ||·|| be the matrix norm. max It is the maximum value among the norms corresponding to all states;

[0090] , , These are the corresponding weighting coefficients;

[0091] Select states with scores below a preset value for marginalization, while retaining key constraint information.

[0092] (4) Based on the adaptive edge-out strategy, implement prior quality monitoring and multiple hypothesis maintenance mechanism to optimize the initial attitude estimation and output the optimal handle attitude;

[0093] Define prior quality indicators for prior quality monitoring. The prior quality indicators are:

[0094]

[0095] in, The number of residual terms corresponding to the prior constraints. Let i be the i-th prior residual vector. This is the prior information matrix generated after marginalization. It is the diagonal block corresponding to the i-th residual block in the inverse of the prior information matrix;

[0096] when or At that time, recalibrate, among which, The minimum prior quality threshold, It is an adjustable parameter. is the trace of the prior covariance matrix.

[0097] Multi-motion hypothesis set Each hypothesis Corresponding to different motion patterns and marginalized priors , and These are the prior information matrix and gradient vector for the m-th motion pattern, respectively.

[0098] Effective priors are synthesized through weighted synthesis.

[0099] Set the decay weights for the prior weight decay strategy over time.

[0100]

[0101] in The decay rate parameter is adaptively adjusted according to motion irregularities. At the marginalization moment; the effective prior constraint synthesis is attenuated:

[0102] The effective prior constraints are attenuated during synthesis:

[0103]

[0104]

[0105] After attenuation This serves as the final prior constraint for sliding window optimization.

[0106] The present invention also relates to a handle positioning device for sparse features, the device comprising a processor and a memory:

[0107] The memory is used to store program code and transmit the program code to the processor;

[0108] The processor is used to execute the handle localization method for sparse features according to the instructions in the program code.

[0109] To evaluate the performance of this method, a benchmark dataset containing real data from virtual reality controllers was constructed, including synchronized visual and inertial data and precise real pose information of the controllers. The data was collected in an indoor environment using PimaxCrystal Light, and the real pose information was obtained using the FZMotion motion capture system.

[0110] Table 1 Control Data Sheet

[0111]

[0112] As shown in Table 1, these controllers exhibit a wide range of motion velocities and angular velocities, with a maximum linear velocity of 7.302 m / s and a maximum angular velocity exceeding 3400 rpm, simulating realistic and challenging user interactions. The proposed method achieves millimeter-level accuracy in all sequences. The root mean square error (RMSE) of position estimation is consistently below 3.67 mm, with controller 04 performing best (left: 0.655 mm, right: 0.877 mm). Even under high-speed motion conditions, such as controller 01, the system maintains robust tracking performance with a maximum error below 51.754 mm. These results fully demonstrate the effectiveness and reliability of the non-rigid visual-inertial measurement unit (VIS) framework in practical virtual reality controller tracking scenarios.

[0113] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0117] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0118] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A handle positioning method for sparse features, characterized in that: The method includes the following steps: S1 constructs a non-rigid multi-sensor joint state estimation model, establishing a non-rigid connection between the camera and the handle's IMU; S2 performs joint optimization of visual reprojection error and IMU pre-integration error within a sliding window to obtain the initial pose estimate of the handle; S3 Based on the number of visible feature points, the geometric degradation state is analyzed through observability analysis, and a motion-aware adaptive edge-forming strategy is established accordingly to dynamically adjust the sliding window; S4 Based on the adaptive edge-shifting strategy, a priori quality monitoring and multiple hypothesis maintenance mechanism are implemented to optimize the initial pose estimation and output the optimal handle pose.

2. The handle positioning method for sparse features according to claim 1, characterized in that: In S1, the non-rigid multi-sensor joint state estimation model satisfies the following: ， in, Let the rotation matrix of the handle be in the world coordinate system. It is a translation vector. For velocity vector, and These are the deviation vectors of the accelerometer and gyroscope, respectively.

3. The handle positioning method for sparse features according to claim 2, characterized in that: In S2, the objective function for joint optimization is: ， in, For IMU pre-integration residuals, For IMU bias residuals, For visual reprojection residuals, and This is the covariance matrix corresponding to the IMU pre-integration residual and the visual reprojection residual. Let i and j be the set of keyframe indices within the sliding window, where i and j are keyframes respectively. Let i be the set of feature points in the i-th frame. For feature point indexes.

4. The handle positioning method for sparse features according to claim 3, characterized in that: The visual reprojection error is associated with the feature projection model, and the 2D pixel coordinates of the κ-th light point on the i-th frame camera image are obtained based on the feature projection model. , ， in, For camera projection model, This refers to the calibration position of the feature point in the handle coordinate system. To measure noise in the image, and The camera's orientation.

5. The handle positioning method for sparse features according to claim 1, characterized in that: In S3, the adaptive edge-shifting strategy for motion perception is to define a motion irregularity metric. , ， in, , and They are acceleration, jerk, and jerk-jerk respectively. , and They are respectively , and The corresponding weights For time windows.

6. The handle positioning method for sparse features according to claim 5, characterized in that: based on Update the sliding window size to, ， in, , These are the upper and lower limits of the window size. This indicates that the sensitivity parameter is being adjusted; and These are the minimum and maximum values ​​for measuring motion irregularity; A state priority scoring function is established, and states to be marginalized are selected based on the scores; the state priority scoring function is associated with time scores, geometric scores, and motion sensitivity scores.

7. The handle positioning method for sparse features according to claim 1, characterized in that: In S4, prior quality indicators are defined for prior quality monitoring. The prior quality indicators are: ， in, The number of residual terms corresponding to the prior constraints. Let i be the i-th prior residual vector. This is the prior information matrix generated after marginalization. It is the diagonal block corresponding to the i-th residual block in the inverse of the prior information matrix; when or At that time, recalibrate, among which, The minimum prior quality threshold, It is an adjustable parameter. is the trace of the prior covariance matrix.

8. The handle positioning method for sparse features according to claim 7, characterized in that: In S4, the set of multiple motion hypotheses Each hypothesis Corresponding to different motion patterns and marginalized priors , and These are the prior information matrix and gradient vector for the m-th motion pattern, respectively. Effective priors are synthesized through weighted synthesis.

9. The handle positioning method for sparse features according to claim 8, characterized in that: Set a prior weight decay strategy over time. ， ， ， in The decay rate parameter is adaptively adjusted according to motion irregularities. This is a moment of marginalization.

10. A handle positioning device for sparse features, characterized in that: The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the handle localization method for sparse features as described in any one of claims 1 to 9 according to instructions in the program code.

Images