Imu-based keyframe tracking method and apparatus, head-mounted device, and medium

Abstract

The application discloses an IMU-based key frame tracking method and device, a head-mounted device and a medium, and relates to the technical field of instant positioning. The method determines a first quaternion corresponding to a current frame collected by a head-mounted device according to IMU data of the head-mounted device; filters a target key frame from a plurality of key frames of a map in which the head-mounted device is located according to the first quaternion; determines matching feature points through a first bag-of-words model of the current frame and a second bag-of-words model of the target key frame; locates a current pose of the head-mounted device based on the matching feature points to obtain a number of deleted outliers; if the number of deleted outliers is less than a preset number, it is determined that key frame tracking is successful, and the current pose is set as a real pose of the head-mounted device. The technical scheme can reduce the amount of calculation in the key frame tracking process, thereby improving the output real-time performance of the tracking and positioning information of the head-mounted device.

Description

Technical Field

[0001] This application belongs to the field of real-time positioning technology, and in particular relates to a keyframe tracking method, device, head-mounted device, and computer-readable storage medium based on IMU. Background Technology

[0002] In the fields of VR (Virtual Reality) and AR (Augmented Reality), when tracking in indoor usage scenarios using SLAM (Simultaneous Localization and Mapping) algorithms, keyframe matching is required. Common keyframe tracking schemes include tracking constant-velocity motion models, tracking reference keyframes, and relocalization.

[0003] However, in traditional keyframe tracking schemes, even with an established map atlas, the large number of keyframes in the atlas necessitates pose calculation based on a vast number of keyframes, followed by pose optimization and keyframe matching verification. This process involves significant computational complexity. In summary, existing keyframe tracking schemes are time-consuming, resulting in low real-time performance of pose output. Summary of the Invention

[0004] The main objective of this application is to provide an IMU-based keyframe tracking method, apparatus, head-mounted device, and computer-readable storage medium. The aim is to reduce the computational load during keyframe matching, thereby improving the accuracy and real-time performance of tracking and positioning information.

[0005] To achieve the above objectives, this application provides an IMU-based keyframe tracking method, which includes:

[0006] Based on the IMU data of the head-mounted device, determine the first quaternion corresponding to the current frame collected by the head-mounted device;

[0007] Based on the first quaternion, the target keyframe is selected from multiple keyframes of the map where the head-mounted device is located.

[0008] Matching feature points are determined by comparing the first bag-of-words model of the current frame with the second bag-of-words model of the target keyframe;

[0009] Based on the matched feature points, the current pose of the head-mounted device is located, and the number of outliers to be deleted is obtained;

[0010] If the number of deleted outliers is less than a preset number, the keyframe tracking is determined to be successful, and the current pose is set to the actual pose of the head-mounted device.

[0011] Optionally, the step of determining the first quaternion corresponding to the current frame acquired by the head-mounted device based on the IMU data of the head-mounted device includes:

[0012] Extract gyroscope data from the IMU data;

[0013] Integrate the gyroscope data to obtain the first quaternion corresponding to the current frame.

[0014] Optionally, the step of filtering out the target keyframe from multiple keyframes of the map where the head-mounted device is located based on the first quaternion includes:

[0015] Traverse each of the keyframes, and calculate the rotation angle difference and vector difference between each keyframe and the current frame based on the first quaternion and the second quaternion of each keyframe;

[0016] Keyframes whose rotation angle difference and vector difference are less than preset angle difference threshold and preset vector difference threshold, respectively, are designated as candidate keyframes;

[0017] The candidate keyframe with the smallest difference in rotation angle among all the candidate keyframes is set as the target keyframe.

[0018] Optionally, the step of determining matching feature points using the first bag-of-words model of the current frame and the second bag-of-words model of the target keyframe includes:

[0019] The feature points in the current frame are converted into a first bag-of-words model using a preset bag-of-words algorithm, wherein the first bag-of-words model includes the feature vectors corresponding to each feature point in the current frame.

[0020] Multiple matching feature points are obtained by matching the feature vectors in the first bag-of-words model with the feature vectors in the second bag-of-words model of the target keyframe.

[0021] Optionally, the step of locating the current pose of the head-mounted device based on the matched feature points and obtaining the number of outliers to be deleted includes:

[0022] The pose of the previous frame corresponding to the current frame is used as the initial pose, and the initial pose is optimized by the matching feature points to obtain the current pose of the head-mounted device.

[0023] The number of outliers to be deleted is determined based on the current pose and each of the matching feature points.

[0024] Optionally, after the step of locating the current pose of the head-mounted device based on the matching feature points and obtaining the number of outliers to be deleted, the method further includes:

[0025] If the number of deleted outliers is not less than the preset number, then keyframe tracking is determined to have failed.

[0026] The target keyframe corresponding to the current frame is tracked using a preset constant velocity motion model algorithm.

[0027] When the target keyframe is successfully tracked, the true pose of the head-mounted device is located based on the target keyframe.

[0028] Optionally, the IMU-based keyframe tracking method further includes:

[0029] If the target keyframe tracking fails, each keyframe will be traversed to obtain the number of feature points that are successfully matched between each keyframe and the current frame.

[0030] Select keyframes with a number of feature points greater than a preset threshold as candidate keyframes;

[0031] The head-mounted device is located using each of the candidate keyframes to obtain the initial pose corresponding to each candidate keyframe.

[0032] Each initial pose is optimized to obtain the true pose of the head-mounted device.

[0033] In addition, to achieve the above objectives, this application also provides an IMU-based keyframe tracking device, which is applied to a head-mounted device;

[0034] The IMU-based keyframe tracking device includes:

[0035] The quaternion calculation module is used to determine the first quaternion corresponding to the current frame collected by the head-mounted device based on the IMU data of the head-mounted device;

[0036] The keyframe filtering module is used to filter out target keyframes from multiple keyframes of the map where the head-mounted device is located based on the first quaternion.

[0037] The feature point matching module is used to determine matching feature points by comparing the first bag-of-words model of the current frame with the second bag-of-words model of the target keyframe;

[0038] The pose localization module is used to locate the current pose of the head-mounted device based on the matched feature points and obtain the number of deleted outliers;

[0039] The tracking and judgment module is used to determine that the keyframe tracking is successful if the number of deleted external points is less than a preset number, and to set the current pose as the true pose of the head-mounted device.

[0040] In addition, to achieve the above objectives, this application also provides a head-mounted device, the head-mounted device comprising: a memory, a processor, and an IMU-based keyframe tracking program stored in the memory and executable on the processor, wherein the IMU-based keyframe tracking program, when executed by the processor, implements the steps of the IMU-based keyframe tracking method as described above.

[0041] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing an IMU-based keyframe tracking program, which, when executed by a processor, implements the steps of the IMU-based keyframe tracking method as described above.

[0042] This application proposes an IMU-based keyframe tracking method, apparatus, head-mounted device, and computer-readable storage medium. First, based on the IMU data of the head-mounted device, a first quaternion corresponding to the current frame acquired by the head-mounted device is determined. Then, based on the first quaternion, a target keyframe is selected from multiple keyframes on the map where the head-mounted device is located. Matching feature points are determined using a first bag-of-words model of the current frame and a second bag-of-words model of the target keyframe. Based on the matching feature points, the current pose of the head-mounted device is located, and the number of outliers to be deleted is obtained. If the number of outliers to be deleted is less than a preset number, keyframe tracking is considered successful, and the current pose is set to the true pose of the head-mounted device.

[0043] Thus, in the technical solution of this application embodiment, when performing positioning and tracking on a head-mounted device, the key frame at that angle is quickly filtered out mainly by the quaternion of the current frame. Compared with the traditional key frame matching scheme, the number of key frames that need to be calculated is effectively reduced, thereby reducing the computational complexity of the SLAM algorithm, reducing the amount of matching calculation, and enabling the key frame to be determined faster and more timely. This reduces the positioning time of the head-mounted device and improves the real-time performance of pose output while ensuring positioning accuracy. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the hardware operating environment of the head-mounted device involved in the embodiments of this application;

[0045] Figure 2 This is a flowchart illustrating the steps of the first embodiment of the IMU-based keyframe tracking method of this application;

[0046] Figure 3This is a schematic diagram illustrating the overall inventive concept of the IMU-based keyframe tracking method of this application;

[0047] Figure 4 This is a flowchart illustrating the steps of the second embodiment of the IMU-based keyframe tracking method of this application;

[0048] Figure 5 This is a schematic diagram of the functional modules of an embodiment of the IMU-based keyframe tracking device of this application.

[0049] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0050] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0051] Reference Figure 1 , Figure 1 This is a schematic diagram of the hardware operating environment of the head-mounted device involved in the embodiments of this application.

[0052] It should be noted that the head-mounted device in this application embodiment can be a camera device, display device, and positioning device that executes the keyframe tracking method based on IMU (Inertial Measurement Unit) in this application. In addition, the terminal device in this application embodiment can also be a computing unit such as a PC, smartphone, or tablet connected to the above-mentioned terminal devices.

[0053] It should be understood that, based on different design needs of actual applications, the terminal device in the embodiments of this application may of course also be other devices or device modules. For example, the terminal device may specifically be a data storage control terminal or a portable computer, or other mobile or non-mobile terminal devices.

[0054] like Figure 1As shown, the head-mounted device may include: a processor 1001, such as a CPU; a communication bus 1002; a user interface 1003; a network interface 1004; and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0055] Those skilled in the art will understand that Figure 1 The head-mounted device structure shown does not constitute a limitation on the head-mounted device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0056] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an IMU-based keyframe tracking program.

[0057] exist Figure 1 In the terminal shown, network interface 1004 is mainly used to connect to the backend server and communicate data with it; user interface 1003 is mainly used to connect to the client and communicate data with it; and processor 1001 can be used to call the IMU-based keyframe tracking program stored in memory 1005 and perform the following operations:

[0058] Based on the IMU data of the head-mounted device, determine the first quaternion corresponding to the current frame collected by the head-mounted device;

[0059] Based on the first quaternion, the target keyframe is selected from multiple keyframes of the map where the head-mounted device is located.

[0060] Matching feature points are determined by comparing the first bag-of-words model of the current frame with the second bag-of-words model of the target keyframe;

[0061] Based on the matched feature points, the current pose of the head-mounted device is located, and the number of outliers to be deleted is obtained;

[0062] If the number of deleted outliers is less than a preset number, the keyframe tracking is determined to be successful, and the current pose is set to the actual pose of the head-mounted device.

[0063] Optionally, the processor 1001 can also be used to call the IMU-based keyframe tracking program stored in the memory 1005 and perform the following operations:

[0064] Extract gyroscope data from the IMU data;

[0065] Integrate the gyroscope data to obtain the first quaternion corresponding to the current frame.

[0066] Optionally, the processor 1001 can also be used to call the IMU-based keyframe tracking program stored in the memory 1005 and perform the following operations:

[0067] Traverse each of the keyframes, and calculate the rotation angle difference and vector difference between each keyframe and the current frame based on the first quaternion and the second quaternion of each keyframe;

[0068] Keyframes whose rotation angle difference and vector difference are less than preset angle difference threshold and preset vector difference threshold, respectively, are designated as candidate keyframes;

[0069] The candidate keyframe with the smallest difference in rotation angle among all the candidate keyframes is set as the target keyframe.

[0070] Optionally, the processor 1001 can also be used to call the IMU-based keyframe tracking program stored in the memory 1005 and perform the following operations:

[0071] The feature points in the current frame are converted into a first bag-of-words model using a preset bag-of-words algorithm, wherein the first bag-of-words model includes the feature vectors corresponding to each feature point in the current frame.

[0072] Multiple matching feature points are obtained by matching the feature vectors in the first bag-of-words model with the feature vectors in the second bag-of-words model of the target keyframe.

[0073] Optionally, the processor 1001 can also be used to call the IMU-based keyframe tracking program stored in the memory 1005, and perform the following operations before loading the map atlas that matches the image data acquired by the head-mounted device:

[0074] The pose of the previous frame corresponding to the current frame is used as the initial pose, and the initial pose is optimized by the matching feature points to obtain the current pose of the head-mounted device.

[0075] The number of outliers to be deleted is determined based on the current pose and each of the matching feature points.

[0076] Optionally, the processor 1001 can also be used to call the IMU-based keyframe tracking program stored in the memory 1005, and after the steps of locating the current pose of the head-mounted device based on the matching feature points and obtaining the number of outliers to be deleted, it further performs the following operations:

[0077] If the number of deleted outliers is not less than the preset number, then keyframe tracking is determined to have failed.

[0078] The target keyframe corresponding to the current frame is tracked using a preset constant velocity motion model algorithm.

[0079] When the target keyframe is successfully tracked, the true pose of the head-mounted device is located based on the target keyframe.

[0080] Optionally, the processor 1001 can also be used to call the IMU-based keyframe tracking program stored in the memory 1005 and perform the following operations:

[0081] If the target keyframe tracking fails, each keyframe will be traversed to obtain the number of feature points that are successfully matched between each keyframe and the current frame.

[0082] Select keyframes with a number of feature points greater than a preset threshold as candidate keyframes;

[0083] The head-mounted device is located using each of the candidate keyframes to obtain the initial pose corresponding to each candidate keyframe.

[0084] Each initial pose is optimized to obtain the true pose of the head-mounted device.

[0085] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the IMU-based keyframe tracking method of this application. In the first embodiment of the IMU-based keyframe tracking method of this application, the method includes:

[0086] Step S10: Determine the first quaternion corresponding to the current frame collected by the head-mounted device based on the IMU data of the head-mounted device;

[0087] In this embodiment of the application, the head-mounted device may be an AR glasses, VR glasses, or VR helmet, etc., which has a built-in IMU unit. Specifically, the IMU unit includes a gyroscope, and the first quaternion corresponding to the current frame can be calculated by integrating the gyroscope data.

[0088] Step S20: Based on the first quaternion, select the target keyframe from multiple keyframes of the map where the head-mounted device is located.

[0089] Additionally, it should be noted that the storage unit of the head-mounted device in this embodiment has pre-stored a map atlas library composed of multiple scene map atlases. When the user inputs SLAM commands, feature point matching is first performed between the image data collected by the head-mounted device and the map atlases in the map atlas library to obtain the map atlas that best matches the image data. This map atlas can be used in the SLAM algorithm for tracking and locating the head-mounted device. This map atlas includes keyframes corresponding to multiple viewpoints. The technical solution in this embodiment requires selecting the target keyframe that is closest to the current frame from each of the keyframes for locating the head-mounted device.

[0090] For example, in the technical solution of this embodiment, the rotation angle difference and vector difference between the first quaternion and the quaternion of each key frame are calculated, thereby selecting the key frame with the smallest difference as the target key frame.

[0091] Step S30: Determine matching feature points by comparing the first bag-of-words model of the current frame with the second bag-of-words model of the target keyframe;

[0092] In this embodiment, the first step is to determine the corresponding bag-of-words model for feature points based on the image data of the current frame. The bag-of-words model is used to vectorize the current frame; that is, it includes multiple feature point vectors corresponding to the current frame. Similarly, the target keyframe also has multiple corresponding feature point vectors. During the matching process, the feature point vectors in the two bag-of-words models are matched sequentially to obtain successfully matched feature point vectors, thereby determining the successfully matched feature points.

[0093] Step S40: Based on the matching feature points, locate the current pose of the head-mounted device and obtain the number of outliers to be deleted;

[0094] In this embodiment, it should be noted that after determining the matching feature points, the correspondence between 3D and 2D points can be solved using a preset PNP (Perspective-n-Point) algorithm, and the current pose of the head-mounted device can be obtained by optimizing the 3D-2D reprojection error. Furthermore, during the pose localization process described above, each matching feature point needs to be screened and deleted, categorized into outer points and inner points. Inner points are the feature points used for pose localization, while outer points are the feature points that need to be discarded or deleted, thus obtaining the number of outer points to be deleted.

[0095] Step S50: If the number of deleted external points is less than a preset number, it is determined that the keyframe tracking is successful, and the current pose is set to the true pose of the head-mounted device.

[0096] In this embodiment, the success of keyframe tracking is primarily determined by the number of outliers deleted. This is because a larger number of outliers deleted during pose localization via feature point matching indicates lower accuracy of the selected matching feature points. This means the feature point matching strategy using the target keyframe is ill-suited to the current tracking and localization process, concluding that the target keyframe selection has failed. In this case, the target keyframe matching should be performed again.

[0097] On the other hand, the smaller the number of deleted outliers and the less than the preset number, the higher the accuracy of the matched feature points, which meets the tracking and positioning requirements, and the target keyframe is successfully tracked. The preset number can be set according to the positioning accuracy requirements in actual applications; a higher preset number results in lower positioning accuracy, and a lower preset number results in higher positioning accuracy.

[0098] In another feasible embodiment, the success of keyframe tracking can be measured by the number of inliers determined during pose localization. Specifically, an inlier number threshold (e.g., 10) can be set. If the number of inliers in the matched feature points is greater than or equal to 10, the keyframe tracking is considered successful; if the number of inliers is less than 10, the keyframe tracking is considered unsuccessful.

[0099] It should be noted that after keyframe tracking fails, other preset traditional keyframe tracking methods can be used to perform keyframe tracking, thereby ensuring that the target keyframe with the required accuracy can be tracked when keyframe tracking through IMU data fails. Therefore, the technical solution of this application embodiment can combine the IMU-based keyframe tracking method proposed in this application with other traditional keyframe tracking methods to improve the real-time performance of the output positioning information while ensuring the accuracy of the output positioning information.

[0100] Furthermore, in a feasible embodiment, the step of determining the first quaternion corresponding to the current frame acquired by the head-mounted device based on the IMU data of the head-mounted device may include:

[0101] Step S11: Extract gyroscope data from the IMU data;

[0102] Step S12: Integrate the gyroscope data to obtain the first quaternion corresponding to the current frame.

[0103] In this embodiment, the gyroscope data from the IMU (Integrated Device Unit) on the head-mounted device is used to calculate the quaternion of the current frame. A quaternion is a hypercomplex number; it can be understood that a quaternion consists of a real number *a* plus three imaginary units *i*, *j*, and *k*, and they have the following relationship: *i*... 2=j2=k2=-1, i°=j°=k°=1, each quaternion is a linear combination of 1, i, j and k, that is, a quaternion can generally be represented as a+bi+cj+dk, where a, b, c and d are real numbers.

[0104] In this embodiment, the geometric meaning of i, j, and k can be understood as a rotation, where i rotation represents a rotation from the positive Z-axis to the positive Y-axis in the plane where the Z-axis intersects the Y-axis, j rotation represents a rotation from the positive X-axis to the positive Z-axis in the plane where the X-axis intersects the Z-axis, and k rotation represents a rotation from the positive Y-axis to the positive X-axis in the plane where the Y-axis intersects the X-axis. -i, -j, and -k represent the opposite rotations of i, j, and k, respectively. Therefore, the first quaternion of the current frame can be used to characterize the current pose of the head-mounted device. Specifically, the IMU data consists of two parts: an accelerometer and a gyroscope. The gyroscope data is used to represent the angular velocity of rotation around the x, y, and z axes of the IMU coordinate system. In this embodiment, the gyroscope data can be integrated to obtain the first quaternion. The expression is as follows:

[0105]

[0106] Where, q 当前帧 It is the first quaternion. It is a real number in the first quaternion. and It is an imaginary number in the first quaternion.

[0107] Furthermore, in a feasible embodiment, step S20, the step of filtering out the target keyframe from multiple keyframes of the map where the head-mounted device is located based on the first quaternion, may include:

[0108] Step S21: Traverse each key frame and calculate the rotation angle difference and vector difference between each key frame and the current frame based on the first quaternion and the second quaternion of each key frame;

[0109] Step S22: Set key frames whose rotation angle difference and vector difference are less than preset angle difference threshold and preset vector difference threshold, respectively, as candidate key frames;

[0110] Step S23: Set the candidate keyframe with the smallest rotation angle difference among all the candidate keyframes as the target keyframe.

[0111] This application provides a method for filtering target keyframes based on the first quaternion of the current frame and the second quaternion of each keyframe. Specifically, the quaternion can generally be represented as a+bi+cj+dk, where a, b, c, and d are real numbers.

[0112] In this embodiment, the geometric meaning of i, j, and k can be understood as a rotation, where i rotation represents a rotation from the positive Z-axis to the positive Y-axis in the plane where the Z-axis intersects the Y-axis, j rotation represents a rotation from the positive X-axis to the positive Z-axis in the plane where the X-axis intersects the Z-axis, and k rotation represents a rotation from the positive Y-axis to the positive X-axis in the plane where the Y-axis intersects the X-axis. -i, -j, and -k represent the opposite rotations of i, j, and k, respectively. Therefore, the quaternion of the current frame can be used to characterize the current pose of the head-mounted device. Specifically, the IMU data consists of two parts: an accelerometer and a gyroscope. The gyroscope data is used to represent the angular velocity of rotation around the x, y, and z axes of the IMU coordinate system. In this embodiment, the gyroscope data can be integrated to obtain the first quaternion. The expression for the first quaternion is as follows:

[0113]

[0114] Where, q 当前帧 It is the first quaternion. It is a real number in the first quaternion. and It is an imaginary number in the first quaternion.

[0115] For example, the second quaternion expression of the keyframe is as follows:

[0116]

[0117] Where q 关键帧 It is the second quaternion. For a real number in the second quaternion, as well as It is an imaginary number in the second quaternion.

[0118] Furthermore, in the embodiments of this application, under the same coordinate system, the angle difference can be calculated using the real numbers in the first quaternion and the second quaternion, and the vector difference can be calculated using the imaginary numbers in the first quaternion and the second quaternion.

[0119] For example, the expression for calculating the angle difference is:

[0120] The expression for calculating the vector difference is:

[0121]

[0122] After calculating the rotation angle difference and vector difference between each keyframe and the current frame, keyframes whose rotation angle difference and vector difference are both less than the preset angle difference threshold and preset vector difference can be selected to obtain candidate keyframes. Finally, based on the rotation angle difference, the target keyframe with the highest quaternion similarity to the current frame among the candidate keyframes is selected, achieving rapid selection of target keyframes. The technical solution of this application embodiment does not require pose calculation for each keyframe separately, nor does it require a large number of tedious and complex calculations, effectively reducing the computational resource consumption and computation time of the positioning and tracking process, and improving the real-time performance of the positioning information output by the head-mounted device.

[0123] Further, step S30, the step of determining matching feature points using the first bag-of-words model of the current frame and the second bag-of-words model of the target keyframe, may include:

[0124] Step S31: Using a preset bag-of-words algorithm, the feature points in the current frame are converted into a first bag-of-words model, wherein the first bag-of-words model includes the feature vectors corresponding to each feature point in the current frame;

[0125] Step S32: Perform feature matching between the feature vectors in the first bag-of-words model and the second bag-of-words model of the target keyframe to obtain multiple matching feature points.

[0126] In this embodiment, the feature vectors in the first bag-of-words model are sequentially matched with the feature vectors in the bag-of-words model corresponding to the target keyframe. If the difference between the two feature vectors is less than a preset vector threshold, the feature vector in the bag-of-words model corresponding to the keyframe can be considered a successfully matched feature vector with the feature vector in the feature point bag-of-words model, thus obtaining the successfully matched feature point corresponding to the target keyframe. In this embodiment, the number of successfully matched feature points represents the similarity between the target keyframe and the current frame.

[0127] It should be noted that the preset bag-of-words algorithm can be the ORB (Oriented FAST and Rotated) image feature extraction algorithm. The ORB algorithm extracts corner points from image data as feature points because corner points, as edge feature points (i.e., feature vectors) of objects in the image data, can effectively represent the features of the image data. Thus, the first bag-of-words model is formed based on the feature vectors corresponding to the edge feature points. Specifically, the FAST (Features from accelerated segment test) algorithm is used to extract corner points as feature points, mainly based on the following principle: if a pixel has a sufficient number of neighboring pixels with significantly different pixel values, then that pixel is likely a corner point. Similarly, the second bag-of-words model of the target keyframe can also be pre-calculated using the same method.

[0128] In this embodiment of the application, after calculating the first bag-of-words model corresponding to the current frame, the feature vectors in the first bag-of-words model and the feature vectors in the second bag-of-words model corresponding to the target keyframe are matched sequentially to obtain multiple successfully matched feature points.

[0129] Further, in step S40, the step of locating the current pose of the head-mounted device based on the matched feature points and obtaining the number of outliers to be deleted may include:

[0130] Step S41: Take the pose of the previous frame corresponding to the current frame as the initial pose, and optimize the initial pose using the matching feature points to obtain the current pose of the head-mounted device.

[0131] Step S42: Determine the number of outliers to be deleted based on the current pose and each of the matching feature points.

[0132] Reference Figure 3 In the technical solution of this application embodiment, IMU data is first input, then gyroscope data is extracted, and then the quaternion of the current frame is calculated. Then, the target key frame with the highest quaternion similarity is selected and the feature points are matched by the bag-of-words model. Then, the pose optimization solution can be performed based on each of the matched feature points. In the process of performing pose optimization solution by matching feature points, a certain number of outliers need to be discarded. Based on the number of outliers discarded and the outlier discarding strategy (the preset number of outliers), the tracking success of the current frame is determined.

[0133] For example, in this embodiment, the pose of the previous frame is used as the current pose. The reprojection error is calculated and optimized using the established correspondence between 2D and 3D points corresponding to the matched feature points and the PNP algorithm, ultimately solving for the current pose of the head-mounted device. The method described above for solving the pose based on matched feature points, the PNP algorithm, and the reprojection error is a relatively mature technical solution in the field and will not be elaborated upon here. Furthermore, based on the pose optimization results, it is determined which points among the matched feature points are outliers. If an outlier is identified, it is deleted, and the number of deleted outliers is determined. If an inlier is identified, a counter is incremented, and the number of inliers is obtained. It should be noted that the number of inliers can also be used as one of the criteria for determining whether keyframe tracking is successful. For example, when the number of inliers is greater than a preset inlier threshold (e.g., 10), keyframe tracking can be determined to be successful, and the current pose can be used as the true pose of the head-mounted device and output.

[0134] In this embodiment, a map set matching the image data acquired by the head-mounted device is first loaded; the current frame quaternion corresponding to the current frame is determined based on the IMU gyroscope data corresponding to the head-mounted device; the feature point bag-of-words model corresponding to the current frame is matched with the keyframes in the map set to determine the map keyframes and their corresponding keyframe quaternions; when the difference between the current frame quaternion and the keyframe quaternion is less than a preset threshold, the head-mounted device is tracked and located based on the map keyframes, and the current device pose is output.

[0135] Thus, in the technical solution of this application embodiment, when performing positioning and tracking on a head-mounted device, the key frame at that angle is quickly filtered out mainly by the quaternion of the current frame. Compared with the traditional key frame matching scheme, this effectively reduces the number of key frames that need to be calculated, thereby reducing the computational complexity of the SLAM algorithm, reducing the amount of matching calculation, and enabling the key frame to be determined faster and more timely. This reduces the positioning time of the head-mounted device, thereby improving the real-time performance of pose output while ensuring positioning accuracy under the limited computing power of the embedded platform.

[0136] Furthermore, based on the first embodiment of the IMU-based keyframe tracking method of this application described above, a second embodiment of the IMU-based keyframe tracking method of this application is proposed.

[0137] In this embodiment, after step S40, which involves locating the current pose of the head-mounted device based on the matching feature points and obtaining the number of outliers to be deleted, the method may further include:

[0138] Step S41: If the number of deleted outer points is not less than the preset number, then the keyframe tracking is determined to have failed.

[0139] Step S42: Track the target keyframe corresponding to the current frame using a preset constant velocity motion model algorithm;

[0140] Step S42: When the target keyframe is successfully tracked, the true pose of the head-mounted device is located based on the target keyframe.

[0141] In this embodiment, to further improve the execution efficiency of the SLAM algorithm, when keyframe tracking fails due to the IMU data-based quaternion calculation method described in this application, it is necessary to combine the constant velocity motion model algorithm from traditional technical solutions to track the target keyframe. First, after deleting an outlier equal to or greater than the preset number, it can be determined that the method of selecting keyframes through quaternion matching has failed. Therefore, a preset constant velocity motion model algorithm needs to be invoked to track the target keyframe closest to the current frame and continue the tracking and positioning of the head-mounted device, outputting the true pose.

[0142] For example, step S70 includes: first, updating the pose of the previous frame; if the camera on the head-mounted device is a stereo RGB-D camera, adding 3D points that have not been created as map points (the stereo camera can obtain depth information) as temporary map points to the map; obtaining the pose of the current frame according to the constant velocity motion model; then performing projection matching based on the map points corresponding to the feature points of the previous frame, and returning the number of successfully matched feature points; if the number of successfully matched feature points is lower than a preset threshold, expanding the search radius; if the number of successfully matched feature points is still lower than the preset threshold, determining that the target keyframe tracking has failed; if the number of successfully matched feature points is not lower than the preset threshold, optimizing the pose of the current frame based on the successfully matched feature points; deleting outliers, and returning whether the matching was successful based on the relationship between the number of remaining inliers matched and the preset number. It should be noted that the step of tracking the target keyframe corresponding to the current frame using the preset constant velocity motion model algorithm is only one feasible implementation method and is not intended to limit the implementation details. Those skilled in the art can use other mature constant velocity motion model algorithms to track and obtain the target keyframe, which is not limited here.

[0143] Furthermore, it can be understood that when target keyframe tracking fails, keyframe tracking can also be performed using a relocation method. The IMU-based keyframe tracking method may further include:

[0144] Step S43: If the target keyframe tracking fails, then each keyframe will be traversed to obtain the number of feature points that are successfully matched between each keyframe and the current frame.

[0145] Step S44: Select key frames with a number of feature points greater than a preset threshold number of feature points and set them as candidate key frames;

[0146] Step S45: The head-mounted device is located using each of the candidate keyframes to obtain the initial pose corresponding to each candidate keyframe.

[0147] Step S46: Perform pose optimization on each of the initial poses to obtain the true pose of the head-mounted device.

[0148] In this embodiment of the application, it should be noted that step S43 can be executed either after step S40 or after step S42. That is to say, the relocation method provided in this embodiment of the application can be executed either after the failure of the keyframe tracking method based on IMU or after the failure of the keyframe tracking method based on constant velocity motion model. In addition, those skilled in the art can also adaptively adjust the execution order of these two traditional keyframe tracking methods according to actual needs, and can also execute them repeatedly or in a loop until the target keyframe that best matches the current frame is obtained.

[0149] For example, steps S43 to S46 include: First, after the target keyframe tracking fails, camera relocalization is triggered; traversing each keyframe of the current map, the set of keyframes corresponding to the current frame and the number of successfully matched feature points corresponding to each keyframe are found through the bag-of-words model; obtaining a preset feature point number threshold, filtering each keyframe, and obtaining a set of candidate keyframes with a feature point number greater than the feature point number threshold; based on the successfully matched feature points between each candidate keyframe and the current frame, the camera pose is iteratively estimated using the RANSAC (Random Sample Consensus) algorithm + EPNP (Non-Iterative PnP algorithm) to obtain the initial pose corresponding to each candidate keyframe; and iteratively optimizing each initial pose by calculating the reprojection error corresponding to each initial pose, finally obtaining the true pose with the smallest reprojection error.

[0150] It should be noted that the step of tracking the target keyframe and pose corresponding to the current frame through the relocation method is only one feasible implementation method and is not intended to limit the implementation details. Those skilled in the art can use other mature relocation algorithms to track and obtain the target keyframe, and no restrictions are imposed here.

[0151] This application provides a method for keyframe matching and tracking after keyframe matching and selection using IMU data and quaternions fails. This method combines a constant velocity motion model and a relocation method to fill the keyframe gap after tracking failure in the keyframe matching scheme using IMU data and quaternions. It improves the entire process of tracking and positioning head-mounted devices, enhances the adaptability of the IMU-based keyframe tracking method, and outputs the current pose data of the head-mounted device with higher accuracy, providing users with a better VR / AR device experience.

[0152] Furthermore, this application also provides an IMU-based keyframe tracking device, please refer to... Figure 5 , Figure 5 This is a functional block diagram of an embodiment of the IMU-based keyframe tracking device of this application, as shown below. Figure 5 As shown, the IMU-based keyframe tracking device of this application includes:

[0153] The quaternion calculation module 10 is used to determine the first quaternion corresponding to the current frame collected by the head-mounted device based on the IMU data of the head-mounted device;

[0154] The keyframe filtering module 20 is used to filter out target keyframes from multiple keyframes of the map where the head-mounted device is located based on the first quaternion.

[0155] The feature point matching module 30 is used to determine matching feature points by comparing the first bag-of-words model of the current frame with the second bag-of-words model of the target keyframe;

[0156] The pose localization module 40 is used to locate the current pose of the head-mounted device based on the matching feature points and obtain the number of deleted outliers;

[0157] The tracking and judgment module 50 is used to determine that the keyframe tracking is successful if the number of deleted external points is less than a preset number, and to set the current pose as the true pose of the head-mounted device.

[0158] Optionally, the quaternion calculation module 10 is used for:

[0159] Extract gyroscope data from the IMU data;

[0160] Integrate the gyroscope data to obtain the first quaternion corresponding to the current frame.

[0161] Optionally, the keyframe filtering module 20 is used for:

[0162] Traverse each of the keyframes, and calculate the rotation angle difference and vector difference between each keyframe and the current frame based on the first quaternion and the second quaternion of each keyframe;

[0163] Keyframes whose rotation angle difference and vector difference are less than preset angle difference threshold and preset vector difference threshold, respectively, are designated as candidate keyframes;

[0164] The candidate keyframe with the smallest difference in rotation angle among all the candidate keyframes is set as the target keyframe.

[0165] Optionally, the feature point matching module 30 is used for:

[0166] The feature points in the current frame are converted into a first bag-of-words model using a preset bag-of-words algorithm, wherein the first bag-of-words model includes the feature vectors corresponding to each feature point in the current frame.

[0167] Multiple matching feature points are obtained by matching the feature vectors in the first bag-of-words model with the feature vectors in the second bag-of-words model of the target keyframe.

[0168] Optionally, the pose positioning module 40 is used for:

[0169] The pose of the previous frame corresponding to the current frame is used as the initial pose, and the initial pose is optimized by the matching feature points to obtain the current pose of the head-mounted device.

[0170] The number of outliers to be deleted is determined based on the current pose and each of the matching feature points.

[0171] Optionally, the IMU-based keyframe tracking device further includes a constant velocity model localization module, which is used for:

[0172] If the number of deleted outliers is not less than the preset number, then keyframe tracking is determined to have failed.

[0173] The target keyframe corresponding to the current frame is tracked using a preset constant velocity motion model algorithm.

[0174] When the target keyframe is successfully tracked, the true pose of the head-mounted device is located based on the target keyframe.

[0175] Optionally, the IMU-based keyframe tracking device further includes a relocation module, which is used for:

[0176] If the target keyframe tracking fails, each keyframe will be traversed to obtain the number of feature points that are successfully matched between each keyframe and the current frame.

[0177] Select keyframes with a number of feature points greater than a preset threshold as candidate keyframes;

[0178] The head-mounted device is located using each of the candidate keyframes to obtain the initial pose corresponding to each candidate keyframe.

[0179] Each initial pose is optimized to obtain the true pose of the head-mounted device.

[0180] The specific implementations of the various functional modules of the IMU-based keyframe tracking device in this application are basically the same as the implementations of the IMU-based keyframe tracking method in the above application, and will not be repeated here.

[0181] This application also provides a computer storage medium storing an IMU-based keyframe tracking program, which, when executed by a processor, implements the steps of the IMU-based keyframe tracking program method as described in any of the above embodiments.

[0182] The specific embodiments of the computer storage medium in this application are basically the same as the embodiments of the keyframe tracking program method based on IMU in the above application, and will not be described in detail here.

[0183] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the IMU-based keyframe tracking method of this application as described in any of the above embodiments, which will not be repeated here.

[0184] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0185] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0186] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a head-mounted device (such as TWS earphones, etc.) to execute the methods described in the various embodiments of this application.

[0187] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A keyframe tracking method based on IMU, characterized in that, The IMU-based keyframe tracking method includes: Based on the IMU data of the head-mounted device, determine the first quaternion corresponding to the current frame collected by the head-mounted device; Based on the first quaternion, the target keyframe is selected from multiple keyframes of the map where the head-mounted device is located. Matching feature points are determined by comparing the first bag-of-words model of the current frame with the second bag-of-words model of the target keyframe; Based on the matched feature points, the current pose of the head-mounted device is located, and the number of outliers to be deleted is obtained; If the number of deleted outliers is less than a preset number, the keyframe tracking is determined to be successful, and the current pose is set to the actual pose of the head-mounted device.

2. The IMU-based keyframe tracking method as described in claim 1, characterized in that, The step of determining the first quaternion corresponding to the current frame acquired by the head-mounted device based on the IMU data of the head-mounted device includes: Extract gyroscope data from the IMU data; Integrate the gyroscope data to obtain the first quaternion corresponding to the current frame.

3. The IMU-based keyframe tracking method as described in claim 1, characterized in that, The step of filtering out target keyframes from multiple keyframes of the map where the head-mounted device is located based on the first quaternion includes: Traverse each of the keyframes, and calculate the rotation angle difference and vector difference between each keyframe and the current frame based on the first quaternion and the second quaternion of each keyframe; Keyframes whose rotation angle difference and vector difference are less than preset angle difference threshold and preset vector difference threshold, respectively, are designated as candidate keyframes; The candidate keyframe with the smallest difference in rotation angle among all the candidate keyframes is set as the target keyframe.

4. The IMU-based keyframe tracking method as described in claim 1, characterized in that, The step of determining matching feature points using the first bag-of-words model of the current frame and the second bag-of-words model of the target keyframe includes: The feature points in the current frame are converted into a first bag-of-words model using a preset bag-of-words algorithm, wherein the first bag-of-words model includes the feature vectors corresponding to each feature point in the current frame. Multiple matching feature points are obtained by matching the feature vectors in the first bag-of-words model with the feature vectors in the second bag-of-words model of the target keyframe.

5. The IMU-based keyframe tracking method as described in claim 1, characterized in that, The step of locating the current pose of the head-mounted device based on the matched feature points and obtaining the number of outliers to be deleted includes: The pose of the previous frame corresponding to the current frame is used as the initial pose, and the initial pose is optimized by the matching feature points to obtain the current pose of the head-mounted device. The number of outliers to be deleted is determined based on the current pose and each of the matching feature points.

6. The IMU-based keyframe tracking method as described in any one of claims 1-5, characterized in that, After the step of locating the current pose of the head-mounted device based on the matched feature points and obtaining the number of outliers to be deleted, the method further includes: If the number of deleted outliers is not less than the preset number, then keyframe tracking is determined to have failed. The target keyframe corresponding to the current frame is tracked using a preset constant velocity motion model algorithm. When the target keyframe is successfully tracked, the true pose of the head-mounted device is located based on the target keyframe.

7. The IMU-based keyframe tracking method as described in claim 6, characterized in that, The IMU-based keyframe tracking method also includes: If the target keyframe tracking fails, each keyframe will be traversed to obtain the number of feature points that are successfully matched between each keyframe and the current frame. Select keyframes with a number of feature points greater than a preset threshold as candidate keyframes; The head-mounted device is located using each of the candidate keyframes to obtain the initial pose corresponding to each candidate keyframe. Each initial pose is optimized to obtain the true pose of the head-mounted device.

8. A keyframe tracking device based on an IMU, characterized in that, The IMU-based keyframe tracking device includes: The quaternion calculation module is used to determine the first quaternion corresponding to the current frame collected by the head-mounted device based on the IMU data of the head-mounted device; The keyframe filtering module is used to filter out target keyframes from multiple keyframes of the map where the head-mounted device is located based on the first quaternion. The feature point matching module is used to determine matching feature points by comparing the first bag-of-words model of the current frame with the second bag-of-words model of the target keyframe; The pose localization module is used to locate the current pose of the head-mounted device based on the matched feature points and obtain the number of deleted outliers; The tracking and judgment module is used to determine that the keyframe tracking is successful if the number of deleted external points is less than a preset number, and to set the current pose as the true pose of the head-mounted device.

9. A head-mounted device, characterized in that, The head-mounted device includes: a memory, a processor, and an IMU-based keyframe tracking program stored in the memory and executable on the processor, wherein the IMU-based keyframe tracking program, when executed by the processor, implements the steps of the IMU-based keyframe tracking method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an IMU-based keyframe tracking program, which, when executed by a processor, implements the steps of the IMU-based keyframe tracking method as described in any one of claims 1 to 7.

Images