Hand gesture tracking method and device

Abstract

The present invention discloses a gesture tracking method and device, wherein the method comprises: wearing a wearable wristband on the wrist, the surface of the wearable wristband is provided with optical pattern marking points, and the wearable wristband is built with an inertial navigation sensor; capturing the motion state of the optical pattern marking points in space in real time, obtaining an optical tracking image, and obtaining the inertial navigation data of the wearable wristband through the inertial navigation sensor; obtaining the 4DoF information of the wearable wristband through computer vision technology according to the optical tracking image and the inertial navigation data, and using the 4DoF information to represent the position movement information of the hand in the three directions of the x-axis, y-axis, and z-axis in three-dimensional space and the rotation information of the hand around the z-axis direction; calculating the other 22DoF information of the hand through the 4DoF information, thereby obtaining the position and posture information of the hand. The present invention can improve the tracking accuracy of the position and posture of the hand.

Description

Technical Field

[0001] The present invention relates to the field of gesture recognition technology, and in particular to a gesture tracking method and device. Background Art

[0002] At present, in the field of VR / AR / MR, there are various methods for gesture recognition. Usually, computer vision tracking and recognition technology or inertial sensor-based motion capture technology is used to solve the gesture recognition problem. However, both computer vision tracking and recognition technology and inertial sensor-based motion capture technology have some problems.

[0003] Based on computer vision tracking and recognition technology, the hand's posture, position and finger joint posture are recognized and restored. Generally, a tracking camera is set up in the environment, or a tracking camera is set up on a VR head-mounted display device, and the position, rotation information and finger joint posture information of the hand under the camera are tracked in real time through image recognition technology. In actual use, some complex gestures of the user, such as crossing the hands, overlapping the hands, etc., will produce less recognition information on the image captured by the camera, resulting in a large error between the gesture recognition result and the user's actual gesture; or when the scene background used by the user is more complex, the segmentation detection of the user's hand on the image is prone to false detection, resulting in unstable accuracy of the user's gesture recognition, etc.

[0004] The use of inertial sensors for hand motion capture is simple to operate, highly portable, not affected by external light, not affected by complex hand gestures, with high gesture recognition capture accuracy and high data sampling rate. However, the fine movements of the hand joints cannot be restored, and the position movement information of the hand cannot be located; and because the hand posture data is obtained by integral calculation using inertial sensors, the hand posture data will have cumulative drift over time; and it is easily affected by the surrounding ferromagnetic environment, which accelerates the cumulative drift error.

[0005] Therefore, whether it is based on computer vision tracking and recognition technology or motion capture technology based on inertial sensors, the recognition accuracy of gestures is unstable and easily affected by the actual application scenarios or surrounding environment. Summary of the invention

[0006] In view of the above problems, an object of the present invention is to provide a gesture tracking method and device to solve the problems existing in the existing gesture recognition technology that the recognition accuracy is unstable and is easily affected by environmental factors.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] One aspect of the present invention is to provide a gesture tracking method, comprising:

[0009] Wearing a wearable wristband on the wrist, wherein the surface of the wearable wristband is provided with optical pattern marking points, and the wearable wristband is built with an inertial navigation sensor;

[0010] Capturing the motion state of the optical pattern marker in space in real time, acquiring an optical tracking image, and acquiring inertial navigation data of the wearable wristband through the inertial navigation sensor;

[0011] According to the optical tracking image and the inertial navigation data, 4DoF information of the wearable wristband is obtained by computer vision technology, and the 4DoF information is used to represent the position movement information of the hand in the three directions of the x-axis, y-axis and z-axis in the three-dimensional space and the rotation information of the hand around the z-axis direction;

[0012] The other 22DoF information of the hand is calculated through the 4DoF information, so as to obtain the position and posture information of the hand.

[0013] Preferably, the step of acquiring 4DoF information of the wearable bracelet by computer vision technology according to the motion state and the inertial navigation data comprises:

[0014] Detecting feature points of the optical pattern markers on the wearable wristband on the optical tracking image;

[0015] Predicting the position coordinates of the wearable wristband on the next frame of optical tracking image according to the detected feature points and the inertial navigation data, and constructing a feature point tracking queue with the detected feature point information and the prediction information of the next frame;

[0016] Acquire image data of the optical tracking image in real time, and acquire the predicted position of the feature point corresponding to each wearable wristband in the feature point tracking queue on the optical tracking image of the current frame according to the image data;

[0017] The 4DoF information of the wearable wristband is obtained according to the predicted position and the corresponding inertial navigation data.

[0018] Preferably, the step of detecting feature points of the optical pattern marker points on the wearable bracelet on the optical tracking image comprises:

[0019] Establishing a feature matching database, wherein the feature matching database includes each optical pattern marker point on the wearable wristband and its corresponding feature vector;

[0020] Acquire an optical tracking image in real time, detect feature points of the optical tracking image, and extract feature vectors corresponding to the feature points;

[0021] The feature points of the optical tracking image are matched with the feature points in the feature matching database to obtain a preset number of feature points with the maximum feature matching degree and continuous corresponding spatial positions.

[0022] Preferably, the FAST detection algorithm is used to detect the feature points of the optical tracking image according to the distribution of the optical pattern marker points on the wearable bracelet; and the feature vector is extracted from each detected feature point using the region extraction method.

[0023] Preferably, the step of matching the feature points of the optical tracking image with the feature points in the feature matching database comprises:

[0024] Align the coordinate system of the inertial navigation sensor built into the wearable wristband with the coordinate system of the tracking camera;

[0025] Obtaining the rotation angle of each optical pattern mark point on the wearable wristband relative to the tracking camera through inertial navigation data, and obtaining the physical position relationship between two adjacent feature points on the optical tracking image;

[0026] spatially sorting the feature points on the optical tracking image according to the physical position relationship;

[0027] According to the spatial order, a sliding window traversal calculation is performed with a preset number of points as a group, and feature point matching is performed with the spatial order corresponding to the feature points in the feature matching database.

[0028] Preferably, the step of predicting the position coordinates of the wearable wristband on the next frame of optical tracking image according to the detected feature points and the inertial navigation data comprises:

[0029] Obtaining the two-dimensional position coordinates of the detected feature points on the optical tracking image;

[0030] Calculate the position and rotation information of the wearable wristband relative to the tracking camera using a PNP algorithm according to the three-dimensional position coordinates and the two-dimensional position coordinates corresponding to the feature point on the wearable wristband;

[0031] The position coordinates of the wearable wristband on the next frame of optical tracking image are predicted according to the position and rotation information and the inertial navigation data.

[0032] Preferably, before the step of obtaining the predicted position of the feature point corresponding to each wearable wristband in the feature point tracking queue on the optical tracking image of the current frame according to the image data, it also includes:

[0033] According to the image data of the optical tracking image, the number of wearable bracelets in the feature point tracking queue is determined,

[0034] If the number of the wearable bracelets is 0, return to the step of detecting feature points of the optical pattern markers on the wearable bracelets on the optical tracking image;

[0035] If the number of the wearable wristbands is 1, return to the step of detecting the feature points of the optical pattern markers on the wearable wristbands on the optical tracking image, and at the same time, perform the step of obtaining the predicted positions of the feature points corresponding to the wearable wristbands on the optical tracking image of the current frame;

[0036] If the number of the wearable bracelets is 2, the step of obtaining the predicted positions of the feature points corresponding to the wearable bracelets on the optical tracking image of the current frame is performed.

[0037] Preferably, after the step of obtaining the predicted position of the feature point corresponding to each wearable wristband in the feature point tracking queue on the optical tracking image of the current frame according to the image data, it also includes:

[0038] Taking the predicted position as the origin center and the area range of a set size as the pixel window;

[0039] In the pixel window, an absolute two-dimensional position of the feature point on the optical tracking image of the current frame is obtained by using an NCC matching algorithm;

[0040] The 4DoF information of the wearable wristband is obtained according to the absolute two-dimensional position and the corresponding inertial navigation data.

[0041] Preferably, after the step of obtaining the 4DoF information of the wearable bracelet by computer vision technology, the method further includes:

[0042] The number of hands appearing in the optical tracking image acquired by the tracking camera is determined. If the number of hands appearing is greater than 2, the distance between the hands and the tracking camera is determined based on the 4DoF information, and the two hands with the smallest distance are selected as gesture tracking targets.

[0043] Another aspect of the present invention is to provide a gesture tracking device, comprising:

[0044] A wearable wristband, wherein an optical pattern mark point is arranged on the surface of the wearable wristband, and an inertial navigation sensor is built in the wearable wristband, and the inertial navigation sensor is used to obtain inertial navigation data of the wearable wristband;

[0045] A tracking camera, used to capture the motion state of the optical pattern marking points in space in real time and obtain an optical tracking image;

[0046] A 4DoF information acquisition module, which acquires 4DoF information of the wearable wristband through computer vision technology according to the optical tracking image and the inertial navigation data, and uses the 4DoF information to represent the position movement information of the hand in the three directions of the x-axis, y-axis and z-axis in the three-dimensional space and the rotation information of the hand around the z-axis direction;

[0047] The hand information acquisition module calculates other 22DoF information of the hand through the 4DoF information, so as to obtain the position and posture information of the hand.

[0048] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0049] The present invention sets some optical pattern marking points on the wearable wristband, combines the inertial navigation sensor built into the wearable wristband, and uses computer vision technology to track the position and flip angle information of the wearable wristband in space as the 4DOF information of the hand. The other 22DoF of the hand is calculated and tracked through the 4DOF information of the hand, which greatly improves the accuracy and stability of the 26DoF information tracking of the hand, improves the accuracy of gesture tracking and recognition, and can accurately locate the position and posture of the hand. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of the gesture tracking method of the present invention;

[0051] Figure 2 It is a schematic diagram of the position of the hand 26DoF in the present invention. DETAILED DESCRIPTION

[0052] The embodiments of the present invention will be described below with reference to the accompanying drawings. It will be appreciated by those skilled in the art that the described embodiments may be modified in various ways or combinations thereof without departing from the spirit and scope of the present invention. Therefore, the drawings and descriptions are illustrative in nature and are not intended to limit the scope of the claims. In addition, in this specification, the drawings are not drawn to scale and the same reference numerals represent the same parts.

[0053] Figure 1 is a flow chart of the gesture tracking method of the present invention, such as Figure 1 As shown, the gesture tracking method of the present invention includes:

[0054] Wearing a wearable wristband on the wrist, wherein an optical pattern mark point is arranged on the surface of the wearable wristband, so as to track the motion state of the wearable wristband through the optical pattern mark point, and an inertial navigation sensor is built in the wearable wristband, so as to obtain inertial navigation data through the inertial navigation sensor, thereby facilitating the acquisition of the spatial position of the optical pattern mark point in the wearable wristband, wherein the inertial navigation sensor may be an inertial measurement unit (IMU);

[0055] The motion state of the optical pattern marker in space is captured in real time, an optical tracking image is obtained, and the inertial navigation data of the wearable wristband is obtained through the inertial navigation sensor, wherein the optical tracking image can be obtained by taking a tracking camera arranged on a helmet display device, and the position of the optical pattern marker can be captured in real time through the optical tracking image; of course, the tracking camera can also be arranged on other tracking devices instead of the helmet display device;

[0056] According to the optical tracking image and the inertial navigation data, 4DoF information of the wearable wristband is obtained by computer vision technology, wherein the 4DoF information of the wearable wristband includes the position and flip angle (angle of rotation around the Z axis) information of the wearable wristband in space, and the 4DoF information of the wearable wristband is used to represent the position movement information of the hand in the three directions of the x-axis, y-axis and z-axis in the three-dimensional space and the rotation information of the hand around the z-axis direction;

[0057] The other 22DoF information of the hand is calculated through the 4DoF information, so as to obtain the position and posture information of the hand. Figure 2 , Figure 2 The schematic diagram of the position of the hand 26DoF in the present invention is shown, and the degree of freedom information of each position of the hand is obtained according to the hand characteristics. By tracking the hand 26DoF information, the hand position and posture can be accurately identified.

[0058] It should be noted that the method of calculating other 22DoF information using the hand 4DoF information is a conventional technology and will not be described in detail in the present invention.

[0059] The following is a detailed description of the application of the gesture tracking method of the present invention in the field of VR / AR / MR as an example. When applied, multiple tracking cameras are built into the head-mounted display device. In order to make the camera tracking range of the wearable bracelet reach 180°*180° (H*V), it is preferred to build 4 tracking cameras into the VR / AR / MR head-mounted display device, and complete the wide-angle tracking by stitching the perspectives of the 4 tracking cameras. The following only takes the tracking of the wearable bracelet by one of the tracking cameras as an example. The tracking of the wearable bracelet by other tracking cameras is roughly the same and will not be repeated.

[0060] In one embodiment, according to the motion state and the inertial navigation data, the step of acquiring 4DoF information of the wearable wristband by computer vision technology includes:

[0061] Detecting feature points of the optical pattern markers on the wearable wristband on the optical tracking image;

[0062] Predicting the position coordinates of the wearable wristband on the next frame of optical tracking image according to the detected feature points and the inertial navigation data, and constructing a feature point tracking queue with the detected feature point information and the prediction information of the next frame;

[0063] Acquire image data of the optical tracking image in real time, and acquire the predicted position of the feature point corresponding to each wearable wristband in the feature point tracking queue on the optical tracking image of the current frame according to the image data;

[0064] The 4DoF information of the wearable wristband is obtained according to the predicted position and the corresponding inertial navigation data.

[0065] Furthermore, the step of detecting feature points of the optical pattern marker points on the wearable wristband on the optical tracking image includes:

[0066] Establish a feature matching database, the feature matching database includes each optical pattern marker point on the wearable bracelet and its corresponding feature vector; specifically, pre-set the distribution of the optical pattern marker points on the wearable bracelet, perform feature point detection through an image feature point detection algorithm, and extract the feature vector of the feature point through a feature point feature vector calculation algorithm, so as to establish the feature vector corresponding to each marker point on the optical tracking image corresponding to the optical pattern marker point on the wearable bracelet. For various motion postures of the wearable bracelet relative to the tracking camera, the feature vectors corresponding to each optical pattern marker point on the wearable bracelet can be obtained. Since the wearable bracelet may exist at multiple angles relative to the VR head-mounted display, there are multiple feature vectors for each optical pattern marker point on the wearable bracelet. The optical pattern marker points and the corresponding feature vectors are stored to construct a feature matching database, which can be saved in a local file according to a certain data structure;

[0067] Acquire an optical tracking image in real time, detect feature points of the optical tracking image, and extract feature vectors corresponding to the feature points, wherein the method of detecting feature points and extracting feature vectors is the same as the method used when constructing a feature matching database;

[0068] The feature points of the optical tracking image are matched with the feature points in the feature matching database to obtain a preset number of feature points with the maximum feature matching degree and continuous corresponding spatial positions.

[0069] It should be noted that the present invention can use FAST detection algorithm, scale-invariant feature transform (SIFT), SURF (Speed ​​Up Robust Features) feature point extraction and other methods to detect feature points. Preferably, the FAST detection algorithm is used, and according to the distribution of optical pattern markers on the wearable bracelet, the feature points of the optical tracking image are detected; and the feature vector is extracted for each detected feature point using the region extraction method. Specifically, for the neighborhood 5*5 window of the feature point, the pixel gradient is calculated for each pixel grayscale value in the window and the pixel grayscale value corresponding to the feature point, and all pixels in the window are traversed in turn to calculate the gradient of all window pixels, and then the gradient value is normalized to obtain the feature vector of the feature point.

[0070] In one embodiment, the step of matching the feature points of the optical tracking image with the feature points in the feature matching database comprises:

[0071] The coordinate system of the inertial navigation sensor built into the wearable wristband is calibrated and aligned with the coordinate system of the tracking camera; by aligning the timestamps of the inertial navigation sensor and the tracking camera, the rotation angle of each optical pattern marker on the current wearable wristband relative to the tracking camera can be obtained through the attitude data of the inertial navigation sensor;

[0072] The rotation angle of each optical pattern mark point on the wearable wristband relative to the tracking camera is obtained through inertial navigation data, and the physical position relationship between two adjacent feature points on the optical tracking image is obtained; according to the physical position relationship between two adjacent optical pattern mark points on the wearable wristband, through the rotation angle and the imaging principle of the camera, it can be confirmed that if the position relationship of two mark points in space is adjacent, the imaging on the image must also be adjacent, so the feature points on the wearable wristband can be matched with the feature points on the optical tracking image, and the corresponding relative positions in space will not change, which can reduce mismatching operations;

[0073] spatially sorting the feature points on the optical tracking image according to the physical position relationship, so as to confirm as much as possible that feature points that are spatially adjacent are also adjacent after being sorted on the optical tracking image;

[0074] According to the spatial order, a preset number of points are grouped together to perform sliding window traversal calculations, and feature point matching is performed in the spatial order corresponding to the feature points in the feature matching database. Among them, the feature point matching can use matching methods such as Hamming distance matching, KNN matching, RANSAC matching (RANdom Sample Consensus), etc., preferably KNN matching method. When a sliding window traversal calculation is performed in a group of four, four feature points with the maximum feature matching degree and continuous corresponding spatial positions can be obtained.

[0075] In one embodiment, the step of predicting the position coordinates of the wearable wristband on the next frame of optical tracking image according to the detected feature points and the inertial navigation data includes:

[0076] Obtaining the two-dimensional position coordinates of the detected feature points on the optical tracking image;

[0077] According to the three-dimensional position coordinates and the two-dimensional position coordinates corresponding to the feature point on the wearable wristband, the position and rotation information of the wearable wristband relative to the tracking camera are calculated using the PNP algorithm (pespective-n-point), that is, the 4DoF information of the wearable wristband in the current frame optical tracking image is obtained;

[0078] The position coordinates of the wearable wristband on the next frame of optical tracking image are predicted according to the position and rotation information and the inertial navigation data.

[0079] In one embodiment, before the step of obtaining the predicted position of the feature point corresponding to each wearable wristband in the feature point tracking queue on the optical tracking image of the current frame according to the image data, it also includes: judging the number of wearable wristbands in the feature point tracking queue according to the image data of the optical tracking image, if the number of the wearable wristbands is 0, returning to execute the step of detecting the feature points of the optical pattern markers on the wearable wristbands on the optical tracking image; if the number of the wearable wristbands is 1, returning to execute the step of detecting the feature points of the optical pattern markers on the wearable wristbands on the optical tracking image, and at the same time executing the step of obtaining the predicted position of the feature point corresponding to the wearable wristband on the optical tracking image of the current frame; if the number of the wearable wristbands is 2, executing the step of obtaining the predicted position of the feature point corresponding to the wearable wristband on the optical tracking image of the current frame.

[0080] Further, after the step of obtaining the predicted position of the feature point corresponding to each wearable wristband in the feature point tracking queue on the optical tracking image of the current frame according to the image data, it also includes:

[0081] Taking the predicted position as the origin center and the area range of a set size as the pixel window, for example, a pixel window of a 5*5 area;

[0082] In the pixel window, an absolute two-dimensional position of the feature point on the optical tracking image of the current frame is obtained by using a Normalized cross correlation (NCC) matching algorithm;

[0083] The 4DoF information of the wearable wristband is obtained according to the absolute two-dimensional position and the corresponding inertial navigation data. Through the above prediction information, it can be obtained that the absolute two-dimensional position of each tracked feature point will not exceed the 5*5 pixel area of ​​the predicted position.

[0084] The gesture tracking method of the present invention is preferably limited to tracking the gestures of the left and right hands of the user, without considering other hands that appear in the tracking field of view. Preferably, after the step of obtaining the 4DoF information of the wearable bracelet by computer vision technology, it also includes:

[0085] The number of hands appearing in the optical tracking image acquired by the tracking camera is determined. If the number of hands appearing is greater than 2, the distance between the hands and the tracking camera is determined based on the 4DoF information, and the two hands with the smallest distance are selected as gesture tracking targets.

[0086] The present invention also provides a gesture tracking device, comprising:

[0087] A wearable wristband, wherein an optical pattern mark point is arranged on the surface of the wearable wristband, and an inertial navigation sensor is built in the wearable wristband, and the inertial navigation sensor is used to obtain inertial navigation data of the wearable wristband;

[0088] A tracking camera, used to capture the motion state of the optical pattern marker in space in real time and obtain an optical tracking image; wherein the tracking camera can be set in a helmet display device or can be an independent camera tracking device;

[0089] A 4DoF information acquisition module, which acquires 4DoF information of the wearable wristband through computer vision technology according to the optical tracking image and the inertial navigation data, and uses the 4DoF information to represent the position movement information of the hand in the three directions of the x-axis, y-axis and z-axis in the three-dimensional space and the rotation information of the hand around the z-axis direction;

[0090] The hand information acquisition module calculates other 22DoF information of the hand through the 4DoF information, so as to obtain the position and posture information of the hand.

[0091] In one embodiment, the 4DoF information acquisition module includes:

[0092] A feature point detection unit, used to detect feature points of the optical pattern markers on the wearable wristband on the optical tracking image;

[0093] A tracking queue construction unit predicts the position coordinates of the wearable wristband on the next frame of optical tracking image according to the detected feature points and the inertial navigation data, and constructs the detected feature point information and the prediction information of the next frame into a feature point tracking queue;

[0094] A position prediction unit, which acquires image data of the optical tracking image in real time, and acquires a predicted position of a feature point corresponding to each wearable wristband in the feature point tracking queue on the optical tracking image of the current frame according to the image data;

[0095] The information acquisition unit acquires 4DoF information of the wearable wristband according to the predicted position and the corresponding inertial navigation data.

[0096] Furthermore, the feature point detection unit detects feature points of the optical pattern marker points on the wearable wristband on the optical tracking image in the following manner, including:

[0097] Establish a feature matching database, the feature matching database includes each optical pattern marker point on the wearable bracelet and its corresponding feature vector; specifically, pre-set the distribution of the optical pattern marker points on the wearable bracelet, perform feature point detection through an image feature point detection algorithm, and extract the feature vector of the feature point through a feature point feature vector calculation algorithm, so as to establish the feature vector corresponding to each marker point on the optical tracking image corresponding to the optical pattern marker point on the wearable bracelet. For various motion postures of the wearable bracelet relative to the tracking camera, the feature vectors corresponding to each optical pattern marker point on the wearable bracelet can be obtained. Since the wearable bracelet may exist at multiple angles relative to the VR head-mounted display, there are multiple feature vectors for each optical pattern marker point on the wearable bracelet. The optical pattern marker points and the corresponding feature vectors are stored to construct a feature matching database, which can be saved in a local file according to a certain data structure;

[0098] Acquire an optical tracking image in real time, detect feature points of the optical tracking image, and extract feature vectors corresponding to the feature points, wherein the method of detecting feature points and extracting feature vectors is the same as the method used when constructing a feature matching database;

[0099] The feature points of the optical tracking image are matched with the feature points in the feature matching database to obtain a preset number of feature points with the maximum feature matching degree and continuous corresponding spatial positions.

[0100] In one embodiment, the tracking queue construction unit predicts the position coordinates of the wearable wristband on the next frame of optical tracking image according to the detected feature points and the inertial navigation data in the following manner, including:

[0101] Obtaining the two-dimensional position coordinates of the detected feature points on the optical tracking image;

[0102] Calculate the position and rotation information of the wearable wristband relative to the tracking camera using a PNP algorithm according to the three-dimensional position coordinates and the two-dimensional position coordinates corresponding to the feature point on the wearable wristband;

[0103] The position coordinates of the wearable wristband on the next frame of optical tracking image are predicted according to the position and rotation information and the inertial navigation data.

[0104] It should be noted that the specific implementation of the gesture tracking device of the present invention is substantially the same as the specific implementation of the gesture tracking method described above, and will not be described in detail herein.

[0105] The gesture tracking device of the present invention can be applied to the VR / AR / MR field. The wearable bracelet is used as a gesture tracker to track the position and posture information of the hand in a three-dimensional environment in real time, which can help solve the problem of high-precision restoration of finger joints. In addition, the present invention can improve the tracking accuracy of the 26DoF information of the hand, improve the stability of gesture tracking, and thus enhance the user experience.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A gesture tracking method, characterized in that: include: Wearing a wearable wristband on the wrist, wherein the surface of the wearable wristband is provided with optical pattern marking points, and the wearable wristband is built with an inertial navigation sensor; Capturing the motion state of the optical pattern marker in space in real time, acquiring an optical tracking image, and acquiring inertial navigation data of the wearable wristband through the inertial navigation sensor; According to the optical tracking image and the inertial navigation data, 4DoF information of the wearable wristband is obtained by computer vision technology, and the 4DoF information is used to represent the position movement information of the hand in the three directions of the x-axis, y-axis and z-axis in the three-dimensional space and the rotation information of the hand around the z-axis direction; Calculate the other 22DoF information of the hand through the 4DoF information, so as to obtain the position and posture information of the hand; Among them, the step of obtaining 4DoF information of the wearable bracelet by computer vision technology according to the motion state and the inertial navigation data includes: Detecting feature points of the optical pattern markers on the wearable wristband on the optical tracking image; Predicting the position coordinates of the wearable wristband on the next frame of optical tracking image according to the detected feature points and the inertial navigation data, and constructing a feature point tracking queue with the detected feature point information and the prediction information of the next frame; Acquire image data of the optical tracking image in real time, and acquire the predicted position of the feature point corresponding to each wearable wristband in the feature point tracking queue on the optical tracking image of the current frame according to the image data; The 4DoF information of the wearable wristband is obtained according to the predicted position and the corresponding inertial navigation data.

2. The gesture tracking method according to claim 1, characterized in that: The step of detecting feature points of the optical pattern markers on the wearable wristband on the optical tracking image comprises: Establishing a feature matching database, wherein the feature matching database includes each optical pattern marker point on the wearable wristband and its corresponding feature vector; Acquire an optical tracking image in real time, detect feature points of the optical tracking image, and extract feature vectors corresponding to the feature points; The feature points of the optical tracking image are matched with the feature points in the feature matching database to obtain a preset number of feature points with the maximum feature matching degree and continuous corresponding spatial positions.

3. The gesture tracking method according to claim 2, characterized in that: The FAST detection algorithm is used to detect the feature points of the optical tracking image according to the distribution of the optical pattern marker points on the wearable bracelet; and the feature vector is extracted from each detected feature point using the region extraction method.

4. The gesture tracking method according to claim 2, characterized in that: The step of matching the feature points of the optical tracking image with the feature points in the feature matching database comprises: Align the coordinate system of the inertial navigation sensor built into the wearable wristband with the coordinate system of the tracking camera; Obtaining the rotation angle of each optical pattern mark point on the wearable wristband relative to the tracking camera through inertial navigation data, and obtaining the physical position relationship between two adjacent feature points on the optical tracking image; spatially sorting the feature points on the optical tracking image according to the physical position relationship; According to the spatial order, a sliding window traversal calculation is performed with a preset number as a group, and feature point matching is performed in the spatial order corresponding to the feature points in the feature matching database.

5. The gesture tracking method according to claim 1, characterized in that: The step of predicting the position coordinates of the wearable wristband on the next frame of optical tracking image according to the detected feature points and the inertial navigation data comprises: Obtaining the two-dimensional position coordinates of the detected feature points on the optical tracking image; Calculate the position and rotation information of the wearable wristband relative to the tracking camera using a PNP algorithm according to the three-dimensional position coordinates and the two-dimensional position coordinates corresponding to the feature point on the wearable wristband; The position coordinates of the wearable wristband on the next frame of optical tracking image are predicted according to the position and rotation information and the inertial navigation data.

6. The gesture tracking method according to claim 1, characterized in that: Before the step of obtaining the predicted position of the feature point corresponding to each wearable wristband in the feature point tracking queue on the optical tracking image of the current frame according to the image data, the method further includes: Determine the number of wearable wristbands in the feature point tracking queue according to the image data of the optical tracking image, and if the number of the wearable wristbands is 0, return to the step of detecting feature points of the optical pattern markers on the wearable wristbands on the optical tracking image; If the number of the wearable wristbands is 1, return to the step of detecting the feature points of the optical pattern markers on the wearable wristbands on the optical tracking image, and at the same time, perform the step of obtaining the predicted positions of the feature points corresponding to the wearable wristbands on the optical tracking image of the current frame; If the number of the wearable bracelets is 2, the step of obtaining the predicted positions of the feature points corresponding to the wearable bracelets on the optical tracking image of the current frame is performed.

7. The gesture tracking method according to claim 1, characterized in that: After the step of obtaining the predicted position of the feature point corresponding to each wearable wristband in the feature point tracking queue on the optical tracking image of the current frame according to the image data, the method further includes: Taking the predicted position as the origin center and the area range of a set size as the pixel window; In the pixel window, an absolute two-dimensional position of the feature point on the optical tracking image of the current frame is obtained by using an NCC matching algorithm; The 4DoF information of the wearable wristband is obtained according to the absolute two-dimensional position and the corresponding inertial navigation data.

8. The gesture tracking method according to claim 1, characterized in that: After the step of obtaining the 4DoF information of the wearable bracelet through computer vision technology, it also includes: The number of hands appearing in the optical tracking image acquired by the tracking camera is determined. If the number of hands appearing is greater than 2, the distance between the hands and the tracking camera is determined based on the 4DoF information, and the two hands with the smallest distance are selected as gesture tracking targets.

9. A gesture tracking device, characterized in that: include: A wearable wristband, wherein an optical pattern mark point is arranged on the surface of the wearable wristband, and an inertial navigation sensor is built in the wearable wristband, and the inertial navigation sensor is used to obtain inertial navigation data of the wearable wristband; A tracking camera, used to capture the motion state of the optical pattern marking points in space in real time and obtain an optical tracking image; A 4DoF information acquisition module, which acquires 4DoF information of the wearable wristband through computer vision technology according to the optical tracking image and the inertial navigation data, and uses the 4DoF information to represent the position movement information of the hand in the three directions of the x-axis, y-axis and z-axis in the three-dimensional space and the rotation information of the hand around the z-axis direction; A hand information acquisition module calculates other 22DoF information of the hand through the 4DoF information, thereby acquiring the position and posture information of the hand; Wherein, the 4DoF information acquisition module includes: A feature point detection unit, used to detect feature points of the optical pattern markers on the wearable wristband on the optical tracking image; A tracking queue construction unit predicts the position coordinates of the wearable wristband on the next frame of optical tracking image according to the detected feature points and the inertial navigation data, and constructs the detected feature point information and the prediction information of the next frame into a feature point tracking queue; A position prediction unit, which acquires image data of the optical tracking image in real time, and acquires a predicted position of a feature point corresponding to each wearable wristband in the feature point tracking queue on the optical tracking image of the current frame according to the image data; The information acquisition unit acquires 4DoF information of the wearable wristband according to the predicted position and the corresponding inertial navigation data.

Images