Hand tracking method, data processing device and head-mounted device

Abstract

The present invention discloses a hand tracking method, a data processing device and a head-mounted device, which obtains real-time 3D gesture poses through a four-eye fisheye camera. 3d , the global detection model Det is used for four-camera hand detection, the 2D skeleton point model Ske is used for 2D hand skeleton point prediction, and Inverse Kinematics is used for multi-eye skeleton point optimization to obtain pose 3d , using the deep learning refine solution to do double checkout to ensure high-quality tracking frame and high-accuracy 3D pose, using and optimizing the hand 3D pose interpolation method to extrapolate the tracking frame, and using the local detection model Local DET The advantage of the present invention is that it improves the stability of tracking, avoids tracking loss and tracking deviation, avoids the occurrence of tracking category errors that lead to problems in subsequent interactions, and greatly improves the quality of the tracking frame.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a hand tracking method, data processing equipment, and head-mounted device. Background Art

[0002] Gesture recognition technology enables gesture-based human-computer interaction by identifying the user's hand posture and movements. Hand tracking is an integral part of this technology solution. With the advancement of computer vision and the increase in edge-to-edge computing power, visual algorithms are gradually expanding into various edge-to-edge scenarios. As a new type of edge-to-edge computing device, VR uses visual algorithms to create an immersive experience. However, VR devices also place high demands on physical interaction. Traditional VR devices rely on controllers for human-machine interaction. As deep learning technology matures, 3D bare-hand interaction is gradually expanding into this scenario and will continue to expand to a growing number of interactive devices.

[0003] Currently, there are two main approaches to target tracking. One approach involves using an algorithm to identify candidate regions for the current target based on the area occupied by the target in the previous frame. The algorithm then calculates the degree of match between the candidate regions and the target area in the previous frame (or other criteria, such as color histogram distribution), selecting the region with the best match as the target's location prediction for the current frame. Another approach involves modeling the target object, finding the region in the current image that is closest to the target model, and then updating the target model.

[0004] Therefore, current hand tracking technologies all track the area occupied by the target. However, due to the complex deformation and posture changes of human hands and diverse features, the correlation between previous and next frames is weak, making area tracking difficult. In addition, the tracking algorithm is usually computationally intensive and the algorithm logic is complex, resulting in serious lag in hand tracking response, which reduces the accuracy of gesture recognition. Summary of the Invention

[0005] The present invention provides a hand tracking method, a data processing device and a head-mounted device, which are used to solve the problems of poor stability of existing hand tracking technology, easy error in hand tracking categories and substandard hand tracking frame quality.

[0006] In order to solve the above technical problems, the present invention discloses the following technical solutions:

[0007] The present invention provides a hand tracking method, which includes a detection frame acquisition step, a coordinate acquisition step, an error optimization step, a tracking frame acquisition step, a tracking frame detection step, and a classification and comparison step.

[0008] The detection frame acquisition step is based on the global detection model of deep learning to poll the four-eye camera to obtain the hand detection frame position information; the coordinate acquisition step is to obtain the hand image through the detection frame, predict the skeleton point information in the hand image based on the skeleton model of deep learning, and obtain the coordinates of at least one skeleton point in the hand image coordinate system; the error optimization step is based on the pose calculation in the head display IMU coordinate system based on inverse kinematics optimization. 3d The error is obtained to get the optimized pose 3d ; The tracking frame acquisition step is to optimize the pose 3d Reprojecting to the four cameras, obtaining the 2D reprojected skeleton points of each camera, and calculating the tracking frame of the hand image at time t based on the 2D reprojected skeleton points of each camera; the tracking frame detection step is based on the local detection model, detecting the tracking frame, excluding the hand-free tracking frame caused by the tracking frame offset, and adjusting the tracking offset to correct the tracking frame in real time; the classification comparison step is based on the classification result comparison function, comparing the corresponding hand classification information given by the local detection model with the corresponding hand classification information given by the global detection model to determine the hand category.

[0009] Furthermore, the formula for the detection frame acquisition step is:

[0010] handdet i ,handinfo i =RoundRobin(Det(Camera i )),i=1,2,3,4

[0011] Among them, handdet i Represents the detection box obtained by the global detection model Det reasoning, handinfo i It indicates that the global detection model Det gives the corresponding hand classification information, RoundRobin indicates polling detection, and i indicates the number of cameras.

[0012] Furthermore, the formula for the coordinate acquisition step is:

[0013] handske i =Ske(image i ),i=1,2,3,4

[0014] Among them, handske i represents the coordinates of 21 skeleton points in the hand image coordinate system obtained by the i-th camera, Ske represents the 2D skeleton point model, image i Represents the hand image captured by the i-th camera.

[0015] Furthermore, the formula for the error optimization step is:

[0016] E=∑|Ins i *T i *pose 3d -handske i | 2

[0017] Among them, E represents the error size, pose 3d Indicates the 3D pose of the 21 bone points of the hand under the head display IMU system, Ins i Denote as the internal parameter of the i-th camera, T i Denote as the external parameter of the i-th camera.

[0018] Furthermore, the formula for the tracking frame acquisition step is:

[0019] handpro j,i =Ins i *T i *pose 3d , i=1,2,3,4;

[0020] TrackBox j =cal(handpro j,i ), i=1,2,3,4; j=1,2,3,4;

[0021] TrackBox res =select(TrackBox j ), i=1,2,3,4; res=1,2,3,4;

[0022] Among them, handpro j,i TrackBox represents the 2D reprojected skeleton points of each camera at time t. j Indicates the tracking box, TrackBox res Represents the filtered tracking box, cal is the box calculation function, Select is the tracking box filtering function, res represents the number of filtered tracking boxes, and j is used to identify the number of tracking boxes.

[0023] Furthermore, the formula for the tracking frame detection step is:

[0024] localhanddet res ,handinfo res =Local DET (TrackBox res ), res = 1, 2, 3, 4

[0025] Among them, localhanddetres Represents the local detection box obtained by local detection model reasoning, handinfo res Indicates left and right hand category information, Local DET Represents a local detection model.

[0026] Furthermore, the formula for the classification comparison step is:

[0027] flag=compare(handinfo res ,handinfo i ), i=1,2,3,4; res=1,2,3,4; i=res

[0028] Among them, flag represents the comparison result, compare represents the classification result comparison function, handinfo res It indicates that the local detection model gives the corresponding hand classification information.

[0029] Furthermore, the tracking frame acquisition step includes a speed calculation step, an interpolation weight calculation step, and a pose calculation step.

[0030] The speed calculation step is to calculate the wrist root node speed at time t based on the acceleration of the wrist root node along the x-axis, y-axis, and z-axis. The formula is:

[0031]

[0032] Wrist acc,t =max(acc x ,acc y ,acc z )

[0033] Among them, Wrist acc,t represents the wrist root node velocity at time t, acc x Indicates the acceleration of the wrist root node along the x-axis, acc y Indicates the acceleration of the wrist root node along the y-axis, acc z Indicates the acceleration of the wrist root node along the z-axis, and max is the maximum value selection function.

[0034] The interpolation weight calculation step is to calculate pose according to the speed of the wrist root node at time t. 3d The hand motion interpolation weight is given by

[0035] weight1=(Ti t -Ti t-2 ) / (Ti t-1 -Ti t-2 )

[0036] weight2=(1+Wrist acc,t-1 ) / (1+Wrist acc,t-2 )

[0037] weight=weight1*weight2

[0038] Among them, weight1 represents the time interval ratio weight of the first three moments, Ti t is the timestamp at time t, Ti t-1 is the timestamp at time t-1, Ti t-2 is the timestamp of time t-2, weight2 represents the acceleration ratio weight of the first two moments, and weight represents the difference weight;

[0039] The pose calculation step is to calculate the 3D pose of the 21 skeletal points of the hand at time t in the head-mounted display IMU system, and its formula is:

[0040] pose 3d,t =weight*pose 3d,t-2 -pose 3d,t-1

[0041] Among them, pose 3d,t represents the 3D pose of the 21 skeleton points of the hand at time t in the head-mounted display IMU system, pose 3d,t-2 The 3D pose of the 21 hand skeleton points at time t-2 under the head display IMU system, pose 3d,t-1 Indicates the 3D pose of the 21 hand skeleton points at time t-1 in the head-mounted display IMU system.

[0042] Furthermore, the tracking frame screening function specifically includes a proportion calculation step, a ratio calculation step, and a distance calculation step.

[0043] The proportion calculation step is to calculate the proportion of bone points that are not within the range of the hand image; the ratio calculation step is to calculate the length ratio of each bone; and the distance calculation step is to calculate the distance between the center of the bone point and the center of the tracking frame.

[0044] Furthermore, the tracking frame detection step further includes a fusion step and a filtering step.

[0045] The fusion step is to obtain the local coordinates of the hand based on the local detection model and fuse them with the tracking frame, and its formula is:

[0046] TrackBox merge =Merge(localhanddet res ,TrackBox res),res=1,2,3,4

[0047] Among them, TrackBox merge Indicates a fusion frame, and Merge indicates fusion.

[0048] The filtering step is to filter the fusion frame, and its formula is:

[0049] TrackBox filter =Filter(TrackBox merge ),merge=1,2,3,4

[0050] Among them, TrackBox filter Indicates the fused frame after filtering, Filter indicates filtering, and merge indicates the number of fused frames.

[0051] The present invention further provides a data processing device comprising a memory and a processor. The memory is used to store executable program code; the processor is used to read the executable program code to run a computer program corresponding to the executable program code to perform at least one step of the hand tracking method.

[0052] The present invention also provides a head-mounted device, comprising a multi-eye camera, wherein the multi-eye camera is electrically connected to

[0053] the data processing device.

[0054] Compared with the prior art, the present invention has at least the following technical effects:

[0055] The present invention provides a hand tracking method, data processing equipment and head-mounted device. The method can be applied to other similar visual algorithms, such as face and eye, to obtain real-time 3D gesture poses through a four-eye fisheye camera. 3d , the global detection model Det is used for four-camera hand detection, the 2D skeleton point model Ske is used for 2D hand skeleton point prediction, and Inverse Kinematics is used for multi-eye skeleton point optimization to obtain pose 3d , using the deep learning refine solution to do double checkout to ensure high-quality tracking frame and high-accuracy 3D pose, using and optimizing the hand 3D pose interpolation method to extrapolate the tracking frame, and using the local detection model Local DET The tracking frame is corrected in real time to improve tracking stability, avoid tracking loss and tracking deviation, and avoid tracking category errors that may cause problems in subsequent interactions, thereby greatly improving the quality of the tracking frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Figure 1 A circuit structure diagram of a head-mounted device according to an embodiment of the present invention;

[0058] Figure 2 This is a flow chart of a hand tracking method according to an embodiment of the present invention;

[0059] Figure 3 This is a flowchart of the steps for obtaining a tracking frame according to an embodiment of the present invention;

[0060] Figure 4 This is a flowchart of the tracking frame screening function according to an embodiment of the present invention;

[0061] Figure 5 A schematic structural diagram of a storage medium according to an embodiment of the present invention;

[0062] Figure 6 This is a low-speed MPJPA effect diagram of another hand tracking method according to an embodiment of the present invention;

[0063] Figure 7 This is a diagram showing the effect of low-speed MPJPA tracking of the hand tracking method according to an embodiment of the present invention;

[0064] Figure 8 This is a high-speed MPJPA effect diagram of other hand tracking methods described in an embodiment of the present invention;

[0065] Figure 9 This is a tracking high-speed MPJPA effect diagram of the hand tracking method described in an embodiment of the present invention.

[0066] The following are marked in the figure:

[0067] 10. Data processing device, 11. Memory, 12. Processor, 20. Head-mounted device, 30. Quadruple camera. DETAILED DESCRIPTION

[0068] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0069] like Figure 1As shown, the present invention provides a head-mounted device 20, comprising a four-eye camera 30 and a data processing device 10 electrically connected to each other, wherein the four-eye camera 30 is used to obtain the three-dimensional position of the user's hand in real time, that is, the 3D gesture pose 3d The data processing device 10 can use the four-eye camera 30 to 3d Real-time tracking and pose 3d to classify.

[0070] like Figure 2 As shown, the present invention also provides a hand tracking method, including step S1) a detection frame acquisition step, step S2) a coordinate acquisition step, step S3) an error optimization step, step S4) a tracking frame acquisition step, step S5) a tracking frame detection step, step S6) a fusion step, step S7) a filtering step and step S8) a classification and comparison step.

[0071] Step S1) Detection frame acquisition step: The global detection model Det based on deep learning polls the hand detection frame position information obtained by the four-eye camera. The formula is:

[0072] handdet i ,handinfo i =RoundRobin(Det(Camera i )),i=1,2,3,4

[0073] Among them, handdet i Represents the detection box obtained by the global detection model Det reasoning, handinfo i It indicates that the global detection model Det gives the corresponding hand classification information. The hand categories are divided into: left hand, right hand, and i indicates the number of cameras;

[0074] Step S2) coordinate acquisition step, obtaining a hand image through the detection frame, predicting the skeleton point information in the hand image based on the skeleton model of deep learning, and obtaining the coordinates of at least one skeleton point in the hand image coordinate system. Preferably, in this embodiment, the coordinates of 21 skeleton points are included, and the formula is:

[0075] handske i =Ske(image i ),i=1,2,3,4

[0076] Among them, handske i Indicates the coordinates of 21 skeleton points in the image coordinate system containing the hand area obtained by the i-th camera, Ske represents the 2D skeleton point model, image i Represents the image containing the hand area obtained by the i-th camera.

[0077] Step S3) Error optimization step, based on inverse kinematics optimization calculation pose in the head display IMU coordinate system 3d The error is obtained to get the optimized pose 3d , whose formula is

[0078] E=∑|Ins i *T i *pose 3d -handske i | 2

[0079] Among them, E represents the error size, pose 3d Indicates the 3D pose of the 21 bone points of the hand under the head display IMU system, Ins i Denote as the internal parameter of the i-th camera, T i Denote it as the external parameter of the i-th camera. The closer the value of E is to 0, the smaller the error is, and the more realistic the 3D pose of the 21 skeletal points of the hand under the head-mounted display IMU system is.

[0080] Step S4) Tracking frame acquisition step, the optimized pose 3d Reprojection to the four-eye camera. The purpose of reprojection is to transform the optimized pose into 3d The disparity map is converted into a depth map to obtain the 2D reprojection skeleton points of each eye camera. The hand image tracking frame at time t is calculated based on the 2D reprojection skeleton points of each eye camera. The tracking frame at time t during hand tracking is the result of time interpolation between time t-1 and time t-2. Preferably, the tracking frame at time t during hand tracking can be the result of time interpolation between more frames. The formula is:

[0081] handpro j,i =Ins i *T i *pose 3d , i=1,2,3,4;

[0082] TrackBox j =cal(handpro j,i ), i=1,2,3,4; j=1,2,3,4;

[0083] TrackBox res =select(TrackBox j ), i=1,2,3,4; res=1,2,3,4;

[0084] Among them, handpro j,i TrackBox represents the 2D reprojected skeleton points of each camera at time t. jIndicates the tracking box, TrackBox res Represents the filtered tracking box, cal is the box calculation function, Select is the tracking box filtering function, res represents the number of filtered tracking boxes, and j is used to identify the number of tracking boxes.

[0085] In this embodiment, the calculation frame function cal is used to obtain the image coordinates of 21 skeleton points obtained by inference of the skeleton point model, and the maximum and minimum coordinate values ​​of the 21 skeleton point image coordinates are selected as the two vertices of the diagonal of the rectangular frame. The tracking frame screening function Select is used to exclude low-quality tracking frames to prevent low-quality tracking frames from affecting the presentation effect of the 21 skeleton points and causing the failure of the final hand interaction.

[0086] like Figure 3 As shown, step S4) the tracking frame acquisition step includes step S41) the speed calculation step, step S42) the interpolation weight calculation step and step S43) the pose calculation step.

[0087] Step S41) Velocity calculation step: Calculate the wrist root node velocity at time t based on the acceleration of the wrist root node along the x-axis, y-axis, and z-axis. The formula is:

[0088]

[0089] Wrist acc,t =max(acc x ,acc y ,acc z )

[0090] In order to normalize the acceleration, a piecewise function is used to represent the acceleration. This piecewise function can quickly attenuate the impact of acceleration on the overall tracking, preventing excessive acceleration from invalidating the interpolation result. This keeps the impact of acceleration on the overall tracking within a certain range. The piecewise function also ensures that the acceleration is continuous at the boundary, avoiding step-like phenomena in the calculated value.

[0091] Among them, Wrist acc,t represents the wrist root node velocity at time t, acc x Indicates the acceleration of the wrist root node along the x-axis, acc y Indicates the acceleration of the wrist root node along the y-axis, acc z Indicates the acceleration of the wrist root node along the z-axis, max is the maximum value selection function, select acc x ,acc y and acc z The maximum of the three accelerations.

[0092] Step S42) interpolation weight calculation step, calculate pose according to the speed of the wrist root node at time t 3d The hand motion interpolation weight is given by

[0093] weight1=(Ti t -Ti t-2 ) / (Ti t-1 -Ti t-2 )

[0094] weight2=(1+Wrist acc,t-1 ) / (1+Wrist acc,t-2 )

[0095] weight=weight1*weight2

[0096] The advantage of calculating the interpolation weight based on the wrist root node velocity is that different interpolation weights calculated based on the wrist root node velocity can avoid delays in pose interpolation that cause tracking frame offsets, without affecting subsequent tracking accuracy and stability. It also avoids the influence of pose interpolation on the stability of the four-lens fisheye camera's frame rate.

[0097] Among them, weight1 is the weight of the time interval ratio of the first three moments, which is used to represent the impact of the current acceleration on the interpolation. Weight1 can smooth the stability of the camera frame rate; Ti t is the timestamp at time t, Ti t-1 is the timestamp at time t-1, Ti t-2 is the timestamp of time t-2, weight2 represents the acceleration ratio weight of the previous two moments, weight represents the difference weight, and weight2 can smooth the impact of acceleration on tracking stability.

[0098] Step S43) The pose calculation step calculates the 3D pose of the 21 skeletal points of the hand at time t in the head-mounted display IMU system. The formula is:

[0099] pose 3d,t =weight*pose 3d,t-2 -pose 3d,t-1

[0100] Among them, pose 3d,t represents the 3D pose of the 21 skeleton points of the hand at time t in the head-mounted display IMU system, pose 3d,t-2 The 3D pose of the 21 hand skeleton points at time t-2 under the head display IMU system, pose 3d,t-1 Indicates the 3D pose of the 21 hand skeleton points at time t-1 in the head-mounted display IMU system.

[0101] like Figure 4 As shown, the tracking frame screening function specifically includes step S91) a proportion calculation step, step S92) a ratio calculation step and step S93) a distance calculation step.

[0102] Step S91) The proportion calculation step calculates the proportion of the skeleton points that are not within the range of the hand image. The more such skeleton points there are, the lower the score corresponding to the tracking frame is, and the worse the quality of the tracking frame is.

[0103] Step S92) Ratio calculation step, calculate the length ratio of each bone segment and exclude outlier bone points. If an outlier bone point appears in a tracking frame, the tracking frame will be excluded.

[0104] Step S93) The distance calculation step calculates the distance between the center of the skeleton point and the center of the tracking frame. The closer the center of the skeleton point is to the center of the tracking frame, the higher the score and the better the quality of the tracking frame.

[0105] Step S5) Tracking frame detection step, based on the local detection model Local DET , detect the filtered tracking frame, exclude the no-hand tracking frame caused by the tracking frame offset, and adjust the tracking offset and correct the tracking frame in real time. The formula is:

[0106] localhanddet res ,handinfo res =Local DET (TrackBox res ), res = 1, 2, 3, 4

[0107] Among them, localhanddet res Represents the local detection box obtained by local detection model reasoning, handinfo res Indicates left and right hand category information, namely left hand, right hand and no hand, Local DET Represents a local detection model.

[0108] Step S6) Fusion step: Based on the local detection model, the local coordinates of the hand are obtained and fused with the tracking frame. The formula is:

[0109] TrackBox merge =Merge(localhanddet res ,TrackBox res ),res=1,2,3,4

[0110] Among them, TrackBox merge Indicates a fusion frame, and Merge indicates fusion.

[0111] Step S7) filtering step, filtering the fusion frame, the formula is:

[0112] TrackBox filter =Filter(TrackBox merge ),merge=1,2,3,4

[0113] Among them, TrackBox filter Indicates the fused frame after filtering, Filter indicates filtering, and merge indicates the number of fused frames.

[0114] S8) Classification comparison step: Based on the classification result comparison function, the corresponding hand classification information given by the local detection model is compared with the corresponding hand classification information given by the global detection model to determine the hand category. The classification result comparison function can eliminate the problem of incorrect classification of left and right hands. At the same time, the local detection model giving no hand classification can alleviate the false detection problem of the global detection model. Its formula is:

[0115] flag=compare(handinfo res ,handinfo i ), i=1,2,3,4; res=1,2,3,4; i=res

[0116] Among them, flag represents the comparison result, compare represents the classification result comparison function, handinfo res Indicates that the local detection model gives the corresponding hand classification information and determines the tracking box TrackBox filter The hand information in is left hand, right hand or no hand.

[0117] like Figure 5 As shown, the present invention also provides a data processing device 10, including a memory 11 and a processor 12, the memory 11 is used to store executable program code, and the processor 12 reads the executable program code to run a computer program corresponding to the executable program code to execute at least one step in the hand tracking method.

[0118] like Figure 6-7 As shown in the figure, the tracking stability evaluation index MPJPA of the hand tracking method described in the present invention is far superior to the tracking stability evaluation index MPJPA of other existing hand tracking methods. Compared with the low-speed tracking MPJPA, under the condition of low-speed movement, the MPJPA of the hand tracking method described in the present invention has fewer mutations and is relatively stable, reflecting that the skeleton point prediction is relatively stable and the tracking is relatively stable.

[0119] like Figure 8-9As shown in the figure, compared with tracking high-speed MPJPA, in the case of high-speed movement, the stability of the tracking system will decrease with the increase of movement speed, and the faster speed will cause the tracking frame to be unstable. However, the weighted interpolation of the hand tracking method described in the present invention can enable the system to give interpolation and correct projection frames relatively stably even at higher speeds, and the local detection continuously corrects the tracking frame so that the MPJPA will not rise sharply under high-speed conditions, reflecting the effectiveness of the optimization method.

[0120] The advantage of the present invention is that it can obtain real-time 3D gesture poses through a four-eye fisheye camera. 3d The global detection model Det is used for four-camera hand detection, the 2D skeleton point model Ske is used for 2D hand skeleton point prediction, and InverseKinematics is used for multi-eye skeleton point optimization to obtain pose. 3d , using the deep learning refine solution to do double checkout to ensure high-quality tracking frame and high-accuracy 3D pose, using and optimizing the hand 3D pose interpolation method to extrapolate the tracking frame, and using the local detection model Local DET Correct the tracking frame in real time to improve tracking stability, avoid tracking loss and tracking deviation, and avoid tracking category errors that may cause problems in subsequent interactions, thereby greatly improving the quality of the tracking frame.

[0121] The above is a detailed introduction to a hand tracking method, data processing device and head-mounted device provided in the embodiments of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the technical solutions and core ideas of the present invention. Ordinary technicians in this field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein with equivalents; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hand tracking method, characterized in that: include: In the detection frame acquisition step, the global detection model based on deep learning polls and detects the hand detection frame position information obtained by the quad-camera; A coordinate acquisition step, acquiring a hand image through the detection frame, predicting skeletal point information in the hand image based on a deep learning skeletal model, and acquiring coordinates of at least one skeletal point in a coordinate system containing the hand image; Error optimization step, based on inverse kinematics optimization to calculate the pose in the head-mounted display IMU system 3d The error is obtained to get the optimized pose 3d ; The formula for the tracking frame acquisition step is: h andpro j,i =Ins i *T i *pose 3d ,i=1,2,3,4; TrackBox j =cal(h andpro j,i ),i=1,2,3,4;j=1,2,3,4; TrackBox res =select(TrackBox j ),i=1,2,3,4;res=1,2,3,4; Among them, h andpro j,i TrackBox represents the 2D reprojected skeleton points of each camera at time t. j Indicates the tracking box, TrackBox res Indicates the filtered tracking frame, cal is the calculation frame function, Select is the tracking frame filtering function, res indicates the number of filtered tracking frames, and j is used to identify the number of tracking frames. Tracking frame acquisition step, the optimized pose 3d Reproject to the four cameras to obtain the 2D reprojected skeleton points of each camera, and calculate the tracking frame of the hand image at time t based on the 2D reprojected skeleton points of each camera; A tracking frame detection step, based on a local detection model, detects the tracking frame, excludes the hand-free tracking frame caused by tracking frame offset, adjusts the tracking offset, and corrects the tracking frame in real time; The formula for the tracking frame detection step is: localh anddet res ,h andinfo res =Local DET (TrackBox res ),res=1,2,3,4 Among them, localh anddet res Represents the local detection box obtained by local detection model reasoning, handinfo res Indicates left and right hand category information, Local DET represents the local detection model; After the tracking frame detection step, the following steps are also included The fusion step is to obtain the local coordinates of the hand based on the local detection model and fuse them with the tracking frame. The formula is TrackBox merge =Merge(localh anddet res ,TrackBox res ),res=1,2,3,4 Among them, TrackBox merge Indicates the fusion box, Merge indicates fusion; and The filtering step is to filter the fusion frame, and the formula is: TrackBox filter =Filter(TrackBox merge ),merge=1,2,3,4 Among them, TrackBix filter Indicates the fusion frame after filtering, Filter indicates filtering, and merge indicates Indicates the number of fused frames; and a classification comparison step, comparing the hand classification information provided by the local detection model with the hand classification information provided by the global detection model based on a classification result comparison function to determine the hand category; The formula for the classification comparison step is: flag=compare(h andinfo res ,h andinfo i ),i=1,2,3,4;res=1,2,3,4;i=res Among them, flag represents the comparison result, compare represents the classification result comparison function, and h andinfo res It indicates that the local detection model gives the corresponding hand classification information.

2. The hand tracking method according to claim 1, wherein: The formula for the detection frame acquisition step is: the hand i ,hand info i =RoundRobin(It(Camera i )),i=1,2,3,4 Among them, h anddet i Represents the detection box inferred by the global detection model Det, h andinfo i It indicates that the global detection model Det gives the corresponding hand classification information, RoundRobin indicates polling detection, and i indicates the number of cameras.

3. The hand tracking method according to claim 1, wherein: The formula for the coordinate acquisition step is: glove i =Spoon(image i ),i=1,2,3,4 Among them, handsk i represents the coordinates of 21 skeleton points in the hand image coordinate system obtained by the i-th camera, Ske represents the 2D skeleton point model, image i Represents the hand image captured by the i-th camera.

4. The hand tracking method according to claim 1, wherein: The formula for the error optimization step is: E=∑|Ins i *T i *pose 3d -h andske i | 2 Among them, E represents the error size, pose 3d Indicates the 3D pose of the 21 bone points of the hand under the head display IMU system, Ins i Denote as the internal parameter of the i-th camera, T i Denote as the external parameter of the i-th camera.

5. The hand tracking method according to claim 1, wherein: The tracking frame acquisition step include: The speed calculation steps are based on the acceleration of the wrist root node along the x-axis, y-axis, and z-axis. Speed, calculate the wrist root node speed at time t, the formula is Wrist acc,t =max(acc x ,acc y ,acc z ) Among them, Wrist acc,t represents the wrist root node velocity at time t, acc x Indicates the acceleration of the wrist root node along the x-axis, acc y Indicates the acceleration of the wrist root node along the y-axis, acc z Represents the acceleration of the wrist root node along the z-axis, and max is the maximum value selection function; Interpolation weight calculation step, calculate pose according to the speed of the wrist root node at time t 3d Hand Movement The dynamic interpolation weight is: weigh t1=(Ti t -Of t-2 ) / (Of t-1 -Of t-2 ) weigh t2=(1+wrist acc,t-1 ) / (1+Wrist acc,t-2 ) weigh t=weigh t1*weigh t2 Among them, weigh t1 represents the time interval ratio weight of the first three moments, Ti t is the timestamp at time t, Ti t-1 is the timestamp at time t-1, Ti t-2 is the timestamp of time t-2, weigh t2 represents the acceleration ratio weight of the first two times, and weigh t represents the difference weight; and The pose calculation step is to calculate the 3D pose of the 21 bone points of the hand at time t in the head-mounted display IMU system. The formula is: pose 3d,t =weigh t*pose 3d,t-2 -pose 3d,t-1 Among them, pose 3d,t represents the 3D pose of the 21 skeleton points of the hand at time t in the head-mounted display IMU system, pose 3d,t-2 The 3D pose of the 21 hand skeleton points at time t-2 under the head display IMU system, pose 3d,t-1 Indicates the 3D pose of the 21 hand skeleton points at time t-1 in the head-mounted display IMU system.

6. The hand tracking method according to claim 1, wherein: The tracking frame filtering function The specific steps include: a proportion calculation step, calculating the proportion of bone points that are not within the range of the hand image; Proportion calculation step, calculate the length ratio of each bone; as well as The distance calculation step calculates the distance between the center of the skeleton point and the center of the tracking frame.

7. A data processing device, characterized in that: include: a memory for storing executable program code; as well as A processor reads the executable program code to run a computer program corresponding to the executable program code to perform at least one step of the hand tracking method according to any one of claims 1 to 6.

8. A head-mounted device, characterized in that: include: The data processing device according to claim 7; as well as A multi-camera is electrically connected to the data processing device.

Images