Gesture tracking method, system, equipment and medium

By obtaining three-dimensional motion information of hand key points in the video and combining with deep neural network to identify hand areas, the existing gesture tracking technology is solved to solve the problems of light and background sensitivity, variability and operation limitations, achieving high accuracy and Flexible gesture recognition.

CN119088201APending Publication Date: 2024-12-06SHENZHEN GUANGJIAN TECH CO LTD +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310656444.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-05
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing gesture tracking technologies are sensitive to lighting and background, variability and operational limitations, resulting in insufficient recognition accuracy and flexibility.

Method used

By obtaining hand key points in different frames in the video, calculate the motion angle and motion amplitude of hand key points in three-dimensional space, determine the complete motion trajectory of left and right hands, use the deep neural network model to identify the hand area and resist interference , reduce the dependence on hand models.

Benefits of technology

It improves the accuracy and flexibility of gesture recognition, reduces the amount of calculation, adapts to different groups of people and operating environments, and enhances operation flexibility.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088201A_ABST
    Figure CN119088201A_ABST
Patent Text Reader

Abstract

The gesture tracking method comprises the following steps: S1, acquiring a first left hand region and a first right hand region in a first frame of image; obtaining a second left hand area and a second right hand area in the second frame image; s2, identifying hand key points of the first left hand area, the first right hand area, the second left hand area and the second right hand area, and determining postures of the left hand and the right hand; s3, comparing the central hand key point positions of the first left hand area and the second left hand area, and determining the motion trail of the left hand; the central hand key point positions of the first right hand area and the second right hand area are compared, and the motion trail of the right hand is determined; s4, calculating a motion included angle of the central hand key point in a three-dimensional space; and S5, determining complete motion tracks of the left hand and the right hand through the motion included angles and the motion amplitudes among the multiple frames of images. The method has the characteristics of good stability, high accuracy, rapidness and high efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] Gesture tracking refers to the use of computer vision technology or sensor technology to identify and track the movement trajectory and gestures of human hands, thereby achieving hand interaction and control. Gesture tracking technology can be widely used in human-computer interaction, virtual reality, smart home and other fields.

[0003] Gesture tracking technology mainly includes the following technologies:

[0004] Gesture recognition technology based on image processing: By collecting hand images, extracting hand features and texture information, and using a classifier model to recognize gestures. Common features include hand shape, color, texture, contour, etc.

[0005] Gesture recognition technology based on depth cameras: By using depth sensors such as Microsoft Kinect, a more accurate three-dimensional hand model can be obtained, combined with algorithms based on machine learning, more accurate gesture tracking and recognition can be achieved.

[0006] Gesture recognition technology based on inertial sensors: By using inertial sensors such as accelerometers and gyroscopes to measure hand acceleration and angular velocity, gestures can be recognized. This technology can achieve gesture recognition without the need for a camera.

[0007] Gesture recognition technology based on traditional sensors: By using traditional sensors such as gloves and wristbands, the hand movement trajectory and posture can be measured to achieve hand interaction and control.

[0008] The disadvantages of the gesture tracking technology in the prior art are as follows:

[0009] Sensitive to lighting and background: Gesture tracking requires collecting and analyzing hand images, but in low light or complex background conditions, the hand images may be blurred or unrecognizable, affecting the tracking and recognition accuracy of gestures.

[0010] Variability: The hand shapes and gestures of different people may vary greatly, and the same gesture may have many different variations, which will increase the difficulty of recognition and the error rate.

[0011] Operational restrictions: Gesture tracking usually requires the use of hands for operation, so a certain distance and angle must be maintained, and it cannot be too close or too twisted, otherwise the recognition effect will be affected. In addition, hands must be kept clean and dry to avoid sweat and dust that interfere with gesture recognition.

[0012] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0013] To this end, the present invention obtains hand key points in different frames in the video, calculates the movement angle of the hand key points in three-dimensional space, and determines the complete movement trajectory of the left and right hands in combination with the movement amplitude, so as to better identify the meaning of gestures and perform various gesture interactions.

[0014] In a first aspect, the present invention provides a gesture tracking method, characterized in that it comprises the following steps:

[0015] Step S1: Acquire a first left-hand region and a first right-hand region in a first frame image; acquire a second left-hand region and a second right-hand region in a second frame image; the first frame image and the second frame image contain depth information;

[0016] Step S2: identifying the hand key points of the first left hand area, the first right hand area, the second left hand area and the second right hand area, and determining the postures of the left hand and the right hand;

[0017] Step S3: comparing the positions of the central hand key points of the first left hand region and the second left hand region to determine the movement trajectory of the left hand; comparing the positions of the central hand key points of the first right hand region and the second right hand region to determine the movement trajectory of the right hand; wherein the central hand key point is the average of one or more of the palm key points;

[0018] Step S4: Calculate the movement angle θ of the central hand key point in the three-dimensional space;

[0019] Step S5: Determine the complete motion trajectory of the left hand and the right hand by analyzing the motion angle θ and the motion amplitude between multiple frames of images.

[0020] Optionally, the gesture tracking method is characterized in that step S1 comprises:

[0021] Step S11: using a first deep neural network model to identify the infrared image and the depth image of the first frame image to obtain a first left-hand area and a first right-hand area;

[0022] Step S12: Use the first deep neural network model to identify the infrared image and the depth image of the second frame image to obtain a second left-hand area and a second right-hand area.

[0023] Optionally, the gesture tracking method is characterized in that, during the training of the first deep neural network model, interfering hand samples are also added to enhance the resistance of the first deep neural network model to interference.

[0024] Optionally, the gesture tracking method is characterized in that, in step S3, according to the posture of the left hand, left hand key points are selected from multiple hand key points in the first left hand area and the second left hand area to determine the movement trajectory of the left hand; according to the posture of the right hand, right hand key points are selected from multiple hand key points in the first right hand area and the second right hand area to determine the movement trajectory of the right hand.

[0025] Optionally, the gesture tracking method is characterized in that step S4 comprises:

[0026] Step S41: calculating the three-dimensional space coordinates P1 of the central hand key point in the first frame image and the three-dimensional space coordinates P2 of the second frame image;

[0027] Step S42: Obtaining the vector of the central hand key point moving from the first frame image to the second frame image

[0028] Step S43: calculating the three-dimensional space coordinates P3 of the central hand key point in the third frame image;

[0029] Step S44: Obtaining the vector of the central hand key point moving from the second frame image to the third frame image

[0030] Step S45: Calculate the vector With vector The angle θ at point P2.

[0031] Optionally, the gesture tracking method is characterized in that step S5 comprises:

[0032] Step S51: using the central hand key points in multiple frames of images, respectively calculating the movement amplitude of the left hand and the right hand in each frame of image;

[0033] Step S52: when the movement amplitude meets the preset requirement, calculating the offset angle θ' of the vector of the central hand key point in the subsequent frame relative to the vector of the central hand key point in the previous frame;

[0034] Step S53: Calculate the sum of multiple offset angles θ', and when the sum of the offset angles θ' is greater than or equal to 360 degrees, determine that the left hand or the right hand completes a closed-loop motion.

[0035] Optionally, the gesture tracking method is characterized in that, when calculating the offset angle, the position of the central hand key point in different frames is first smoothed, and then the offset angle is calculated.

[0036] In a second aspect, the present invention provides a gesture tracking system, for implementing any of the above-mentioned gesture tracking methods, characterized in that it comprises:

[0037] A hand region module, used to obtain a first left hand region and a first right hand region in a first frame image; and obtain a second left hand region and a second right hand region in a second frame image; the first frame image and the second frame image contain depth information;

[0038] A key point module, used to identify the hand key points of the first left hand area, the first right hand area, the second left hand area and the second right hand area, and determine the postures of the left hand and the right hand;

[0039] A motion trajectory module, used for comparing the positions of the central hand key points of the first left hand area and the second left hand area to determine the motion trajectory of the left hand; and comparing the positions of the central hand key points of the first right hand area and the second right hand area to determine the motion trajectory of the right hand; wherein the central hand key point is the average of one or more of the palm key points;

[0040] A motion angle module is used to calculate the motion angle θ of the central hand key point in three-dimensional space;

[0041] The judgment module is used to determine the complete movement trajectory of the left hand and the right hand through the movement angle θ and the movement amplitude between multiple frames of images.

[0042] In a third aspect, the present invention provides a gesture tracking device, characterized in that it includes:

[0043] processor;

[0044] a memory storing executable instructions of the processor;

[0045] The processor is configured to perform any one of the steps of the gesture tracking method described above by executing the executable instructions.

[0046] In a fourth aspect, the present invention provides a computer-readable storage medium for storing a program, characterized in that when the program is executed, the steps of any one of the aforementioned gesture tracking methods are implemented.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention can accurately measure the movement of the palm, better identify the meaning of the palm movement, and perform various effective interactions by extracting the left hand area, the right hand area and key points of different frame images.

[0049] The present invention does not need to establish a complete hand model, but only needs to identify the key information of the hand for identification and processing, which greatly reduces the amount of calculation, is conducive to improving the processing speed, and reduces the requirement for computing power.

[0050] The present invention does not require high standards for the discrimination of hand shapes and gestures, and only needs to obtain a rough outline. Therefore, it has high adaptability to different groups of people, is conducive to obtaining stable analysis results, and improves recognition accuracy.

[0051] The present invention utilizes gesture information in three-dimensional space for analysis, greatly reducing restrictions on the operator's hand direction, distance, etc., allowing the operator to have higher operating flexibility and operating space, and has good adaptability to various application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0053] Figure 1 A flowchart of a gesture tracking method according to an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of a first left-hand area and a first right-hand area in an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of a motion trajectory in an embodiment of the present invention;

[0056] Figure 4 This is a flowchart of steps for obtaining a hand region in an embodiment of the present invention;

[0057] Figure 5 This is a flow chart of the steps of calculating the motion angle in an embodiment of the present invention;

[0058] Figure 6 is a schematic diagram of a motion angle in an embodiment of the present invention;

[0059] Figure 7A flowchart of steps for obtaining a complete motion trajectory in an embodiment of the present invention;

[0060] Figure 8 A schematic diagram of a motion trajectory in an embodiment of the present invention;

[0061] Fig. 9 is a structural schematic diagram of a gesture tracking system in an embodiment of the present invention;

[0062] Fig.10 is a schematic structural diagram of a gesture tracking device in an embodiment of the present invention; and

[0063] Fig.11 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0065] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0066] A gesture tracking method provided by an embodiment of the present invention is intended to solve the problems existing in the prior art.

[0067] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0068] The present invention obtains the key points of the hands in different frames in the video, calculates the movement angles of the key points of the hands in three-dimensional space, and determines the complete movement trajectories of the left and right hands in combination with the movement amplitude, so as to better identify the meaning of the gestures and perform various gesture interactions, with the characteristics of good stability, high accuracy, fastness and high efficiency.

[0069] Figure 1 FIG. 1 is a flowchart of a method for tracking gestures in an embodiment of the present invention. Figure 1 As shown, the steps of a gesture tracking method in an embodiment of the present invention include:

[0070] Step S1: Acquire a first left-hand region and a first right-hand region in a first frame image; and acquire a second left-hand region and a second right-hand region in a second frame image.

[0071] In this step, the first frame image and the second frame image contain depth information. The first frame image and the second frame image can be RGBD images, or can be depth images composed of an infrared image and depth information. Of course, they can also be depth images composed of a speckle pattern or an image in another format and depth information, and this embodiment does not limit this. This specification will be described later using infrared images and depth maps as examples, but those skilled in the art will appreciate that the present invention is applicable to the combination of any two images, and the combination of any two images is also within the scope of protection of the present invention. When obtaining the first frame image and the second frame image, a depth camera or other depth sensor can be used to obtain a depth image of the current scene. Figure 2 As shown in FIG. 1 , the left hand and the right hand appear in the same image at the same time. This step identifies the left hand area and the right hand area.

[0072] In some embodiments, when determining the first left hand area and the second right hand area, three sub-steps are required: hand detection, left and right hand screening, and hand area determination:

[0073] S11: Perform hand detection: Use a hand detection algorithm in the depth image, such as a hand detection algorithm based on a neural network or other computer vision algorithms, to detect each hand in the image. Hand detection can be performed using an RGB image, an infrared image, or a depth image. Of course, hand detection can also be performed using a combination of any two images. The hand detection algorithm can use any algorithm, and only needs to identify the outline of the hand. This step does not require high accuracy for hand recognition.

[0074] S12: Screening left and right hands: For each detected hand, use a computer vision algorithm to determine whether it is a left hand or a right hand. For example, the position and direction of the hand can be used to determine whether the hand is a left hand or a right hand. It can also be determined based on the relative position of the two hands. The left hand and the right hand represent different meanings, so they need to be distinguished. For example, when the left hand is clenched into a fist and the right hand is palm, the left hand does not move and the right hand draws a circle, which means that the distinction between the left hand and the right hand is important for identifying the meaning of the gesture.

[0075] S13: Determine the hand region: For each instance of the left and right hands, a computer vision algorithm may be used to determine the hand region. For example, an image segmentation algorithm may be used to separate the hand region. This step determines the left hand region and the right hand region, respectively.

[0076] Step S2: Identify the hand key points of the first left hand area, the first right hand area, the second left hand area and the second right hand area, and determine the postures of the left hand and the right hand.

[0077] In this step, key points are extracted from each hand area to determine the posture of the left and right hands. When extracting the key points of the hand, a computer vision method, such as a convolutional neural network (CNN) or a support vector machine (SVM), is used to extract key points from the palm image. These feature points may include finger tips, joints, and the center of the palm. When determining the posture of the left and right hands, the left and right sides and directions of the palm can be determined by combining the identified key points of the hand. For example, if the key point of the thumb is on the right side of the palm, it is considered to be the right hand, and if the thumb points to the left side of the palm, it is considered to be the left hand. Posture judgment requires combining the coordinates of multiple feature points, such as the coordinates of the wrist, elbow, and palm center, to determine the rotation and tilt of the palm. It should be noted that the judgment of the left and right hands can be performed in this step or in step S1.

[0078] Step S3: Compare the positions of the central hand key points of the first left hand area and the second left hand area to determine the movement trajectory of the left hand; compare the positions of the central hand key points of the first right hand area and the second right hand area to determine the movement trajectory of the right hand.

[0079] In this step, the central hand key point is the average of one or more of the palm key points. The central hand key point is the hand key point closest to the wrist, such as the key point at the base of the hand. The central hand key point can also be the average of multiple hand key points. For example, when the hand is clenched into a fist, only some hand key points can be obtained. Then, the obtained hand key points are averaged, and the central hand key point obtained has better stability. The central hand key point has better stability than other hand key points and can better evaluate the movement trajectory of the palm.

[0080] In some embodiments, according to the posture of the left hand, left hand key points are selected from multiple hand key points in the first left hand area and the second left hand area to determine the movement trajectory of the left hand; according to the posture of the right hand, right hand key points are selected from multiple hand key points in the first right hand area and the second right hand area to determine the movement trajectory of the right hand.

[0081] Step S4: Calculate the movement angle θ of the central hand key point in three-dimensional space.

[0082] In this step, first obtain the position of the center hand key point in the camera coordinate system, which can be represented by a ternary coordinate system, where each point consists of x, y, and z coordinates. Then calculate the vector: use the two coordinates of the center hand key point to calculate the vector. When calculating the vector angle, use the standard formula for vectors to calculate the angle between the two vectors. The vector angle can be calculated using the cosine function, sine function, or scalar product formula. Finally, convert radians to degrees: the vector angle is expressed in radians by default, and it needs to be converted to an angle (in degrees).

[0083] Step S5: Determine the complete motion trajectory of the left hand and the right hand by analyzing the motion angle θ and the motion amplitude between multiple frames of images.

[0084] In this step, the continuous position change of the central hand key point in the three-dimensional space is obtained by using the position of the central hand key point in multiple frames of images. By comparing adjacent positions, the motion angle θ and the motion amplitude can be obtained. The motion angle can be used to determine the form of the movement, and the motion amplitude can be used to determine the effectiveness of the movement. When the movement amplitude is small, it is judged to remain still; only when the movement amplitude is within the preset range is it judged as a valid movement. The image can be processed in real time through the motion angle, so that the calculation of the motion trajectory can be performed in real time, which is more efficient and can effectively solve the problem of poor timeliness in gesture recognition. At the same time, by judging the motion angle in three-dimensional space, the palm movement can be identified more accurately, and the requirements for the palm posture and orientation are greatly reduced. Figure 3 As shown in the figure, in three-dimensional space, the position movement of the palm can be clearly represented by a vector, so that the motion angle θ and the motion amplitude can be calculated at the same time, which greatly facilitates the calculation. The motion trajectory is judged by the motion angle, so that the motion trajectory does not need to be completely closed, and the complete closed-loop motion can be identified, thereby improving the tolerance of gesture recognition.

[0085] Figure 4 FIG. 1 is a flowchart of a step of obtaining a hand region in an embodiment of the present invention. Figure 4 As shown, in an embodiment of the present invention, a step of obtaining a hand region includes:

[0086] Step S11: using a first deep neural network model to identify the infrared image and the depth image of the first frame image to obtain a first left-hand area and a first right-hand area.

[0087] In this step, the infrared image and the depth image are passed as input to the first deep neural network model. Here we can use the first deep neural network model for identification and distinction, which is pre-trained to obtain the ability to identify and classify the hand area. By calling the model for inference, the model output result is obtained. For example, a binary output vector representing the left hand or the right hand. There are many ways to interpret and visualize the results, for example, the area can be visualized using methods such as heat maps. Finally, based on the results of the model output, the hand area in the first frame image is determined. The output results are the first left hand area and the first right hand area. The first deep neural network model is a deep neural network, which can be any of a fully connected neural network (FCN), a convolutional neural network (CNN), a recurrent neural network (RNN), etc.

[0088] Step S12: Use the first deep neural network model to identify the infrared image and the depth image of the second frame image to obtain a second left-hand area and a second right-hand area.

[0089] In this step, the specific steps are the same as those in step S11 and will not be repeated here.

[0090] This embodiment uses a deep neural network to identify the infrared image and the depth image to obtain the left-hand area and the right-hand area. By utilizing the advantages of the deep neural network, the recognition of the left-hand area and the right-hand area is achieved with greater stability.

[0091] In some embodiments, when training the first deep neural network model, interference hand samples are also added to enhance the resistance of the first deep neural network model to interference. Interference hand samples refer to one or more additional hands in addition to the hand that needs to be identified, such as a hand that is far away, a single hand, etc. By adding interference hands, the ability of the first deep neural network model to match and identify two hands can be improved, and the anti-interference ability during gesture recognition can be improved, so that this embodiment has higher stability when applied to scenes with a large number of people.

[0092] Figure 5 FIG. 1 is a flow chart of the steps for calculating the motion angle in an embodiment of the present invention. Figure 5 As shown, in an embodiment of the present invention, a step of calculating a motion angle includes:

[0093] Step S41: Calculate the three-dimensional space coordinates P1 of the central hand key point in the first frame image and the three-dimensional space coordinates P2 in the second frame image.

[0094] In this step, a camera coordinate system is established to obtain the three-dimensional spatial coordinates P1 and P2 of the hand key point in the first frame image and the second frame image. All coordinates in this embodiment refer to the coordinates in the camera coordinate system. The hand key point can be the central hand key point in the above embodiment.

[0095] Step S42: Obtaining the vector of the central hand key point moving from the first frame image to the second frame image

[0096] In this step, the vector It is a vector in three-dimensional space, which can represent the motion characteristics of the key points of the hand.

[0097] Step S43: Calculate the three-dimensional space coordinates P3 of the central hand key point in the third frame image.

[0098] In this step, the coordinates P3 of the key points of the hand in the third frame image are obtained. Compared with the previous embodiment, this embodiment performs calculations based on the data of three frames of images. The first frame image, the second frame image and the third frame image in this embodiment are three frames of images obtained in chronological order. The first frame image, the second frame image and the third frame image can be three frames of images obtained continuously, or three frames of images obtained in sequence at fixed intervals.

[0099] Step S44: Obtaining the vector of the central hand key point moving from the second frame image to the third frame image

[0100] In this step, the vector is the displacement of the third frame image relative to the second frame image.

[0101] Step S45: Calculate the vector With vector The angle θ at point P2.

[0102] In this step, if Figure 6 As shown, the vector angle can be calculated using the cosine function, sine function, or scalar product formula.

[0103] If we use the cosine function to calculate the angle between vectors:

[0104]

[0105] here represents the dot product of two vectors, and and They respectively represent the modulus (i.e. length) of the two vectors.

[0106] If we use the sine function to calculate the angle between the vectors:

[0107]

[0108] here Represents the cross product of two vectors.

[0109] If we use the scalar product to calculate the angle between the vectors:

[0110]

[0111] here and They respectively represent the modulus (i.e. length) of the two vectors.

[0112] This embodiment uses the position information in the three frames of images to obtain a vector, and then calculates the vector to accurately obtain the motion angle.

[0113] Figure 7 FIG. 1 is a flow chart of steps for obtaining a complete motion trajectory in an embodiment of the present invention. Figure 7 As shown, in an embodiment of the present invention, a step of obtaining a complete motion trajectory includes:

[0114] Step S51: using the central hand key points in multiple frames of images, respectively calculating the movement amplitude of the left hand and the right hand in each frame of image.

[0115] In this step, the movement amplitude of the left and right hands in each frame refers to the position change of the left and right hands in the current frame relative to the previous frame. The position change is represented by the displacement of the central hand key point. The movement amplitude is the distance moved by the central hand key point in three-dimensional space, that is, the modulus of the displacement vector, such as The range of motion was calculated separately for the left and right hands.

[0116] Step S52: When the movement amplitude meets the preset requirement, the offset angle θ' of the vector of the central hand key point in the subsequent frame relative to the vector of the central hand key point in the previous frame is calculated.

[0117] In this step, the movement amplitude is first compared with the preset requirement. The preset requirement is a preset value for measuring the palm movement amplitude, which can be a fixed pixel or a value related to the depth value of the central hand key point. When the preset requirement is a pixel, it can be any pixel such as 5 pixels, 10 pixels, 20 pixels, 30 pixels, etc., and the pixel at this time is positively correlated with the resolution of the depth camera that obtains the image. When the preset requirement is related to the depth value of the central hand key point, the larger the depth value of the central hand key point, the smaller the value of the preset requirement; conversely, the smaller the depth value of the central hand key point, the larger the value of the preset requirement. For example, when the depth value of the central hand key point is 100, the preset requirement value is 30; when the depth value of the central hand key point is 150, the preset requirement value is 20. The change in the value of the preset requirement is not linearly related to the depth value of the central hand key point.

[0118] Different from the motion angle obtained in step S4, the offset angle θ' obtained in this step also has a direction, that is, in addition to positive values, there are also negative values. Figure 8 As shown in FIG. 1 , when the palm is moving, if there is shaking, the movement angle will change dramatically. However, this step can filter out the influence of short-term shaking by introducing the direction into the calculation.

[0119] Step S53: Calculate the sum of multiple offset angles θ', and when the sum of the offset angles θ' is greater than or equal to 360 degrees, determine that the left hand or the right hand completes a closed-loop motion.

[0120] In this step, multiple offset angles θ' are summed, and when the sum is greater than or equal to 360 degrees, it is determined that one closed-loop motion is completed. If there are multiple movements in the same direction, the sum of the offset angles θ' is compared with an integer multiple of 360 degrees to determine the number of closed-loop motions completed. For example, if the offset angle θ' remains unchanged along the clockwise motion direction and the sum of the offset angles θ' reaches 720 degrees, it is determined that two closed-loop motions are completed; if the sum of the offset angles θ' reaches 1080 degrees, it is determined that three closed-loop motions are completed.

[0121] In some embodiments, when calculating the offset angle, the positions of the central hand key points in different frames are first smoothed, and then the offset angle is calculated.

[0122] In this step, considering the factors such as sudden changes and jitters in the motion trajectory of the central hand key point, the position of the central hand key point is smoothed, and then the offset angle is calculated, which can make the judgment of the closed-loop motion more accurate. Figure 8As shown, position 1 and position 3 are connected, and point 2' on line segment 13 that is closest to 2 is taken as the adjusted central hand key point; position 2' and position 4 are connected, and point 3' on line segment 2'4 that is closest to 3 is taken as the adjusted central hand key point, and adjustments are made in sequence to obtain the positions of all adjusted central hand key points. Of course, the positions of the hand key points can also be smoothed twice or more to further filter out disturbance factors and improve accuracy. When performing smoothing, a variety of methods can be used, such as obtaining the point on the line segment that is closest to the central hand key point in the previous example, or using the midpoint of the line segment, or any other method.

[0123] When implementing this embodiment, the meaning of various hand gestures and motion states can be freely defined. For example, when the left hand is clenched into a fist and the right hand is open, the system is started; the left hand is still, and the right hand draws a circle clockwise to indicate zooming in. In the scheme of turning the left or right hand in circles, the first circle may not be counted, but is only used to evaluate user behavior and make mode selections. When designing gestures, it is necessary to avoid common gestures and easily confusing gestures. For example, the thumbs-up is a common action, which can be set to be meaningless; the "OK" gesture can only be explicitly set to "confirm" or the number "3". When the left or right hand is turning in a circle, if the turning behavior is interrupted, a timer is started, and when the preset time is reached and there is no movement, it is judged that the instruction has ended.

[0124] Fig. 9 FIG. 1 is a schematic diagram of a gesture tracking system according to an embodiment of the present invention. Fig. 9 As shown, a gesture tracking system in an embodiment of the present invention includes:

[0125] A hand region module, used to obtain a first left hand region and a first right hand region in a first frame image; and obtain a second left hand region and a second right hand region in a second frame image; the first frame image and the second frame image contain depth information;

[0126] A key point module, used to identify the hand key points of the first left hand area, the first right hand area, the second left hand area and the second right hand area, and determine the postures of the left hand and the right hand;

[0127] A motion trajectory module, used for comparing the positions of the central hand key points of the first left hand area and the second left hand area to determine the motion trajectory of the left hand; and comparing the positions of the central hand key points of the first right hand area and the second right hand area to determine the motion trajectory of the right hand; wherein the central hand key point is the average of one or more of the palm key points;

[0128] A motion angle module is used to calculate the motion angle θ of the central hand key point in three-dimensional space;

[0129] The judgment module is used to determine the complete movement trajectory of the left hand and the right hand through the movement angle θ and the movement amplitude between multiple frames of images.

[0130] Specifically, the hand area module obtains the image and identifies the palm area. The key point module identifies the hand key points in the palm area and then determines the hand posture. The motion trajectory module determines the central hand key point among the hand key points and determines the motion trajectory of the left and right hands. The motion angle module calculates the motion angle θ of the central hand key point. The judgment module processes the motion angle θ and the motion amplitude between multiple frames of images to obtain the overall motion state, so that the complete motion trajectory of the palm can be analyzed.

[0131] This embodiment obtains the hand key points of different frames in the video, calculates the movement angle of the hand key points in three-dimensional space, and determines the complete movement trajectory of the left and right hands in combination with the movement amplitude, so as to better identify the meaning of gestures and perform various gesture interactions, with the characteristics of good stability, high accuracy, fast and efficient.

[0132] The present invention also provides a gesture tracking device in an embodiment, including a processor and a memory storing executable instructions of the processor, wherein the processor is configured to execute the steps of a gesture tracking method by executing the executable instructions.

[0133] As mentioned above, this embodiment obtains the hand key points of different frames in the video, calculates the movement angle of the hand key points in the three-dimensional space, and determines the complete movement trajectory of the left and right hands in combination with the movement amplitude, so as to better identify the meaning of the gestures and perform various gesture interactions, with the characteristics of good stability, high accuracy, fast and efficient.

[0134] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0135] Fig.10 is a schematic diagram of the structure of a gesture tracking device in an embodiment of the present invention. Fig.10 The electronic device 600 according to this embodiment of the present invention is described. Fig.10 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0136] like Fig.10As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0137] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various exemplary embodiments of the present invention described in the above-mentioned gesture tracking method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0138] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0139] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0140] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0141] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although Fig.10Not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0142] In an embodiment of the present invention, a computer-readable storage medium is also provided for storing a program, and the steps of a gesture tracking method are implemented when the program is executed. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above-mentioned gesture tracking method section of this specification.

[0143] As shown above, this embodiment obtains the hand key points of different frames in the video, calculates the movement angle of the hand key points in the three-dimensional space, and determines the complete movement trajectory of the left and right hands in combination with the movement amplitude, so as to better identify the meaning of the gestures and perform various gesture interactions, with the characteristics of good stability, high accuracy, fast and efficient.

[0144] Fig.11 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Fig.11 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0145] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0146] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0147] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0148] This embodiment obtains the hand key points of different frames in the video, calculates the movement angle of the hand key points in three-dimensional space, and determines the complete movement trajectory of the left and right hands in combination with the movement amplitude, so as to better identify the meaning of gestures and perform various gesture interactions, with the characteristics of good stability, high accuracy, fast and efficient.

[0149] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.

[0150] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A gesture tracking method, characterized in that: The steps include: Step S1: Acquire a first left-hand region and a first right-hand region in a first frame image; acquire a second left-hand region and a second right-hand region in a second frame image; the first frame image and the second frame image contain depth information; Step S2: identifying the hand key points of the first left hand area, the first right hand area, the second left hand area and the second right hand area, and determining the postures of the left hand and the right hand; Step S3: comparing the positions of the central hand key points of the first left hand region and the second left hand region to determine the movement trajectory of the left hand; comparing the positions of the central hand key points of the first right hand region and the second right hand region to determine the movement trajectory of the right hand; wherein the central hand key point is the average of one or more of the palm key points; Step S4: Calculate the movement angle θ of the central hand key point in the three-dimensional space; Step S5: Determine the complete motion trajectory of the left hand and the right hand by analyzing the motion angle θ and the motion amplitude between multiple frames of images.

2. A gesture tracking method according to claim 1, characterized in that: Step S1 includes: Step S11: using a first deep neural network model to identify the infrared image and the depth image of the first frame image to obtain a first left-hand area and a first right-hand area; Step S12: Use the first deep neural network model to identify the infrared image and the depth image of the second frame image to obtain a second left-hand area and a second right-hand area.

3. A gesture tracking method according to claim 2, characterized in that: During the training of the first deep neural network model, interference hand samples are also added to enhance the resistance of the first deep neural network model to interference.

4. A gesture tracking method according to claim 1, characterized in that: In step S3, according to the posture of the left hand, left hand key points are selected from multiple hand key points in the first left hand area and the second left hand area to determine the movement trajectory of the left hand; according to the posture of the right hand, right hand key points are selected from multiple hand key points in the first right hand area and the second right hand area to determine the movement trajectory of the right hand.

5. A gesture tracking method according to claim 1, characterized in that: Step S4 includes: Step S41: calculating the three-dimensional space coordinates P1 of the central hand key point in the first frame image and the three-dimensional space coordinates P2 of the second frame image; Step S42: Obtaining the vector of the central hand key point moving from the first frame image to the second frame image Step S43: calculating the three-dimensional space coordinates P3 of the central hand key point in the third frame image; Step S44: Obtaining the vector of the central hand key point moving from the second frame image to the third frame image Step S45: Calculate the vector With vector The angle θ at point P2.

6. A gesture tracking method according to claim 1, characterized in that: The step S5 comprises: Step S51: using the central hand key points in multiple frames of images, respectively calculating the movement amplitude of the left hand and the right hand in each frame of image; Step S52: when the movement amplitude meets the preset requirement, calculating the offset angle θ' of the vector of the central hand key point in the subsequent frame relative to the vector of the central hand key point in the previous frame; Step S53: Calculate the sum of multiple offset angles θ', and when the sum of the offset angles θ' is greater than or equal to 360 degrees, determine that the left hand or the right hand completes a closed-loop motion.

7. A gesture tracking method according to claim 6, characterized in that: When calculating the offset angle, the positions of the central hand key points in different frames are first smoothed, and then the offset angle is calculated.

8. A gesture tracking system, used to implement the gesture tracking method according to any one of claims 1 to 7, characterized in that: include: A hand region module, used for acquiring a first left hand region and a first right hand region in a first frame image; Acquire a second left-hand area and a second right-hand area in a second frame of image; The first frame image and the second frame image contain depth information; A key point module, used to identify the hand key points of the first left hand area, the first right hand area, the second left hand area and the second right hand area, and determine the postures of the left hand and the right hand; A motion trajectory module, used for comparing the positions of the central hand key points of the first left hand area and the second left hand area to determine the motion trajectory of the left hand; and comparing the positions of the central hand key points of the first right hand area and the second right hand area to determine the motion trajectory of the right hand; wherein the central hand key point is the average of one or more of the palm key points; A motion angle module is used to calculate the motion angle θ of the central hand key point in three-dimensional space; The judgment module is used to determine the complete movement trajectory of the left hand and the right hand through the movement angle θ and the movement amplitude between multiple frames of images.

9. A gesture tracking device, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to perform the steps of the gesture tracking method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the gesture tracking method according to any one of claims 1 to 7 are implemented.