Depth-guided gesture tracking method, system, equipment and medium

Abstract

A depth-guided gesture tracking method is characterized by comprising the following steps: S1, acquiring a first frame of image, and obtaining an initial hand region according to a preset depth range; s2, performing hand recognition on the initial hand region to obtain a first hand region; s3, recognizing a hand key point and a hand posture of the first hand region; s4, determining the position of a central hand key point of the first hand region; s5, calculating a motion included angle theta between the central hand key point and the previous frame of image in the three-dimensional space; and S6, determining a complete movement track of the hand according to the sum of the movement included angles theta, the position of the central hand key point and the movement amplitude. The method has the characteristics of real-time performance, good stability, high accuracy, rapidness and high efficiency.

Description

Background Art

[0002] Gesture tracking refers to the use of computer vision technology or sensor technology to identify and track the movement trajectory and gestures of human hands, thereby achieving hand interaction and control. Gesture tracking technology can be widely used in human-computer interaction, virtual reality, smart home and other fields.

[0003] Gesture tracking technology mainly includes the following technologies:

[0004] Gesture recognition technology based on image processing: By collecting hand images, extracting hand features and texture information, and using a classifier model to recognize gestures. Common features include hand shape, color, texture, contour, etc.

[0005] Gesture recognition technology based on depth camera: By using depth sensors, a more accurate three-dimensional hand model can be obtained, combined with algorithms based on machine learning, more accurate gesture tracking and recognition can be achieved.

[0006] Gesture recognition technology based on inertial sensors: By using inertial sensors such as accelerometers and gyroscopes to measure hand acceleration and angular velocity, gestures can be recognized. This technology can achieve gesture recognition without the need for a camera.

[0007] Gesture recognition technology based on traditional sensors: By using traditional sensors such as gloves and wristbands, the hand movement trajectory and posture can be measured to achieve hand interaction and control.

[0008] Gesture tracking technology is widely used in human-computer interaction, virtual reality, smart home and other fields, but it still faces problems such as recognition accuracy, latency, and variability, and needs continuous improvement and optimization.

[0009] The disadvantages of the gesture tracking technology in the prior art are as follows:

[0010] Sensitive to lighting and background: Gesture tracking requires collecting and analyzing hand images, but in low light or complex background conditions, the hand images may be blurred or unrecognizable, affecting the tracking and recognition accuracy of gestures.

[0011] Variability: The hand shapes and gestures of different people may vary greatly, and the same gesture may have many different variations, which will increase the difficulty of recognition and the error rate.

[0012] Latency: Gesture tracking usually requires processing multiple frames of images to determine the gesture, so there is a certain delay, which may affect real-time performance and user experience.

[0013] Operational restrictions: Gesture tracking usually requires the use of hands for operation, so a certain distance and angle must be maintained, and it cannot be too close or too twisted, otherwise the recognition effect will be affected. In addition, hands must be kept clean and dry to avoid sweat and dust that interfere with gesture recognition.

[0014] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0015] To this end, the present invention obtains the first frame image in the video, and guides it according to the depth range to obtain the first hand area, and then calculates the movement angle of the hand key points in the three-dimensional space through the hand key points of different frames, and determines the complete movement trajectory of the hand in combination with the movement amplitude, so as to better identify the meaning of the gesture and perform various gesture interactions.

[0016] In a first aspect, the present invention provides a depth-guided gesture tracking method, characterized in that it comprises the following steps:

[0017] Step S1: Acquire a first frame image, and obtain an initial hand area according to a preset depth range; wherein the first frame image includes a one-to-one corresponding depth map and a second image; and the second image is one of an RGB map, a spot map, and an infrared map;

[0018] Step S2: performing hand recognition on the initial hand region to obtain a first hand region;

[0019] Step S3: identifying the hand key points and hand postures of the first hand area;

[0020] Step S4: determining the position of the central hand key point of the first hand region; wherein the central hand key point is the average of one or more of the palm key points;

[0021] Step S5: Calculate the motion angle θ between the central hand key point in the three-dimensional space and the previous frame image;

[0022] Step S6: Determine the complete movement trajectory of the hand by calculating the sum of the movement angle θ, the position of the central hand key point and the movement amplitude.

[0023] Optionally, the depth-guided gesture tracking method is characterized in that step S2 comprises:

[0024] Step S21: performing hand recognition on the initial hand region to obtain a first region;

[0025] Step S22: completing the incomplete hand in the first region to obtain a second region;

[0026] Step S23: Compare the second area with the first hand area in the second frame image to obtain the position of the completely obscured hand, and then merge it with the second area to obtain the first hand area; wherein the second frame image is the previous frame of the first frame image.

[0027] Optionally, the depth-guided gesture tracking method is characterized in that step S3 comprises:

[0028] Step S31: identifying the hand key points and hand postures of the first hand area;

[0029] Step S32: performing correlation analysis on the plurality of first hand regions to determine whether different first hand regions belong to the same person;

[0030] Step S33: based on the comparison between the hand posture and the preset posture, the first hand region having a posture different from the preset posture is eliminated.

[0031] Optionally, the depth-guided gesture tracking method is characterized by further comprising:

[0032] Step S34: Compare with the first several frames before the first frame image to determine the valid first hand area.

[0033] Optionally, the depth-guided gesture tracking method is characterized in that step S5 comprises:

[0034] Step S51: calculating the three-dimensional space coordinates P1 of the central hand key point in the first frame image and the three-dimensional space coordinates P2 of the second frame image;

[0035] Step S52: Obtaining the vector of the central hand key point moving from the second frame image to the second frame image

[0036] Step S53: calculating the three-dimensional space coordinates P3 of the central hand key point in the third frame image; wherein the third frame image is the previous frame of the second frame image;

[0037] Step S54: Obtaining the vector of the central hand key point moving from the third frame image to the second frame image

[0038] Step S55: Calculate and obtain vector With vector The angle θ at point P2.

[0039] Optionally, the depth-guided gesture tracking method is characterized in that step S6 comprises:

[0040] Step S61: Calculate the motion amplitude of each frame of image using the central hand key point in multiple frames of image;

[0041] Step S62: when the movement amplitude meets the preset requirement, calculating the offset angle θ' of the vector of the central hand key point in the subsequent frame relative to the vector of the central hand key point in the previous frame;

[0042] Step S63: Calculate the sum of multiple offset angles θ', and when the sum of the offset angles θ' is greater than or equal to 360 degrees, determine that the left hand or the right hand completes a closed-loop motion.

[0043] Optionally, the depth-guided gesture tracking method is characterized in that, when calculating the offset angle, the position of the central hand key point in different frames is first smoothed, and then the offset angle is calculated.

[0044] In a second aspect, the present invention provides a depth-guided gesture tracking system, which is used to implement any of the depth-guided gesture tracking methods described above, and is characterized by comprising:

[0045] An initial region module is used to obtain a first frame image and obtain an initial hand region according to a preset depth range; wherein the first frame image includes a one-to-one corresponding depth map and a second image; and the second image is one of an RGB map, a spot map, and an infrared map;

[0046] A hand region module, used for performing hand recognition on the initial hand region to obtain a first hand region;

[0047] A key point module, used to identify hand key points and hand postures of the first hand area;

[0048] A motion trajectory module, used to determine the position of a central hand key point of the first hand region; wherein the central hand key point is the average of one or more of the palm key points;

[0049] A motion angle module is used to calculate the motion angle θ between the central hand key point in the three-dimensional space and the previous frame image;

[0050] The judgment module is used to determine the complete movement trajectory of the hand through the sum of the movement angle θ, the position of the central hand key point and the movement amplitude.

[0051] In a third aspect, the present invention provides a depth-guided gesture tracking device, characterized in that it includes:

[0052] processor;

[0053] a memory storing executable instructions of the processor;

[0054] The processor is configured to perform any of the steps of the depth-guided gesture tracking method described above by executing the executable instructions.

[0055] In a fourth aspect, the present invention provides a computer-readable storage medium for storing a program, characterized in that when the program is executed, the steps of any one of the aforementioned depth-guided gesture tracking methods are implemented.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] The present invention guides the identification of the first hand area according to a preset depth range, reduces the identification range, reduces the amount of calculation, is conducive to improving the calculation speed, and reduces the requirement for computing power.

[0058] The present invention can accurately measure the movement of the palm, better identify the meaning of the palm movement, and perform various effective interactions by extracting the first hand area and the central hand key points of different frame images.

[0059] The present invention does not need to establish a complete hand model, but only needs to identify the key information of the hand for identification and processing, which greatly reduces the amount of calculation, is conducive to improving the processing speed, and reduces the requirement for computing power.

[0060] The present invention does not require high standards for the discrimination of hand shapes and gestures, and only needs to obtain a rough outline. Therefore, it has high adaptability to different groups of people, is conducive to obtaining stable analysis results, and improves recognition accuracy.

[0061] The present invention utilizes gesture information in three-dimensional space for analysis, greatly reducing restrictions on the operator's hand direction, distance, etc., allowing the operator to have higher operating flexibility and operating space, and has good adaptability to various application scenarios.

[0062] The present invention can obtain hand posture and movement status in real time with small calculation amount, can quickly track and respond to gestures, shorten reaction time, reduce delay, and improve user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0064] Figure 1 A flowchart of the steps of a depth-guided gesture tracking method according to an embodiment of the present invention;

[0065] Figure 2 This is a schematic diagram of a first hand region in an embodiment of the present invention;

[0066] Figure 3 A schematic diagram of a motion trajectory in an embodiment of the present invention;

[0067] Figure 4 This is a flowchart of a step of obtaining a first hand region in an embodiment of the present invention;

[0068] Figure 5 This is a flowchart of steps for identifying key points and hand gestures in an embodiment of the present invention;

[0069] Figure 6 Another flowchart of steps for identifying hand key points and hand gestures in an embodiment of the present invention;

[0070] Figure 7 This is a flow chart of the steps of calculating the motion angle in an embodiment of the present invention;

[0071] Figure 8 is a schematic diagram of a motion angle in an embodiment of the present invention;

[0072] Fig. 9 A flowchart of steps for obtaining a complete motion trajectory in an embodiment of the present invention;

[0073] Fig.10 A schematic diagram of a motion trajectory in an embodiment of the present invention;

[0074] Fig.11 is a structural schematic diagram of a depth-guided gesture tracking system in an embodiment of the present invention;

[0075] Fig.12 is a schematic structural diagram of a depth-guided gesture tracking device in an embodiment of the present invention; and

[0076] Fig.13 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0078] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0079] An embodiment of the present invention provides a depth-guided gesture tracking method, which aims to solve the problems existing in the prior art.

[0080] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0081] The present invention obtains the first frame image in the video, and guides it according to the depth range to obtain the first hand area, and then calculates the movement angle of the hand key points in the three-dimensional space through the hand key points in different frames, and determines the complete movement trajectory of the hand in combination with the movement amplitude, so as to better identify the meaning of the gesture and perform various gesture interactions, with the characteristics of good stability, high accuracy, fast and efficient.

[0082] Figure 1 FIG. 1 is a flowchart of a depth-guided gesture tracking method according to an embodiment of the present invention. Figure 1 As shown, the steps of a depth-guided gesture tracking method in an embodiment of the present invention include:

[0083] Step S1: Acquire the first frame image and obtain the initial hand area according to a preset depth range.

[0084] In this step, the first frame image includes a one-to-one corresponding depth map and a second image; the second image is one of an RGB image, a spot image, and an infrared image. When obtaining the first frame image, a depth camera or other depth sensor can be used to obtain a depth image of the current scene. Figure 2 As shown, the RGBD image contains depth information and image information of the hand. The depth map and the second image are aligned at the pixel level. By selecting the depth map, the corresponding range on the second image can be directly obtained.

[0085] Step S2: performing hand recognition on the initial hand region to obtain a first hand region.

[0086] In this step, hand recognition is performed on the second image of the initial hand region to obtain the first hand region. When performing hand recognition, various methods for hand recognition in the prior art can be used, such as deep learning neural network, feature matching, semantic recognition, etc.

[0087] In some embodiments, when determining the initial hand region, three sub-steps are required: hand detection, left and right hand screening, and hand region determination:

[0088] S201: Perform hand detection: Use a hand detection algorithm in the depth image, such as a hand detection algorithm based on a neural network or other computer vision algorithms, to detect each hand in the image. Hand detection can be performed using an RGB image, an infrared image, or a depth image. Of course, hand detection can also be performed using a combination of any two images. The hand detection algorithm can use any algorithm, and only needs to identify the outline of the hand. This step does not require high accuracy for hand recognition.

[0089] S202: Screening left and right hands: For each detected hand, use a computer vision algorithm to determine whether it is a left hand or a right hand. For example, the position and direction of the hand can be used to determine whether the hand is a left hand or a right hand. It can also be determined based on the relative positions of the two hands. The left hand and the right hand represent different meanings, so they need to be distinguished. For example, when the left hand is clenched into a fist and the right hand is palm, the left hand does not move and the right hand draws a circle, which means that the distinction between the left hand and the right hand is of great significance for identifying the meaning of the gesture.

[0090] S203: Determine the hand region: For each instance of the left and right hands, a computer vision algorithm may be used to determine the hand region. For example, an image segmentation algorithm may be used to separate the hand region. This step determines the left hand region and the right hand region, respectively.

[0091] Step S3: Identify the hand key points and hand postures of the first hand area.

[0092] In this step, key points are extracted from the first hand area to determine the hand posture. When extracting the hand key points, a computer vision method, such as a convolutional neural network (CNN) or a support vector machine (SVM), is used to extract key points from the palm image. These feature points may include finger tips, joints, and the center of the palm. When determining the posture of the left and right hands, the left and right and direction of the palm can be determined in combination with the identified hand key points. For example, if the key point of the thumb is on the right side of the palm, it is considered to be the right hand, and if the thumb points to the left side of the palm, it is considered to be the left hand. Posture judgment requires combining the coordinates of multiple feature points, such as the coordinates of the wrist, elbow, and palm center, to determine the rotation and tilt of the palm. It should be noted that the judgment of the left and right hands can be performed in this step or in step S2.

[0093] Step S4: Determine the central hand key point position of the first hand area.

[0094] In this step, the central hand key point is the average of one or more of the palm key points. The central hand key point is the hand key point closest to the wrist, such as the key point at the base of the hand. The central hand key point can also be the average of multiple hand key points. For example, when the hand is clenched into a fist, only some hand key points can be obtained. Then, the obtained hand key points are averaged, and the central hand key point obtained has better stability. The central hand key point has better stability than other hand key points and can better evaluate the movement trajectory of the palm.

[0095] Step S5: Calculate the movement angle θ between the central hand key point in the three-dimensional space and the previous frame image.

[0096] In this step, first obtain the position of the central hand key point in the camera coordinate system, which can be represented by a ternary coordinate system, where each point consists of x, y, and z coordinates. Then calculate the vector: use the two coordinates of the central hand key point to calculate the vector. When calculating the vector angle, use the standard formula for the vector to calculate the angle between the two vectors. The vector angle can be calculated using the cosine function, sine function, or scalar product formula. Finally, convert radians to angles: the vector angle is expressed in radians by default and needs to be converted to angles (in degrees). Since the central hand key point in the previous frame image is known, it can be directly used for calculations with the central hand key point of the current frame.

[0097] Step S6: Determine the complete movement trajectory of the hand by calculating the sum of the movement angle θ, the position of the central hand key point and the movement amplitude.

[0098] In this step, the continuous position change of the central hand key point in the three-dimensional space is obtained by using the position of the central hand key point in the current frame and the previous frame image. By comparing adjacent positions, the motion angle θ and the motion amplitude can be obtained. The motion angle can be used to determine the form of the movement, and the motion amplitude can be used to determine the effectiveness of the movement. When the movement amplitude is small, it is judged to remain still; only when the movement amplitude is within the preset range, it is judged to be a valid movement. The image can be processed in real time through the motion angle, so that the calculation of the motion trajectory can be performed in real time, which is more efficient and can effectively solve the problem of poor timeliness in gesture recognition. At the same time, by judging the motion angle in three-dimensional space, the palm movement can be identified more accurately, and the requirements for the palm posture and orientation are greatly reduced. Figure 3 As shown in the figure, in three-dimensional space, the position movement of the palm can be clearly represented by a vector, so that the motion angle θ and the motion amplitude can be calculated at the same time, which greatly facilitates the calculation. The motion trajectory is judged by the motion angle, so that the motion trajectory does not need to be completely closed, and the complete closed-loop motion can be identified, thereby improving the tolerance of gesture recognition.

[0099] Figure 4 FIG. 1 is a flow chart of steps for obtaining a first hand region in an embodiment of the present invention. Figure 4 As shown, in an embodiment of the present invention, a step of obtaining a first hand region is as follows:

[0100] Step S21: performing hand recognition on the initial hand region to obtain a first region.

[0101] In this step, the hand recognition model is used for recognition to obtain a first region. The first region is a region directly recognized by the hand recognition model. The first regions recognized by different hand recognition models may be different. The first region is a region in a three-dimensional space.

[0102] Step S22: completing the incomplete hand in the first region to obtain a second region.

[0103] In this step, the hand that may be partially blocked is completed to simulate a complete hand posture. The hand simulated in this step is an area in the three-dimensional space. The second area is an area in the three-dimensional space. This step generates complete hand information based on partial hand information.

[0104] Step S23: Compare the second area with the first hand area in the second frame image to obtain the position of the completely blocked hand, and then merge it with the second area to obtain the first hand area.

[0105] In this step, the second frame image is the previous frame of the first frame image. Different from the recovery of partially blocked hands in step S23, this step recovers completely blocked hands. Since the hand may be blocked for a short time by the object in front when it moves, this step compares the image of the previous frame with the current frame image to identify the disappeared hand area, and regards the position as the position in the current frame, and merges it with the second area to obtain the first hand area.

[0106] This embodiment can more comprehensively obtain the first hand area by restoring partially occluded and fully occluded hands, thereby having more reliable recognition performance when facing occlusion. Compared with the hand recognition scheme in the prior art, it has better adaptability to dynamic occlusion and can track and interactively recognize gestures more stably.

[0107] Figure 5 FIG. 1 is a flow chart of steps for identifying key points and hand gestures in an embodiment of the present invention. Figure 5 As shown, in an embodiment of the present invention, a step of identifying hand key points and hand postures includes:

[0108] Step S31: Identify the hand key points and hand postures of the first hand area.

[0109] In this step, a variety of methods can be used to identify the hand key points and hand postures of the first hand area. For example, the first deep neural network model is used for identification. Here we can use the first deep neural network model for identification and distinction, and the model is pre-trained to obtain the recognition and classification capabilities of the hand area. By calling the model for inference, the result of the model output is obtained. For example, a binary output vector representing the hand. There are many ways to interpret and visualize the results, for example, the area can be visualized using methods such as heat maps. Finally, based on the results of the model output, the hand area in the first frame image is determined. The output result is the first hand area. The first deep neural network model is a deep neural network, which can be any of a fully connected neural network (Fully Connected Neural Network, referred to as FCN), a convolutional neural network (Convolutional Neural Network, referred to as CNN), a recurrent neural network (Recurrent Neural Network, referred to as RNN), etc.

[0110] Step S32: performing correlation analysis on the multiple first hand regions to determine whether different first hand regions belong to the same person.

[0111] In this step, for the multiple first hand regions obtained in the previous step, correlation analysis is performed on the multiple first hand regions to determine whether different hands belong to the same person. This step can be processed using a second deep neural network model, and by processing the left and right hands in different postures, it can be identified whether different hand regions belong to the same person. By analyzing multiple first hand regions, valid first hand regions can be identified and invalid first hand regions can be filtered out.

[0112] Step S33: based on the comparison between the hand posture and the preset posture, the first hand region having a posture different from the preset posture is eliminated.

[0113] In this step, the processing is combined with the correlation in the previous step. When the left hand and the right hand belong to the same person, they will be processed as a whole. For example, if one hand is in a preset posture and the other hand is in a meaningless posture, the validity of the gesture can be judged. The preset posture is divided into a single-hand posture and a two-hand posture. If there are two hands and it is the same as the two-hand posture, it is judged to be valid. If only one hand is in the preset posture, it is compared with the single-hand posture, and the different first hand areas are eliminated.

[0114] This embodiment recognizes hand key points and hand postures through a deep neural network, analyzes the correlation between multiple first hand areas, and then compares them with preset postures, which greatly improves the accuracy of judging the effectiveness of hand postures, improves anti-interference ability, and has stronger stability.

[0115] In some embodiments, when training the first deep neural network model, interference hand samples are also added to enhance the resistance of the first deep neural network model to interference. Interference hand samples refer to one or more additional hands in addition to the hand that needs to be identified, such as a hand that is far away, a single hand, etc. By adding interference hands, the ability of the first deep neural network model to match and identify two hands can be improved, and the anti-interference ability during gesture recognition can be improved, so that this embodiment has higher stability when applied to scenes with a large number of people.

[0116] Figure 6 FIG. 2 is another flow chart of steps for identifying key points and hand gestures in an embodiment of the present invention. Figure 6 As shown, another step of identifying hand key points and hand gestures in the embodiment of the present invention further includes:

[0117] Step S34: Compare with the first several frames before the first frame image to determine the valid first hand area.

[0118] In this step, considering that the number of first hand areas identified in the previous step may still be interfered, the first hand areas of the current frame are compared with the previous frame to identify continuous and stable first hand areas, thereby filtering out interfering hands that appear for a short time and identifying first hand areas that interact continuously and stably, thereby improving anti-interference performance.

[0119] Figure 7 FIG. 1 is a flow chart of the steps for calculating the motion angle in an embodiment of the present invention. Figure 7 As shown, in an embodiment of the present invention, a step of calculating a motion angle includes:

[0120] Step S51: Calculate the three-dimensional space coordinates P1 of the central hand key point in the first frame image and the three-dimensional space coordinates P2 in the second frame image.

[0121] In this step, the second frame image is the previous frame of the first frame image to establish the camera coordinate system. The three-dimensional space coordinates P1 and P2 of the hand key point in the first frame image and the second frame image are obtained. In this embodiment, all coordinates refer to the coordinates in the camera coordinate system. The hand key point can be the central hand key point in the aforementioned embodiment.

[0122] Step S52: Obtaining the vector of the central hand key point moving from the second frame image to the second frame image

[0123] In this step, the vector It is a vector in three-dimensional space, which can represent the motion characteristics of the key points of the hand.

[0124] Step S53: Calculate the three-dimensional space coordinates P3 of the central hand key point in the third frame image.

[0125] In this step, the third frame image is the previous frame of the second frame image. The coordinates P3 of the key points of the hand in the third frame image are obtained. Compared with the previous embodiment, this embodiment performs calculations based on the data of three frames of images. The first frame image, the second frame image and the third frame image in this embodiment are three frames of images obtained in chronological order. The first frame image, the second frame image and the third frame image can be three frames of images obtained continuously or three frames of images obtained in sequence at fixed intervals.

[0126] Step S54: Obtaining the vector of the central hand key point moving from the third frame image to the second frame image

[0127] In this step, the vector is the displacement of the third frame image relative to the second frame image.

[0128] Step S55: Calculate and obtain vector With vector The angle θ at point P2.

[0129] In this step, if Figure 8 As shown, the vector angle can be calculated using the cosine function, sine function, or scalar product formula.

[0130] If we use the cosine function to calculate the angle between vectors:

[0131]

[0132] here represents the dot product of two vectors, and and They respectively represent the modulus (i.e. length) of the two vectors.

[0133] If we use the sine function to calculate the vector angle:

[0134]

[0135] here Represents the cross product of two vectors.

[0136] If we use the scalar product to calculate the angle between the vectors:

[0137]

[0138] here and They respectively represent the modulus (i.e. length) of the two vectors.

[0139] This embodiment uses the position information in the three frames of images to obtain a vector, and then calculates the vector to accurately obtain the motion angle.

[0140] Fig. 9 FIG. 1 is a flow chart of steps for obtaining a complete motion trajectory in an embodiment of the present invention. Fig. 9 As shown, in an embodiment of the present invention, a step of obtaining a complete motion trajectory includes:

[0141] Step S61: Calculate the motion amplitude of each frame of image using the central hand key point in multiple frames of image.

[0142] In this step, the movement amplitude of the left and right hands in each frame refers to the position change of the left and right hands in the current frame relative to the previous frame. The position change is represented by the displacement of the central hand key point. The movement amplitude is the distance moved by the central hand key point in three-dimensional space, that is, the modulus of the displacement vector, such as

[0143] Step S62: When the movement amplitude meets the preset requirement, the offset angle θ' of the vector of the central hand key point in the subsequent frame relative to the vector of the central hand key point in the previous frame is calculated.

[0144] In this step, the movement amplitude is first compared with the preset requirement. The preset requirement is a preset value for measuring the palm movement amplitude, which can be a fixed pixel or a value related to the depth value of the central hand key point. When the preset requirement is a pixel, it can be any pixel such as 5 pixels, 10 pixels, 20 pixels, 30 pixels, etc., and the pixel at this time is positively correlated with the resolution of the depth camera that obtains the image. When the preset requirement is related to the depth value of the central hand key point, the larger the depth value of the central hand key point, the smaller the value of the preset requirement; conversely, the smaller the depth value of the central hand key point, the larger the value of the preset requirement. For example, when the depth value of the central hand key point is 100, the preset requirement value is 30; when the depth value of the central hand key point is 150, the preset requirement value is 20. The change in the value of the preset requirement is not linearly related to the depth value of the central hand key point.

[0145] Different from the motion angle obtained in step S5, the offset angle θ' obtained in this step also has a direction, that is, in addition to positive values, there are also negative values. Fig.10 As shown in FIG. 1 , when the palm is moving, if there is shaking, the movement angle will change dramatically. However, this step can filter out the influence of short-term shaking by introducing the direction into the calculation.

[0146] Step S63: Calculate the sum of multiple offset angles θ', and when the sum of the offset angles θ' is greater than or equal to 360 degrees, determine that the left hand or the right hand completes a closed-loop motion.

[0147] In this step, multiple offset angles θ' are summed, and when the sum is greater than or equal to 360 degrees, it is determined that one closed-loop motion is completed. If there are multiple movements in the same direction, the sum of the offset angles θ' is compared with an integer multiple of 360 degrees to determine the number of closed-loop motions completed. For example, if the offset angle θ' remains unchanged along the clockwise motion direction and the sum of the offset angles θ' reaches 720 degrees, it is determined that two closed-loop motions are completed; if the sum of the offset angles θ' reaches 1080 degrees, it is determined that three closed-loop motions are completed.

[0148] In some embodiments, when calculating the offset angle, the positions of the central hand key points in different frames are first smoothed, and then the offset angle is calculated.

[0149] In this step, considering the factors such as sudden changes and jitters in the motion trajectory of the central hand key point, the position of the central hand key point is smoothed, and then the offset angle is calculated, which can make the judgment of the closed-loop motion more accurate. Fig.10As shown, position 1 and position 3 are connected, and point 2' on line segment 13 that is closest to 2 is taken as the adjusted central hand key point; position 2' and position 4 are connected, and point 3' on line segment 2'4 that is closest to 3 is taken as the adjusted central hand key point, and adjustments are made in sequence to obtain the positions of all adjusted central hand key points. Of course, the positions of the hand key points can also be smoothed twice or more to further filter out disturbance factors and improve accuracy. When performing smoothing, a variety of methods can be used, such as obtaining the point on the line segment that is closest to the central hand key point in the previous example, or using the midpoint of the line segment, or any other method.

[0150] When implementing this embodiment, the meaning of various hand gestures and motion states can be freely defined. For example, when the left hand is clenched into a fist and the right hand is open, the system is started; the left hand is still, and the right hand draws a circle clockwise to indicate zooming in. In the scheme of turning the left or right hand in circles, the first circle may not be counted, but is only used to evaluate user behavior and make mode selections. When designing gestures, it is necessary to avoid common gestures and easily confusing gestures. For example, the thumbs-up is a common action, which can be set to be meaningless; the "OK" gesture can only be explicitly set to "confirm" or the number "3". When the left or right hand is turning in a circle, if the turning behavior is interrupted, a timer is started, and when the preset time is reached and there is no movement, it is judged that the instruction has ended.

[0151] Fig.11 FIG. 1 is a schematic diagram of a structure of a depth-guided gesture tracking system according to an embodiment of the present invention. Fig.11 As shown, a depth-guided gesture tracking system in an embodiment of the present invention includes:

[0152] An initial region module is used to obtain a first frame image and obtain an initial hand region according to a preset depth range; wherein the first frame image includes a one-to-one corresponding depth map and a second image; and the second image is one of an RGB map, a spot map, and an infrared map;

[0153] A hand region module, used for performing hand recognition on the initial hand region to obtain a first hand region;

[0154] A key point module, used to identify hand key points and hand postures of the first hand area;

[0155] A motion trajectory module, used to determine the position of a central hand key point of the first hand region; wherein the central hand key point is the average of one or more of the palm key points;

[0156] A motion angle module is used to calculate the motion angle θ between the central hand key point in the three-dimensional space and the previous frame image;

[0157] The judgment module is used to determine the complete movement trajectory of the hand through the sum of the movement angle θ, the position of the central hand key point and the movement amplitude.

[0158] Specifically, the initial area module obtains the initial hand area using a preset depth range. The hand area module identifies the initial hand area, which has higher accuracy and less computational complexity than direct image recognition. The key point module identifies the hand key points in the palm area and determines the hand posture. The motion trajectory module determines the central hand key point among the hand key points and determines the motion trajectory of the left and right hands. The motion angle module calculates the motion angle θ of the central hand key point. The judgment module processes the motion angle θ and the motion amplitude between multiple frames of images to obtain the overall motion state, so that the complete motion trajectory of the palm can be analyzed.

[0159] This embodiment obtains the first frame image in the video, and guides it according to the depth range to obtain the first hand area, and then calculates the movement angle of the hand key points in the three-dimensional space through the hand key points of different frames, and determines the complete movement trajectory of the hand in combination with the movement amplitude, so as to better identify the meaning of the gesture and perform various gesture interactions, with the characteristics of good stability, high accuracy, fast and efficient.

[0160] The present invention also provides a depth-guided gesture tracking device in an embodiment, including a processor and a memory storing executable instructions of the processor, wherein the processor is configured to execute the steps of a depth-guided gesture tracking method by executing the executable instructions.

[0161] As mentioned above, this embodiment obtains the first frame image in the video, and guides it according to the depth range to obtain the first hand area, and then calculates the movement angle of the hand key points in the three-dimensional space through the hand key points of different frames, and determines the complete movement trajectory of the hand in combination with the movement amplitude, so as to better identify the meaning of the gesture and perform various gesture interactions, with the characteristics of good stability, high accuracy, fast and efficient.

[0162] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0163] Fig.12 is a schematic diagram of the structure of a depth-guided gesture tracking device in an embodiment of the present invention. Fig.12 The electronic device 600 according to this embodiment of the present invention is described. Fig.12 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0164] like Fig.12 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0165] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various exemplary embodiments of the present invention described in the above-mentioned depth-guided gesture tracking method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0166] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 6201 and / or a cache storage unit 6202 , and may further include a read-only storage unit (ROM) 6203 .

[0167] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0168] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0169] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although Fig.12 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0170] In an embodiment of the present invention, a computer-readable storage medium is also provided for storing a program, and when the program is executed, the steps of a method for deeply guided gesture tracking are implemented. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above-mentioned method for deeply guided gesture tracking of this specification.

[0171] As shown above, this embodiment obtains the first frame image in the video, and guides it according to the depth range to obtain the first hand area, and then calculates the movement angle of the hand key points in the three-dimensional space through the hand key points of different frames, and determines the complete movement trajectory of the hand in combination with the movement amplitude, so as to better identify the meaning of the gesture and perform various gesture interactions, with the characteristics of good stability, high accuracy, fast and efficient.

[0172] Fig.13 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Fig.13 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0173] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0174] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0175] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0176] This embodiment obtains the first frame image in the video, and guides it according to the depth range to obtain the first hand area, and then calculates the movement angle of the hand key points in the three-dimensional space through the hand key points of different frames, and determines the complete movement trajectory of the hand in combination with the movement amplitude, so as to better identify the meaning of the gesture and perform various gesture interactions, with the characteristics of good stability, high accuracy, fast and efficient.

[0177] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.

[0178] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A depth-guided gesture tracking method, characterized in that: The steps include: Step S1: Acquire a first frame image, and obtain an initial hand area according to a preset depth range; wherein the first frame image includes a one-to-one corresponding depth map and a second image; and the second image is one of an RGB map, a spot map, and an infrared map; Step S2: performing hand recognition on the initial hand region to obtain a first hand region; Step S3: identifying the hand key points and hand postures of the first hand area; Step S4: determining the position of the central hand key point of the first hand region; wherein the central hand key point is the average of one or more of the palm key points; Step S5: Calculate the motion angle θ between the central hand key point in the three-dimensional space and the previous frame image; Step S6: Determine the complete movement trajectory of the hand by calculating the sum of the movement angle θ, the position of the central hand key point and the movement amplitude.

2. A depth-guided gesture tracking method according to claim 1, characterized in that: Step S2 includes: Step S21: performing hand recognition on the initial hand region to obtain a first region; Step S22: completing the incomplete hand in the first region to obtain a second region; Step S23: Compare the second area with the first hand area in the second frame image to obtain the position of the completely obscured hand, and then merge it with the second area to obtain the first hand area; wherein the second frame image is the previous frame of the first frame image.

3. A depth-guided gesture tracking method according to claim 1, characterized in that: Step S3 includes: Step S31: identifying the hand key points and hand postures of the first hand area; Step S32: performing correlation analysis on the plurality of first hand regions to determine whether different first hand regions belong to the same person; Step S33: based on the comparison between the hand posture and the preset posture, the first hand region having a posture different from the preset posture is eliminated.

4. A depth-guided gesture tracking method according to claim 3, characterized in that: Also includes: Step S34: Compare with the first several frames before the first frame image to determine the valid first hand area.

5. The depth-guided gesture tracking method according to claim 1, characterized in that: Step S5 includes: Step S51: calculating the three-dimensional space coordinates P1 of the central hand key point in the first frame image and the three-dimensional space coordinates P2 of the second frame image; Step S52: Obtaining the vector of the central hand key point moving from the second frame image to the second frame image Step S53: calculating the three-dimensional space coordinates P3 of the central hand key point in the third frame image; wherein the third frame image is the previous frame of the second frame image; Step S54: Obtaining the vector of the central hand key point moving from the third frame image to the second frame image Step S55: Calculate and obtain vector With vector The angle θ at point P2.

6. A depth-guided gesture tracking method according to claim 1, characterized in that: The step S6 comprises: Step S61: Calculate the motion amplitude of each frame of image using the central hand key point in multiple frames of image; Step S62: when the movement amplitude meets the preset requirement, calculating the offset angle θ' of the vector of the central hand key point in the subsequent frame relative to the vector of the central hand key point in the previous frame; Step S63: Calculate the sum of multiple offset angles θ', and when the sum of the offset angles θ' is greater than or equal to 360 degrees, determine that the left hand or the right hand completes a closed-loop motion.

7. A depth-guided gesture tracking method according to claim 6, characterized in that: When calculating the offset angle, the positions of the central hand key points in different frames are first smoothed, and then the offset angle is calculated.

8. A depth-guided gesture tracking system, used to implement the depth-guided gesture tracking method according to any one of claims 1 to 7, characterized in that: include: An initial region module is used to obtain a first frame image and obtain an initial hand region according to a preset depth range; wherein the first frame image includes a one-to-one corresponding depth map and a second image; and the second image is one of an RGB map, a spot map, and an infrared map; A hand region module, used for performing hand recognition on the initial hand region to obtain a first hand region; A key point module, used to identify hand key points and hand postures of the first hand area; A motion trajectory module, used to determine the position of a central hand key point of the first hand region; wherein the central hand key point is the average of one or more of the palm key points; A motion angle module is used to calculate the motion angle θ between the central hand key point in the three-dimensional space and the previous frame image; The judgment module is used to determine the complete movement trajectory of the hand through the sum of the movement angle θ, the position of the central hand key point and the movement amplitude.

9. A depth-guided gesture tracking device, characterized in that include: processor; a memory storing executable instructions of the processor; The processor is configured to perform the steps of the depth-guided gesture tracking method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the depth-guided gesture tracking method according to any one of claims 1 to 7 are implemented.

Images