Hand Tracking Method, Device, Equipment and Computer Storage Medium

Abstract

The present application discloses a hand tracking method, apparatus, device and computer storage medium. The hand tracking method includes obtaining a user action video; identifying a plurality of hands included in a target frame image to obtain feature information corresponding to the plurality of hands; determining whether there is a target hand that meets a target screening condition among the plurality of hands according to the feature information; and when it is determined that there is a target hand, determining the target hand as the hand to be tracked. According to the embodiments of the present application, while ensuring the accuracy of hand tracking, the limitation on user gesture manipulation can be reduced, and the user experience can be improved.

Description

Technical Field

[0001] The present application relates to computer vision technology, and more particularly to a hand tracking method, apparatus, device and computer storage medium. Background Art

[0002] With the development of computer vision technology, more and more intelligent electronic devices can use cameras to track human movements and behaviors, and receive user control instructions for the device by capturing the user's movements and gestures. In this way, users can control the device within a certain distance without the need for control devices such as remote controls or traditional input devices such as mouse and keyboard. For example, users can call up a specific function in the device system with a simple gesture.

[0003] Typically, in the process of tracking gesture movements, the camera may capture multiple hands, such as two hands of the same user, or multiple hands of multiple users. How to determine the target hand that actually needs to be tracked from the multiple hands captured becomes a problem that needs to be solved urgently.

[0004] The existing hand tracking method is to distinguish the target hand by the iconic accessories worn on the target hand, or to track the target hand by setting a preset control area and making the target hand perform gesture control in the preset control area to ensure that there is only one hand in the area. This results in limited gesture control and reduces the user experience. Summary of the invention

[0005] The embodiments of the present application provide a hand tracking method, apparatus, device and computer storage medium, which can solve the problem that gesture control is limited and user experience is poor in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a hand tracking method, the method comprising:

[0007] Obtaining a user action video; wherein the user action video is an action video containing multiple hands;

[0008] Identify multiple hands included in a target frame image, and obtain feature information corresponding to the multiple hands; wherein the target frame image is any frame image except the first frame image in the user action video;

[0009] Determining whether there is a target hand that meets a target screening condition among the multiple hands according to the feature information; wherein the target screening condition includes at least two conditions of being located in a region corresponding to the predicted position, being of the same left and right hand type as the target tracked hand, and being of the same gesture type as the target tracked hand, and the target tracked hand is a tracked hand determined in a frame image previous to the target frame image;

[0010] When it is determined that the target hand exists, the target hand is determined as the tracked hand.

[0011] In an optional implementation manner, the determining whether there is a target hand that meets the target screening condition among the multiple hands according to the feature information includes:

[0012] According to the feature information, determine whether there is a first hand among the multiple hands that meets the first condition in the target screening condition;

[0013] When it is determined that the first hand exists and the number of the first hands is one, the first hand is determined as the tracked hand;

[0014] When it is determined that the first hand exists and the number of the first hands is multiple, according to the feature information, determine whether there is a target hand among the multiple first hands that meets other conditions in the target screening condition except the first condition.

[0015] In an optional implementation manner, the feature information includes position information;

[0016] The determining whether there is a first hand among the multiple hands that meets the first condition in the target screening condition according to the feature information includes:

[0017] Obtain the position information of the tracked hand in at least two frames of images before the target frame image;

[0018] According to the position information of the tracked hand in the at least two frames of images, determine the motion feature of the tracked hand;

[0019] Determine the corresponding area of the predicted position in the target frame image according to the motion feature;

[0020] According to the position information corresponding to the multiple hands respectively, determine whether there is a first hand among the multiple hands that is located in the corresponding area of the predicted position.

[0021] In an optional implementation manner, the feature information further includes left - right hand type information;

[0022] The determining whether there is a target hand among the multiple first hands that meets other conditions in the target screening condition except the first condition according to the feature information includes:

[0023] According to the left - right hand type information corresponding to the multiple first hands respectively, determine whether there is a second hand among the multiple first hands that has the same left - right hand type as the target tracked hand;

[0024] When it is determined that the target hand exists, determining the target hand as the tracked hand includes:

[0025] When it is determined that the second hand exists, determining the second hand as the tracked hand.

[0026] In an optional implementation manner, the feature information further includes gesture type information;

[0027] After determining whether there is a second hand among the multiple first hands whose left - right hand type is the same as that of the target tracked hand according to the left - right hand type information corresponding to the multiple first hands, the method further includes:

[0028] When any condition in the first target condition is satisfied, determining whether there is a third hand among the multiple second hands whose gesture type is the same as that of the target tracked hand according to the gesture type information corresponding to the multiple second hands respectively;

[0029] Wherein, the first target condition includes:

[0030] The condition for determining the first hand as the second hand when it is determined that the second hand does not exist;

[0031] The condition for determining that the second hand exists and the number of the second hands is multiple;

[0032] When it is determined that the second hand exists, determining the second hand as the tracked hand includes:

[0033] When it is determined that the third hand exists, determining the third hand as the tracked hand.

[0034] In an optional implementation manner, the feature information further includes an identification probability value;

[0035] When it is determined that the third hand exists, determining the third hand as the tracked hand includes:

[0036] When any condition in the second target condition is satisfied, calculating the overlap degree between the area where the multiple third hands are located and the area corresponding to the predicted position according to the position information corresponding to the multiple third hands respectively;

[0037] Calculating the score values corresponding to the multiple third hands respectively according to the overlap degree and the identification probability value;

[0038] Obtaining the fourth hand with the highest score value from the multiple third hands;

[0039] Determine the fourth hand as the hand to be tracked;

[0040] Among them, the second target condition includes:

[0041] When it is determined that the third hand does not exist, the condition for determining the second hand as the third hand;

[0042] The condition for determining that the third hand exists and the number of the third hands is multiple.

[0043] In an alternative embodiment, the calculating the score values respectively corresponding to the multiple third hands according to the coincidence degree and the recognition probability value includes:

[0044] Calculate the product between the coincidence degree and the recognition probability value corresponding to each third hand among the multiple third hands respectively, and determine the product as the score value corresponding to each third hand.

[0045] In an alternative embodiment, the feature information further includes hand key point position information;

[0046] Before determining whether there is a third hand with the same gesture type as the target hand to be tracked among the multiple second hands according to the gesture type information respectively corresponding to the multiple second hands, the method further includes:

[0047] Determine the gesture type information respectively corresponding to the multiple second hands according to the hand key point position information respectively corresponding to the multiple second hands.

[0048] In a second aspect, an embodiment of the present application provides a hand tracking device, and the hand tracking device includes:

[0049] An acquisition module, configured to acquire a user action video; wherein, the user action video is an action video including multiple hands;

[0050] An identification module, configured to identify multiple hands included in a target frame image, and obtain feature information corresponding to the multiple hands; wherein, the target frame image is any frame image except the first frame image in the user action video;

[0051] A screening module, configured to determine whether there is a target hand that meets a target screening condition among the multiple hands according to the feature information; wherein, the target screening condition includes at least two of being located in a region corresponding to a predicted position, having the same left / right hand type as the target hand to be tracked, and having the same gesture type as the target hand to be tracked, and the target hand to be tracked is the hand to be tracked determined in the previous frame image of the target frame image;

[0052] A first determination module, configured to determine the target hand as the hand to be tracked when it is determined that the target hand exists.

[0053] In an optional implementation manner, the screening module includes:

[0054] A first screening sub-module, configured to determine whether there is a first hand among the multiple hands that meets the first condition in the target screening conditions according to the feature information;

[0055] A first determination sub-module, configured to determine the first hand as the hand to be tracked when it is determined that the first hand exists and the number of the first hands is one;

[0056] A second screening sub-module, configured to determine whether there is a target hand among the multiple first hands that meets other conditions in the target screening conditions except the first condition according to the feature information when it is determined that the first hand exists and the number of the first hands is multiple.

[0057] In an optional implementation manner, the feature information includes position information;

[0058] The first screening sub-module includes:

[0059] An information acquisition unit, configured to acquire the position information of the hand to be tracked in at least two frames of images before the target frame image;

[0060] A feature determination unit, configured to determine the motion feature of the hand to be tracked according to the position information of the hand to be tracked in the at least two frames of images;

[0061] A region determination unit, configured to determine the corresponding region of the predicted position in the target frame image according to the motion feature;

[0062] A target determination unit, configured to determine whether there is a first hand among the multiple hands that is located in the corresponding region of the predicted position according to the position information corresponding to the multiple hands respectively.

[0063] In an optional implementation manner, the feature information further includes left / right hand type information;

[0064] The second screening sub-module includes:

[0065] A first determination unit, configured to determine whether there is a second hand among the multiple first hands that has the same left / right hand type as the target hand to be tracked according to the left / right hand type information corresponding to the multiple first hands respectively;

[0066] The first determination module includes:

[0067] A second determination sub-module, configured to determine the second hand as the hand to be tracked when it is determined that the second hand exists.

[0068] In an alternative embodiment, the feature information further includes gesture type information;

[0069] The second screening sub-module further includes:

[0070] A second determination unit, configured to determine whether there is a third hand among the multiple second hands that has the same gesture type as the target hand to be tracked according to the gesture type information corresponding to the multiple second hands, after determining whether there is a second hand among the multiple first hands that has the same left / right hand type as the target hand to be tracked according to the left / right hand type information corresponding to the multiple first hands respectively, and when any condition in the first target condition is satisfied;

[0071] Wherein, the first target condition includes:

[0072] The condition for determining the first hand as the second hand when it is determined that the second hand does not exist;

[0073] The condition for determining that the second hand exists and the number of the second hands is multiple;

[0074] The second determination sub-module includes:

[0075] A third determination unit, configured to determine the third hand as the hand to be tracked when it is determined that the third hand exists.

[0076] In an alternative embodiment, the feature information further includes an identification probability value;

[0077] The third determination unit includes:

[0078] A first calculation sub-unit, configured to calculate the overlap degree between the area where the multiple third hands are located and the area corresponding to the predicted position according to the position information corresponding to the multiple third hands when any condition in the second target condition is satisfied;

[0079] A second calculation sub-unit, configured to calculate the score values corresponding to the multiple third hands according to the overlap degree and the identification probability value;

[0080] An acquisition sub-unit, configured to acquire a fourth hand with the highest score value from the multiple third hands;

[0081] A determination sub-unit, configured to determine the fourth hand as the hand to be tracked;

[0082] Wherein, the second target condition includes:

[0083] The condition for determining the second hand part as the third hand part when it is determined that the third hand part does not exist;

[0084] The condition for determining that the third hand part exists and the number of the third hand parts is multiple.

[0085] In an alternative embodiment, the second calculation subunit is specifically configured to:

[0086] Calculate the product of the coincidence degree and the recognition probability value corresponding to each third hand part among the multiple third hand parts respectively, and determine the product as the score value corresponding to each third hand part.

[0087] In an alternative embodiment, the feature information further includes hand key point position information;

[0088] The hand tracking device further includes:

[0089] A second determination module, configured to determine the gesture type information corresponding to each of the multiple second hand parts according to the hand key point position information corresponding to each of the multiple second hand parts before determining whether there is a third hand part with the same gesture type as the target hand to be tracked among the multiple second hand parts according to the gesture type information corresponding to each of the multiple second hand parts.

[0090] In a third aspect, an embodiment of the present application provides an electronic device, which includes: a processor and a memory storing computer program instructions;

[0091] When the processor executes the computer program instructions, the hand tracking method described in any one of the embodiments of the first aspect is implemented.

[0092] In a fourth aspect, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the hand tracking method described in any one of the embodiments of the first aspect is implemented.

[0093] In the hand tracking method, device, equipment and computer storage medium in the embodiments of the present application, by identifying multiple hands included in a target frame image in a user action video, using the feature information corresponding to the multiple hands, and comprehensively considering at least two screening conditions among hand position, left / right hand type, and gesture type, a target hand that meets the above at least two screening conditions is screened out from the identified multiple hands. In this way, while ensuring the accuracy of hand tracking, the limitation on the user's gesture manipulation can be reduced, and the user experience can be improved. Description of the Drawings

[0094] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0095] Figure 1 is one of the schematic flowcharts of the hand tracking method provided by an embodiment of the present application;

[0096] Figure 2 is the second of the schematic flowcharts of the hand tracking method provided by an embodiment of the present application;

[0097] Figure 3 is the schematic structural diagram of the hand tracking device provided by another embodiment of the present application;

[0098] Figure 4 is the schematic structural diagram of the electronic device provided by yet another embodiment of the present application. Detailed Embodiments

[0099] The following will describe in detail the features and exemplary embodiments of various aspects of the present application. To make the purpose, technical solutions, and advantages of the present application clearer, the following further describes the present application in detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than limiting the present application. For those skilled in the art, the present application can be implemented without some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0100] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article, or device. Without further limitation, the elements defined by the statement "including..." do not exclude the existence of additional identical elements in the process, method, article, or device including the said elements.

[0101] To solve the problems of the prior art, embodiments of the present application provide a hand tracking method, apparatus, device, and computer storage medium. The hand tracking method can be applied to scenarios where multiple hands are manipulated to track the hand to be tracked. First, the hand tracking method provided by the embodiments of the present application will be introduced below.

[0102] Figure 1 The flowchart of the hand tracking method provided by an embodiment of the present application is shown. As Figure 1 shown, the hand tracking method may specifically include the following steps:

[0103] First, in step 110, obtain a user action video; wherein, the user action video is an action video containing multiple hands.

[0104] Secondly, in step 120, identify multiple hands included in the target frame image to obtain feature information corresponding to the multiple hands; wherein, the target frame image is any frame image other than the first frame image in the user action video.

[0105] Next, in step 130, determine whether there is a target hand that meets the target screening conditions among the multiple hands according to the feature information; wherein, the target screening conditions include at least two of being located within the corresponding area of the predicted position, having the same left / right hand type as the target hand to be tracked, and having the same gesture type as the target hand to be tracked, and the target hand to be tracked is the hand to be tracked determined in the previous frame image of the target frame image.

[0106] Finally, in step 140, when it is determined that there is a target hand, determine the target hand as the hand to be tracked.

[0107] Thus, by identifying multiple hands included in the target frame image of the user action video, using the feature information corresponding to the multiple hands, and comprehensively considering at least two screening conditions among hand position, left / right hand type, and gesture type, the target hand that meets the above at least two screening conditions is screened out from the identified multiple hands. In this way, while ensuring the accuracy of hand tracking, the limitation on user gesture manipulation can be reduced, and the user experience can be improved.

[0108] The above steps will be described in detail below, as follows:

[0109] First, regarding step 110, in the embodiments of the present application, the user action video may be an action video containing multiple hands collected by a camera. Among them, the multiple hands may be two hands of one user or multiple hands of multiple users, which is not limited herein.

[0110] Secondly, regarding step 120, in the embodiments of the present application, each frame image in the user action video is extracted in chronological order, and each frame image is preprocessed, such as operations like scaling and enhancement. Each frame image may include multiple hands, and the target frame image may be any frame image in the user action video except the first frame image. The feature information may include the position information of multiple hands, gesture type information, key point position information, left and right hand type information, etc. Specifically, artificial intelligence (AI) can be used to identify and obtain the feature information corresponding to multiple hands.

[0111] In a specific example, in the case of collecting a user action video containing multiple hands through a camera, each frame image in the user action video is extracted in chronological order, and each frame image is input into a deep neural network model. The deep neural network model identifies multiple hands included in any frame image in the user action video and outputs information such as the position information, gesture type information, key point position information, and left and right hand type information corresponding to multiple hands.

[0112] Next, regarding step 130, in the embodiments of the present application, the predicted position may be the current hand position predicted based on the movement trajectories of the tracked hand positions in at least two frame images before the target frame image. The gesture types may be, for example, a fist and a palm. The execution order of the screening process corresponding to each screening condition is not limited here.

[0113] In a specific example, after obtaining information such as the positions and gesture types of multiple hands in the user action video through AI recognition, the hands located within the area corresponding to the predicted position among multiple hands are determined according to the above information, and then the left and right hand types and gesture types of these hands are compared with the target tracked hand, so as to filter out the hands with the same left and right hand types and gesture types as the target tracked hand, where the target tracked hand is the tracked hand determined in the previous frame image of the target frame image.

[0114] Finally, regarding step 140, in the embodiments of the present application, when it is determined that there are target hands, the target hands are determined as the tracked hands. The number of target hands may be one or multiple, which is not limited here.

[0115] Based on this, in an alternative embodiment, step 130 may specifically include:

[0116] According to the feature information, determine whether there is a first hand among multiple hands that meets the first condition in the target screening condition;

[0117] When it is determined that there is a first hand and the number of the first hands is one, determine the first hand as the tracked hand;

[0118] When it is determined that there is a first hand part and the number of first hand parts is multiple, according to the feature information, it is determined whether there is a target hand part among the multiple first hand parts that satisfies other conditions except the first condition in the target screening conditions.

[0119] Here, it can be sequentially determined whether there is a hand part that satisfies each condition in the target screening conditions. During the screening process, if there is only one hand part that satisfies the conditions, this hand part can be determined as the tracked hand part. If there are multiple hand parts that satisfy the conditions, the screening can continue according to other conditions. Among them, when the number of first hand parts that satisfy the first condition is multiple, the screening process of other conditions can continue, and so on.

[0120] In this way, when the hand part that uniquely satisfies the conditions is screened out, the screening process can be exited in time, improving the hand part tracking efficiency.

[0121] Based on this, in an alternative embodiment, the feature information includes position information. The step of determining whether there is a first hand part that satisfies the first condition in the target screening conditions according to the feature information may specifically include:

[0122] Obtain the position information of the tracked hand part in at least two frames of images before the target frame image;

[0123] According to the position information of the tracked hand part in at least two frames of images, determine the motion characteristics of the tracked hand part;

[0124] Determine the corresponding area of the predicted position in the target frame image according to the motion characteristics;

[0125] According to the position information corresponding to multiple hand parts respectively, determine whether there is a first hand part among the multiple hand parts that is located in the corresponding area of the predicted position.

[0126] Here, the position information may be, for example, the information obtained by positioning an object using a deep neural network model. The motion characteristics of the tracked hand part can be determined according to the position information of the tracked hand part in the previous at least two frames of images, such as the motion speed, acceleration, and motion distance of the tracked hand part. Through these information, a motion trajectory can be obtained, so as to determine the possible position area of the tracked hand part in the target frame image, that is, the corresponding area of the predicted position. The first hand part may be one or more hand parts in the target frame image that are located in the corresponding area of the predicted position.

[0127] Exemplarily, if the tracked hand is moving at a constant speed in at least two frames of images before the target frame image, a motion trajectory corresponding to the tracked hand can be determined, and the area corresponding to the possible position of the tracked hand in the target frame image can be predicted according to the motion speed of the tracked hand. If there is a hand in the target frame image within the area corresponding to the predicted position, it is determined as the tracked hand.

[0128] In this way, through the position information of the tracked hand in at least two frames of images before the target frame image, the position corresponding to the tracked hand in the target frame image can be predicted, which is convenient for determining whether there is a hand in the multiple hands within the area corresponding to the predicted position, thereby improving the accuracy of hand tracking.

[0129] Based on this, in an alternative embodiment, the feature information further includes left / right hand type information;

[0130] The step of determining whether there is a target hand among the multiple first hands that meets other conditions except the first condition in the target screening condition according to the feature information may specifically include:

[0131] According to the left / right hand type information corresponding to the multiple first hands respectively, determine whether there is a second hand among the multiple first hands with the same left / right hand type as the target tracked hand;

[0132] When it is determined that there is a target hand, determining the target hand as the tracked hand may specifically include:

[0133] When it is determined that there is a second hand, determine the second hand as the tracked hand.

[0134] Here, before screening by left / right hand type, hand screening and gesture type screening can be performed according to the position information first, which is not limited here. The left / right hand type information can be determined by using a deep neural network model, for example. The second hand can be the hand among the first hands with the same left / right hand type as the target tracked hand, and there can be one or more such second hands.

[0135] Exemplarily, when there is no second hand, this detection is not performed, and the first hand is still determined as the tracked hand, or the first hand is used as the second hand to continue the subsequent screening process.

[0136] In this way, by comparing the left / right hand types of the first hand and the target tracked hand, hands with the same left / right hand type as the tracked hand in the previous frame image of the target frame image can be screened out, narrowing the screening range, thereby further improving the accuracy of hand tracking.

[0137] Based on this, in an alternative embodiment, the feature information further includes gesture type information;

[0138] After determining whether there is a second hand in the multiple first hands that has the same left - right hand type as the target tracked hand according to the left - right hand type information corresponding to the multiple first hands, the hand tracking method provided by the embodiments of the present application may further include:

[0139] When any of the conditions in the first target condition is satisfied, determine whether there is a third hand in the multiple second hands that has the same gesture type as the target tracked hand according to the gesture type information corresponding to the multiple second hands respectively;

[0140] Wherein, the first target condition includes:

[0141] The condition for determining the first hand as the second hand when it is determined that there is no second hand;

[0142] The condition for determining that there is a second hand and the number of second hands is multiple;

[0143] When it is determined that there is a second hand, determining the second hand as the tracked hand may specifically include:

[0144] When it is determined that there is a third hand, determining the third hand as the tracked hand.

[0145] Here, the gesture type information can be determined by using a deep neural network model, for example, it can be the characteristics of the hand shape, such as making a fist, crossing the palm and fingers, etc. The third hand can be the hand in the multiple second hands that has the same gesture type as the target tracked hand, and it can be one or multiple.

[0146] Exemplarily, when there is no second hand and the first hand is determined as the second hand, or when there is a second hand and the number of second hands is multiple, the left - right hand type can be screened to determine whether there is a third hand that has the same left - right hand type as the tracked hand in the previous frame image. If there is no third hand, this detection is not performed, and the second hand is still determined as the tracked hand, or the second hand is used as the third hand to continue the subsequent screening process.

[0147] In this way, by comparing the gesture types of the second hand and the target tracked hand, the hand that has the same gesture type as the tracked hand in the previous frame image of the target frame image can be screened out, narrowing the screening range, so as to improve the accuracy of hand tracking of the tracked hand in the case of multiple hand manipulations.

[0148] Based on this, in an alternative embodiment, the feature information further includes an identification probability value;

[0149] When it is determined that there is a third hand, determining the third hand as the hand to be tracked may specifically include:

[0150] When any condition in the second target condition is met, calculate the coincidence degree between the regions where multiple third hands are located and the regions corresponding to the predicted positions according to the position information corresponding to the multiple third hands respectively;

[0151] Calculate the score values corresponding to the multiple third hands respectively according to the coincidence degree and the recognition probability value;

[0152] Obtain the fourth hand with the highest score value from the multiple third hands;

[0153] Determine the fourth hand as the hand to be tracked;

[0154] Among them, the second target condition includes:

[0155] When it is determined that there is no third hand, the condition for determining the second hand as the third hand;

[0156] The condition for determining that there is a third hand and the number of third hands is multiple.

[0157] Here, the recognition probability value can be the probability value that the deep neural network model recognizes the target region image as a hand, or the probability value obtained by multiplying this probability value by a coefficient. Among them, the coefficient can be determined by the distance between the center position of the third hand and the center position of the region corresponding to the predicted position. The farther the distance, the smaller the coefficient. The coincidence degree can be determined according to the ratio between the intersection and the union of the region where the third hand is located and the region corresponding to the predicted position. The larger the ratio, the higher the coincidence degree. The fourth hand can be the hand with the highest score value obtained from the multiple third hands.

[0158] Exemplarily, when there is no third hand, the detection of the gesture type may not be performed. At this time, the second hand can be determined as the third hand, and the screening can continue according to the score value, and the fourth hand with the highest score value among them is determined as the hand to be tracked. Or, when there is a third hand and the number of third hands is multiple, the screening can be performed according to the score value, and the fourth hand with the highest score value among them is determined as the hand to be tracked.

[0159] In this way, by calculating the score values corresponding to the multiple third hands respectively according to the coincidence degree and the recognition probability value, the hand with the highest coincidence degree with the region corresponding to the predicted position can be screened out, further improving the accuracy of hand tracking of the hand to be tracked.

[0160] Based on this, in an alternative embodiment, calculating the score values corresponding to the multiple third hands respectively according to the coincidence degree and the recognition probability value may further include:

[0161] Calculate the product of the overlap degree and the recognition probability value corresponding to each of the multiple third hands respectively, and determine the product as the score value corresponding to each third hand.

[0162] Here, the calculation formula of the score value can be Score = IoU * P, where IoU can be the overlap degree between the area where each third hand is located and the corresponding area of the predicted position, P can be the recognition probability value corresponding to each third hand, and Score can be the score value corresponding to each third hand.

[0163] In this way, the score values corresponding to the multiple third hands can be accurately calculated.

[0164] Based on this, in an alternative embodiment, the feature information further includes hand key point position information;

[0165] Before determining whether there is a third hand among the multiple second hands whose gesture type is the same as that of the target tracked hand according to the gesture type information corresponding to the multiple second hands respectively, the method may further include:

[0166] Determine the gesture type information corresponding to the multiple second hands according to the hand key point position information corresponding to the multiple second hands respectively.

[0167] Here, the hand key point position information may be the joint position information of the hand. In addition, the hand key point position information can also be used to determine the hand position information and left / right hand type information corresponding to the multiple second hands, etc.

[0168] In this way, by recognizing the hand key point position information corresponding to the multiple second hands respectively, the accuracy of gesture type recognition can be improved.

[0169] To better describe the entire solution, based on the above embodiments, a specific example is given.

[0170] For example, as Figure 2 shown in the flowchart of the hand tracking method. The hand tracking method may include S201 - S212, which will be explained in detail below.

[0171] S201, Image acquisition.

[0172] In a specific example, a user action video is obtained through a camera, and then each frame of the image is extracted and pre - processed, such as operations like scaling and enhancement.

[0173] S202, AI detection and recognition.

[0174] In a specific example, each frame of the image is fed into a target detection and localization model, such as using a deep neural network model, to locate the positions of all hands in the image.

[0175] S203. Predict the position of the hand being tracked for the target.

[0176] In a specific example, obtain the position information of the hand being tracked in at least two frames of images before the target frame image, determine the motion characteristics of the hand being tracked, and determine the corresponding region of the predicted position in the target frame image according to the motion characteristics.

[0177] S204. Take the hand within the corresponding region of the predicted position as the first hand.

[0178] In a specific example, determine the hand among multiple hands that is within the corresponding region of the predicted position according to the position information corresponding to each of the multiple hands, and take it as the first hand.

[0179] S205. Detect whether there is a second hand with the same left / right hand type as the hand being tracked for the target. If so, execute S206; if not, execute S207.

[0180] In a specific example, determine whether there is a second hand with the same left / right hand type as the hand being tracked for the target among the first hands according to the left / right hand type information corresponding to each of the first hands.

[0181] S206. Determine whether the number of second hands is multiple. If so, execute S208; if not, execute S212.

[0182] In a specific example, if there is a second hand with the same left / right hand type as the hand being tracked for the target among the first hands, determine whether the number of second hands is multiple.

[0183] S207. Ignore this detection and take the first hand as the second hand.

[0184] In a specific example, if there is no second hand with the same left / right hand type as the hand being tracked for the target among the first hands, ignore this detection and continue to take the first hand as the second hand.

[0185] S208. Detect whether there is a third hand with the same gesture type as the hand being tracked for the target. If so, execute S209; if not, execute S210.

[0186] In a specific example, determine whether there is a third hand with the same gesture type as the hand being tracked for the target among the multiple second hands according to the gesture type information corresponding to each of the multiple second hands.

[0187] S209. Determine whether the number of third hands is multiple. If so, execute S211; if not, execute S212.

[0188] In a specific example, if there is a third hand among multiple second hands that has the same gesture type as the target hand to be tracked, determine whether the number of third hands is more than one.

[0189] S210. Ignore this detection and use the second hand as the third hand.

[0190] In a specific example, if there is no third hand among multiple second hands that has the same gesture type as the target hand to be tracked, ignore this detection and use the second hand as the third hand.

[0191] S211. Calculate the score values respectively corresponding to multiple third hands.

[0192] In a specific example, according to the position information respectively corresponding to multiple third hands, calculate the overlap degree between the regions where multiple third hands are located and the regions corresponding to the predicted positions, and calculate the score values respectively corresponding to multiple third hands according to the overlap degree and the recognition probability values.

[0193] S212. Output the detection result.

[0194] In a specific example, the hand with the largest score value respectively corresponding to multiple third hands can be used as the hand to be tracked. Additionally, if there is one second hand among the first hands that has the same left / right hand type as the target hand to be tracked, the second hand can be used as the hand to be tracked. If there is one third hand among the second hands that has the same gesture type as the target hand to be tracked, the third hand can be used as the hand to be tracked.

[0195] Thus, by recognizing multiple hands included in the target frame image in the user action video, using the feature information corresponding to the multiple hands, and comprehensively considering at least two screening conditions among hand position, left / right hand type, and gesture type, the target hand that meets the above at least two screening conditions is screened out from the recognized multiple hands. In this way, while ensuring the accuracy of hand tracking, the limitation on the user's gesture manipulation can be reduced, and the user experience can be improved.

[0196] Figure 3 FIG. is a schematic structural diagram of a hand tracking device shown according to an exemplary embodiment.

[0197] As Figure 3 shown, the hand tracking device 300 may include:

[0198] An acquisition module 301, configured to acquire a user action video; wherein, the user action video is an action video including multiple hands.

[0199] An identification module 302, configured to identify multiple hands included in a target frame image, and obtain feature information corresponding to the multiple hands; wherein, the target frame image is any frame image in a user action video except the first frame image;

[0200] A screening module 303, configured to determine whether there is a target hand among the multiple hands that meets a target screening condition according to the feature information; wherein, the target screening condition includes at least two of being located within a corresponding area of a predicted position, having the same left / right hand type as a target tracked hand, and having the same gesture type as the target tracked hand, and the target tracked hand is the tracked hand determined in the previous frame image of the target frame image;

[0201] A first determination module 304, configured to, when it is determined that there is a target hand, determine the target hand as the tracked hand.

[0202] In an optional implementation manner, the screening module 303 includes:

[0203] A first screening sub-module, configured to determine whether there is a first hand among the multiple hands that meets a first condition in the target screening condition according to the feature information;

[0204] A first determination sub-module, configured to, when it is determined that there is a first hand and the number of first hands is one, determine the first hand as the tracked hand;

[0205] A second screening sub-module, configured to, when it is determined that there is a first hand and the number of first hands is multiple, determine whether there is a target hand among the multiple first hands that meets other conditions in the target screening condition except the first condition according to the feature information.

[0206] In an optional implementation manner, the feature information includes position information;

[0207] The first screening sub-module may specifically include:

[0208] An information acquisition unit, configured to acquire the position information of the tracked hand in at least two frame images before the target frame image;

[0209] A feature determination unit, configured to determine the motion feature of the tracked hand according to the position information of the tracked hand in at least two frame images;

[0210] A region determination unit, configured to determine the corresponding region of the predicted position in the target frame image according to the motion feature;

[0211] A target determination unit, configured to determine whether there is a first hand among the multiple hands that is located within the corresponding region of the predicted position according to the position information corresponding to the multiple hands respectively.

[0212] In an alternative embodiment, the feature information further includes left / right hand type information;

[0213] The second screening sub-module may include:

[0214] A first determination unit, configured to determine whether there is a second hand in the multiple first hands that has the same left / right hand type as the target tracked hand according to the left / right hand type information respectively corresponding to the multiple first hands;

[0215] The first determination module 304 includes:

[0216] A second determination sub-module, configured to, when it is determined that there is a second hand, determine the second hand as the tracked hand.

[0217] In an alternative embodiment, the feature information further includes gesture type information;

[0218] The second screening sub-module may further include:

[0219] A second determination unit, configured to, after determining whether there is a second hand in the multiple first hands that has the same left / right hand type as the target tracked hand according to the left / right hand type information respectively corresponding to the multiple first hands, and when any condition in the first target condition is satisfied, determine whether there is a third hand in the multiple second hands that has the same gesture type as the target tracked hand according to the gesture type information respectively corresponding to the multiple second hands;

[0220] Wherein, the first target condition includes:

[0221] The condition for determining the first hand as the second hand when it is determined that there is no second hand;

[0222] The condition for determining that there is a second hand and the number of second hands is multiple;

[0223] The second determination sub-module includes:

[0224] A third determination unit, configured to, when it is determined that there is a third hand, determine the third hand as the tracked hand.

[0225] In an alternative embodiment, the feature information further includes an identification probability value;

[0226] The third determination unit may include:

[0227] A first calculation sub-unit, configured to, when any condition in the second target condition is satisfied, calculate the coincidence degree between the area where the multiple third hands are located and the area corresponding to the predicted position according to the position information respectively corresponding to the multiple third hands;

[0228] A second calculation subunit, configured to calculate score values respectively corresponding to multiple third hands according to the degree of coincidence and the recognition probability value;

[0229] An acquisition subunit, configured to acquire a fourth hand with the highest score value from multiple third hands;

[0230] A determination subunit, configured to determine the fourth hand as the hand to be tracked;

[0231] Wherein, the second target condition includes:

[0232] When it is determined that there is no third hand, the condition for determining the second hand as the third hand;

[0233] The condition for determining that there is a third hand and the number of third hands is multiple.

[0234] In an alternative embodiment, the second calculation subunit is specifically configured to:

[0235] Calculate the product between the degree of coincidence and the recognition probability value respectively corresponding to each third hand among multiple third hands, and determine the product as the score value corresponding to each third hand.

[0236] In an alternative embodiment, the feature information further includes hand key point position information;

[0237] The hand tracking device 300 may further include:

[0238] A second determination module, configured to determine the gesture type information respectively corresponding to multiple second hands according to the hand key point position information respectively corresponding to multiple second hands before determining whether there is a third hand with the same gesture type as the target hand to be tracked among multiple second hands according to the gesture type information respectively corresponding to multiple second hands.

[0239] Thus, by recognizing multiple hands included in the target frame image in the user action video, using the feature information corresponding to the multiple hands, and comprehensively considering at least two screening conditions among the hand position, left / right hand type, and gesture type, the target hand that meets the above at least two screening conditions is screened out from the recognized multiple hands. In this way, while ensuring the accuracy of hand tracking, the limitation on the user's gesture manipulation can be reduced, and the user experience can be improved.

[0240] Figure 4 Shows the hardware structure schematic diagram of the electronic device provided in the embodiment of the present application.

[0241] The electronic device may include a processor 401 and a memory 402 storing computer program instructions.

[0242] Specifically, the above-mentioned processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be an integrated circuit configured to implement one or more of the embodiments of the present application.

[0243] The memory 402 may include a mass storage for data or instructions. By way of example and not limitation, the memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disc, a magneto-optical disc, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In a suitable case, the memory 402 may include removable or non-removable (or fixed) media. In a suitable case, the memory 402 may be internal or external to the integrated gateway disaster recovery device. In a specific embodiment, the memory 402 is a non-volatile solid-state memory.

[0244] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present application.

[0245] The processor 401 reads and executes the computer program instructions stored in the memory 402 to implement any one of the hand tracking methods in the above embodiments.

[0246] In one example, the electronic device may further include a communication interface 403 and a bus 410. Among them, as Figure 4 shown, the processor 401, the memory 402, and the communication interface 403 are connected through the bus 410 and complete communication with each other.

[0247] The communication interface 403 is mainly used to implement communication between the modules, devices, units, and / or devices in the embodiments of the present application.

[0248] The bus 410 includes hardware, software, or both, and couples the components of the online data flow metering device to each other. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable bus or a combination of two or more of these. Where appropriate, the bus 410 may include one or more buses. Although the embodiments of the present application describe and illustrate specific buses, the present application contemplates any suitable bus or interconnect.

[0249] The electronic device can implement the hand tracking method in the embodiments of the present application based on identifying multiple hands included in the target frame image in the user action video, so as to achieve the combination Figure 1 and Figure 3 the hand tracking method and device described.

[0250] In addition, in combination with the hand tracking method in the above embodiments, the embodiments of the present application can be implemented by providing a computer storage medium. Computer program instructions are stored on the computer storage medium; when the computer program instructions are executed by a processor, any one of the online data flow metering methods in the above embodiments is implemented.

[0251] It should be clear that the present application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present application.

[0252] The functional blocks shown in the above-described structural block diagrams can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, and so on. When implemented in software, the elements of the present application are programs or code segments for performing the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted via a data signal carried in a carrier wave over a transmission medium or a communication link. A "machine-readable medium" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and so on. The code segment can be downloaded via a computer network such as the Internet, an intranet, and so on.

[0253] It should also be noted that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be executed in the order mentioned in the embodiments, or different from the order in the embodiments, or several steps can be executed simultaneously.

[0254] Aspects of the present application have been described above with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each block in the flowcharts and / or block diagrams, and the combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the functions / actions specified in one or more blocks of the flowcharts and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each block in the block diagrams and / or flowcharts, and the combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware for performing the specified functions or actions, or by a combination of dedicated hardware and computer instructions.

[0255] As described above, this is only the specific implementation manner of the present application. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein. It should be understood that the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present application.

Claims

1. A hand tracking method, characterized in that, Including: Obtaining a user action video; wherein, the user action video is an action video including multiple hands. Identifying multiple hands included in a target frame image to obtain feature information corresponding to the multiple hands; wherein, the target frame image is any frame image other than the first frame image in the user action video, and the feature information includes position information, left / right hand type information, gesture type information, and an identification probability value. According to the feature information, determining whether there is a first hand among the multiple hands that satisfies a first condition, where the first condition is being within the area corresponding to the predicted position in the target frame image. When it is determined that there is the first hand and the number of the first hands is multiple, according to the left / right hand type information corresponding to the multiple first hands respectively, determining whether there is a second hand among the multiple first hands that has the same left / right hand type as the target hand to be tracked. When any condition in a first target condition is satisfied, according to the gesture type information corresponding to the multiple second hands respectively, determining whether there is a third hand among the multiple second hands that has the same gesture type as the target hand to be tracked; wherein, the target hand to be tracked is the hand to be tracked determined in the previous frame image of the target frame image. When any condition in a second target condition is satisfied, according to the position information corresponding to the multiple third hands respectively, calculating the overlap degree between the area where the multiple third hands are located and the area corresponding to the predicted position. According to the overlap degree and the identification probability value, calculating score values corresponding to the multiple third hands respectively. Obtaining a fourth hand with the highest score value from the multiple third hands. Determining the fourth hand as the hand to be tracked. Wherein, the first target condition includes: the condition of determining the first hand as the second hand when it is determined that there is no second hand; the condition of determining that there is a second hand and the number of the second hands is multiple. The second target condition includes: The condition of determining the second hand as the third hand when it is determined that there is no third hand. The condition of determining that there is a third hand and the number of the third hands is multiple.

2. The method according to claim 1, wherein After determining whether there is a first hand among the multiple hands that satisfies the first condition according to the feature information, the method further includes: When it is determined that there is the first hand and the number of the first hands is one, determining the first hand as the hand to be tracked.

3. The method according to claim 2, wherein Determining whether there is a first hand among the multiple hands that satisfies the first condition according to the feature information includes: Obtaining the position information of the hand to be tracked in at least two frame images before the target frame image. According to the position information of the hand to be tracked in the at least two frame images, determining the motion feature of the hand to be tracked. According to the motion feature, determining the area corresponding to the predicted position in the target frame image. According to the position information corresponding to the multiple hands respectively, determining whether there is a first hand among the multiple hands that is within the area corresponding to the predicted position.

4. The method according to claim 3, characterized in that, Calculating the score values respectively corresponding to multiple third hands according to the overlap degree and the recognition probability value includes: Calculating the product between the overlap degree and the recognition probability value corresponding to each third hand in multiple third hands respectively, and determining the product as the score value corresponding to each third hand.

5. The method according to claim 3, characterized in that, The feature information further includes hand key point position information; Before determining whether there is a third hand in multiple second hands that has the same gesture type as the gesture type of the target tracked hand according to the gesture type information respectively corresponding to multiple second hands, the method further includes: Determining the gesture type information respectively corresponding to multiple second hands according to the hand key point position information respectively corresponding to multiple second hands.

6. A hand tracking device, characterized in that, The device includes: An acquisition module, configured to acquire a user action video; wherein, the user action video is an action video including multiple hands; An identification module, configured to identify multiple hands included in a target frame image, and obtain feature information corresponding to the multiple hands; wherein, the target frame image is any frame image other than the first frame image in the user action video, and the feature information includes position information, left / right hand type information, gesture type information, and recognition probability value; A first screening sub-module, configured to determine whether there is a first hand that meets a first condition among the multiple hands according to the feature information, where the first condition is to be within the area corresponding to the predicted position of the target frame image; A first determination unit, configured to, when it is determined that there is the first hand and the number of the first hands is multiple, determine whether there is a second hand among the multiple first hands that has the same left / right hand type as the left / right hand type of the target tracked hand according to the left / right hand type information respectively corresponding to the multiple first hands; A second determination unit, configured to, when any condition in a first target condition is met, determine whether there is a third hand among the multiple second hands that has the same gesture type as the gesture type of the target tracked hand according to the gesture type information respectively corresponding to the multiple second hands; wherein, the target tracked hand is the tracked hand determined in the previous frame image of the target frame image; A first calculation sub-unit, configured to, when any condition in a second target condition is met, calculate the overlap degree between the area where multiple third hands are located and the area corresponding to the predicted position according to the position information respectively corresponding to the multiple third hands; A second calculation sub-unit, configured to calculate the score values respectively corresponding to multiple third hands according to the overlap degree and the recognition probability value; An acquisition sub-unit, configured to acquire a fourth hand with the highest score value from multiple third hands; A determination sub-unit, configured to determine the fourth hand as the tracked hand; Wherein, the first target condition includes: the condition of determining the first hand as the second hand when it is determined that there is no second hand; the condition of determining that there is a second hand and the number of the second hands is multiple; The second target condition includes: The condition of determining the second hand as the third hand when it is determined that there is no third hand; Determine the conditions for the existence of the third hand part and the number of the third hand parts being multiple.

7. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the hand tracking method according to any one of claims 1-5 is implemented.

8. A computer storage medium, characterized in that, Computer program instructions are stored on the computer storage medium, and when the computer program instructions are executed by a processor, the hand tracking method according to any one of claims 1-5 is implemented.

Images