Gesture action recognition method and device and electronic equipment

Abstract

The invention relates to a gesture action recognition method and device and electronic equipment. The method comprises the steps that continuous multi-frame hand shape images containing hand shape information are acquired; identifying a plurality of preset hand shape data points in the current hand shape image, and determining position data of a preset center target point in the hand shape; determining a palm direction of a palm corresponding to the hand type in the current hand type image based on the position data of the plurality of centered target points; if the palm direction accords with a preset palm direction, determining a matched target hand type from a plurality of preset hand types to be selected based on hand type data of hand type data points in the current hand type image, and obtaining hand type geometric features of the target hand type in the current hand type image; and if the acquired continuous multiple frames of hand shape images all meet the condition of matching the target hand shape, obtaining a gesture action recognition result according to the hand shape geometric features in the two adjacent frames of hand shape images. Therefore, by adopting the scheme, the accuracy of gesture action recognition can be improved.

Description

Technical Field

[0001] This application relates to the technical field of action image recognition, and in particular, to a gesture action recognition method, device, and electronic device. Background Art

[0002] With the development of the field of image recognition technology, the application of biological limb action recognition is becoming more and more extensive and the requirements are becoming more stringent; for example, in the field of biological security recognition, strict requirements should be imposed on limb action recognition to improve the security of recognition.

[0003] Gesture action recognition is a kind of image recognition technology based on computer vision. This technology captures and analyzes the actions of the human hand through a camera, so as to recognize specific gestures. The related traditional technical solutions mainly use the CNN neural network (convolutional neural network) to perform inference classification of gestures to achieve gesture recognition.

[0004] However, the applicant found during the implementation process that the traditional technology at least has the problem of low accuracy in gesture action recognition. Summary of the Invention

[0005] Based on this, the purpose of this application aims to at least solve one of the above technical defects, especially the technical defect of low accuracy in gesture action recognition in the prior art. This application provides a gesture action recognition method, device, and electronic device.

[0006] In a first aspect, this application provides a gesture action recognition method, which includes:

[0007] When the recognition device recognizes the existence of a hand shape, obtain a continuous multi-frame hand shape image containing hand shape information;

[0008] Identify multiple preset hand shape data points in the current hand shape image, and determine multiple preset center target points and the position data of the center target points in the hand shape from the multiple preset hand shape data points; where the current hand shape image is any one of the hand shape images;

[0009] Based on the position data of the multiple center target points, determine the palm direction of the palm corresponding to the hand shape in the current hand shape image;

[0010] If the palm direction conforms to the preset palm direction, then based on the hand shape data of the hand shape data points in the current hand shape image, determine the matching target hand shape from multiple preset candidate hand shapes, and obtain the hand shape geometric features of the target hand shape in the current hand shape image;

[0011] If it is obtained that the continuous multi-frame hand shape images all meet the condition of matching the target hand shape, then obtain the gesture action recognition result according to the hand shape geometric features in two adjacent hand shape images.

[0012] In one embodiment, based on the position data of multiple centered target points, determining the palm direction of the hand corresponding to the hand shape in the current hand shape image includes:

[0013] Obtaining the direction vector of the palm corresponding to the hand shape according to the position data of a preset number of centered target points;

[0014] When the direction vector meets the preset direction vector, determining the flipping angle of the palm movement;

[0015] Based on the flipping angle, determining the palm direction.

[0016] In one embodiment, the centered target points include the wrist point and the root point of the middle finger;

[0017] Determining the flipping angle of the palm movement includes:

[0018] According to the position data of the wrist point, translating in the preset coordinate axis direction to obtain the first translated position data;

[0019] Based on the first translated position data, the position data of the wrist point, and the position data of the root point of the middle finger, obtaining the flipping angle.

[0020] In one embodiment, if the flipping angle is within the preset angle range, it indicates that the palm direction conforms to the preset palm direction; wherein, the preset angle range includes 45° - 135°.

[0021] In one embodiment, obtaining the hand shape geometric features of the target hand shape in the current hand shape image includes:

[0022] Determining the proportion data of the target hand shape in the current hand shape image and determining the position center point data of the target hand shape;

[0023] Based on the proportion data of the target hand shape and / or the position center point data of the target hand shape, obtaining the hand shape geometric features.

[0024] In one embodiment, according to the hand shape geometric features in two adjacent frames of hand shape images, obtaining the gesture action recognition result includes:

[0025] According to the proportion data of the previous frame of hand shape image and the proportion data of the next frame of hand shape image, obtaining the hand shape position change result;

[0026] According to multiple hand shape position change results, determining the gesture action recognition result.

[0027] In one embodiment, according to the hand shape geometric features in two adjacent frames of hand shape images, obtaining the gesture action recognition result includes:

[0028] Translate the position center point data of the target hand shape in the previous frame of hand shape image to obtain the second translation position data;

[0029] Based on the second translation position data, the center point position data of the previous frame of hand shape image, and the center point position data of the next frame of hand shape image, obtain the hand shape position change result;

[0030] Based on multiple hand shape position change results, determine the gesture action recognition result.

[0031] In one embodiment, based on the hand shape data of the hand shape data points in the current hand shape image, determine the matching target hand shape from multiple preset candidate hand shapes, including:

[0032] Based on the position data of multiple hand shape data points, determine the position relationship between multiple fingers;

[0033] Based on the position relationship between multiple fingers and the preset finger position relationships of multiple candidate hand shapes, determine the matching target hand shape from multiple candidate hand shapes.

[0034] In a second aspect, the present application provides a gesture action recognition device, which includes:

[0035] A hand shape image module, configured to obtain a continuous multi-frame hand shape image containing hand shape information when the recognition device recognizes that there is a hand shape;

[0036] A hand shape data point module, configured to recognize multiple preset hand shape data points in the current hand shape image, and determine multiple preset center target points and the position data of the center target points in the hand shape from the multiple preset hand shape data points; where the current hand shape image is any frame of hand shape image;

[0037] A palm direction module, configured to determine the palm direction of the hand shape corresponding to the hand shape in the current hand shape image based on the position data of multiple center target points;

[0038] A target hand shape module, configured to, if the palm direction meets the preset palm direction, based on the hand shape data of the hand shape data points in the current hand shape image, determine the matching target hand shape from multiple preset candidate hand shapes, and obtain the hand shape geometric features of the target hand shape in the current hand shape image;

[0039] A recognition result module, configured to, if the continuous multi-frame hand shape images all meet the condition of matching the target hand shape, obtain the gesture action recognition result according to the hand shape geometric features in two adjacent frames of hand shape images.

[0040] In a third aspect, the present application provides an electronic device, including a memory and a processor, where the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0041] As can be seen from the above technical solutions, the embodiments of the present application have the following advantages:

[0042] The gesture action recognition method, device and electronic device provided by the present application can determine the palm direction of the hand corresponding to the hand shape by continuously recognizing multiple frames of hand shape images and using the position data of the central target point in the hand shape of each frame of hand shape image. In this way, the hand shapes that meet the palm direction can be screened by the palm direction, that is, the hand shapes that do not meet the palm direction can be excluded, ensuring the accuracy of gesture action recognition. Further, the target hand shape can be determined through the hand shape data points of each frame of hand shape image. When the conditions for a coherent gesture action are met through further screening of multiple frames of hand shape images, the gesture action recognition result can be obtained based on the geometric features of multiple consecutive frames of hand shapes. Thus, compared with the traditional technology, the present application screens the hand shapes that meet the palm direction through hand shape images, and can recognize the actions of consecutive hand shapes to accurately recognize gesture actions. Therefore, through multiple conditional restrictions on gesture action recognition, the accuracy of gesture action recognition is improved, and it does not rely on the recognition and pre-training of neural networks, and can also reduce the cost of gesture action recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0044] Figure 1 It is a schematic flowchart of a gesture action recognition method provided by an embodiment of the present application;

[0045] Figure 2 It is a schematic diagram of preset hand shape data points of a hand shape image provided by an embodiment of the present application;

[0046] Figure 3 It is a schematic flowchart of the step of determining the palm direction provided by an embodiment of the present application;

[0047] Figure 4 It is a schematic flowchart of the step of obtaining the geometric features of the hand shape provided by an embodiment of the present application;

[0048] Figure 5 It is a schematic flowchart of another gesture action recognition method provided by an embodiment of the present application;

[0049] Figure 6 It is a schematic structural diagram of a gesture action recognition device provided by an embodiment of the present application;

[0050] Figure 7 Schematic diagram of the internal structure of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0051] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0052] With the development of the field of image recognition technology, the application of biological limb movement recognition is becoming more and more widespread and the requirements are becoming more stringent; for example, in the field of biological security recognition, strict requirements should be imposed on limb movement recognition to improve the security of recognition.

[0053] Gesture movement recognition is an image recognition technology based on computer vision. This technology captures and analyzes the movements of the human hand through a camera to identify specific gestures. The related traditional technical solutions mainly use a CNN neural network (convolutional neural network) for gesture inference classification to achieve gesture recognition. However, there are at least the following defects in using the neural network technology for gesture movement recognition:

[0054] Training an AI model requires a large amount of materials and training time; usually, inference classification is performed through a CNN neural network (convolutional neural network), and generally there are corresponding classification results. However, if a gesture movement that does not belong to the training set appears, it is easy to make a misjudgment error; the methods of traditional technologies usually only perform inference classification recognition on a single gesture picture, but have no recognition effect on continuous hand movements such as stretching forward and swinging left and right, and the recognition accuracy is too low.

[0055] Based on this, the present application provides a gesture movement recognition method, device and electronic device to solve the problem of at least low accuracy of gesture movement recognition in traditional technologies. Through multiple condition restrictions, the present application can perform multiple analyses on hand shape images, does not rely on the recognition and pre-training of neural networks, and can perform action recognition on continuous hand shapes, thereby improving the accuracy of gesture movement recognition and reducing the cost of gesture movement recognition.

[0056] In an exemplary embodiment, Figure 1 Schematic flow chart of the gesture movement recognition method provided by an embodiment of the present application, as Figure 1As shown in the figure, this embodiment takes the application of this method to a terminal as an example. It can be understood that this method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. This method can include the following steps: S101 to S105. Among them:

[0057] S101. When the recognition device recognizes that there is a hand shape, obtain a continuous multi-frame hand shape image containing hand shape information.

[0058] Among them, the recognition device can refer to a device used for gesture recognition. For example, it can be a terminal. The hand shape can be hand shape data read and recognized through calculation. For example, the hand shape data in the image frame can be read through the MediaPipe algorithm. The hand shape image can refer to an image frame with a hand shape.

[0059] Exemplarily, the recognition device terminal can collect images through a camera. When it is determined that there is a hand shape in the collected image, the image containing the hand shape can be obtained, and it can be further determined whether multiple consecutive images contain a hand shape. If so, a continuous multi-frame hand shape image containing the hand shape can be obtained. For example, the terminal can read the camera stream, read an image frame every 200 milliseconds, and the read image is judged by the MediaPipe algorithm whether there is hand shape data in the image frame. If so, the terminal can determine the coordinate data of a total of 21 preset hand shape data points included in the hand shape data.

[0060] S102. Recognize multiple preset hand shape data points in the current hand shape image, and determine multiple preset center target points and the position data of the center target points in the hand shape from the multiple preset hand shape data points; where the current hand shape image is any one of the hand shape images.

[0061] Among them, the preset hand shape data points can be data points preset according to the hand shape. For example, the preset hand shape data points can include the fingertip point of the index finger, the root point of the index finger, the fingertip point of the thumb, etc. The center target point can refer to the target point centered in the hand shape. For example, if divided according to the direction from the thumb to the little finger, then the fingertip point of the middle finger, the root point of the middle finger, and the wrist point can be used as the center target points, and the palm can be divided into two equal parts according to the above direction; if according to the direction from the wrist to the fingertip, then the root point of the index finger, the root point of the middle finger, the root point of the ring finger, etc. can be used as the center target points.

[0062] The position data may refer to coordinate data in an image. The current hand-shaped image can be any one of a series of consecutive hand-shaped images, that is, any hand-shaped image can be processed in the manner provided in this embodiment.

[0063] Exemplarily, for each hand-shaped image, the terminal can identify multiple preset hand-shaped data points in the hand-shaped image through the MediaPipe algorithm. And according to a preset rule, at least three preset center target points can be determined from the multiple preset hand-shaped data points, as well as the position data of these center target points. In this way, the direction of the palm corresponding to the hand in the hand-shaped image can be determined by using the coordinate data of at least three center target points, and further, it can be judged whether it meets the hand shape requirements through the direction of the palm, improving the recognition accuracy.

[0064] Optionally, Figure 2 is a schematic diagram of preset hand-shaped data points of a hand-shaped image provided by an embodiment of the present application; as Figure 2 shown, the MediaPipe algorithm can read 21 three-dimensional coordinate points (x, y, z) including:

[0065] 0, wrist; 1, base of thumb; 2, first joint of thumb; 3, second joint of thumb; 4, tip of thumb; 5, base of index finger; 6, first joint of index finger; 7, second joint of index finger; 8, tip of index finger; 9, base of middle finger; 10, first joint of middle finger; 11, second joint of middle finger; 12, tip of middle finger; 13, base of ring finger; 14, first joint of ring finger; 15, second joint of ring finger; 16, tip of ring finger; 17, base of little finger; 18, first joint of little finger; 19, second joint of little finger; 20, tip of little finger.

[0066] Schematically, 12, 11, 10, 9, 0 can be used as a set of center target points. 3, 5, 9, 13, 17 can also be used as another set of center target points. As an example, in this embodiment, 12, 9, 0 are used as the main center target points. In this way, through these three target points of the tip of the middle finger, the base of the finger, and the wrist point, a plane can be formed, and this plane can roughly divide the palm into two, and the left and right movement range of the middle finger is small, which can reduce the influence of different gesture movements on the reference points and improve the accuracy.

[0067] S103. Determine the palm direction of the palm corresponding to the hand in the current hand-shaped image based on the position data of multiple center target points.

[0068] Among them, the palm direction may refer to the orientation of the palm center.

[0069] Exemplarily, the terminal can obtain the orientation of the palm center of the palm corresponding to the hand in the current hand-shaped image according to the position data of three center target points.

[0070] Optionally, the terminal can calculate the normal vector of the palm corresponding to the hand shape based on the coordinate data of the three centered target points, and determine whether the normal vector meets the direction requirements. If it meets, it can further determine whether the angle of the palm orientation meets the direction requirements. If it meets, it can perform the next processing action. Otherwise, if any of the above requirements is not met, the hand shape image frame can be directly discarded. In this way, the gesture action with the correct hand shape direction can be accurately recognized, thereby improving the accuracy of gesture action recognition.

[0071] S104. If the palm direction meets the preset palm direction, based on the hand shape data of the hand shape data points in the current hand shape image, determine the matching target hand shape from multiple preset candidate hand shapes, and obtain the hand shape geometric features of the target hand shape in the current hand shape image.

[0072] Among them, the preset palm direction can refer to the preset palm direction requirements. For example, the preset palm direction can be the direction with the palm facing the camera or the direction with the palm facing away from the camera. The preset palm direction can be adjusted in real time. For example, during gesture action recognition, the orientation of the palm can be randomly required. In this way, by randomly adjusting the recognition and verification requirements in real time, the accuracy of gesture action recognition can be improved, and the security of gesture action verification can also be improved.

[0073] The candidate hand shapes can refer to the hand shape data preset for hand shape recognition requirements. For example, it can be required that the gesture action is in the state of five fingers spread out during recognition, then the preset candidate hand shapes can include the candidate hand shapes in the state of five fingers spread out. The target hand shape can refer to the hand shape that matches the hand shape data of the hand shape data points among multiple candidate hand shapes.

[0074] The hand shape geometric features can refer to the geometric features of the hand shape in the hand shape image. For example, the hand shape geometric features can include the geometric center position feature of the hand shape in the hand shape image, the hand shape size feature, the hand shape shape feature, the proportion of the hand shape in the image, etc. The hand shape geometric features can be used to judge the changes of the hand shape in multiple consecutive hand shape images.

[0075] Exemplarily, the terminal can determine whether the palm direction meets the preset palm direction. If it does not meet, there is no need to perform gesture recognition anymore, and it can be considered that the hand shape does not meet the recognition requirements, and there may be a hand movement that is not a gesture recognition action, thereby improving the accuracy of gesture action recognition. If the terminal determines that the palm direction meets the preset palm direction, it can screen out the target hand shape that matches the hand shape data of the hand shape data points in the current hand shape image from multiple preset candidate hand shapes, and can determine the hand shape geometric features of the target hand shape in the current hand shape image.

[0076] S105. If it is determined that consecutive multiple frames of hand gesture images all meet the condition of matching the target hand gesture, then based on the hand gesture geometric features in two adjacent frames of hand gesture images, obtain the gesture action recognition result.

[0077] The gesture action recognition result may refer to the recognition result of a gesture action. For example, the gesture action recognition result may indicate that a left - right waving action, a forward - backward stretching action, etc. are recognized.

[0078] Exemplarily, the terminal can determine whether consecutive multiple frames of hand gesture images all meet the condition of matching the target hand gesture. If the target hand gesture is matched in all consecutive multiple frames of hand gesture images, then based on the hand gesture geometric features of the previous frame and the next frame in two adjacent frames of hand gesture images, the target gesture action can be recognized. The target gesture action refers to the recognized gesture action. In this way, the continuous hand gesture action can be recognized through consecutive multiple frames of hand gesture images, reducing the time for the neural network to recognize gesture actions, improving the response efficiency of gesture action recognition, and also improving the accuracy of continuous hand gesture action recognition.

[0079] In this embodiment, by continuously recognizing multiple frames of hand gesture images, the palm direction of the hand corresponding to the hand gesture in each frame can be determined through the position data of the centered target point in the hand gesture of each frame. In this way, the hand gestures that meet the palm direction can be screened through the palm direction, that is, the hand gestures that do not meet the palm direction can be excluded, ensuring the accuracy of gesture action recognition. Further, through the hand gesture data points of each frame of hand gesture image, the matching target hand gesture is determined. When further screening the hand gestures that meet the conditions of continuous gesture actions through multiple frames of hand gesture images, the gesture action recognition result is obtained according to the geometric features of consecutive multiple frames of hand gestures. In this way, compared with the traditional technology, in this application, the hand gestures that meet the palm direction are screened through hand gesture images, and continuous hand gestures can be recognized for accurate recognition of gesture actions. Thus, through multiple - condition restrictions on gesture action recognition, the accuracy of gesture action recognition is improved, and it does not rely on the recognition and pre - training of the neural network, and can also reduce the cost of gesture action recognition.

[0080] In an exemplary embodiment, Figure 3 is a schematic flowchart of the steps for determining the palm direction provided by the embodiment of the present application. As Figure 3 shown, on the basis of Figure 1 , the steps of the gesture action recognition method are further described exemplarily. In the step of S103, based on the position data of multiple centered target points, the palm direction of the hand corresponding to the hand gesture in the current hand gesture image is determined, which may specifically include S301 to S303, where:

[0081] S301. Obtain the direction vector of the hand corresponding to the palm according to the position data of a preset number of centered target points.

[0082] S302. Determine the flipping angle of the palm movement when the direction vector meets the preset direction vector.

[0083] S303. Determine the palm direction based on the flipping angle.

[0084] Herein, the preset number may refer to the number of center target points that can determine the direction vector preset. For example, the direction vector of the plane where the palm is located can be determined through three center target points. The direction vector may refer to the normal vector of the plane where the palm is located. The preset direction vector may refer to the pre-set normal vector, and the preset direction vector is adjusted in real time when the gesture action recognition instruction is started. For example, when recognizing a gesture action, the required direction vector can be randomly set. In this way, by randomly adjusting the recognition and verification requirements in real time, the accuracy of gesture action recognition can be improved, and the security of biometric security recognition and verification can also be improved.

[0085] The flipping angle may refer to the included angle between the plane where the palm is located and the plane where the camera is located.

[0086] Exemplarily, the terminal can obtain the direction vector of the plane where the palm is located according to the three-dimensional coordinate data of the three center target points. If the direction vector meets the preset direction vector, then the flipping angle of the palm movement can be further determined. The specific palm direction can be obtained according to the flipping angle, and it can be determined whether the palm direction meets the preset palm direction by judging whether the flipping angle meets the preset angle.

[0087] Schematically, the normal vector calculation can be performed on the three center target points of the wrist point A1, the fingertip B1 of the middle finger, and the root B2 of the middle finger as follows in expressions (1) and (2):

[0088]

[0089]

[0090]

[0091] Wherein, xM1 is the x-axis coordinate of M1, yM1 is the y-axis coordinate of M1, zM1 is the y-axis coordinate of M1, and M = A1, B1, B2. n is the normal vector of the plane where the palm is located.

[0092] As an example, if the specification for gesture action recognition is preset as the palm facing the camera, then if the z component of the normal vector n is negative, it indicates that the palm is facing the camera, otherwise it does not meet the specification and is directly discarded waiting for the next frame of hand shape image.

[0093] If the z-component of the normal vector n is negative, it indicates that the palm is facing the camera, which conforms to the specification of gesture action recognition. Otherwise, it does not conform to the specification and is directly discarded waiting for the next image frame.

[0094] In this embodiment, by first determining whether the normal vector of the plane where the palm is located meets the requirements of the preset direction vector, if it does not meet the requirements, the hand shape image can be directly discarded, indicating that the gesture action does not meet the preset requirements. In this way, it can be initially recognized whether a gesture action needs to be recognized, improving the recognition accuracy. If it meets the requirements of the preset direction vector, the flipping angle of the palm movement can be further used to determine whether the palm direction meets the preset angle. If it does not meet the requirements, it also indicates that the gesture action does not meet the preset requirements. If it meets the requirements, the recognition of the gesture action can be further realized, thereby further improving the accuracy of gesture action recognition. The embodiment of the present application determines through multiple conditions, first determines the general direction of the palm, and then determines the palm direction through a small angle, thereby greatly improving the accuracy of gesture action recognition and reducing the misjudgment of the state caused by the user using the hand for other reasons.

[0095] In an exemplary embodiment, the centered target points include the wrist point and the root point of the middle finger;

[0096] Determining the flipping angle of the palm movement includes:

[0097] According to the position data of the wrist point, translate it in the preset coordinate axis direction to obtain the first translated position data;

[0098] Based on the first translated position data, the position data of the wrist point, and the position data of the root point of the middle finger, obtain the flipping angle.

[0099] Among them, the preset coordinate axis can refer to the coordinate axis with a preset direction, such as the x-axis direction. The first translated position data can refer to the position data obtained by translating the wrist point.

[0100] Exemplarily, the position of the wrist point A1 can be translated along the x-axis to obtain point A2, and through A2, A1, and B3, the plane where the cross-section of the camera is located can be calculated, and further the flipping angle can be calculated.

[0101] In this embodiment, by using the wrist point, the root point of the middle finger, and the point obtained by translating the wrist point along the preset coordinate axis direction, the flipping angle can be accurately and effectively determined, providing judgment data for the judgment of gesture action recognition, thereby improving the accuracy of gesture action recognition.

[0102] Optionally, if the flipping angle is within the preset angle range, it indicates that the palm direction conforms to the preset palm direction; among them, the preset angle range includes 45° - 135°.

[0103] Among them, the preset angle range can be a pre-set angle range, and this angle range can be set for the angle between the plane where the palm is located and the plane where the cross-section of the camera is located.

[0104] Exemplarily, the flipping angle (included angle) can be converted to the range between -180° and 180°. If the included angle degree conforms to the range of 45° - 135°, it is considered to conform to the specification (the palm direction conforms to the preset palm direction), otherwise it does not conform to the specification and is directly discarded waiting for the next image frame.

[0105] In this embodiment, by setting the angle range of 45° - 135°, the gesture actions in the specific orientation range can be effectively recognized, so that the misrecognition of gesture actions caused by the user using the hand for other reasons than gesture recognition can be reduced, and thus the accuracy of gesture action recognition can be improved.

[0106] In an exemplary embodiment, Figure 4 is a schematic flowchart of the steps for obtaining the hand shape geometric features provided by the embodiment of the present application. As Figure 4 shown, on the basis of Figure 1 , the steps of the gesture action recognition method are further exemplarily described. Among them, in the step of S104, obtaining the hand shape geometric features of the target hand shape in the current hand shape image can specifically include: S401 to S402, where:

[0107] S401. Determine the proportion data of the target hand shape in the current hand shape image, and determine the position center point data of the target hand shape.

[0108] S402. Based on the proportion data of the target hand shape and / or the position center point data of the target hand shape, obtain the hand shape geometric features.

[0109] Among them, the proportion data can be area proportion, pixel proportion, space proportion and other data. The position center point data can refer to the coordinate data of the geometric center of the target hand shape.

[0110] Exemplarily, the terminal can determine the proportion of the target hand shape in the current hand shape image, and obtain the geometric features of the hand shape according to this proportion situation.

[0111] The terminal can also determine the position data of the geometric center point of the target hand shape, and obtain the geometric features of the hand shape according to the position data of this geometric center point.

[0112] The terminal can also comprehensively obtain the geometric features of the hand shape according to the proportion data of the target hand shape and the position data of the geometric center point of the target hand shape.

[0113] In this embodiment, the geometric features of the hand shape can be determined by the occupied area and / or the position center of the target hand shape, which can provide the basis for the hand shape data features for continuous gesture action recognition, thereby facilitating the improvement of the accuracy of gesture action recognition.

[0114] In an exemplary embodiment, in step S105, according to the geometric features of the hand shape in two adjacent frames of hand shape images, the gesture action recognition result can be obtained, which specifically may include:

[0115] Based on the occupancy data of the previous frame of hand shape image and the occupancy data of the next frame of hand shape image, obtain the hand shape position change result;

[0116] Based on multiple hand shape position change results, determine the gesture action recognition result.

[0117] Among them, the previous frame of hand shape image and the next frame of hand shape image are two adjacent consecutive frames of hand shape images. The hand shape position change result may refer to the data change between the front and rear frame images.

[0118] Exemplarily, for the occupancy data of each frame of hand shape image, it can be recorded into the queue data L1. In this way, the terminal can obtain the change result of the hand shape position according to the occupancy data of the previous and next frames of hand shape images. For example, if the occupancy data of the target hand shape in the next frame of hand shape graph is larger than the occupancy data of the hand shape graph in the previous frame data, and exceeds a certain occupancy difference threshold, it can be determined that the hand shape position change result is that the hand extends forward (moves towards the camera direction), and conversely, the hand shape position change result is that the hand retracts backward (moves in the opposite direction of the camera). As an example, through the hand shape position change results of multiple front and rear frames, it can be determined whether the gesture action makes repeated actions of reaching forward and retracting backward.

[0119] In this embodiment, the hand shape position change result is obtained based on the occupancy data of the previous frame of hand shape image and the occupancy data of the next frame of hand shape image. In this way, the hand shape position change result can be accurately judged, thereby improving the accuracy of continuous gesture action recognition.

[0120] In an exemplary embodiment, in step S105, according to the geometric features of the hand shape in two adjacent frames of hand shape images, the gesture action recognition result can be obtained, which specifically may include:

[0121] Translate according to the position center point data of the target hand shape in the previous frame of hand shape image to obtain the second translation position data;

[0122] Based on the second translation position data, the center point position data of the previous frame of hand shape image, and the center point position data of the next frame of hand shape image, obtain the hand shape position change result;

[0123] Determine the gesture action recognition result based on the change results of multiple hand shape positions.

[0124] Among them, the second translation position data may refer to the position data obtained by translating the center point of the position of the target hand shape.

[0125] Exemplarily, the image center point C1 of the previous frame of data is translated one point C2 to the right, and the image center point D1 of the current data. By calculating the included angle of these three points C2, C1, and D1, and converting the range of the included angle to between -180° and 180°, according to the distribution of the angle of the included angle in the four quadrants, the displacement situation corresponding to the gesture is obtained.

[0126] For example, in the picture P1, through hand shape recognition, the position corresponding to the target hand shape F1 can be obtained, and according to the size of the hand shape, it is cropped. The length and width H1, W1 of the cropped image can be calculated. The coordinates (X1, Y1) of the upper left corner point of the hand shape F1 in the picture A can be calculated. By calculating, the center point C1 of the hand shape F1 = (X1 + W1 / 2, Y1 + H1 / 2) is obtained. Translating C1 to the right gives C2 (C2 = (X1 + (W1) / 2 + 1, Y1 + (H1) / 2)). Then continue to read the queue data, and read the hand shape image frame with the hand shape F2, and crop it according to the size of the hand shape. The length and width H2, W2 of the cropped image can be calculated. The coordinates (X2, Y2) of the upper left corner point of the hand shape F2 in the picture P2 can be calculated. By calculating, the center point D1 of the hand shape F2 (D1 = (X2 + (W2) / 2, Y2 + (H2) / 2)) is obtained.

[0127] Taking C1 as the origin, calculate the included angle of these three points C2, C1, and D1, and judge in which interval the point C2 is relative to the point C1. The division of the interval can be to divide the rectangular coordinate system into up, down, left, and right. Among them, up corresponds to 46° - 135°, down corresponds to 226° (-134°) - 315° (-45°), left corresponds to 136° - 225° (-135°), and right corresponds to 316° (-44°) - 45°. The above example applies the range of 0 - 359°. For easy understanding, the angles in the brackets are the corresponding angles converted to the range of -180° - 180°. By judging which area the specific angle is located in, the moving direction of the gesture (the change result of the hand shape position) can be obtained.

[0128] In this embodiment, through the geometric center position data of the previous frame of hand shape image, its translation data, and the geometric center position data of the next frame, the moving change results of the gesture in different directions of up, down, left, and right can be effectively and accurately obtained, thereby improving the accuracy of gesture action recognition.

[0129] Optionally, for the hand shape state (target hand shape), the proportion of the hand shape in the screen, and the position center point of each frame of hand shape image, they can be recorded in the queue data L1.

[0130] Read the data in the queue L1, remove the unrecognizable data content. If the collected hand shape does not meet the threshold (for example, the threshold conditions set for 3 gesture actions), directly clear the queue and wait for the next image frame. If it meets the threshold conditions, calculate the displacement action of the gesture based on the previous and next frames. Record the displacement action of the first valid data as the start. When reading the second valid data, judge the gesture action based on the previous valid data, and generate new queue data L2 according to the data sequence of the obtained position change, stretching change, image gesture content, etc.

[0131] Through the queue data L2, match according to the preset gesture actions to obtain the recognition result of the matching gesture actions.

[0132] As an example, a left - and - right waving action provided by an embodiment of the present application is shown in Table 1.

[0133] Table 1 Left - and - right waving action rules

[0134]

[0135] In this way, the embodiment of the present application provides an implementation method for judging the stretching of the hand shape and the rules of up - down, left - right position movement.

[0136] In an exemplary embodiment, for the step S104, based on the hand shape data of the hand shape data points in the current hand shape image, to determine the matching target hand shape from a plurality of preset candidate hand shapes, it may specifically include:

[0137] Determine the positional relationship between multiple fingers according to the position data of multiple hand shape data points;

[0138] Determine the matching target hand shape from multiple candidate hand shapes according to the positional relationship between multiple fingers and the preset finger positional relationships of multiple candidate hand shapes.

[0139] Among them, the hand shape data points may include the data points of each finger, such as Figure 2 the recognized hand shape data points shown. The positional relationship may refer to the positional distance relationship between fingers.

[0140] Exemplarily, the terminal can obtain the positional relationship between fingers according to the coordinate data of the recognized hand shape data points, and can find the matching finger positional relationship from the preset finger positional relationships of multiple candidate hand shapes according to the positional relationship between multiple fingers, and use the hand shape corresponding to the matching finger positional relationship as the target hand shape.

[0141] Optionally, perform algorithmic recognition on hand shapes that conform to the specification. As an example, the following are some of the hand shape rules extracted and described:

[0142] OK gesture: It is required to satisfy that the distance between the fingertips of the thumb and the index finger is close enough (which can be judged by setting a threshold), and the y-components of the fingertips of the remaining three fingers need to be smaller than the y-components of the corresponding finger root points.

[0143] Five fingers spread apart: It is required to satisfy that the distance from the thumb fingertip to the wrist is greater than the distance from the first segment of the thumb to the wrist, which is greater than the distance from the thumb root to the wrist, and the y-components of the fingertips of the remaining four fingers need to be smaller than the y-components of the corresponding finger root points.

[0144] Five fingers closed: It is required to satisfy that the distance from the thumb fingertip to the middle finger root is less than the distance from the first segment of the thumb to the middle finger root, and the y-components of the fingertips of the remaining four fingers need to be greater than the y-components of the corresponding finger root points.

[0145] Unrecognizable: All hand shape states that cannot meet the rules are marked as unrecognizable.

[0146] As an example, if any of the above rules can be satisfied, the hand shape that meets the corresponding rule can be used as the target hand shape.

[0147] In this embodiment, the positional relationship between multiple fingers is determined through the position data of multiple hand shape data points; and the matching target hand shape can be determined from multiple candidate hand shapes according to the positional relationship between multiple fingers and the preset finger positional relationship of multiple candidate hand shapes, thereby improving the accuracy of hand shape recognition.

[0148] In an exemplary embodiment, Figure 5 is a schematic flowchart of another gesture action recognition method provided by the embodiments of the present application. As Figure 5 shown, on the basis of Figure 1 further exemplary descriptions are made on the steps of the gesture action recognition method. Specifically, the gesture action recognition method may include S501 to S510, where:

[0149] S501. Read the camera stream;

[0150] S502. Read the hand shape data in the image frame through the MediaPipe algorithm;

[0151] S503. Determine whether there is a hand shape; if so, execute S504; if not, directly end the recognition of the gesture action;

[0152] S504. Determine whether the palm and finger orientations meet the requirements; if so, execute S505; if not, directly end the recognition of the gesture action.

[0153] S505. Gesture state recognition; specifically: the target hand shape can be determined from the candidate hand shapes.

[0154] S506. Record data such as the hand shape state, position in the image, size ratio, time stamp, etc. of the hand shape into the queue L1.

[0155] S507. Determine whether the number of actions in the queue L1 meets the specified threshold and the time of the actions is within a certain range; if not, execute S508, if so, execute S509.

[0156] S508. Clear the data with expired time stamps in the queue L1.

[0157] S509. Read the queue and calculate the corresponding enumerated actions based on multiple gesture states and position information.

[0158] S510. Return the recognition result.

[0159] S511. Empty the queue.

[0160] Exemplarily, the steps of S501 to S510 can be implemented through the above-mentioned various embodiments and combinations of multiple embodiments. The specific limitations in this embodiment can refer to the limitations of the various embodiments of the gesture action recognition method in the above text, which will not be elaborated here.

[0161] In this embodiment, since the gesture recognition rule is implemented based on the corresponding point positions in combination with the corresponding algorithm and does not require training, it saves the R & D time and cost. Restriction rules are added to only allow the effective recognition of gestures within a specific range and with a specific orientation, greatly improving the recognition accuracy and reducing the misjudgment of the state caused by the user's use of the hand for other reasons. The recognition of coherent actions can be performed.

[0162] It should be understood that although the steps in the flowcharts involved in the above-mentioned embodiments are shown in sequence according to the arrows, these steps do not necessarily need to be executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily need to be executed at the same moment, but can be executed at different moments. The execution order of these steps or stages does not necessarily need to be sequential, but can be executed alternately or in turn with at least a part of the steps or stages in other steps or other steps.

[0163] The gesture action recognition device provided in the embodiments of the present application will be described below. The gesture action recognition device and the above-mentioned gesture action recognition method have the same inventive concept. The implementation solutions provided by this device to solve problems are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more embodiments of the gesture action recognition device provided below can refer to the limitations on the gesture action recognition method in the above text. The gesture action recognition device described below and the gesture action recognition method described above can be referred to each other correspondingly, and will not be elaborated here.

[0164] In an exemplary embodiment, Figure 6 is a schematic structural diagram of a gesture action recognition device provided in the embodiments of the present application. As Figure 6 shown, the gesture action recognition device 60 includes: a hand shape image module 610, a hand shape data point module 620, a palm direction module 630, a target hand shape module 640, and a recognition result module 650, where:

[0165] The hand shape image module 610 is configured to obtain a continuous plurality of frames of hand shape images containing hand shape information when the recognition device recognizes that there is a hand shape.

[0166] The hand shape data point module 620 is configured to recognize a plurality of preset hand shape data points in the current hand shape image, and determine a plurality of preset center target points and the position data of the center target points in the hand shape from the plurality of preset hand shape data points; wherein, the current hand shape image is any one frame of hand shape image.

[0167] The palm direction module 630 is configured to determine the palm direction of the hand shape corresponding to the hand shape in the current hand shape image based on the position data of the plurality of center target points.

[0168] The target hand shape module 640 is configured to, if the palm direction meets the preset palm direction, determine a matching target hand shape from a plurality of preset candidate hand shapes based on the hand shape data of the hand shape data points in the current hand shape image, and obtain the hand shape geometric features of the target hand shape in the current hand shape image.

[0169] The recognition result module 650 is configured to, if a plurality of consecutive frames of hand shape images all meet the condition of matching the target hand shape, obtain the gesture action recognition result according to the hand shape geometric features in two adjacent frames of hand shape images.

[0170] In an exemplary embodiment, the palm direction module 630 is configured to obtain a direction vector of the palm corresponding to the hand shape according to the position data of a preset number of center target points; when the direction vector meets the preset direction vector, determine the flipping angle of the palm movement; and determine the palm direction based on the flipping angle.

[0171] In an exemplary embodiment, the centered target points include the wrist point and the base point of the middle finger. The palm direction module 730 is configured to obtain first translation position data by translating in a preset coordinate axis direction according to the position data of the wrist point; and obtain a flipping angle based on the first translation position data, the position data of the wrist point, and the position data of the base point of the middle finger.

[0172] In an exemplary embodiment, if the flipping angle is within a preset angle range, it indicates that the palm direction conforms to the preset palm direction; wherein, the preset angle range includes 45° - 135°.

[0173] In an exemplary embodiment, the target hand shape module 740 is configured to determine the proportion data of the target hand shape in the current hand shape image and determine the position center point data of the target hand shape; and obtain hand shape geometric features based on the proportion data of the target hand shape and / or the position center point data of the target hand shape.

[0174] In an exemplary embodiment, the recognition result module 750 is configured to obtain the hand shape position change result according to the proportion data of the previous frame of hand shape image and the proportion data of the next frame of hand shape image; and determine the gesture action recognition result according to multiple hand shape position change results.

[0175] In an exemplary embodiment, the recognition result module 750 is configured to translate to obtain second translation position data according to the position center point data of the target hand shape in the previous frame of hand shape image; obtain the hand shape position change result according to the second translation position data, the center point position data of the previous frame of hand shape image, and the center point position data of the next frame of hand shape image; and determine the gesture action recognition result according to multiple hand shape position change results.

[0176] In an exemplary embodiment, the target hand shape module 740 is configured to determine the position relationship between multiple fingers according to the position data of multiple hand shape data points; and determine the matching target hand shape from multiple candidate hand shapes according to the position relationship between multiple fingers and the preset finger position relationships of multiple candidate hand shapes.

[0177] In an exemplary embodiment, the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by one or more processors, the one or more processors are caused to execute the steps of any one of the gesture action recognition methods in the above embodiments.

[0178] In an exemplary embodiment, the present application further provides a computer device, in which a computer program is stored. When the computer-readable instructions are executed by one or more processors, the one or more processors are caused to execute the steps of any one of the gesture action recognition methods in the above embodiments.

[0179] In an exemplary embodiment, the present application further provides a computer program product, including a computer program which, when executed by a processor, implements the steps of the gesture recognition method in any of the above embodiments.

[0180] Schematically, as Figure 7 shown, Figure 7 is a schematic internal structure diagram of a computer device provided by an embodiment of the present application. The computer device 700 may be provided as a server. Referring to Figure 7 , the computer device 700 includes a processing component 702, which further includes one or more processors, and memory resources represented by a memory 701 for storing instructions executable by the processing component 702, such as application programs. The application programs stored in the memory 701 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 702 is configured to execute instructions to perform the text recognition method in any of the above embodiments.

[0181] The computer device 700 may further include a power supply component 703 configured to perform power management of the computer device 700, a wired or wireless network interface 704 configured to connect the computer device 700 to a network, and an input / output (I / O) interface 705. The computer device 700 may operate based on an operating system stored in the memory 701, such as Windows Server TM, Mac OS XTM, Unix TM, Linux TM, Free BSDTM or the like.

[0182] Those skilled in the art can understand that Figure 7 the structure shown in

[0183] is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different component layout.

[0184] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0185] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0186] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A gesture recognition method, characterized in that: The method comprises: When the recognition device recognizes the existence of a hand shape, a continuous plurality of frames of hand shape images containing hand shape information are acquired; Identify a plurality of preset hand shape data points in a current hand shape image, and determine a plurality of preset center target points in the hand shape and position data of the center target points from the plurality of preset hand shape data points; wherein the current hand shape image is any frame of hand shape image; Based on the position data of the plurality of centered target points, determining the palm direction of the palm corresponding to the hand shape in the current hand shape image; If the palm direction meets the preset palm direction, a matching target hand shape is determined from a plurality of preset candidate hand shapes based on the hand shape data of the hand shape data points in the current hand shape image, and hand shape geometric features of the target hand shape in the current hand shape image are obtained; If multiple consecutive frames of hand shape images are obtained and all of them meet the condition of matching the target hand shape, the hand gesture recognition result is obtained according to the hand shape geometric features in the hand shape images of two adjacent frames.

2. The method according to claim 1, characterized in that The determining, based on the position data of the plurality of centered target points, the palm direction of the palm corresponding to the hand shape in the current hand shape image comprises: Obtaining a direction vector of the palm corresponding to the hand shape according to the position data of the preset number of the centered target points; When the direction vector satisfies a preset direction vector, determining a flip angle of the palm activity; Based on the flip angle, the palm direction is determined.

3. The method according to claim 2, characterized in that The center target point includes the wrist point and the base point of the middle finger; The determining the flip angle of the palm activity includes: According to the position data of the wrist point, translate in a preset coordinate axis direction to obtain first translation position data; The flip angle is obtained based on the first translation position data, the position data of the wrist point, and the position data of the base of the middle finger.

4. The method according to claim 2 or 3, characterized in that: If the flip angle is within a preset angle range, it indicates that the palm direction conforms to the preset palm direction; wherein the preset angle range includes 45°-135°.

5. The method according to claim 1, characterized in that The step of obtaining the hand shape geometric features of the target hand shape in the current hand shape image includes: Determine the proportion data of the target hand shape in the current hand shape image, and determine the position center point data of the target hand shape; The hand shape geometric features are obtained based on the proportion data of the target hand shape and / or the position center point data of the target hand shape.

6. The method according to claim 5, characterized in that The step of obtaining a hand gesture recognition result based on the hand shape geometric features in two adjacent frames of the hand shape images includes: Obtaining a hand position change result according to the proportion data of the hand shape image in the previous frame and the proportion data of the hand shape image in the next frame; The gesture action recognition result is determined according to the multiple hand shape position change results.

7. The method according to claim 5 or 6, characterized in that: The step of obtaining a hand gesture recognition result based on the hand shape geometric features in two adjacent frames of the hand shape images includes: According to the position center point data of the target hand shape of the hand shape image in the previous frame, translate to obtain second translation position data; Obtaining the hand position change result according to the second translation position data, the center point position data of the hand shape image in a previous frame, and the center point position data of the hand shape image in a next frame; The gesture action recognition result is determined according to the multiple hand shape position change results.

8. The method according to claim 1, characterized in that The determining of a matching target hand shape from a plurality of preset candidate hand shapes based on the hand shape data of the hand shape data points in the current hand shape image comprises: Determining the positional relationship between the plurality of fingers according to the positional data of the plurality of hand shape data points; According to the positional relationship between the multiple fingers and the preset finger positional relationship of the multiple candidate hand shapes, a matching target hand shape is determined from the multiple candidate hand shapes.

9. A gesture recognition device, characterized in that: The device comprises: A hand shape image module is used to obtain a continuous multi-frame hand shape image containing hand shape information when the recognition device recognizes the existence of a hand shape; A hand shape data point module, used for identifying a plurality of preset hand shape data points in a current hand shape image, and determining a plurality of preset center target points in the hand shape and position data of the center target points from the plurality of preset hand shape data points; wherein the current hand shape image is any frame of hand shape image; A palm direction module, used for determining the palm direction of the palm corresponding to the hand shape in the current hand shape image based on the position data of the plurality of centered target points; A target hand shape module is used to determine a matching target hand shape from a plurality of preset candidate hand shapes based on the hand shape data of the hand shape data points in the current hand shape image if the palm direction matches the preset palm direction, and to obtain the hand shape geometric features of the target hand shape in the current hand shape image; The recognition result module is used to obtain a plurality of consecutive frames of hand shape images that all meet the conditions of matching the target hand shape, and then obtain the gesture action recognition result according to the hand shape geometric features in the hand shape images of two adjacent frames.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images