Hand gesture recognition methods, devices, electronic equipment and storage media

Abstract

The application provides a hand-raising gesture recognition method and device, electronic equipment and a storage medium. The hand-raising gesture validity recognition method performs gesture recognition on a to-be-recognized image through a trained target detection model to obtain a to-be-detected image in which a hand-raising gesture exists, further detects the positions of each key point of a target palm in the to-be-detected image, and if the positional relationship of each key point meets at least N of M characteristics determined according to the hand-raising gesture, the recognized hand-raising gesture is a valid gesture. After the hand-raising gesture is recognized through the target detection model, when the positional relationship of the key points meets multiple attributes of the hand-raising gesture, it is determined that the recognized hand-raising gesture is indeed valid. Therefore, the recognition accuracy of the hand-raising gesture can be further improved, the influence of misrecognition of the neural network mode can be avoided, and the application scenario has certain accuracy requirements for gesture recognition.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction, and more particularly to a method, device, electronic device, and storage medium for recognizing hand gestures. Background Technology

[0002] Due to the development of computer vision technology, gesture control based on computer vision has become a widely used interaction method in the field of human-computer interaction. Gesture control typically uses a camera to track human movements and sends control commands to the device based on the human's posture. This allows for hands-free remote control of devices within a certain range, eliminating the need for remote controls or traditional input / output devices such as mice and keyboards. Currently, the most common method for gesture recognition is to learn through deep neural networks and then use the trained neural network model to recognize gestures in images. However, due to various reasons, such as inaccurate training sample labeling, insufficient training quantity, differences between different samples, or limitations in the accuracy of the neural network model itself, even well-trained neural network models often experience misrecognition. Summary of the Invention

[0003] Based on the above situation, the main purpose of this application is to provide a hand gesture recognition method, device, electronic device and storage medium, which further confirms the validity of the hand gesture by key points and the features of the hand gesture after the neural network recognizes the hand gesture, so as to improve the recognition accuracy.

[0004] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0005] A method for recognizing hand gestures, the method comprising:

[0006] The trained target detection model performs gesture recognition on the image to be identified to obtain the image to be detected, which contains a raised hand gesture.

[0007] Key point detection is performed on the target hand in the image to be detected to determine the position of each key point of the target hand in the image to be detected.

[0008] When the positional relationship of each key point satisfies at least N of the preset M attributes, the hand gesture identified in the image to be detected is confirmed as a valid gesture; wherein, the M attributes include attributes of different types of standard hand gestures, M and N are integers, and N≤M.

[0009] Optionally, the result of the gesture recognition includes a detection box of the target hand, and obtaining the image to be detected includes:

[0010] Based on the gesture recognition results, the image to be recognized containing the raised hand gesture is determined as the first image;

[0011] The image to be detected is obtained by cropping a sub-image containing the target hand from the first image based on the detection box of the target hand, wherein the detection box covers the target hand.

[0012] Optionally, the types of the M attributes include size, vertical direction, horizontal direction, and orientation.

[0013] The size attribute represents the size of the target hand.

[0014] The vertical direction attribute describes the vertical positional relationship of each finger of the target hand.

[0015] The horizontal direction attribute indicates the positional relationship of each finger of the target hand in the horizontal direction;

[0016] The orientation attribute indicates the angle at which the target palm faces the direction being photographed.

[0017] Optionally, the size attribute includes attribute 1 and attribute 2;

[0018] The attribute 1 is: among the coordinates of each key point, the maximum horizontal difference is greater than or equal to half of the horizontal width of the image to be detected, and the maximum vertical difference is greater than or equal to half of the vertical height of the image to be detected;

[0019] The attribute 2 is: the relationship between the height h and the width w of the target palm is: h>w and h<2w, wherein the maximum vertical difference is the height h and the maximum horizontal difference is the width w.

[0020] Optionally, the key points include the key points of the palm base, the key points of each finger joint and fingertip, and the vertical direction attributes include attribute 3, attribute 4 and attribute 5;

[0021] The attribute 3 is: the palm heel key point is located at the bottom of each of the key points;

[0022] The attribute 4 is: among the key points of the index fingertip, the middle fingertip, and the ring fingertip, at least one is located at the top of each of the key points;

[0023] Attribute 5 is: the positions of all key points on the little finger are sequentially upward in the vertical direction.

[0024] Optionally, the key points include the key points of the palm base, the key points of each finger joint and fingertip, and the horizontal attributes include attribute 6, attribute 7 and attribute 8.

[0025] Attribute 6 is: the key point of the thumb tip is located at the leftmost or rightmost of each key point;

[0026] The attribute 7 is: the angle between the line connecting the key point at the base of the index finger and the key point at the base of the little finger and the horizontal direction is less than a preset value;

[0027] The attribute 8 is that the key points at the base of the little finger, the ring finger, the middle finger, and the index finger are positioned sequentially to the left or right in the horizontal direction.

[0028] Optionally, the key points include the palm base key points, the key points of each finger joint and fingertip, and the orientation attribute includes attribute 9;

[0029] The attribute 9 is: the relationship between the distance d1 between the key point at the base of the index finger and the key point at the base of the little finger and the distance d2 between the key point at the base of the palm and the key point at the base of the ring finger is: d2 < 1.5d1.

[0030] This application embodiment also provides a hand gesture recognition device, the device comprising:

[0031] The gesture pre-recognition module is used to perform gesture recognition on the image to be recognized using a trained target detection model to obtain the image to be detected, wherein the image to be detected contains a raised hand gesture.

[0032] The key point detection module is used to detect key points of the target hand in the image to be detected and determine the position of each key point of the target hand in the image to be detected.

[0033] The validity confirmation module is used to confirm that the hand-raising gesture identified in the image to be detected is valid when the positional relationship of each key point satisfies no less than N of the preset M features. The M attributes include attributes of different types of standard hand-raising gestures, M and N are integers, and N≤M.

[0034] Optionally, the gesture pre-recognition module obtains the image to be detected, including:

[0035] The image to be recognized containing the raised hand gesture is identified as the first image;

[0036] The image to be detected is obtained by cropping a sub-image containing the target hand from the first image based on the detection bounding box of the target hand.

[0037] Optionally, in the validity verification module, the types of the M attributes include size attribute, vertical direction attribute, horizontal direction attribute, and orientation attribute, wherein,

[0038] The size attribute represents the size of the target hand.

[0039] The vertical direction attribute describes the vertical positional relationship of each finger of the target hand.

[0040] The horizontal direction attribute indicates the positional relationship of each finger of the target hand in the horizontal direction;

[0041] The orientation attribute indicates the angle at which the target palm faces the direction being photographed.

[0042] Optionally, in the validity confirmation module, the size attribute includes attribute 1 and attribute 2;

[0043] The attribute 1 is: among the coordinates of each key point, the maximum horizontal difference is greater than or equal to half of the horizontal width of the image to be detected, and the maximum vertical difference is greater than or equal to half of the vertical height of the image to be detected;

[0044] The attribute 2 is: the relationship between the height h and the width w of the target palm is: h>w and h<2w, wherein the maximum vertical difference is the height h and the maximum horizontal difference is the width w.

[0045] Optionally, in the validity confirmation module, the key points include the key points of the palm base, the key points of each finger joint and fingertip, and the vertical direction attributes include attribute 3, attribute 4 and attribute 5;

[0046] The attribute 3 is: the palm heel key point is located at the bottom of each of the key points;

[0047] The attribute 4 is: among the key points of the index fingertip, the middle fingertip, and the ring fingertip, at least one is located at the top of each of the key points;

[0048] Attribute 5 is: the positions of all key points on the little finger are sequentially upward in the vertical direction.

[0049] Optionally, in the validity confirmation module, the key points include the key points of the palm base, the key points of each finger joint and fingertip, and the horizontal attributes include attribute 6, attribute 7 and attribute 8.

[0050] Attribute 6 is: the key point of the thumb tip is located at the leftmost or rightmost of each key point;

[0051] The attribute 7 is: the angle between the line connecting the key point at the base of the index finger and the key point at the base of the little finger and the horizontal direction is less than a preset value;

[0052] The attribute 8 is that the key points at the base of the little finger, the ring finger, the middle finger, and the index finger are positioned sequentially to the left or right in the horizontal direction.

[0053] Optionally, in the validity confirmation module, the key points include the palm base key points, the key points of each finger joint and fingertip, and the orientation attribute includes attribute 9;

[0054] The attribute 9 is: the relationship between the distance d1 between the key point at the base of the index finger and the key point at the base of the little finger and the distance d2 between the key point at the base of the palm and the key point at the base of the ring finger is: d2 < 1.5d1.

[0055] This application also provides an electronic device, including a processor, which is used to implement the hand gesture recognition method described above.

[0056] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the hand gesture recognition method described above.

[0057] Beneficial effects:

[0058] This application employs the aforementioned method, using a trained object detection model to perform gesture recognition on the image to be identified, obtaining an image to be detected containing a raised hand gesture. Further, the positions of various key points of the target hand in the image to be detected are detected. If the positional relationships of these key points satisfy at least N of the M attributes determined based on the raised hand gesture, then the identified raised hand gesture is considered valid. After the raised hand gesture is identified by the object detection model, if the positional relationships of the key points satisfy multiple attributes of the raised hand gesture, the identified raised hand gesture is considered valid. This further improves the accuracy of raised hand gesture recognition and avoids the impact of misidentification by the neural network model, making it particularly suitable for application scenarios with certain accuracy requirements for gesture recognition.

[0059] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description

[0060] The preferred embodiments of the present invention will now be described with reference to the accompanying drawings. In the drawings:

[0061] Figure 1 The diagram shown is a flowchart of a hand gesture recognition method according to one embodiment of this application;

[0062] Figure 2 The diagram shown is a schematic representation of 21 key points of a target hand in one embodiment of this application;

[0063] Figure 3 The diagram shown is a coordinate system established for key points of each finger of the target hand in one embodiment of this application.

[0064] Figure 4 The figure shown is a functional block diagram of a hand gesture recognition device according to one embodiment of this application;

[0065] Figure 5 The diagram shown is a structural schematic of an electronic device used to implement the embodiments disclosed in this application. Detailed Implementation

[0066] To provide a more detailed description of the technical solutions of this application and to facilitate a better understanding of this application, specific embodiments of this application are described below in conjunction with the accompanying drawings. However, it should be understood that all illustrative embodiments and their descriptions are used to explain this application and do not constitute the sole limitation of this application.

[0067] In this application, terms such as "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0068] Current gesture control typically requires the continuous recognition of multiple different gestures to obtain a complete control gesture. The applicant found that for various consecutive gestures, raising a hand is usually the start or end action. Therefore, if the hand-raising gesture is misrecognized, it can easily lead to the misrecognition of other gestures as well. Furthermore, the hand-raising gesture is used most frequently in various usage scenarios, and it is very similar to many unconscious gestures of users, which are also easily misrecognized as hand-raising gestures. Therefore, in gesture control, the accuracy requirement for recognizing hand-raising gestures is higher than that for other gestures.

[0069] To facilitate understanding of the various embodiments disclosed in this application, the hand gesture recognition methods disclosed in the various embodiments of this application will first be described in detail. The execution entity of the hand gesture recognition methods provided in the various embodiments of this application can generally be a computing device. This computing device can be implemented as software or as a combination of software and hardware. The computing device can be integrated into an electronic device with a certain computing capability. This electronic device may include, for example, a terminal device, a server, or other processing devices. The terminal device can be a mobile terminal such as a mobile phone, laptop computer, digital broadcast receiver, PDA (Personal Digital Assistant), PAD (Tablet PC), PMP (Portable Multimedia Player), in-vehicle terminal (e.g., in-vehicle navigation terminal), etc., as well as a fixed terminal such as a digital TV, desktop computer, interactive smart display device, etc. In some possible implementations, the hand gesture recognition method can be implemented by a processor calling computer-readable instructions stored in memory.

[0070] Please refer to Figure 1 The diagram shown is a flowchart illustrating a hand gesture recognition method provided in an embodiment of this application. The method includes steps S100-S300:

[0071] S100, the trained target detection model performs gesture recognition on the image to be identified to obtain the image to be detected, wherein the image to be detected contains a raised hand gesture;

[0072] S200, perform key point detection on the target hand in the image to be detected, and determine the position of each key point of the target hand in the image to be detected;

[0073] S300, when the positional relationship of each key point satisfies at least N of the preset M attributes, the hand gesture identified in the image to be detected is confirmed as a valid gesture; wherein, the M attributes include attributes of different types of standard hand gestures, M and N are integers, and N≤M.

[0074] This application employs the above-described method. For the image to be detected containing a raised hand gesture identified by a pre-trained object detection model, the positions of various key points of the target hand in the image are further detected. If the positional relationships of these key points satisfy at least N of the M attributes determined based on the raised hand gesture, the identified raised hand gesture is considered valid. After the raised hand gesture is identified by the object detection model, when the positional relationships of the key points satisfy multiple attributes of the raised hand gesture, the identified raised hand gesture is deemed a valid gesture, and the gesture recognition result is output. This further improves the accuracy of raised hand gesture recognition and avoids the impact of misidentification by the neural network model, making it particularly suitable for application scenarios with certain accuracy requirements for gesture recognition.

[0075] The following provides a detailed explanation of steps S100-S300:

[0076] In step S100, in gesture recognition, video signals are typically obtained by capturing the actions of the user controlling the electronic device through a camera. In scenarios requiring detection, the video signal data acquired by the camera is usually stored in system memory in frames or blocks. The processor can retrieve images frame by frame from memory and preprocess each frame, including operations such as scaling and enhancement, to obtain the image to be recognized. The target detection model can be a pre-trained neural network model. For example, neural network structures such as YOLO-v1, YOLO-v2, YOLO-v3, YOLO-v4, SSD, and FPN can be used. After labeling a large number of sample images, the aforementioned neural network models are trained. The trained target detection model can identify the hand gesture in the image to be recognized to determine whether the gesture type in the image to be recognized is a raised hand gesture. If so, the image to be recognized is the image to be detected, or a sub-image containing the target hand is extracted from the image to be recognized as the image to be detected. Understandably, gesture recognition results include the type of gesture and a detection box to locate the specific position of the gesture, with the palm usually included within the detection box.

[0077] Furthermore, in a preferred embodiment, step S100, obtaining the image to be detected includes:

[0078] S101, based on the gesture recognition result, the image to be recognized containing the raised hand gesture is determined as the first image;

[0079] S102, based on the detection box of the target hand, a sub-image containing the target hand is cropped from the first image to obtain the image to be detected, wherein the detection box covers the target hand.

[0080] In steps S101 and S102 above, when the target detection model performs gesture recognition on the image to be recognized, it generates a detection box when it recognizes a palm. This detection box is usually a rectangle that covers the palm. If a raised hand gesture is recognized in any image to be recognized, the image to be recognized is determined as the first image. Based on the detection box determined during gesture recognition, a sub-image containing the target palm is cropped from the first image, and the cropped sub-image is used as the image to be detected.

[0081] Using the above embodiment, after identifying the first image containing the raised hand gesture, a sub-image containing the target hand is cropped from the first image based on the detection box of the palm as the image to be detected. In this way, when performing key point detection, it is not necessary to detect the entire image to be identified, which can reduce the amount of calculation and speed up the detection speed.

[0082] It is understandable that when generating detection boxes here, it is also possible to simply determine the relevant boundary data of the detection boxes without displaying the detection boxes on the image.

[0083] In step S200, the key points of the human hand are typically the bone nodes. Key point detection can be performed using a pre-trained key point detection model, which can be a deep neural network model. For example, deep neural network models such as ResNet, MobileNet, and NASNet can be used.

[0084] The keypoint detection model can have a varying number of keypoint detection models; for example, it can have 21 keypoint detection patterns, such as... Figure 2 The diagram shown illustrates the 21 most common key points of a hand. Subsequent embodiments of this application will use 21 key points output by the key point detection model as an example for illustration. However, this does not mean that only 21 key points can be used in each embodiment of this application. The hand gesture recognition method of this application can be applied to any number of key point detection models; for example, it can also use 14 key points. During the training of the key point detection model, the sample images can be processed as follows: Figure 2 The labeled keypoints are then used to train the keypoint detection model. Once trained, the keypoint detection model will output the positions of 21 keypoints for each input detection image.

[0085] In step S300, each gesture has unique attributes depending on the posture, which can be reflected by the positional relationship of different key points on the palm. The standard hand-raising gesture is as follows: the palm is extended, the four fingers point upwards, the thumb is spread, and the palm or back of the hand faces the shooting direction. In the standard hand-raising gesture, the size of the palm, the direction of the fingers in the vertical and horizontal directions, and the orientation of the palm all demonstrate attributes different from other gestures. Whether a particular hand possesses these attributes can be confirmed by the positional relationship of different key points on that hand. Therefore, when the positional relationship of the key points of the target hand in the image to be detected satisfies at least N of the M attributes of the hand-raising gesture, the hand-raising gesture detected by the target detection model can be considered a valid gesture, rather than a false detection.

[0086] In this embodiment, the M attributes of the hand-raising gesture can be selected from a variety of attributes based on the varying accuracy of the target detection model in recognizing the hand-raising gesture. The M attributes can include size, vertical direction, horizontal direction, and orientation. The values ​​of M and N also depend on the type and accuracy of the target detection model used. Higher accuracy allows for a smaller value of M, and a closer relationship between N and M; conversely, lower accuracy results in a larger value of M and a greater difference between N and M. For example, if the target detection model has no difference in accuracy for recognizing different types of hand-raising gesture attributes and has high accuracy, then M can be 4, N can be 3, and the M attributes can include one each from size, vertical direction, horizontal direction, and orientation.

[0087] In the image to be detected, an xy-plane coordinate system can be established to better locate the positions of key points in the target palm, and the relationships between the key points can be calculated using the coordinates to confirm whether they meet a certain attribute. For example... Figure 3 As shown, it is for Figure 2 Example diagram of establishing a coordinate system using the palm of the hand.

[0088] The size attribute condition refers to the size attribute of the target hand. For example, the size of the entire hand usually occupies more than half of the image to be detected, the width and height of a normal hand usually meet a preset relationship, and the height of a normal hand and the length of the middle finger usually meet a certain relationship. All of the above attributes can be verified by calculating the position coordinates of each key point.

[0089] The vertical direction attribute condition refers to the vertical position attribute of each finger of the target palm. For example, each finger points vertically upward, the middle finger is usually the longest, and the palm base is at the bottom. The above attributes can be verified by calculating the position coordinates of each key point.

[0090] The horizontal direction attribute refers to the position attribute of each finger of the target palm in the horizontal direction. For example, the thumb or little finger is at the outermost edge, the roots of the three middle fingers are basically aligned (the line connecting them is almost horizontal), and the little finger, ring finger, middle finger, and index finger are from left to right or from right to left, etc. The above attributes can be verified by calculating the position coordinates of each key point.

[0091] The orientation attribute refers to the angle attribute of the target palm facing the direction of the shot. For example, the palm or back of the hand is usually facing the subject. The above attributes can be verified by calculating the position coordinates of each key point. For example, if the palm or back of the hand is facing the subject, the distance from the base of the index finger to the base of the little finger usually satisfies a certain relationship with the distance from the base of the palm to the base of the other fingers.

[0092] In an alternative embodiment, the size attribute includes attribute 1 and attribute 2.

[0093] The attribute 1 is: among the coordinates of each key point, the maximum horizontal difference is greater than or equal to half of the horizontal width of the image to be detected, and the maximum vertical difference is greater than or equal to half of the vertical height of the image to be detected;

[0094] The attribute 2 is: the relationship between the height h and the width w of the target palm is: h>w and h<2w, wherein the maximum vertical difference is the height h and the maximum horizontal difference is the width w.

[0095] In this embodiment, attributes 1 and 2 are selected as two of the size attributes to confirm the hand-raising gesture detected by the object detection model. Among all size attributes, the proportion of the hand size in the image to be detected and the relationship between the width and height of the hand are the two most representative attributes. If the size of a hand occupies a very small proportion in the image to be detected, it indicates that the hand may not actually be a hand, or that there is a problem with the detection box of the object detection model, resulting in problems with the cropped sub-image. If the relationship between the width and height of the hand is inconsistent with that of a normal hand, it indicates that the hand may not be a normal hand, or that the palm or back of the hand has a certain degree of inward curl, and it is not a hand-raising gesture.

[0096] Still with Figure 3 For example, as shown in the figure, among all the key points, key point 4 is the leftmost key point, and key point 20 is the left and right key points. Therefore, the maximum horizontal difference, which is the width of the palm, w = X. 20 -X4, key point 12 is the topmost key point, key point 0 is the left and right key points, then the maximum vertical difference, that is, the hand height h = Y0 - X 12 .

[0097] Understandable Figure 3 The coordinate system is in pixels. This way, after obtaining the palm width w and palm height h, they can be directly compared with the width and height of the image to be detected without conversion.

[0098] In one optional embodiment, the key points include the palm base key point, the key points of each finger joint and fingertip, and the vertical direction attributes include attribute 3, attribute 4 and attribute 5.

[0099] The attribute 3 is: the palm heel key point is located at the bottom of each of the key points;

[0100] The attribute 4 is: among the key points of the index fingertip, the middle fingertip, and the ring fingertip, at least one is located at the top of each of the key points;

[0101] Attribute 5 is: the positions of all key points on the little finger are sequentially upward in the vertical direction.

[0102] In this embodiment, attributes 3, 4, and 5 are selected as three representative attributes in the vertical direction to confirm the hand-raising gesture detected by the target detection model. Among all the vertical direction attributes, the palm heel position, the fingertip position of the middle three fingers, and the direction of the four fingers are the most representative. The palm heel key point is at the bottom of all key points and can be determined by comparing the y-coordinate values ​​of all key points. If the key point is not at the bottom, it means that it is not a hand-raising gesture. If the fingers are not curled but are vertically upward, then usually at least one of the fingertips of the middle three fingers is at the top of all key points. This can also be determined by comparing the y-coordinate values ​​of all key points. At the same time, if it is a hand-raising gesture, then the other four fingers, except for the thumb, all point vertically upward. In this embodiment, since the other fingers have already been confirmed in attribute 4, the little finger is selected as the finger to confirm attribute 5. If the positions of all key points on the little finger are sequentially upward in the vertical direction, it means that the finger pointing vertically upward is satisfied.

[0103] Still with Figure 3 For example, as shown in the figure, among all the key points, key point 0 is the palm heel key point. Comparing the y-coordinates of all key points, if key point 0 is at the bottom, then attribute 3 is satisfied. Key points 8, 12, and 16 are the fingertip key points of the middle three fingers, respectively. Comparing the y-coordinates of all key points, if at least one of key points 8, 12, and 16 is at the top, then attribute 4 is satisfied. Key points 17, 18, 19, and 20 are all the key points on the little finger (including the knuckle key points and the fingertip key points). If the y-coordinates of key points 17, 18, 19, and 20 increase, it means that the little finger is vertically upward, satisfying attribute 5.

[0104] In one optional embodiment, the key points include the palm base key point, the key points of each finger joint and fingertip, and the horizontal attributes include attribute 6, attribute 7 and attribute 8.

[0105] Attribute 6 is: the key point of the thumb tip is located at the leftmost or rightmost of each key point;

[0106] The attribute 7 is: the angle between the line connecting the key point at the base of the index finger and the key point at the base of the little finger and the horizontal direction is less than a preset value;

[0107] The attribute 8 is that the key points at the base of the little finger, the key points at the base of the ring and index fingers, the key points at the base of the middle finger, and the key points at the tip of the index finger are positioned sequentially to the left or to the right in the horizontal direction.

[0108] In this embodiment, attributes 6, 7, and 8 are selected as three representative attributes in the horizontal direction to confirm the hand-raising gesture detected by the target detection model. Among all the horizontal attributes, the positions of the key points of the thumb and the relationship between the positions of the bases of the fingers are the most representative. The key point of the thumb tip is located at the leftmost or rightmost of the key points, which can be determined by comparing the coordinate values ​​of all key points in the x-direction. If the key point is not at the leftmost or rightmost, it means that the thumb is not extended at this time, and it is not a hand-raising gesture; otherwise, it meets the attributes of a hand-raising gesture. If the angle between the line connecting the key points of the base of the index finger and the key point of the base of the little finger and the horizontal direction is not within the preset range, that is, less than the preset value, it means that the palm is likely tilted and pointing in a certain direction, rather than the vertically upward raising fingers. Here, by statistically analyzing the hand-raising gestures of different users, the preset value is usually less than 30°. If the angle between the line connecting the key points of the base of the index finger and the key point of the base of the little finger and the horizontal direction is between 0° and 30°, it is considered to meet attribute 7. At the same time, if the key points at the base of the little finger, the base of the ring and index fingers, the base of the middle finger, and the tip of the index finger are positioned sequentially to the left or right in the horizontal direction, it indicates that attribute 8 is met, meaning that the fingers of the palm are basically in a vertically upward state.

[0109] Still with Figure 3 For example, as shown in the figure, key point 4 is the tip of the thumb. Comparing the x-coordinates of all key points, if key point 4 is located at the far left or far right, then attribute 6 is satisfied. Key points 5, 9, 13, and 17 are the root key points of the other four fingers. If the x-coordinates of key points 5, 9, 13, and 17 increase to the right (or to the left if it is the back of the hand), then it means that the fingers are normally arranged and vertically upward, which best satisfies attribute 8. If the angle formed by the line connecting key points 5 and 17 and the x-axis is between 0° and 30°, then it means that the fingers are basically vertically upward, and the root key points are basically aligned in the x-direction.

[0110] In one optional embodiment, the key points include the palm base key point, the key points of each finger joint and fingertip, and the orientation attribute includes attribute 9.

[0111] The attribute 9 is: the relationship between the distance d1 between the key point at the base of the index finger and the key point at the base of the little finger and the distance d2 between the key point at the base of the palm and the key point at the base of the ring finger is: d2 < 1.5d1.

[0112] In this embodiment, attribute 9 can confirm whether the palm is facing the shooting direction when it is photographed. Statistical analysis of different sample images confirms that when the palm is facing the shooting direction and a hand gesture is made, d2 < 1.5d1 is usually satisfied.

[0113] Still with Figure 3 For example, as shown in the figure, key point 5 is the key point at the base of the index finger, key point 17 is the key point at the base of the little finger, key point 13 is the key point at the base of the index finger, and key point 0 is the key point at the base of the palm. If the distance d1 between key point 5 and key point 17 and the distance d2 between key point 0 and key point 13 satisfy d2 < 1.5d1, then the orientation of the palm can be considered to satisfy the orientation attribute of the raised hand gesture.

[0114] It is understandable that in other embodiments, the distance dx from the palm root key point to the other finger root key points can also be calculated in relation to the above-mentioned d1. The main condition is that dx < 1.5d1 is also satisfied, and it can also be considered that the orientation of the palm satisfies the orientation attribute of the hand raising gesture.

[0115] It is understandable that there is no order among the above attributes, and each type of attribute can be replaced with other attributes that can represent that type of attribute.

[0116] In a specific application scenario, M can be 9 and N can be 7. In this case, after the hand-raising gesture is determined by the target detection model and 21 key points of the palm are detected, the positional relationship of the 21 key points can be used to confirm that at least 7 of the above 9 attributes are satisfied, and then the hand-raising gesture can be confirmed as a valid gesture.

[0117] Please refer to Figure 4 The figure shows a functional module diagram of a hand gesture validity recognition device provided in an embodiment of this application. As shown in the figure, the hand gesture recognition device 100 includes:

[0118] The gesture pre-recognition module 110 is used to perform gesture recognition on the image to be recognized through a trained target detection model to obtain the image to be detected, wherein the image to be detected contains a hand-raising gesture.

[0119] The key point detection module 120 is used to detect key points in the palm of the hand in the image to be detected, and to determine the position of each key point of the palm in the image to be detected.

[0120] The validity confirmation module 130 is used to confirm that the hand gesture identified in the image to be detected is valid when the positional relationship of each key point satisfies no less than N of the preset M features. The M attributes include attributes of different types of standard hand gestures, M and N are integers, and N≤M.

[0121] In this embodiment, the gesture pre-recognition module 110 obtains the image to be detected, including:

[0122] The image to be recognized containing the raised hand gesture is identified as the first image;

[0123] The image to be detected is obtained by cropping a sub-image containing the target hand from the first image based on the detection bounding box of the target hand.

[0124] In an optional embodiment, in the validity verification module 130, the types of the M attributes include size attribute, vertical direction attribute, horizontal direction attribute, and orientation attribute, wherein,

[0125] The size attribute represents the size of the target hand.

[0126] The vertical direction attribute describes the vertical positional relationship of each finger of the target hand.

[0127] The horizontal direction attribute indicates the positional relationship of each finger of the target hand in the horizontal direction;

[0128] The orientation attribute indicates the angle at which the target palm faces the direction being photographed.

[0129] In an optional embodiment, in the validity verification module 130, the size attribute includes attribute 1 and attribute 2;

[0130] The attribute 1 is: among the coordinates of each key point, the maximum horizontal difference is greater than or equal to half of the horizontal width of the image to be detected, and the maximum vertical difference is greater than or equal to half of the vertical height of the image to be detected;

[0131] The attribute 2 is: the relationship between the height h and the width w of the target palm is: h>w and h<2w, wherein the maximum vertical difference is the height h and the maximum horizontal difference is the width w.

[0132] In an optional embodiment, in the validity confirmation module 130, the key points include the key points of the palm base, the key points of each finger joint and fingertip, and the vertical direction attributes include attribute 3, attribute 4 and attribute 5;

[0133] The attribute 3 is: the palm heel key point is located at the bottom of each of the key points;

[0134] The attribute 4 is: among the key points of the index fingertip, the middle fingertip, and the ring fingertip, at least one is located at the top of each of the key points;

[0135] Attribute 5 is: the positions of all key points on the little finger are sequentially upward in the vertical direction.

[0136] In an optional embodiment, in the validity confirmation module 130, the key points include the key points of the palm base, the key points of each finger joint and fingertip, and the horizontal attributes include attribute 6, attribute 7 and attribute 8.

[0137] Attribute 6 is: the key point of the thumb tip is located at the leftmost or rightmost of each key point;

[0138] The attribute 7 is: the angle between the line connecting the key point at the base of the index finger and the key point at the base of the little finger and the horizontal direction is less than a preset value;

[0139] The attribute 8 is that the key points at the base of the little finger, the ring finger, the middle finger, and the index finger are positioned sequentially to the left or right in the horizontal direction.

[0140] In an optional embodiment, in the validity confirmation module 130, the key points include the palm base key points, the key points of each finger joint and fingertip, and the orientation attribute includes attribute 9;

[0141] The attribute 9 is: the relationship between the distance d1 between the key point at the base of the index finger and the key point at the base of the little finger and the distance d2 between the key point at the base of the palm and the key point at the base of the ring finger is: d2 < 1.5d1.

[0142] The hand gesture recognition device 100 in the above embodiments has functions that can be used to execute the methods described in the above method embodiments. Its specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0143] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium.

[0144] The following is for reference. Figure 5 The diagram illustrates a structural schematic of an electronic device 50 suitable for implementing the embodiments disclosed in this application. The electronic device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs, desktop computers, and interactive smart display devices. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed in this application.

[0145] like Figure 5As shown, the electronic device 50 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to different application code stored in the memory 501. The memory 501 may include random access memory (RAM) and read-only memory (ROM). The RAM 503 stores various programs and data required for the operation of the electronic device 50. The memory 501, processor 502, and communication interface 503 can be connected to each other via wires, for example, via a bus 504. The communication interface 503 allows the electronic device 50 to communicate wirelessly or wiredly with other devices to exchange data.

[0146] In the embodiments of this application, when the processor 502 executes the application code stored in the memory 501, it can execute, for example... Figure 1 The hand gesture recognition method provided in the embodiments.

[0147] Although the electronic device 50 with various devices is shown in the figure, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively.

[0148] Specifically, based on the embodiments disclosed above in this application, reference is made to... Figure 1 The processes described in the flowcharts of the embodiments can be implemented as computer programs. For example, embodiments of this application also provide a computer program product including a computer program carried on a computer-readable medium, the computer program comprising methods for executing... Figure 1 The flowchart of the embodiment describes the program code for the method. In such an embodiment, the computer program can be downloaded and installed from a network via communication interface 503, or installed from memory 508. When the computer program is executed by processor 502, it performs the functions defined in the method of the above embodiment.

[0149] It should be noted that the aforementioned computer-readable media may include, but is not limited to: volatile memory, such as random access memory (RAM); memory 802 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 502 may also include combinations of the above types of memory.

[0150] In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0151] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0152] In an optional embodiment, this application also provides a computer storage medium that can be used for computer software instructions, including a computer program, which, when executed by a processor, performs the hand gesture recognition method as described in the above embodiments. This storage medium includes, but is not limited to, flash memory, hard disk, and solid-state drive.

[0153] Those skilled in the art will understand that, without conflict, the above-mentioned preferred solutions can be freely combined and superimposed.

[0154] It should be understood that the above-described embodiments are merely exemplary and not restrictive. Any obvious or equivalent modifications or substitutions made by those skilled in the art regarding the above details without departing from the basic principles of this application shall be included within the scope of the claims of this application.

Claims

1. A hand raising gesture recognition method, characterized by, The method comprises: performing gesture recognition on a to-be-recognized image by using a trained target detection model to obtain a to-be-detected image, wherein a hand-raising gesture exists in the to-be-detected image; performing key point detection on a target palm in the to-be-detected image to determine positions of each key point of the target palm in the to-be-detected image; when a positional relationship of the each key point meets not less than N of M preset attributes, confirming that the hand-raising gesture recognized in the to-be-detected image is a valid gesture; wherein the M attributes comprise attributes of different types of standard hand-raising gestures, M and N are integers, and N is less than M; types of the M attributes comprise a horizontal direction attribute, a size attribute, a vertical direction attribute, and an orientation attribute, wherein the horizontal direction attribute represents a positional relationship of each finger of the target palm in a horizontal direction; the size attribute represents a size attribute of the target palm; the vertical direction attribute represents a positional relationship of each finger of the target palm in a vertical direction; and the orientation attribute represents an angle at which the target palm faces a shooting direction; the each key point comprises a palm root key point, key points of each finger knuckle, and a key point of a fingertip; and the horizontal direction attribute comprises attribute 6, attribute 7, and attribute 8; the attribute 6 is that a fingertip key point of a thumb is located at a leftmost or rightmost side of the each key point; the attribute 7 is that an included angle between a line connecting a root key point of an index finger and a root key point of a little finger and a horizontal direction is less than a preset value; the attribute 8 is that positions of the root key point of the little finger, a root key point of a ring finger, a root key point of a middle finger, and the root key point of the index finger are sequentially leftward or sequentially rightward in the horizontal direction.

2. The hand raising gesture recognition method of claim 1, wherein, a result of the gesture recognition comprises a detection frame of the target palm, and obtaining the to-be-detected image comprises: determining, according to the result of the gesture recognition, a to-be-recognized image in which the hand-raising gesture exists as a first image; cropping, according to the detection frame of the target palm, a sub-image containing the target palm from the first image to obtain the to-be-detected image, wherein the detection frame covers the target palm.

3. The hand raising gesture recognition method of claim 1, wherein, the size attribute comprises attribute 1 and attribute 2; the attribute 1 is that, in coordinates of the each key point, a horizontal maximum difference is greater than or equal to half of a horizontal width of the to-be-detected image, and a vertical maximum difference is greater than or equal to half of a vertical height of the to-be-detected image; the attribute 2 is that a relationship between a height h and a width w of the target palm is h > w and h < 2w, wherein the vertical maximum difference is the height h, and the horizontal maximum difference is the width w.

4. The hand raising gesture recognition method of claim 1, wherein, the each key point comprises a palm root key point, key points of each finger knuckle, and a key point of a fingertip; and the vertical direction attribute comprises attribute 3, attribute 4, and attribute 5; the attribute 3 is that the palm root key point is located at a lowermost side of the each key point; the attribute 4 is that, among a fingertip key point of an index finger, a fingertip key point of a middle finger, and a fingertip key point of a ring finger, at least one is located at an uppermost side of the each key point; the attribute 5 is that positions of all key points on a little finger are sequentially upward in the vertical direction.

5. The hand raising gesture recognition method of claim 1, wherein, The respective key points include a heel key point, respective finger knuckles, and a fingertip key point, and the orientation attribute includes attribute 9. The attribute 9 is that a distance d1 between the index finger root key point and the little finger root key point and a distance d2 between the heel key point and the ring finger root key point satisfy a relationship d2 < 1.5d1.

6. A hand raising gesture recognition apparatus, characterized by, The device comprises: a gesture pre-recognition module configured to perform gesture recognition on a to-be-recognized image by using a trained target detection model to obtain a to-be-detected image, the to-be-detected image including a hand-raising gesture; a key point detection module configured to perform key point detection on a target palm in the to-be-detected image to determine positions of respective key points of the target palm in the to-be-detected image; an effectiveness confirmation module configured to confirm that the hand-raising gesture recognized in the to-be-detected image is valid when a positional relationship of the respective key points satisfies not less than N of M preset attributes, wherein the M attributes include attributes of different types of standard hand-raising gestures, M and N are integers, and N ≤ M; types of the M attributes include a horizontal direction attribute, a size attribute, a vertical direction attribute, and an orientation attribute, wherein the horizontal direction attribute represents a positional relationship of respective fingers of the target palm in a horizontal direction, the size attribute represents a size attribute of the target palm, the vertical direction attribute represents a positional relationship of respective fingers of the target palm in a vertical direction, and the orientation attribute represents an angle at which the target palm faces a shooting direction; the respective key points include a heel key point, respective finger knuckles, and a fingertip key point, and the horizontal direction attribute includes attribute 6, attribute 7, and attribute 8; the attribute 6 is that a fingertip key point of a thumb is located at a leftmost or rightmost side of the respective key points; the attribute 7 is that a line connecting the index finger root key point and the little finger root key point and a horizontal direction form an angle less than a preset value; and the attribute 8 is that the little finger root key point, a ring finger root key point, a middle finger root key point, and the index finger root key point are located in a horizontal direction in a left-to-right or right-to-left order.

7. An electronic device comprising a processor, characterized in that The processor is configured to implement the hand-raising gesture recognition method according to any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program, when executed by the processor, performs the hand-raising gesture recognition method according to any one of claims 1-5.

Images