Method, system and electronic device for gesture recognition

Abstract

Embodiments of the present invention provide a gesture recognition method, system, and electronic device, relating to the field of behavior recognition technology. The method includes: acquiring an image to be recognized; performing human body recognition on the image to be recognized to obtain a human body image of the object to be recognized, and detecting human body key points in the human body image; utilizing wrist key points among the human body key points to determine a hand image of the object to be recognized from the image to be recognized, and detecting hand key points in the image to be recognized; and determining a gesture recognition result for the object to be recognized based on the positional relationship between the hand key points and designated arm key points among the human body key points. Compared with the prior art, the solution provided by the embodiments of the present invention can improve the accuracy of gesture recognition.

Description

Technical Field

[0001] The present invention relates to the field of behavior recognition technology, and in particular to a gesture action recognition method, system and electronic equipment. Background Art

[0002] In daily life, people often use body language to communicate, so as to convey information without being disturbed by external noise or distance. For example, during a football match, due to the noisy environment outside the stadium, the referee often uses body language to officiate or make decisions.

[0003] When people use body movements to communicate, gestures become an indispensable part of their communication. The so-called gestures generally refer to the various movements and postures formed by people using their arms and hands in specific situations.

[0004] In related technologies, when performing gesture recognition, the method generally used is to identify the acquired gesture by comparing the acquired arm motion trajectory with the standard arm motion trajectory corresponding to the preset gesture. However, due to different living habits and physical fitness of users, the motion trajectories of gestures performed by different users can vary greatly. Therefore, the use of the above-mentioned related technologies may result in misidentification and omission of gestures, thereby reducing the recognition accuracy of gestures.

[0005] Based on this, there is an urgent need for a gesture recognition method to improve the recognition accuracy of gestures. Summary of the Invention

[0006] The purpose of the embodiments of the present invention is to provide a gesture recognition method, system, and electronic device to improve the accuracy of gesture recognition. The specific technical solution is as follows:

[0007] In a first aspect, an embodiment of the present invention provides a method for identifying gestures, the method comprising:

[0008] Obtain the image to be recognized;

[0009] Performing human body recognition on the image to be recognized to obtain a human body image of the object to be recognized, and detecting human body key points in the human body image;

[0010] Using the wrist key points among the human body key points, determine the hand image of the object to be identified from the image to be identified, and detect the hand key points in the image to be identified;

[0011] Based on the positional relationship between the hand key points and the designated key points about the arm among the human body key points, a gesture action recognition result of the object to be recognized is determined.

[0012] Optionally, in a specific implementation, obtaining the image to be identified includes:

[0013] A video to be identified is obtained, and each video frame in the video to be identified is determined as an image to be identified.

[0014] Optionally, in a specific implementation, determining the gesture recognition result of the object to be recognized based on the positional relationship between the hand key points and the designated key points about the arm among the body key points includes:

[0015] Determining an initial recognition result corresponding to the video frame based on a positional relationship between the hand key points and a designated key point about the arm among the human body key points;

[0016] The gesture recognition result of the object to be recognized is determined by using the respective initial recognition results and the association relationship between the respective initial recognition results.

[0017] Optionally, in a specific implementation, determining the gesture recognition result of the object to be recognized by utilizing the initial recognition results and the association relationship between the initial recognition results includes:

[0018] For each video frame, cache the initial recognition result corresponding to the video frame, and determine whether the initial recognition result corresponding to the video frame is associated with a specified recognition result; wherein the specified recognition result is: each initial recognition result cached before the initial recognition result corresponding to the video frame;

[0019] If not, the designated recognition result is used to determine the gesture recognition result of the object to be recognized, and a designated deletion process is performed on the designated recognition result.

[0020] Optionally, in a specific implementation, determining whether the initial recognition result corresponding to the video frame is associated with the specified recognition result includes:

[0021] Determine whether the sum of the number of the initial recognition result and the specified recognition result corresponding to the video frame is not greater than a preset value;

[0022] or,

[0023] Determining whether the initial recognition result corresponding to the video frame is associated with the last initial recognition result in the specified recognition results;

[0024] If not, the initial recognition result corresponding to the video frame is not associated with the specified recognition result.

[0025] Optionally, in a specific implementation, the performing a designated deletion process on the designated recognition result includes:

[0026] Clear the specified recognition result; or,

[0027] Delete the first cached initial recognition result among the specified recognition results.

[0028] Optionally, in a specific implementation, before determining the hand image of the object to be identified from the image to be identified using the wrist key points among the human body key points, the method further includes:

[0029] Obtaining the detection confidence of the key points of the human body;

[0030] If the detection confidence is greater than a preset confidence threshold, the wrist key point among the human body key points is used to determine a hand image of the object to be identified from the image to be identified.

[0031] Optionally, in a specific implementation, the determining of a hand image of the object to be identified from the image to be identified by using the wrist key point among the human body key points includes:

[0032] In the image to be identified, taking the wrist key point among the key points of the human body as a starting point and determining an image area of ​​a preset size along a preset direction as a hand image of the object to be identified;

[0033] or,

[0034] In the image to be identified, with the wrist key point as the starting point, an image area of ​​a specified size is determined along the direction from the elbow key point among the human body key points to the wrist key point, as the hand image of the object to be identified; wherein the specified size is determined based on the length between the elbow key point and the wrist key point.

[0035] Optionally, in a specific implementation, the method further includes:

[0036] Output the gesture action recognition result.

[0037] In a second aspect, an embodiment of the present invention further provides a gesture recognition system, the system comprising:

[0038] An image acquisition module, used to acquire an image to be identified;

[0039] A processor, configured to perform human body recognition on the image to be recognized, obtain a human body image of the object to be recognized, and detect key points of the human body in the human body image;

[0040] Using the wrist key points among the human body key points, determine the hand image of the object to be identified from the image to be identified, and detect the hand key points in the image to be identified;

[0041] Based on the positional relationship between the hand key points and the designated key points about the arm among the human body key points, a gesture action recognition result of the object to be recognized is determined.

[0042] Optionally, in a specific implementation, the image acquisition module is specifically configured to:

[0043] A video to be identified is obtained, and each video frame in the video to be identified is determined as an image to be identified.

[0044] Optionally, in a specific implementation, the processor is specifically configured to:

[0045] Determining an initial recognition result corresponding to the video frame based on a positional relationship between the hand key points and a designated key point about the arm among the human body key points;

[0046] The gesture recognition result of the object to be recognized is determined by using the respective initial recognition results and the association relationship between the respective initial recognition results.

[0047] Optionally, in a specific implementation, the processor is specifically configured to:

[0048] For each video frame, cache the initial recognition result corresponding to the video frame, and determine whether the initial recognition result corresponding to the video frame is associated with a specified recognition result; wherein the specified recognition result is: each initial recognition result cached before the initial recognition result corresponding to the video frame;

[0049] If not, the designated recognition result is used to determine the gesture recognition result of the object to be recognized, and a designated deletion process is performed on the designated recognition result.

[0050] Optionally, in a specific implementation, the processor is specifically configured to:

[0051] Determine whether the sum of the number of the initial video result and the specified recognition result corresponding to the video frame is not greater than a preset value;

[0052] or,

[0053] Determining whether the initial recognition result corresponding to the video frame is associated with the last initial recognition result in the specified recognition results;

[0054] If not, the initial recognition result corresponding to the video frame is not associated with the specified recognition result.

[0055] Optionally, in a specific implementation, the processor is specifically configured to:

[0056] Clear the specified recognition result; or,

[0057] Delete the first cached initial recognition result among the specified recognition results.

[0058] Optionally, in a specific implementation, the processor is further configured to:

[0059] Before determining the hand image of the object to be identified from the image to be identified using the wrist key point among the human body key points, obtaining the detection confidence of the human body key point;

[0060] If the detection confidence is greater than a preset confidence threshold, the wrist key point among the human body key points is used to determine a hand image of the object to be identified from the image to be identified.

[0061] Optionally, in a specific implementation, the processor is specifically configured to:

[0062] In the image to be identified, taking the wrist key point among the key points of the human body as a starting point and determining an image area of ​​a preset size along a preset direction as a hand image of the object to be identified;

[0063] or,

[0064] In the image to be identified, with the wrist key point as the starting point, an image area of ​​a specified size is determined along the direction from the elbow key point among the human body key points to the wrist key point, as the hand image of the object to be identified; wherein the specified size is determined based on the length between the elbow key point and the wrist key point.

[0065] Optionally, in a specific implementation, the processor is further configured to:

[0066] Output the gesture action recognition result.

[0067] In a third aspect, an embodiment of the present invention provides a gesture recognition device, the device comprising:

[0068] An image acquisition module, used to acquire an image to be identified;

[0069] A human body key point acquisition module is used to perform human body recognition on the image to be recognized, obtain a human body image of the object to be recognized, and detect human body key points in the human body image;

[0070] a hand key point acquisition module, configured to determine a hand image of the object to be identified from the image to be identified by using the wrist key point among the human body key points, and to detect the hand key points in the image to be identified;

[0071] The result acquisition module is used to determine the gesture recognition result of the object to be recognized based on the positional relationship between the hand key points and the designated key points about the arm among the human body key points.

[0072] Optionally, in a specific implementation, the image acquisition module is specifically configured to:

[0073] A video to be identified is obtained, and each video frame in the video to be identified is determined as an image to be identified.

[0074] Optionally, in a specific implementation, the result acquisition module includes:

[0075] An initial result acquisition submodule, configured to determine an initial recognition result corresponding to the video frame based on a positional relationship between the hand key points and a designated key point about the arm among the human body key points;

[0076] The result acquisition submodule is used to determine the gesture recognition result of the object to be recognized by using the initial recognition results and the association relationship between the initial recognition results.

[0077] Optionally, in a specific implementation, the result acquisition submodule includes:

[0078] A first cache unit is configured to cache, for each video frame, an initial recognition result corresponding to the video frame;

[0079] a judgment unit, configured to judge whether the initial recognition result corresponding to the video frame is associated with a specified recognition result; if not, triggering a first result acquisition unit; wherein the specified recognition result is: each initial recognition result cached before the initial recognition result corresponding to the video frame;

[0080] The first result acquisition unit is configured to determine a gesture recognition result of the object to be recognized using the specified recognition result;

[0081] The deletion unit is used to perform a designated deletion process on the designated recognition result.

[0082] Optionally, in a specific implementation, the judgment unit is specifically configured to:

[0083] Determine whether the sum of the number of the initial video result and the specified recognition result corresponding to the video frame is not greater than a preset value;

[0084] or,

[0085] Determining whether the initial recognition result corresponding to the video frame is associated with the last initial recognition result in the specified recognition results;

[0086] If not, the initial recognition result corresponding to the video frame is not associated with the specified recognition result.

[0087] Optionally, in a specific implementation, the deleting unit is specifically configured to:

[0088] Clear the specified recognition result; or,

[0089] Delete the first cached initial recognition result among the specified recognition results.

[0090] Optionally, in a specific implementation, the device further includes:

[0091] a detection confidence module, configured to obtain a detection confidence of the human body key points before determining a hand image of the object to be identified from the image to be identified using the wrist key points among the human body key points;

[0092] If the detection confidence is greater than a preset confidence threshold, the hand key point acquisition module is triggered.

[0093] Optionally, in a specific implementation, the hand key point acquisition module is specifically configured to:

[0094] In the image to be identified, taking the wrist key point among the key points of the human body as a starting point and determining an image area of ​​a preset size along a preset direction as a hand image of the object to be identified;

[0095] or,

[0096] In the image to be identified, with the wrist key point as the starting point, an image area of ​​a specified size is determined along the direction from the elbow key point among the human body key points to the wrist key point, as the hand image of the object to be identified; wherein the specified size is determined based on the length between the elbow key point and the wrist key point.

[0097] Optionally, in a specific implementation, the device further includes:

[0098] The result output module is used to output the gesture action recognition result.

[0099] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0100] Memory for storing computer programs;

[0101] The processor is configured to implement the steps of any one of the gesture action recognition methods provided in the first aspect when executing the program stored in the memory.

[0102] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any gesture action recognition method provided in the first aspect are implemented.

[0103] In a sixth aspect, an embodiment of the present invention provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the steps of any one of the gesture action recognition methods provided in the first aspect.

[0104] Beneficial effects of the embodiments of the present invention:

[0105] As can be seen from the above, by applying the solution provided by the embodiment of the present invention, when performing gesture recognition, first, an image to be recognized is obtained, and human body recognition is performed on the above image to be recognized to obtain a human body image of the object to be recognized, and then, the human body key points in the above human body image are detected. Thus, the wrist key points among the above human body key points can be used to determine the hand image of the object to be recognized from the above image to be recognized, and then, the hand key points in the above image to be recognized are detected, and finally, based on the positional relationship between the above hand key points and the designated key points about the arm among the above human body key points, the gesture recognition result of the object to be recognized is determined.

[0106] Based on this, by applying the solution provided by the embodiment of the present invention, when performing gesture recognition, the designated key points on the arm of the human body key points of the object to be recognized and the key points on the hand of the object to be recognized can be determined through human body recognition and key point detection. Then, based on the positional relationship between the designated key points and the hand key points, the gesture recognition result of the object to be recognized can be determined. In this way, since the positional relationship between the designated key points on the arm and the hand key points corresponding to each gesture can be consistent for different users, the accuracy of gesture recognition for different users can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0108] Figure 1 A schematic diagram of a flow chart of a first gesture action recognition method provided by an embodiment of the present invention;

[0109] Figure 2 is a schematic diagram of key points of a human body detected in a human body image;

[0110] FIG3( a ) is a schematic diagram of hand image detection according to a specific embodiment of the present invention;

[0111] FIG3( b ) is a schematic diagram of key points of a hand according to a specific embodiment of the present invention;

[0112] Figure 4 is a schematic diagram of an image area of ​​a preset size;

[0113] Figure 5 A schematic flow chart of a second gesture recognition method provided by an embodiment of the present invention;

[0114] Figure 6 A schematic diagram of a flow chart of a third gesture action recognition method provided by an embodiment of the present invention;

[0115] FIG7( a ) is a schematic diagram of a first gesture of a basketball referee;

[0116] FIG7( b ) is a schematic diagram of the second gesture of a basketball referee;

[0117] Figure 7(c) is a schematic diagram of the third gesture of a basketball referee;

[0118] Figure 7(d) is a schematic diagram of the fourth gesture of a basketball referee;

[0119] FIG7( e ) is a schematic diagram of the fifth gesture of a basketball referee;

[0120] Figure 8 A schematic flow chart of a fourth gesture recognition method provided by an embodiment of the present invention;

[0121] Figure 9 A schematic flow chart of a fifth gesture recognition method provided by an embodiment of the present invention;

[0122] Figure 10 A schematic flow chart of a sixth gesture recognition method provided by an embodiment of the present invention;

[0123] Figure 11 A schematic diagram of a specific embodiment of the present invention;

[0124] Figure 12 A schematic diagram of the structure of a gesture recognition system provided by an embodiment of the present invention;

[0125] Figure 13 A schematic structural diagram of a gesture recognition device provided by an embodiment of the present invention;

[0126] Figure 14 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0127] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of the present invention.

[0128] In the related art, when performing gesture recognition, the method adopted is usually: by comparing the obtained arm motion trajectory with the standard arm motion trajectory corresponding to the preset gesture action, the obtained gesture action is recognized. However, due to the different living habits and physical fitness of users, the motion trajectories of gesture actions performed by different users will be quite different. Therefore, the use of the above-mentioned related technology may result in misidentification and missed recognition of gesture actions, thereby reducing the recognition accuracy of gesture movements. Based on this, there is an urgent need for a gesture recognition method to improve the recognition accuracy of gesture actions.

[0129] In order to solve the above technical problems, an embodiment of the present invention provides a gesture action recognition method.

[0130] The method is applicable to various application scenarios requiring gesture recognition. For example, during a basketball game, it is necessary to recognize the referee's gestures to officiate or referee the game. Another example is using gestures to control a mobile phone to control installed applications. Furthermore, the method can be applied to various electronic devices such as laptops, tablets, and mobile phones (hereinafter referred to as electronic devices). Therefore, the embodiments of the present invention do not limit the application scenarios and execution of the method.

[0131] A gesture recognition method provided by an embodiment of the present invention may include the following steps:

[0132] Obtain the image to be recognized;

[0133] Performing human body recognition on the image to be recognized to obtain a human body image of the object to be recognized, and detecting human body key points in the human body image;

[0134] Using the wrist key points among the human body key points, determine the hand image of the object to be identified from the image to be identified, and detect the hand key points in the image to be identified;

[0135] Based on the positional relationship between the hand key points and the designated key points about the arm among the human body key points, a gesture action recognition result of the object to be recognized is determined.

[0136] As can be seen from the above, by applying the solution provided by the embodiment of the present invention, when performing gesture recognition, first, an image to be recognized is obtained, and human body recognition is performed on the above image to be recognized to obtain a human body image of the object to be recognized, and then, the human body key points in the above human body image are detected. Thus, the wrist key points among the above human body key points can be used to determine the hand image of the object to be recognized from the above image to be recognized, and then, the hand key points in the above image to be recognized are detected, and finally, based on the positional relationship between the above hand key points and the designated key points about the arm among the above human body key points, the gesture recognition result of the object to be recognized is determined.

[0137] Based on this, by applying the solution provided by the embodiment of the present invention, when performing gesture recognition, the designated key points on the arm of the human body key points of the object to be recognized and the key points on the hand of the object to be recognized can be determined through human body recognition and key point detection. Then, based on the positional relationship between the designated key points and the hand key points, the gesture recognition result of the object to be recognized can be determined. In this way, since the positional relationship between the designated key points on the arm and the hand key points corresponding to each gesture can be consistent for different users, the accuracy of gesture recognition for different users can be improved.

[0138] A gesture recognition method provided by an embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0139] Figure 1 A flow chart of a gesture recognition method provided by an embodiment of the present invention is shown as follows: Figure 1 As shown, the method includes the following steps S101-S104.

[0140] S101: Acquire an image to be recognized.

[0141] When performing gesture recognition on an object to be recognized, first, an image including the object to be recognized may be acquired as the image to be recognized.

[0142] For example, when recognizing the gestures of a basketball referee in a basketball game, first, an image including the basketball referee can be obtained as an image to be recognized.

[0143] S102: Performing human body recognition on the image to be recognized, obtaining a human body image of the object to be recognized, and detecting human body key points in the human body image.

[0144] After obtaining the image to be identified, human body recognition can be performed on the image to be identified, thereby obtaining a human body image of the object to be identified. Furthermore, key point detection can be performed on the human body image to obtain human body key points in the human body image.

[0145] Optionally, a target detection algorithm may be used to perform human recognition on the image to be recognized, for example, the YOLOv5 algorithm (target detection algorithm), the SSD (Single Shot MultiBox Detector) algorithm, and the like.

[0146] Optionally, key point detection can be performed on the human body image using a key point detection algorithm, such as a HRNet (High Resolution Net) algorithm, a CPM (Convolutional Pose Machine) algorithm, or the like.

[0147] For example, after obtaining the image to be identified including the basketball referee, the YOLOv5 algorithm is used to perform human body recognition on the image to be identified to obtain a human body image of the basketball referee. Then, the HRNet algorithm can be used to further perform key point detection on the above human body image to obtain the human body key points in the human body image of the basketball referee.

[0148] For example, Figure 2 As shown, it is a schematic diagram of the human body key points detected in the human body image, wherein points 0-17 are respectively a human body key point, for example, point 4 and point 7 are wrist key points among the human body key points, point 3 and point 6 are elbow key points among the human body key points, point 2 and point 5 are shoulder key points, etc.

[0149] in, Figure 2 The serial numbers of the key points of the human body are only an example and not a limitation. Any method that can mark the key points of the human body falls within the protection scope of the embodiments of the present invention.

[0150] S103: using the wrist key points among the human body key points, determining the hand image of the object to be identified from the image to be identified, and detecting the hand key points in the image to be identified.

[0151] After obtaining the human body key points in the above-mentioned human body image, the wrist key points can be determined from the above-mentioned human body key points. Thus, the above-mentioned wrist key points can be used to determine the hand image of the object to be identified from the above-mentioned image to be identified. Furthermore, the hand key points in the above-mentioned image to be identified can be obtained by performing key point detection on the above-mentioned hand image.

[0152] Optionally, a key point detection algorithm may be used to detect the hand image, such as the HRNet algorithm, the CPM algorithm, and the like.

[0153] For example, Figure 3(a) is a schematic diagram of hand image detection according to a specific embodiment of the present invention. Figure 3(b) is a schematic diagram of the hand key points in the hand image in Figure 3(a). Points 0-20 are each a hand key point, for example, point 4 is the thumb tip key point, point 8 is the index finger tip key point, point 12 is the middle finger tip key point, and so on.

[0154] The serial numbers of the key points of the hand in FIG3( b ) are merely examples and are not limiting. Any method of marking the key points of the hand falls within the protection scope of the embodiments of the present invention.

[0155] For example, when obtaining Figure 2 As shown in FIG3( a ), after detecting the human body key points in the human body image of the basketball coach, the wrist key points are determined as point 4 and point 7 from the above human body key points. Thus, the wrist key points, key point 4 and key point 7, can be used to determine the hand image of the basketball coach from the image to be identified. By detecting the hand image of the basketball coach as shown in FIG3( a ), the hand key points of the basketball coach as shown in FIG3( b ) are obtained.

[0156] Optionally, in a specific implementation, the above step S103 may include the following step 1031:

[0157] Step 1031: In the image to be identified, starting from the wrist key point among the key points of the human body, an image region of a preset size is determined along a preset direction as a hand image of the object to be identified.

[0158] In this specific implementation method, the extension direction of the arm of the object to be identified and the size of the hand area of ​​the object to be identified can be first determined based on the scene reflected by the image to be identified, wherein the above extension direction can be used as a preset direction, and the above hand area size can be used as a preset size.

[0159] In this way, after obtaining the wrist key point among the key points of the human body, and using the position coordinates of the above-mentioned wrist key point in the image to be identified, in the image to be identified, taking the position coordinates of the above-mentioned wrist key point as the starting point, along the above-mentioned preset direction, using the above-mentioned preset size, the position coordinates of the hand image of the object to be identified are determined. Thus, according to the position coordinates of the hand image determined above, the hand image of the object to be identified is determined, and the size of the hand image is the above-mentioned preset size.

[0160] For example, Figure 4 As shown, Figure 4 Schematic diagram of an image region of a preset size, wherein point 401 is a wrist key point, the direction indicated by the arrow is a preset direction, image region 400 is an image of the hand of the object to be identified, and the size of image region 400 is a preset size.

[0161] Optionally, in a specific implementation, the above step S103 may include the following step 1032:

[0162] Step 1032: In the image to be identified, starting from the wrist key point and following the direction from the elbow key point to the wrist key point in the human body key points, determine an image region of a specified size as a hand image of the object to be identified;

[0163] The specified size is based on the length between the elbow keypoint and the wrist keypoint.

[0164] In this specific implementation, according to the growth law of the human skeleton, usually, the length of the user's arm and hand can be in a certain ratio, and during the movement of the user's arm and hand, the user's hand can extend outward along the direction from the elbow to the wrist. Therefore, after obtaining the wrist key point among the human body key points, the direction in which the elbow key point points to the wrist key point, as well as the length between the elbow key point and the wrist key point, can be further determined. Then, based on the above length, the specified size of the hand can be calculated using the empirical value of the ratio of the length of the human arm and hand. In this way, in the image to be identified, the position coordinates of the hand image of the object to be identified can be determined along the above direction using the above specified size, and the hand image of the object to be identified can be determined based on the position coordinates of the hand image determined above, and the size of the hand image is the above specified size.

[0165] For example, based on the length between the elbow and wrist keypoints in the human body keypoints, the specified size can be calculated as 30 cm * 30 cm. The wrist keypoints in the human body keypoints are X1 and X2, while the elbow keypoints in the human body keypoints are Y1 and Y2. Wrist keypoint X1 and elbow keypoint Y1 are on one arm, while wrist keypoint X2 and elbow keypoint Y2 are on the other arm. Starting from wrist keypoint X1, the positional information of the hand image of the object to be identified is determined along the direction from elbow keypoint Y1 to wrist keypoint X1 in the human body keypoints, according to the human body proportions. Thus, the image area of ​​30 cm * 30 cm is determined as a hand image of the object to be identified.

[0166] Starting from the wrist key point X2, along the direction of the elbow key point Y2 in the human body key points pointing to the wrist key point X2, the position information of the hand image of the object to be identified is determined according to the human body proportions. Thus, the image area of ​​30 cm * 30 cm in size is determined as another hand image of the object to be identified.

[0167] S104: Determine a gesture recognition result of the object to be recognized based on the positional relationship between the hand key points and the designated key points about the arm in the human body key points.

[0168] Since each gesture can include both arm and hand movements, the user's arm and hand can exhibit a specific positional relationship for each gesture. Therefore, the detected positional relationship between the arm and hand of the subject to be identified can be used to determine the gesture performed by the subject to be identified. Furthermore, since the positional relationship between the arm and hand can be characterized by the positional relationship between the hand key points and designated key points related to the arm among the body key points, the gesture recognition result for the subject to be identified can be determined based on the positional relationship between the hand key points and designated key points related to the arm among the body key points.

[0169] Optionally, a correspondence between positional relationships and gesture actions may be pre-established, wherein the positional relationship in the correspondence refers to a positional relationship between a hand key point and a designated key point on the arm among the key points of the human body.

[0170] For example, as shown in Table 1, it is a schematic diagram of a partial correspondence between position relationships and gesture actions.

[0171] Table 1

[0172] Positional relationship Gestures α 9 β 10 γ 11

[0173] Among them, if in the image to be identified, the positional relationship between the hand key points and the designated key points about the arm among the human body key points is α, then the gesture action of the object to be identified can be determined to be gesture action 9; if in the image to be identified, the positional relationship between the hand key points and the designated key points about the arm among the human body key points is β, then the gesture action of the object to be identified can be determined to be gesture action 10; if in the image to be identified, the positional relationship between the hand key points and the designated key points about the arm among the human body key points is γ, then the gesture action of the object to be identified can be determined to be gesture action 11.

[0174] For example, as shown in Table 2, it is another schematic diagram of the partial correspondence between position relationships and gesture actions.

[0175] Table 2

[0176] About the specified key point position C of the arm About the specified key point position D of the arm Hand key point position A Gesture Action 1 Gesture Action 2 Hand key point position B Gesture Action 3 Gesture Action 4

[0177] Among them, if in the image to be identified, the hand key point is located at position A, and the designated key point about the arm is located at position C, then the gesture action of the object to be identified can be gesture action 1; if in the image to be identified, the hand key point is located at position A, and the designated key point about the arm is located at position D, then the gesture action of the object to be identified can be determined to be gesture action 2; if in the image to be identified, the hand key point is located at position B, and the designated key point about the arm is located at position C, then the gesture action of the object to be identified can be determined to be gesture action 3; if in the image to be identified, the hand key point is located at position B, and the designated key point about the arm is located at position D, then the gesture action of the object to be identified can be gesture action 4.

[0178] In this way, after using the obtained human body key points to determine the designated key points about the arms among the above human body key points, the gesture action recognition results of the above object to be identified can be determined based on the positional relationship between the above hand key points and the above designated key points and the pre-established correspondence between the positional relationship and the gesture action.

[0179] Optionally, gesture actions may include arm posture and hand posture. Therefore, first, the hand posture of the object to be identified can be determined based on the hand key points, and then, the arm posture of the object to be identified can be determined based on the arm key points. Thus, the positional relationship between the hand posture and the arm posture of the object to be identified can be determined based on the hand key points and the arm key points. Then, the above positional relationship can be used to obtain the gesture action recognition result of the object to be identified.

[0180] In many cases, a complete gesture action is a dynamic action composed of multiple continuous static actions, and a dynamic action may involve multiple images. Therefore, when recognizing a gesture action completed by performing a dynamic action, multiple images to be recognized may be involved.

[0181] Based on this, optionally, in a specific implementation, such as Figure 5 As shown, the above step S104 may include the following steps S1041-S1042:

[0182] S1041: Determine an initial recognition result corresponding to the video frame based on a positional relationship between the hand key points and designated key points about the arm among the human body key points;

[0183] S1042: Determine a gesture recognition result of the object to be recognized by using the initial recognition results and the association relationship between the initial recognition results.

[0184] Optionally, the initial recognition result may be the positional relationship between the hand key points and the designated key points on the arm among the human body key points.

[0185] In this specific implementation, multiple images to be recognized can be obtained. For each image to be recognized, a gesture recognition result for the object to be recognized is determined as the initial recognition result corresponding to the image to be recognized. The initial recognition results and the correlations between them can then be used to ultimately determine the gesture recognition result for the object to be recognized.

[0186] Among them, for a gesture action completed by executing a static action, it can be determined using an initial recognition result; and for a gesture action completed by executing a dynamic action composed of multiple static actions, it can be determined using multiple initial recognition results with an associated relationship.

[0187] For example, there are three images to be recognized: image m, image n, and image f. For image m, the initial recognition result corresponding to image m is determined to be M; for image n, the initial recognition result corresponding to image n is determined to be N; and for image f, the initial recognition result corresponding to image f is determined to be F.

[0188] Among them, the initial recognition result M is associated with the initial recognition result N, and the initial recognition result F is not associated with the initial recognition result M or the initial recognition result N. Therefore, based on the initial recognition result M and the initial recognition result N, the gesture action recognition result can be determined to be: gesture action a, and based on the initial recognition result F, the gesture action recognition result can be determined to be: gesture action b.

[0189] Optionally, in a specific implementation, such as Figure 6 As shown, a gesture recognition method provided by an embodiment of the present invention may further include the following step S105:

[0190] S105: Outputting the gesture action recognition result.

[0191] In this specific implementation, after the gesture recognition result is obtained, the obtained gesture action recognition result can be output.

[0192] Optionally, the gesture action recognition result is output as text.

[0193] Optionally, the gesture recognition results are displayed as images.

[0194] Optionally, the gesture action recognition result obtained can be output through voice broadcast.

[0195] For example, taking a basketball referee as an example, Figure 7(a)-Figure 7(e) is the output gesture recognition result.

[0196] As shown in Figure 7(a), the gesture of the basketball referee is: raise the right index finger and bend it downward from the wrist once. The gesture recognition result in Figure 7(a) is: one point.

[0197] As shown in Figure 7(b), the gesture of the basketball referee is: raise the index finger and middle finger of the right hand and bend them downward from the wrist once. The gesture recognition result in Figure 7(b) is: two points.

[0198] As shown in Figure 7(c), the gesture of the basketball referee is: raising the thumb, index finger and middle finger of the right hand at the same time. The gesture recognition result in Figure 7(c) is: three-point attempt.

[0199] As shown in Figure 7(d), the gesture of the basketball referee is: raising the thumb, index finger and middle finger of both hands at the same time. The gesture recognition result in Figure 7(d) is: a three-point shot is successful.

[0200] As shown in Figure 7(e), the gesture of the basketball referee is: crossing both hands in front of the body once. The gesture recognition result in Figure 7(e) is: cancel the score.

[0201] Since the object to be identified may be in motion when the image of the object to be identified is captured, the detection accuracy of the key points of the human body in the acquired image to be identified may be low, and thus the accuracy of the gesture action recognition result obtained may be low.

[0202] Based on this, optionally, in a specific implementation, such as Figure 8 As shown, Figure 8 The flowchart of a gesture recognition method provided by an embodiment of the present invention may further include the following steps S106-S107:

[0203] S106: Obtaining the detection confidence of the key points of the human body;

[0204] S107: If the detection confidence is greater than the preset confidence threshold, execute step S103.

[0205] In this specific implementation, before using the wrist key point among the human body key points to determine the hand image of the object to be identified from the image to be identified, the detection confidence of each human body key point in the image to be identified can be obtained.

[0206] Among them, the detection confidence is used to characterize the credibility of the human key points in the image to be identified. For each human key point, the greater the detection confidence of the human key point, the greater the possibility that the human key point is located at the corresponding position on the object to be identified.

[0207] Optionally, when performing key point detection on the human body image of the object to be identified, the detection confidence of each detected key point can be determined simultaneously.

[0208] Therefore, before using the wrist key point among the human body key points to determine the hand image of the object to be identified from the image to be identified, the relationship between the detection confidence of the acquired human body key point and the preset confidence threshold can be first determined.

[0209] Among them, if the detection confidence of the human body key points is greater than the preset confidence threshold, it can be said that the accuracy of the detected human body key points is high, and it is more likely that they are located at the corresponding position on the object to be identified. Therefore, the wrist key points in the human body key points can be further used to determine the hand image of the object to be identified from the image to be identified.

[0210] In addition, the above-mentioned preset confidence threshold can be set according to the needs of actual applications. For example, if the recognition accuracy requirement for gesture actions is low in actual applications, the above-mentioned preset confidence threshold can be set to a smaller value, such as 0.6, 0.5, etc.; if the recognition accuracy requirement for gesture actions is high in actual applications, the above-mentioned preset confidence threshold can be set to a larger value, such as 0.8, 0.9, etc.; in this regard, the embodiment of the present invention does not take specific action.

[0211] Optionally, if the image to be identified is each video frame in a video, then for each video frame, when the detection confidence of the human key points detected in the video frame is not greater than a preset confidence threshold, the video frame can be ignored, and the next video frame can be traversed to perform gesture action recognition again.

[0212] Optionally, if the image to be identified is an independent image, when the detection confidence of the human body key points detected in the image to be identified is not greater than a preset confidence threshold, the image to be identified can be ignored and a new image to be identified can be obtained for gesture action recognition.

[0213] Optionally, the above step S107 may include the following step 1071:

[0214] Step 1071: If the detection confidence of the wrist key point among the key points of the human body is greater than a preset confidence threshold, step S103 is executed.

[0215] Since the wrist key point among the various human body key points is an important key point used in gesture recognition, the detection confidence of the above-mentioned wrist key point can affect the accuracy of the final gesture recognition result. Therefore, before using the wrist key point among the human body key points to determine the hand image of the object to be identified from the image to be identified, you can first determine the relationship between the detection confidence of the wrist key point among the obtained human body key points and the preset confidence threshold. When the detection confidence of the wrist key point is greater than the preset confidence threshold, continue to use the wrist key point among the human body key points to determine the hand image of the object to be identified from the image to be identified.

[0216] As can be seen from the above, by applying the solution provided by the embodiment of the present invention, when performing gesture recognition, first, an image to be recognized is obtained, and human body recognition is performed on the above image to be recognized to obtain a human body image of the object to be recognized, and then, the human body key points in the above human body image are detected. Thus, the wrist key points among the above human body key points can be used to determine the hand image of the object to be recognized from the above image to be recognized, and then, the hand key points in the above image to be recognized are detected, and finally, based on the positional relationship between the above hand key points and the designated key points about the arm among the above human body key points, the gesture recognition result of the object to be recognized is determined.

[0217] Based on this, by applying the solution provided by the embodiment of the present invention, when performing gesture recognition, the designated key points on the arm of the human body key points of the object to be recognized and the key points on the hand of the object to be recognized can be determined through human body recognition and key point detection. Then, based on the positional relationship between the designated key points and the hand key points, the gesture recognition result of the object to be recognized can be determined. In this way, since the positional relationship between the designated key points on the arm and the hand key points corresponding to each gesture can be consistent for different users, the accuracy of gesture recognition for different users can be improved.

[0218] For a continuous video of an object to be identified, the gestures of the object to be identified are identified, for example, the gestures of a basketball coach are identified in a basketball game.

[0219] Optionally, in a specific implementation, such as Figure 9 As shown, a gesture action recognition method provided by an embodiment of the present invention may include the following steps S901-S904:

[0220] S901: Acquire a video to be identified, and determine each video frame in the video to be identified as an image to be identified;

[0221] S902: Performing human body recognition on the image to be recognized, obtaining a human body image of the object to be recognized, and detecting human body key points in the human body image;

[0222] S903: using the wrist key points among the human body key points, determining the hand image of the object to be identified from the image to be identified, and detecting the hand key points in the image to be identified;

[0223] S904: Determine a gesture recognition result of the object to be recognized based on the positional relationship between the hand key points and the designated key points about the arm in the human body key points.

[0224] In this specific implementation, after obtaining the video to be identified, each video frame in the video to be identified can be traversed. Thus, when each video frame of the video to be identified is traversed, the video frame is determined as the image to be identified. Then, for the video frame, human body recognition can be performed on the video frame, thereby obtaining a human body image of the video frame. Then, the human body image can be further key-point detected to obtain the human body key points in the human body image. In this way, the wrist key points can be determined from the human body key points. Thus, the position information of the wrist key points can be used to determine the position information of the hand image of the object to be identified from the image to be identified. Then, the hand image of the object to be identified can be determined. Furthermore, the hand key points in the image to be identified can be obtained by performing key-point detection on the hand image. Finally, after determining the designated key points of the arm in the human body key points using the obtained human body key points, the gesture recognition result of the object to be identified can be determined based on the positional relationship between the hand key points and the designated key points.

[0225] In this way, after obtaining the gesture recognition result of the object to be identified for the video frame, the next video frame of the video to be identified can be traversed, and the gesture recognition result of the object to be identified for each video frame in the video to be identified can be obtained in a cycle.

[0226] For each video frame, steps S902 to S904 are performed in the same manner as steps S102 to S104 above, and are not described in detail here.

[0227] In many cases, a complete gesture action is a dynamic action composed of multiple continuous static actions, and a dynamic action can involve multiple video frames. Therefore, when identifying a gesture action completed by executing a dynamic action, the recognition results corresponding to each video frame can be cached, and then the gesture action recognition result of the object to be identified can be determined based on the recognition results corresponding to multiple video frames.

[0228] Optionally, in a specific implementation, in each of the above specific implementations, if Figure 10 As shown, the above step S104, using each initial recognition result and the correlation relationship between each initial recognition result, determines the gesture recognition result of the object to be recognized, which may include the following steps S1001-S1003:

[0229] S1001: For each video frame, cache the initial recognition result corresponding to the video frame;

[0230] S1002: Determine whether the initial recognition result corresponding to the video frame is associated with the specified recognition result; if not, execute step S1003;

[0231] The designated recognition result is: each initial recognition result cached before the initial recognition result corresponding to the video frame;

[0232] S1003: using the designated recognition result, determining the gesture recognition result of the object to be recognized, and performing designated deletion processing on the designated recognition result.

[0233] In this specific implementation, when performing gesture action recognition, for each video frame, the initial recognition result corresponding to the video frame can be cached first, and it can be determined whether the initial recognition result corresponding to the video frame is associated with the initial recognition results cached before the initial recognition result corresponding to the video frame.

[0234] Among them, each initial recognition result cached before the initial recognition result corresponding to the video frame can be called a designated recognition result. Therefore, it can be determined whether the initial recognition result corresponding to the video frame is associated with the designated recognition result.

[0235] If the initial recognition result corresponding to the video frame is not associated with the specified recognition result, it can be said that the above initial recognition result and the above specified recognition result do not belong to the same gesture action. Therefore, the above specified recognition result can be used to determine the gesture action recognition result of the object to be identified, and the specified recognition result can be deleted.

[0236] If the initial recognition result corresponding to the video frame is associated with the specified recognition result, it can be said that the above initial recognition result and the above specified recognition result belong to the same gesture action. At this time, the initial recognition result corresponding to the video frame can be used as the new specified recognition result, thereby completing the gesture action recognition of the video frame and continuing to perform gesture action recognition on the next image to be recognized.

[0237] For example, there are three video frames, including video frame m, video frame n, and video frame f. The initial recognition result corresponding to video frame m is M; the initial recognition result corresponding to video frame n is N; and the initial recognition result corresponding to video frame f is F.

[0238] The initial recognition result M is the initial recognition result corresponding to the first video frame m. Both the initial recognition result M and the initial recognition result N have been cached, and the initial recognition result M and the initial recognition result N are associated.

[0239] After obtaining the initial recognition result F, cache the initial recognition result F, and determine whether the initial recognition result F is associated with the initial recognition result M and the initial recognition result N that have been cached; if the initial recognition result F is associated with the initial recognition result M and the initial recognition result N that have been cached, it can be said that the initial recognition result F and the initial recognition result M and the initial recognition result N that have been cached belong to the same gesture action, and the initial recognition result F is cached; if the initial recognition result F is not associated with the initial recognition result M and the initial recognition result N that have been cached, it can be said that the initial recognition result F and the initial recognition result M and the initial recognition result N that have been cached do not belong to the same gesture action, therefore, based on the initial recognition result M and the initial recognition result N, the gesture action recognition result can be determined to be: gesture action a, and based on the initial recognition result F, the gesture action recognition result can be determined to be: gesture action b.

[0240] Optionally, in a specific implementation, the above step S1002 may include the following step 21.

[0241] Step 21: Determine whether the sum of the number of the initial recognition result and the designated recognition result corresponding to the video frame is not greater than a preset value; if not, the initial recognition result corresponding to the video frame is not associated with the designated recognition result.

[0242] In this specific implementation, after caching the initial recognition result corresponding to each video frame, it is determined whether the sum of the initial recognition result corresponding to the video frame and the specified recognition result is not greater than a preset value.

[0243] If the sum of the number of the initial recognition result and the specified recognition result corresponding to the video frame is not greater than a preset value, the above-mentioned initial recognition result and the above-mentioned specified recognition result belong to the same gesture action, that is, the initial recognition result corresponding to the video frame is associated with the specified recognition result.

[0244] If the sum of the number of the initial recognition results and the specified recognition results corresponding to the video frame is greater than a preset value, the above-mentioned initial recognition results and the above-mentioned specified recognition results do not belong to the same gesture action, that is, the initial recognition results corresponding to the video frame are not associated with the specified recognition results.

[0245] For example, the preset value is 10, and the sum of the number of specified recognition results is 9. For each video frame, after the initial recognition result corresponding to the video frame is cached, if the sum of the number of the initial recognition result corresponding to the video frame and the number of specified recognition results is equal to 10, then the initial recognition result corresponding to the video frame is associated with the specified recognition result.

[0246] For another example, the preset threshold is 10, and the sum of the number of specified recognition results is 8. For each video frame, after caching the initial recognition result corresponding to the video frame, if the sum of the initial recognition result corresponding to the video frame and the number of specified recognition results is 9, and 9<10, then the initial recognition result and the specified recognition result belong to the same gesture action, that is, the initial recognition result corresponding to the video frame is associated with the specified recognition result.

[0247] Accordingly, in a specific implementation, the above step S1002 may include the following step 22.

[0248] Step 22: Determine whether the initial recognition result corresponding to the video frame is associated with the last initial recognition result in the designated recognition results; if not, the initial recognition result corresponding to the video frame is not associated with the designated recognition result.

[0249] In this specific implementation, there is a correlation between the various initial recognition results in the specified recognition results. Therefore, to determine whether the initial recognition result corresponding to the obtained video frame is correlated with the specified recognition result, it is sufficient to determine whether the initial recognition result corresponding to the obtained video frame is correlated with the last initial recognition result in the specified recognition result.

[0250] Therefore, after caching the initial recognition result corresponding to each video frame, it is determined whether the initial recognition result corresponding to the video frame is associated with the last initial recognition result in the specified recognition results.

[0251] If the initial recognition result corresponding to the video frame is associated with the last initial recognition result in the specified recognition results, then the initial recognition result and the specified recognition result belong to the same gesture action, that is, the initial recognition result corresponding to the video frame is associated with the specified recognition result.

[0252] If the initial recognition result corresponding to the video frame is not associated with the last initial recognition result in the specified recognition results, then the above initial recognition result and the above specified recognition result do not belong to the same gesture action, that is, the initial recognition result corresponding to the video frame is not associated with the specified recognition result.

[0253] Optionally, if the range of change in the positional relationship between the hand key points and the specified key points about the arm in the human body key points represented by the initial recognition result corresponding to the video frame and the last initial recognition result in the specified recognition results does not exceed a preset range, then the initial recognition result of the video frame is associated with the last initial recognition result in the specified recognition results.

[0254] Optionally, if the video playback time interval between the initial recognition result corresponding to the video frame and the last initial recognition result in the specified recognition results falls within a preset time range, then the initial recognition result of the video frame is associated with the last initial recognition result in the specified recognition results.

[0255] In a complete video stream, there may be multiple gestures performed by the object to be identified. Therefore, after obtaining the gesture recognition result of the object to be identified, it is necessary to delete the specified recognition result used to determine the gesture recognition result.

[0256] For example, there are 5 designated recognition results, including designated recognition result 1, designated recognition result 2, designated recognition result 3, designated recognition result 4 and designated recognition result 5, and the above 5 designated recognition results are associated with each other, that is, the above 5 designated recognition results belong to the same gesture action, and the order between the above 5 designated recognition results is: designated recognition result 1, designated recognition result 2, designated recognition result 3, designated recognition result 4 and designated recognition result 5.

[0257] For a new video frame, if the range of change in the positional relationship between the initial recognition result of the video frame and the designated key points about the arm among the hand key points and human body key points represented in the designated recognition result 5 does not exceed the preset range, then the initial recognition result of the video frame is associated with the designated recognition result 5.

[0258] Optionally, in a specific implementation, in the above step S1003, the step of performing a designated deletion process on the designated recognition result may include the following step 23:

[0259] Step 23: Clear the specified recognition results.

[0260] In this specific implementation, after the gesture recognition result of the object to be recognized is determined, the specified recognition result can be cleared.

[0261] Accordingly, in a specific implementation, in the above step S1003, the step of performing a designated deletion process on the designated recognition result may include the following step 24:

[0262] Step 24: Delete the first cached initial recognition result among the specified recognition results.

[0263] In this specific implementation, after the gesture recognition result of the object to be recognized is determined, the first cached initial recognition result among the specified recognition results may be deleted.

[0264] For example, if the number of initial recognition results that can be cached is preset to 10, then after the initial recognition result corresponding to the 11th video frame is cached, the initial recognition result corresponding to the first video frame in the specified recognition results needs to be deleted.

[0265] For ease of understanding, Figure 11 FIG. 1 is a flow chart of a specific embodiment of the present invention.

[0266] in, Figure 11 The stream obtained in the present invention is to obtain the video to be identified, Figure 11 The image in is the image to be identified in the present invention, Figure 11 The target detection in the present invention is to perform human body recognition on the image to be recognized, and obtain a human body image of the object to be recognized. Figure 11 The human body posture estimation in the present invention is to detect the human body key points in the human body image, Figure 11 The hand key point detection in the present invention is to use the wrist key point in the human body key point to determine the hand image of the object to be identified from the image to be identified, and detect the hand key points in the image to be identified. Figure 11 The single-frame gesture recognition in the present invention is based on the positional relationship between the key points of the hand and the designated key points of the arm in the key points of the human body, and determines the initial recognition result corresponding to the video frame. Figure 11 The buffer in is the cache initial recognition result in the present invention, Figure 11 The buffer in the frame T is used to determine whether the sum of the initial recognition result and the specified recognition result corresponding to the video frame is not greater than a preset value in the present invention. Figure 11 The multi-frame gesture recognition in the present invention uses each initial recognition result and the correlation between each initial recognition result to determine the gesture action recognition result of the object to be recognized. Figure 11The first image in the elimination buffer is the initial recognition result first cached in the deletion specified recognition result of the present invention. Figure 11 The output gesture recognition result in is the output gesture action recognition result in the present invention.

[0267] When performing gesture action recognition, first, a stream is taken from the video stream and an image is obtained. Target detection is performed on the image to obtain a human body image in the image. Thus, human body key points can be determined from the above human body image through human body posture estimation. Then, hand key points in the above human body key points are determined by hand key point detection, and then a single-frame gesture recognition result of the image is obtained. Furthermore, the obtained single-needle gesture recognition result is input into a buffer, and it is determined whether the sum of the number of each single-frame gesture recognition result in the above buffer is not greater than a preset value T. If so, the initial recognition result corresponding to the video frame is associated with each pre-cached initial recognition result, and each initial recognition result in the buffer is output as a gesture recognition result, and the first image in the buffer is eliminated at the same time. If not, return to the stream taking step and re-execute the above steps.

[0268] Corresponding to the gesture action recognition method provided by the above embodiment of the present invention, the embodiment of the present invention also provides a gesture action recognition system.

[0269] Figure 12 A structural diagram of a gesture action recognition system provided by an embodiment of the present invention is shown in FIG. Figure 12 As shown, the system may include:

[0270] An image acquisition module 1210 is used to acquire an image to be recognized;

[0271] The processor 1220 is configured to perform human body recognition on the image to be recognized, obtain a human body image of the object to be recognized, and detect key points of the human body in the human body image;

[0272] Using the wrist key points among the human body key points, determine the hand image of the object to be identified from the image to be identified, and detect the hand key points in the image to be identified;

[0273] Based on the positional relationship between the hand key points and the designated key points about the arm among the human body key points, a gesture action recognition result of the object to be recognized is determined.

[0274] As can be seen from the above, by applying the solution provided by the embodiment of the present invention, when performing gesture recognition, first, an image to be recognized is obtained, and human body recognition is performed on the above image to be recognized to obtain a human body image of the object to be recognized, and then, the human body key points in the above human body image are detected. Thus, the wrist key points among the above human body key points can be used to determine the hand image of the object to be recognized from the above image to be recognized, and then, the hand key points in the above image to be recognized are detected, and finally, based on the positional relationship between the above hand key points and the designated key points about the arm among the above human body key points, the gesture recognition result of the object to be recognized is determined.

[0275] Based on this, by applying the solution provided by the embodiment of the present invention, when performing gesture recognition, the designated key points on the arm of the human body key points of the object to be recognized and the key points on the hand of the object to be recognized can be determined through human body recognition and key point detection. Then, based on the positional relationship between the designated key points and the hand key points, the gesture recognition result of the object to be recognized can be determined. In this way, since the positional relationship between the designated key points on the arm and the hand key points corresponding to each gesture can be consistent for different users, the accuracy of gesture recognition for different users can be improved.

[0276] Optionally, in a specific implementation, the image acquisition module 1210 is specifically configured to:

[0277] A video to be identified is obtained, and each video frame in the video to be identified is determined as an image to be identified.

[0278] Optionally, in a specific implementation, the processor 1220 is specifically configured to:

[0279] Determining an initial recognition result corresponding to the video frame based on a positional relationship between the hand key points and a designated key point about the arm among the human body key points;

[0280] The gesture recognition result of the object to be recognized is determined by using the respective initial recognition results and the association relationship between the respective initial recognition results.

[0281] Optionally, in a specific implementation, the processor 1220 is specifically configured to:

[0282] For each video frame, cache the initial recognition result corresponding to the video frame, and determine whether the initial recognition result corresponding to the video frame is associated with a specified recognition result; wherein the specified recognition result is: each initial recognition result cached before the initial recognition result corresponding to the video frame;

[0283] If not, the designated recognition result is used to determine the gesture recognition result of the object to be recognized, and a designated deletion process is performed on the designated recognition result.

[0284] Optionally, in a specific implementation, the processor 1220 is specifically configured to:

[0285] Determine whether the sum of the number of the initial video result and the specified recognition result corresponding to the video frame is not greater than a preset value;

[0286] or,

[0287] Determining whether the initial recognition result corresponding to the video frame is associated with the last initial recognition result in the specified recognition results;

[0288] If not, the initial recognition result corresponding to the video frame is not associated with the specified recognition result.

[0289] Optionally, in a specific implementation, the processor 1220 is specifically configured to:

[0290] Clear the specified recognition result; or,

[0291] Delete the first cached initial recognition result among the specified recognition results.

[0292] Optionally, in a specific implementation, the processor 1220 is further configured to:

[0293] Before determining the hand image of the object to be identified from the image to be identified using the wrist key point among the human body key points, obtaining the detection confidence of the human body key point;

[0294] If the detection confidence is greater than a preset confidence threshold, the wrist key point among the human body key points is used to determine a hand image of the object to be identified from the image to be identified.

[0295] Optionally, in a specific implementation, the processor 1220 is specifically configured to:

[0296] In the image to be identified, taking the wrist key point among the key points of the human body as a starting point and determining an image area of ​​a preset size along a preset direction as a hand image of the object to be identified;

[0297] or,

[0298] In the image to be identified, with the wrist key point as the starting point, an image area of ​​a specified size is determined along the direction from the elbow key point among the human body key points to the wrist key point, as the hand image of the object to be identified; wherein the specified size is determined based on the length between the elbow key point and the wrist key point.

[0299] Optionally, in a specific implementation, the processor 1220 is further configured to:

[0300] Output the gesture action recognition result.

[0301] Corresponding to the gesture recognition method provided by the above embodiment of the present invention, the embodiment of the present invention further provides a gesture recognition device.

[0302] Figure 13 A schematic diagram of a gesture recognition device according to an embodiment of the present invention is shown in FIG. Figure 13 As shown, the device may include the following modules:

[0303] An image acquisition module 1310 is used to acquire an image to be recognized;

[0304] A human body key point acquisition module 1320 is configured to perform human body recognition on the image to be recognized, obtain a human body image of the object to be recognized, and detect human body key points in the human body image;

[0305] A hand key point acquisition module 1330 is configured to determine a hand image of the object to be identified from the image to be identified using the wrist key point among the human body key points, and detect the hand key points in the image to be identified;

[0306] The result acquisition module 1340 is used to determine the gesture recognition result of the object to be recognized based on the positional relationship between the hand key points and the designated key points about the arm among the human body key points.

[0307] As can be seen from the above, by applying the solution provided by the embodiment of the present invention, when performing gesture recognition, first, an image to be recognized is obtained, and human body recognition is performed on the above image to be recognized to obtain a human body image of the object to be recognized, and then, the human body key points in the above human body image are detected. Thus, the wrist key points among the above human body key points can be used to determine the hand image of the object to be recognized from the above image to be recognized, and then, the hand key points in the above image to be recognized are detected, and finally, based on the positional relationship between the above hand key points and the designated key points about the arm among the above human body key points, the gesture recognition result of the object to be recognized is determined.

[0308] Based on this, by applying the solution provided by the embodiment of the present invention, when performing gesture recognition, the designated key points on the arm of the human body key points of the object to be recognized and the key points on the hand of the object to be recognized can be determined through human body recognition and key point detection. Then, based on the positional relationship between the designated key points and the hand key points, the gesture recognition result of the object to be recognized can be determined. In this way, since the positional relationship between the designated key points on the arm and the hand key points corresponding to each gesture can be consistent for different users, the accuracy of gesture recognition for different users can be improved.

[0309] Optionally, in a specific implementation, the image acquisition module 1310 is specifically configured to:

[0310] A video to be identified is obtained, and each video frame in the video to be identified is determined as an image to be identified.

[0311] Optionally, in a specific implementation, the result acquisition module 1340 includes:

[0312] An initial result acquisition submodule, configured to determine an initial recognition result corresponding to the video frame based on a positional relationship between the hand key points and a designated key point about the arm among the human body key points;

[0313] The result acquisition submodule is used to determine the gesture recognition result of the object to be recognized by using the initial recognition results and the association relationship between the initial recognition results.

[0314] Optionally, in a specific implementation, the result acquisition submodule includes:

[0315] A first cache unit is configured to cache, for each video frame, an initial recognition result corresponding to the video frame;

[0316] a judgment unit, configured to judge whether the initial recognition result corresponding to the video frame is associated with a specified recognition result; if not, triggering a first result acquisition unit; wherein the specified recognition result is: each initial recognition result cached before the initial recognition result corresponding to the video frame;

[0317] The first result acquisition unit is configured to determine the gesture recognition result of the object to be recognized by using the specified recognition result, and trigger the deletion unit;

[0318] The deleting unit is used to perform designated deletion processing on the designated recognition result.

[0319] Optionally, in a specific implementation, the judgment unit is specifically configured to:

[0320] Determine whether the sum of the number of the initial video result and the specified recognition result corresponding to the video frame is not greater than a preset value;

[0321] or,

[0322] Determining whether the initial recognition result corresponding to the video frame is associated with the last initial recognition result in the specified recognition results;

[0323] If not, the initial recognition result corresponding to the video frame is not associated with the specified recognition result.

[0324] Optionally, in a specific implementation, the deleting unit is specifically configured to:

[0325] Clear the specified recognition result; or,

[0326] Delete the first cached initial recognition result among the specified recognition results.

[0327] Optionally, in a specific implementation, the device further includes:

[0328] a detection confidence module, configured to obtain a detection confidence of the human body key points before determining a hand image of the object to be identified from the image to be identified using the wrist key points among the human body key points;

[0329] If the detection confidence is greater than a preset confidence threshold, the hand key point acquisition module 1330 is triggered.

[0330] Optionally, in a specific implementation, the hand key point acquisition module 1330 is specifically configured to:

[0331] In the image to be identified, taking the wrist key point among the key points of the human body as a starting point and determining an image area of ​​a preset size along a preset direction as a hand image of the object to be identified;

[0332] or,

[0333] In the image to be identified, with the wrist key point as the starting point, an image area of ​​a specified size is determined along the direction from the elbow key point among the human body key points to the wrist key point, as the hand image of the object to be identified; wherein the specified size is determined based on the length between the elbow key point and the wrist key point.

[0334] Optionally, in a specific implementation, the device further includes:

[0335] The result output module is used to output the gesture action recognition result.

[0336] Corresponding to the above-mentioned gesture recognition method provided by the embodiment of the present invention, the embodiment of the present invention also provides an electronic device, such as Figure 14 As shown, it includes a processor 1401, a communication interface 1402, a memory 1403 and a communication bus 1404, wherein the processor 1401, the communication interface 1402, and the memory 1403 communicate with each other through the communication bus 1404.

[0337] Memory 1403, used for storing computer programs;

[0338] The processor 1401 is configured to implement the steps of the gesture recognition method provided by the embodiment of the present invention when executing the program stored in the memory 1403 .

[0339] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0340] The communication interface is used for communication between the above electronic device and other devices.

[0341] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0342] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0343] In another embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned gesture action recognition methods are implemented.

[0344] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute the steps of any one of the gesture action recognition methods in the above embodiments.

[0345] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0346] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0347] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, since the system embodiments, electronic device embodiments, computer-readable storage medium embodiments, and computer program product embodiments are generally similar to the method embodiments, their descriptions are relatively simple. For related portions, reference can be made to the descriptions of the method embodiments.

[0348] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A gesture recognition method, characterized in that: The method comprises: Acquire a video to be identified, traverse each video frame in the video to be identified, and when traversing each video frame in the video to be identified, determine the video frame as an image to be identified; Performing human body recognition on the image to be recognized to obtain a human body image of the object to be recognized, and detecting human body key points in the human body image; Using the wrist key points among the human body key points, determine the hand image of the object to be identified from the image to be identified, and detect the hand key points in the image to be identified; For each video frame, determining an initial recognition result corresponding to the video frame based on a positional relationship between the hand key points and designated key points about the arm among the human body key points; For each video frame, cache the initial recognition result corresponding to the video frame, and determine whether the initial recognition result corresponding to the video frame is associated with a specified recognition result; wherein the specified recognition result is: each initial recognition result cached before the initial recognition result corresponding to the video frame; If not, the designated recognition result is used to determine the gesture recognition result of the object to be recognized, and a designated deletion process is performed on the designated recognition result.

2. The method according to claim 1, characterized in that The determining whether the initial recognition result corresponding to the video frame is associated with the specified recognition result includes: Determine whether the sum of the number of the initial recognition result and the specified recognition result corresponding to the video frame is not greater than a preset value; or, Determining whether the initial recognition result corresponding to the video frame is associated with the last initial recognition result in the specified recognition results; If not, the initial recognition result corresponding to the video frame is not associated with the specified recognition result.

3. The method according to claim 1, characterized in that The performing designated deletion processing on the designated recognition result includes: Clear the specified recognition result; or, Delete the first cached initial recognition result among the specified recognition results.

4. The method according to claim 1, wherein Before determining the hand image of the object to be identified from the image to be identified by using the wrist key point among the human body key points, the method further includes: Obtaining the detection confidence of the key points of the human body; If the detection confidence is greater than a preset confidence threshold, the wrist key point among the human body key points is used to determine a hand image of the object to be identified from the image to be identified.

5. The method according to claim 1, wherein The method of determining a hand image of the object to be identified from the image to be identified by using the wrist key point among the human body key points includes: In the image to be identified, taking the wrist key point among the key points of the human body as a starting point and determining an image area of ​​a preset size along a preset direction as a hand image of the object to be identified; or, In the image to be identified, with the wrist key point as the starting point, an image area of ​​a specified size is determined along the direction from the elbow key point among the human body key points to the wrist key point, as the hand image of the object to be identified; wherein the specified size is determined based on the length between the elbow key point and the wrist key point.

6. The method according to claim 1, wherein The method further comprises: Output the gesture action recognition result.

7. A gesture recognition system, characterized in that: The system comprises: An image acquisition module is used to acquire a video to be identified, traverse each video frame in the video to be identified, and when traversing each video frame in the video to be identified, determine the video frame as an image to be identified; A processor, configured to perform human body recognition on the image to be recognized, obtain a human body image of the object to be recognized, and detect key points of the human body in the human body image; Using the wrist key points among the human body key points, determine the hand image of the object to be identified from the image to be identified, and detect the hand key points in the image to be identified; For each video frame, determining an initial recognition result corresponding to the video frame based on a positional relationship between the hand key points and designated key points about the arm among the human body key points; For each video frame, cache the initial recognition result corresponding to the video frame, and determine whether the initial recognition result corresponding to the video frame is associated with a specified recognition result; wherein the specified recognition result is: each initial recognition result cached before the initial recognition result corresponding to the video frame; If not, the designated recognition result is used to determine the gesture recognition result of the object to be recognized, and a designated deletion process is performed on the designated recognition result.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 6 when executing a program stored in a memory.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images