Gesture recognition method and system

Abstract

The invention discloses a gesture recognition method and system. The method comprises the following steps: obtaining a human body image corresponding to a video frame and position information of a plurality of set human body key points; obtaining a first-level gesture classification result based on position information of a plurality of set human body key points corresponding to the video frame; obtaining an upper body image corresponding to the video frame and human hand detection information corresponding to the video frame; obtaining a second-level gesture classification result based on the human hand detection information; obtaining a hand image corresponding to the video frame and position information of a plurality of set hand key points corresponding to the video frame; and obtaining a third-level gesture classification result based on the first-level gesture classification result, the second-level gesture classification result and the position information of the plurality of set hand key points, and taking the third-level gesture classification result as a gesture category. By utilizing the scheme of the invention, the problem of gesture recognition at a relatively long distance can be solved, more gestures can be distinguished, and the recognition efficiency and the recognition accuracy of gesture recognition are improved.

Description

Technical Field

[0001] The present application generally relates to the field of computer vision technology. More specifically, the present application relates to a gesture recognition method and system. Background Art

[0002] In the current field of gesture recognition, traditional methods are mainly divided into two categories: one is the gesture recognition method based on wearable device bio-electromagnetic information, and the other is the gesture recognition method based on camera visual information. The method based on electromyographic information uses wearable devices to collect the electromyographic information of the human body when making gestures, and then uses neural networks to train and classify these information features. The gesture recognition method based on camera visual information can be further divided into methods based on traditional image processing and methods based on deep learning. Traditional image processing methods mainly determine the action area through the difference between video frames, and then extract motion features for classification through machine learning and other methods. Deep learning methods mainly focus on hand key point detection, and perform gesture recognition through key point information combined with spatial features and temporal features.

[0003] Gesture recognition methods based on bio-electromechanical information of wearable devices are not convenient enough because they rely on wearable devices, and gesture recognition is easily disturbed, resulting in unsmooth human-computer interaction. Methods based on traditional image processing mainly rely on traditional feature extraction and machine learning classification, and their accuracy and speed are far behind deep learning methods. Existing deep learning methods, such as 3D neural networks, graph neural networks, and recurrent neural networks, can extract temporal features, but due to their large number of parameters and slow speed, they are difficult to meet the commercial needs of daily gesture recognition. 2D neural networks are mainly used for target detection and hand key point detection. The former has low accuracy, and the latter is prone to missed detection or false detection at long distances or in complex backgrounds.

[0004] In view of this, there is an urgent need to provide a gesture recognition solution to significantly improve the accuracy and speed of gesture recognition. Summary of the invention

[0005] In order to at least solve one or more of the technical problems mentioned above, the present application proposes a gesture recognition solution in multiple aspects.

[0006] In a first aspect, the present application provides a gesture recognition method, comprising: step S1: processing a corresponding video frame in a video stream including a target to be detected to obtain a human body image corresponding to the video frame and position information of multiple set human body key points corresponding to the video frame; step S2: performing a first-level gesture classification based on the position information of multiple set human body key points corresponding to the video frame to obtain a first-level gesture classification result; step S3: processing the human body image corresponding to the video frame to obtain an upper body human body image corresponding to the video frame and human hand detection information corresponding to the video frame; step S4: performing a second-level gesture classification based on the human hand detection information to obtain a second-level gesture classification result; step S5: processing the upper body human body image corresponding to the video frame to obtain a hand image corresponding to the video frame and position information of multiple set human hand key points corresponding to the video frame; step S6: performing a third-level gesture classification based on the first-level gesture classification result, the second-level gesture classification result and the position information of multiple set human hand key points to obtain a third-level gesture classification result, and using the third-level gesture classification result as a gesture category.

[0007] In some embodiments, the video frame satisfies multiple preset conditions, and the multiple preset conditions include: a first preset condition: there is a human body image in the video frame; a second preset condition: key point detection of the human body image corresponding to the video frame can obtain position information of multiple set human body key points, wherein, during the execution of step S1, it is determined whether the video frame satisfies the first preset condition and the second preset condition; a third preset condition: a first-level gesture classification result obtained after performing a first-level gesture classification on the position information of multiple set human body key points corresponding to the video frame is included in the first preset gesture category set, wherein, during the execution of step S2, it is determined whether the video frame satisfies The third preset condition; the fourth preset condition: there is a hand image in the upper body human image corresponding to the video frame, wherein, during the execution of step S3, it is determined whether the video frame satisfies the fourth preset condition; the fifth preset condition: after performing the second-level gesture classification on the human hand detection information corresponding to the video frame, the second-level gesture classification result is included in the second preset gesture category set, wherein, during the execution of step S4, it is determined whether the video frame satisfies the fifth preset condition; the sixth preset condition: processing the hand image corresponding to the video frame can obtain the position information of multiple set human hand key points, wherein, during the execution of step S5, it is determined whether the video frame satisfies the sixth preset condition.

[0008] In some embodiments, in response to a video frame not satisfying its corresponding preset condition during the execution of a corresponding step, the processing of the current frame in the video stream is terminated, and the next frame in the video stream is processed starting from step S1.

[0009] In some embodiments, step S1 includes: performing human body detection processing on the corresponding video frames in the video stream including the target to be detected to obtain human body position information; cropping the video frames based on the human body position information to obtain a human body image corresponding to the video frame; performing human body key point detection processing on the human body image corresponding to the video frame to obtain the position information of multiple set human body key points corresponding to the video frame.

[0010] In some embodiments, the multiple set human body key points include: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle.

[0011] In some embodiments, the step S2 includes: filtering out the position information of the human body key points belonging to the upper body from the position information of multiple set human body key points; and obtaining the first level gesture classification result by performing the first level gesture classification on the position information of the human body key points belonging to the upper body.

[0012] In some embodiments, step S3 includes: cropping the human body image corresponding to the video frame based on the position information of multiple set human body key points to obtain the upper body human body image corresponding to the video frame; performing hand detection on the upper body human body image corresponding to the video frame to obtain hand detection information corresponding to the video frame.

[0013] In some embodiments, the human hand detection information includes: the position of the human hand, the direction of the human hand, and the left and right hand judgment results.

[0014] In some embodiments, step S5 includes: cropping the upper body image corresponding to the video frame based on the hand detection information to obtain the hand image corresponding to the video frame; performing hand key point detection on the hand image corresponding to the video frame to obtain the position information of multiple set hand key points corresponding to the video frame.

[0015] In a second aspect, the present application provides a gesture recognition system, which uses the gesture recognition method as described in any embodiment of the first aspect to perform gesture recognition, and the system includes: a human body information acquisition module, which is used to process the corresponding video frame in the video stream including the target to be detected to obtain the human body image corresponding to the video frame and the position information of multiple set human body key points corresponding to the video frame; a first-level gesture classification module, which is used to perform a first-level gesture classification based on the position information of multiple set human body key points corresponding to the video frame to obtain a first-level gesture classification result; a human hand detection module, which is used to process the human body image corresponding to the video frame to obtain the upper hand information corresponding to the video frame. A half-body human image and the hand detection information corresponding to the video frame; a second-level gesture classification module, used to perform second-level gesture classification based on the hand detection information to obtain a second-level gesture classification result; a hand key point detection module, used to process the upper-body human image corresponding to the video frame to obtain a hand image corresponding to the video frame and the position information of multiple set hand key points corresponding to the video frame; a third-level gesture classification module, used to perform third-level gesture classification based on the first-level gesture classification result, the second-level gesture classification result and the position information of multiple set hand key points to obtain a third-level gesture classification result, and use the third-level gesture classification result as the gesture category.

[0016] Through the gesture recognition scheme provided above, the embodiment of the present application can solve the problem of gesture recognition at a long distance by obtaining the human body image corresponding to the video frame and the position information of multiple set human body key points corresponding to the video frame. Obtaining human hand detection information is helpful to distinguish more gestures. By sequentially obtaining the first-level gesture classification results and the second-level gesture classification results, a third-level gesture classification is performed based on the first-level gesture classification results, the second-level gesture classification results and the position information of multiple set human hand key points to obtain the third-level gesture classification results. The gestures can be roughly classified first and then finely classified, and the scope can be gradually narrowed until the most matching gesture category is found, thereby improving the recognition efficiency and recognition accuracy of gesture recognition.

[0017] Furthermore, in some embodiments, by judging whether the video frame satisfies the preset conditions corresponding to each step when executing each step, the corresponding step will be executed only when the video frame satisfies the preset conditions corresponding to the corresponding step. This can accurately screen out frames containing specific content, thereby improving the accuracy of subsequent gesture classification. At the same time, it helps to eliminate irrelevant background information or other interference factors, thereby reducing the probability of misrecognition.

[0018] Furthermore, in some embodiments, when a video frame does not meet its corresponding preset conditions during the execution of corresponding steps, the processing of the current frame in the video stream is terminated, and the next frame in the video stream is processed from the beginning, which can skip the frames that do not meet the conditions, avoid unnecessary waste of computing resources, improve the recognition efficiency of gesture recognition, and ensure the accuracy and timeliness of the gesture category recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein: Figure 1 An exemplary flow chart of a gesture recognition method according to an embodiment of the present application is shown; Figure 2 An exemplary flow chart of obtaining a human body image corresponding to a video frame and position information of a plurality of set human body key points according to an embodiment of the present application is shown; Figure 3 An exemplary structural block diagram of a gesture recognition system according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0021] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present application indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0022] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this application specification and claims, unless the context clearly indicates otherwise, the singular forms of "a", "an" and "the" are intended to include plural forms. It should also be further understood that the term "and / or" used in this application specification and claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0023] As used in this specification and claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0024] The specific implementation of the present application is described in detail below with reference to the accompanying drawings.

[0025] Figure 1 An exemplary flow chart of a gesture recognition method according to an embodiment of the present application is shown.

[0026] like Figure 1 As shown, in step S1, a corresponding video frame in a video stream including a target to be detected is processed to obtain a human body image corresponding to the video frame and position information of a plurality of set human body key points corresponding to the video frame.

[0027] In the embodiment of the present application, the specific process involved in step S1 can be found in Figure 2 .

[0028] Figure 2 An exemplary flow chart of obtaining a human body image corresponding to a video frame and position information of a plurality of set human body key points according to an embodiment of the present application is shown.

[0029] like Figure 2 As shown, in step S11, a human body detection process is performed on a corresponding video frame in a video stream including a target to be detected to obtain human body position information. In step S12, a cropping process is performed on the video frame based on the human body position information to obtain a human body image corresponding to the video frame. In step S13, a human body key point detection process is performed on the human body image corresponding to the video frame to obtain position information of a plurality of set human body key points corresponding to the video frame.

[0030] In an embodiment of the present application, a trained human body detection model is used to perform human body detection processing on corresponding video frames in a video stream including a target to be detected.

[0031] In some embodiments of the present application, the PicoDet-S model is used as a human detection model, which can obtain the specific position of the human target in the video frame, and the PicoDet-S model as a lightweight model has the advantages of fast speed and accuracy, and is suitable for deployment and landing applications. In other embodiments of the present application, the human detection model can also use other lightweight human detection models other than the PicoDet-S model, such as the rtm-det human detection model, which is not limited in this application.

[0032] In an embodiment of the present application, in the process of cropping the video frame based on the human body position information, various existing technologies can be used, and the present application is not limited here. For example, first, a cropping area is determined based on the acquired human body position information. The cropping area can be a rectangular frame centered on the human body. Then, using image processing technology, the determined cropping area is cropped to obtain a video frame portion containing only the human body. Then, the cropped video frame is further processed, such as adjusting the resolution, enhancing the image quality, etc., to meet the needs of subsequent processing steps.

[0033] In an embodiment of the present application, in the process of performing human key point detection on the human image corresponding to the video frame, a trained human key point detection model is used to perform human key point detection on the human image corresponding to the video frame.

[0034] In some embodiments of the present application, the PP-TinyPose lightweight model is used as the human key point detection model to perform human key point detection processing on the human image corresponding to the video frame. In other embodiments of the present application, the human key point detection model may also use other lightweight models other than the PP-TinyPose lightweight model, for example, the rtm-pose human key point detection model, which is not limited in the present application.

[0035] In an embodiment of the present application, by using a trained human key point detection model to perform human key point detection processing on a human image corresponding to the video frame, position information of a plurality of set human key points can be obtained. Specifically, the plurality of set human key points include: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle.

[0036] By setting multiple key points of the human body to be detected, the overall posture and local details of the human body can be accurately identified. These key points can reflect the spatial position relationship of the human body and provide important position information for subsequent gesture classification tasks.

[0037] In an embodiment of the present application, during the execution of step S11, it is necessary to determine whether the current video frame meets the first preset condition. The first preset condition is that there is a human body image in the video frame. In response to the current video frame not meeting the first preset condition, the processing of the current frame in the video stream is terminated, and step S11 is re-executed for the next frame in the video stream until a certain frame in the video stream meets the first preset condition. That is, step S11 can only be executed if the aforementioned video frame meets the first preset condition.

[0038] In an embodiment of the present application, when executing step S13, it is necessary to determine whether the current video frame meets the second preset condition. Among them, the second preset condition is: key point detection of the human body image corresponding to the video frame can obtain the position information of multiple set human body key points. In response to the current video frame not meeting the second preset condition, the processing process of the current frame in the video stream is terminated, and the next frame in the video stream is processed from step S11 until a frame in the video stream meets both the first preset condition and the second preset condition. That is, step S13 can only be executed if the aforementioned video frame meets both the first preset condition and the second preset condition.

[0039] Since it is necessary to determine whether the current video frame meets the first preset condition during the execution of step S11, it is necessary to determine whether the current video frame meets the second preset condition during the execution of step S13. Therefore, it is determined whether the video frame meets the first preset condition and the second preset condition during the execution of step S1.

[0040] After executing step S1, in step S2, a first-level gesture classification is performed based on the position information of a plurality of set human body key points corresponding to the video frame to obtain a first-level gesture classification result.

[0041] In the embodiment of the present application, during the execution of step S2, first, the position information of the human body key points belonging to the upper body is screened out from the position information of the plurality of set human body key points. Then, the first-level gesture classification result is obtained by performing the first-level gesture classification on the position information of the human body key points belonging to the upper body.

[0042] Specifically, the key points of the upper body include shoulders, elbows, wrists and other key points.

[0043] In the embodiment of the present application, by performing the first-level gesture classification on the position information of the key points of the upper body, the key points of the upper body can be focused on to reduce unnecessary calculations. At the same time, since the movements of the upper body, especially the arms, elbows, wrists, and fingers, are the main components of gestures, focusing on the key points of the upper body helps to capture and understand gestures more accurately.

[0044] In an embodiment of the present application, a first-level classifier that has been trained is used to perform first-level gesture classification based on the position information of the key points of the upper body. Specifically, the first-level classifier is trained using the historically set position information of the key points of the human body.

[0045] In an embodiment of the present application, the first-level gesture classification results may include coarse-grained gesture classification results such as “single arm extended horizontally”, “two arms raised”, and “elbow bent”.

[0046] In an embodiment of the present application, during the execution of step S2, it is necessary to determine whether the current video frame satisfies the third preset condition. The third preset condition is: the first-level gesture classification result obtained after the first-level gesture classification of the position information of multiple set human body key points corresponding to the video frame is included in the first preset gesture category set. In response to the current video frame not satisfying the third preset condition, the processing of the current frame in the video stream is terminated, and the next frame in the video stream is processed from step S1 until a frame in the video stream satisfies the first preset condition, the second preset condition and the third preset condition at the same time. That is, step S2 can only be executed when the aforementioned video frame satisfies the first preset condition, the second preset condition and the third preset condition at the same time.

[0047] In the embodiment of the present application, the first preset gesture category set is set according to actual needs and historical experience, and the present application does not limit it here.

[0048] In some embodiments of the present application, the first preset gesture category set does not include invalid gestures. When the first-level gesture classification result is an invalid gesture such as arm drooping, it is determined that the current video frame does not meet the third preset condition.

[0049] After executing step S2, in step S3, the human body image corresponding to the video frame is processed to obtain the upper body human body image corresponding to the video frame and the human hand detection information corresponding to the video frame.

[0050] In an embodiment of the present application, during the execution of step S3, first, the human body image corresponding to the video frame is cropped based on the position information of multiple set human body key points to obtain the upper body human body image corresponding to the video frame. Then, the upper body human body image corresponding to the video frame is subjected to human hand detection processing to obtain human hand detection information corresponding to the video frame.

[0051] In an embodiment of the present application, in the process of cropping the human body image corresponding to the video frame based on the position information of multiple set human body key points, the position information of the human body key points belonging to the upper body is screened out from the position information of the multiple set human body key points, and the human body image corresponding to the video frame is cropped based on the position information of the human body key points belonging to the upper body. In the process of cropping the human body image corresponding to the video frame based on the position information of the human body key points belonging to the upper body, various existing technologies can be used, and the present application is not limited here. For example, first, according to the position information of the human body key points belonging to the upper body, a cropping area is determined. The cropping area can be a rectangular frame centered on the upper body of the human body. Then, using image processing technology, the determined cropping area is cropped, so as to obtain a video frame portion containing only the upper body of the human body. Then, the cropped video frame is further processed, such as adjusting the resolution, enhancing the image quality, etc., to meet the needs of subsequent processing steps.

[0052] In an embodiment of the present application, a hand detection model is used to perform hand detection processing on the upper body image corresponding to the video frame to obtain hand detection information including the position of the hand, the direction of the hand, and the left and right hand judgment results.

[0053] By obtaining hand detection information including the position of the hand, the direction of the hand, and the left-right hand judgment results, it can assist in distinguishing similar gestures (such as "like" and "down like") and reduce the probability of misjudgment. At the same time, by obtaining hand detection information including the position of the hand, the direction of the hand, and the left-right hand judgment results, it can achieve accurate recognition of complex gestures, which is conducive to distinguishing more gestures. ‌ In some embodiments of the present application, the palm_detection model of the mediapipe framework can be selected as the hand detection model. The palm_detection model can detect the position of the hand, the direction of the hand, and the left and right hand judgment results. As a lightweight model, the palm_detection model has the advantages of being fast and accurate, and suitable for deployment and application. In other embodiments of the present application, the hand detection model can also use other lightweight hand detection models other than the PicoDet-S model, such as picdet, rtmdet and other hand detection models, which are not limited in this application.

[0054] In an embodiment of the present application, during the execution of step S3, it is necessary to determine whether the current video frame satisfies the fourth preset condition. The fourth preset condition is: there is a hand image in the upper body human image corresponding to the video frame. In response to the current video frame not satisfying the fourth preset condition, the processing of the current frame in the video stream is terminated, and the next frame in the video stream is processed from step S1 until a frame in the video stream satisfies the first preset condition, the second preset condition, the third preset condition and the fourth preset condition at the same time. That is, step S3 can only be executed when the aforementioned video frame satisfies the first preset condition, the second preset condition, the third preset condition and the fourth preset condition at the same time.

[0055] After executing step S3, in step S4, second-level gesture classification is performed based on the human hand detection information to obtain a second-level gesture classification result.

[0056] In some embodiments of the present application, a trained second-layer classifier is used to perform second-level gesture classification based on human hand detection information. Specifically, the second-layer classifier is trained using historical human hand detection information.

[0057] In an embodiment of the present application, the second-level gesture classification results may include coarse-grained gesture classification results such as “right hand vertical palm facing inward” and “left hand horizontal palm facing downward”.

[0058] In an embodiment of the present application, during the execution of step S4, it is necessary to determine whether the current video frame meets the fifth preset condition. Among them, the fifth preset condition is: the second-level gesture classification result obtained after the second-level gesture classification of the hand detection information corresponding to the video frame is included in the second preset gesture category set. In response to the current video frame not meeting the fifth preset condition, the processing process of the current frame in the video stream is terminated, and the next frame in the video stream is processed from step S1 until a frame in the video stream meets the first preset condition, the second preset condition, the third preset condition, the fourth preset condition and the fifth preset condition at the same time. That is, step S4 can only be executed when the aforementioned video frame meets the first preset condition, the second preset condition, the third preset condition, the fourth preset condition and the fifth preset condition at the same time.

[0059] In the embodiment of the present application, the second preset gesture category set is set according to actual needs and historical experience, and the present application does not limit it here.

[0060] After executing step S4, in step S5, the upper body human body image corresponding to the video frame is processed to obtain the hand image corresponding to the video frame and the position information of multiple set human hand key points corresponding to the video frame.

[0061] In the embodiment of the present application, during the execution of step S5, first, the upper body image corresponding to the video frame is cropped based on the hand detection information to obtain the hand image corresponding to the video frame. Then, the hand key point detection process is performed on the hand image corresponding to the video frame to obtain the position information of multiple set hand key points corresponding to the video frame.

[0062] In an embodiment of the present application, in the process of cropping the upper body human body image corresponding to the video frame based on the hand detection information, various existing technologies can be used, and the present application is not limited here. For example, first, determine the cropping area based on the hand detection information. The cropping area can be a rectangular frame centered on the hand. Then, use image processing technology to crop the determined cropping area to obtain a video frame portion containing only the hand. Then, the cropped video frame is further processed, such as adjusting the resolution, enhancing the image quality, etc., to meet the needs of subsequent processing steps.

[0063] In an embodiment of the present application, a hand key point detection model is used to perform hand key point detection processing on the hand image corresponding to the video frame to obtain position information of the hand key points set including the palm, fingers, joints, etc.

[0064] By setting the key points of the hand that need to be detected, richer spatiotemporal information can be provided, which helps to more accurately describe the details of the gesture, thereby facilitating the subsequent accurate classification of the gesture.

[0065] In some embodiments of the present application, the hand_landmark model of the mediapipe framework is selected as the hand key point detection model. As a lightweight model, the hand_landmark model has the advantages of being fast and accurate, and suitable for deployment and application. In other embodiments of the present application, the hand key point detection model may also be other lightweight hand key point detection models other than the hand_landmark model, and the present application does not limit this.

[0066] In an embodiment of the present application, during the execution of step S5, it is necessary to determine whether the current video frame meets the sixth preset condition. Among them, the sixth preset condition is: processing the hand image corresponding to the video frame can obtain the position information of multiple set human hand key points. In response to the current video frame not meeting the sixth preset condition, the processing process of the current frame in the video stream is terminated, and the next frame in the video stream is processed from step S1 until a frame in the video stream meets the first preset condition, the second preset condition, the third preset condition, the fourth preset condition, the fifth preset condition and the sixth preset condition at the same time. That is, step S5 can only be executed when the aforementioned video frame meets the first preset condition, the second preset condition, the third preset condition, the fourth preset condition, the fifth preset condition and the sixth preset condition at the same time.

[0067] After executing step S5, in step S6, a third-level gesture classification is performed based on the first-level gesture classification results, the second-level gesture classification results and the position information of multiple set human hand key points to obtain the third-level gesture classification results, and the third-level gesture classification results are used as the gesture category.

[0068] In an embodiment of the present application, a trained third-level classifier is used to perform third-level gesture classification based on the first-level gesture classification results, the second-level gesture classification results, and the position information of multiple set human hand key points. Specifically, the third-level classifier is trained using historical first-level gesture classification result data, historical second-level gesture classification result data, and historical set human hand key point position information.

[0069] In the embodiment of the present application, the third-level gesture classification results include basic gestures, digital gestures, symbol gestures, etc. Specifically, basic gestures include fist, open hand, scissors hand (V-shaped), OK gesture, etc. Digital gestures include numbers 1-10. Symbol gestures include thumbs up, heart, pause gesture, etc.

[0070] In one embodiment of the present application, when the first-level gesture classification result is "right arm raised, elbow bent", the second-level gesture classification result is "right hand vertical, palm facing inward", and by analyzing the position information of multiple set key points of the human hand, it is found that the index finger is straight and the other fingers are bent, the third-level gesture classification result is "number 1".

[0071] In summary, through the gesture recognition scheme as described above, through the gesture recognition scheme provided above, the embodiment of the present application can solve the problem of gesture recognition at a long distance by obtaining the human body image corresponding to the video frame and the position information of multiple set human body key points corresponding to the video frame. Obtaining human hand detection information is conducive to distinguishing more gestures. By successively obtaining the first-level gesture classification results and the second-level gesture classification results, a third-level gesture classification is performed based on the first-level gesture classification results, the second-level gesture classification results and the position information of multiple set human hand key points to obtain the third-level gesture classification results. The gestures can be roughly classified first and then finely classified, and the scope can be gradually narrowed until the most matching gesture category is found, thereby improving the recognition efficiency and recognition accuracy of gesture recognition.

[0072] Furthermore, in some embodiments, by judging whether the video frame satisfies the preset conditions corresponding to each step when executing each step, the corresponding step will be executed only when the video frame satisfies the preset conditions corresponding to the corresponding step. This can accurately screen out frames containing specific content, thereby improving the accuracy of subsequent gesture classification. At the same time, it helps to eliminate irrelevant background information or other interference factors, thereby reducing the probability of misrecognition.

[0073] Furthermore, in some embodiments, when a video frame does not meet its corresponding preset conditions during the execution of corresponding steps, the processing of the current frame in the video stream is terminated, and the next frame in the video stream is processed from the beginning, which can skip the frames that do not meet the conditions, avoid unnecessary waste of computing resources, improve the recognition efficiency of gesture recognition, and ensure the accuracy and timeliness of the gesture category recognition results.

[0074] The embodiment of the present application further provides a gesture recognition system, which may use the aforementioned gesture recognition method to perform gesture recognition, or may use other methods to perform gesture recognition, which is not limited in the present application.

[0075] Figure 3 FIG. 4 is a block diagram of an exemplary structure of a gesture recognition system according to an embodiment of the present application.

[0076] like Figure 3 As shown, the gesture recognition system includes a human body information acquisition module 31, a first-level gesture classification module 32, a human hand detection module 33, a second-level gesture classification module 34, a human hand key point detection module 35, and a third-level gesture classification module 36. In the embodiment of the present application, the human body information acquisition module 31, the first-level gesture classification module 32, the human hand detection module 33, the second-level gesture classification module 34, the human hand key point detection module 35, and the third-level gesture classification module 36 can be separate units or integrated in the same integrated circuit, and the present application does not limit this.

[0077] Specifically, the human body information acquisition module 31 is used to process the corresponding video frame in the video stream including the target to be detected to obtain the human body image corresponding to the video frame and the position information of multiple set human body key points corresponding to the video frame.

[0078] Specifically, the first-level gesture classification module 32 is used to perform first-level gesture classification based on the position information of multiple set human body key points corresponding to the video frame to obtain a first-level gesture classification result.

[0079] Specifically, the human hand detection module 33 is used to process the human body image corresponding to the video frame to obtain the upper body human body image corresponding to the video frame and the human hand detection information corresponding to the video frame.

[0080] Specifically, the second-level gesture classification module 34 is used to perform second-level gesture classification based on the human hand detection information to obtain a second-level gesture classification result.

[0081] Specifically, the hand key point detection module 35 is used to process the upper body human body image corresponding to the video frame to obtain the hand image corresponding to the video frame and the position information of multiple set hand key points corresponding to the video frame.

[0082] Specifically, the third-level gesture classification module 36 is used to perform third-level gesture classification based on the first-level gesture classification results, the second-level gesture classification results and the position information of the key points of the human hand, obtain the third-level gesture classification results, and use the third-level gesture classification results as the gesture category.

[0083] When the gesture recognition system adopts the aforementioned gesture recognition method to perform gesture recognition, the human body information acquisition module 31 performs the aforementioned step S1, the first-level gesture classification module 32 performs the aforementioned step S2, the human hand detection module 33 performs the aforementioned step S3, the second-level gesture classification module 34 performs the aforementioned step S4, the human hand key point detection module 35 performs the aforementioned step S5, and the third-level gesture classification module 36 performs the aforementioned step S6. The specific execution process can be referred to the above, and will not be repeated here.

[0084] Although multiple embodiments of the present application have been shown and described herein, it is obvious to those skilled in the art that such embodiments are provided only by way of example. Those skilled in the art can think of many changes, modifications and alternatives without departing from the thought and spirit of the present application. It should be understood that in the process of practicing the present application, various alternatives to the embodiments of the present application described herein can be adopted. The attached claims are intended to limit the scope of protection of the present application, and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A gesture recognition method, characterized in that: include: Step S1: Processing a corresponding video frame in a video stream including a target to be detected to obtain a human body image corresponding to the video frame and position information of a plurality of set human body key points corresponding to the video frame; Step S2: performing first-level gesture classification based on position information of multiple set human body key points corresponding to the video frame to obtain a first-level gesture classification result; Step S3: Processing the human body image corresponding to the video frame to obtain the upper body human body image corresponding to the video frame and the human hand detection information corresponding to the video frame; Step S4: performing second-level gesture classification based on the hand detection information to obtain a second-level gesture classification result; Step S5: processing the upper body image corresponding to the video frame to obtain the hand image corresponding to the video frame and position information of multiple set human hand key points corresponding to the video frame; Step S6: Perform a third-level gesture classification based on the first-level gesture classification result, the second-level gesture classification result and the position information of multiple set human hand key points to obtain a third-level gesture classification result, and use the third-level gesture classification result as the gesture category.

2. The gesture recognition method according to claim 1, characterized in that: The video frame satisfies a plurality of preset conditions, wherein the plurality of preset conditions include: The first preset condition: there is a human body image in the video frame; Second preset condition: performing key point detection on a human body image corresponding to a video frame can obtain position information of a plurality of set human body key points, wherein, during the execution of step S1, it is determined whether the video frame satisfies the first preset condition and the second preset condition; Third preset condition: the first-level gesture classification result obtained after performing the first-level gesture classification on the position information of the plurality of set human body key points corresponding to the video frame is included in the first preset gesture category set, wherein, during the execution of step S2, it is determined whether the video frame meets the third preset condition; Fourth preset condition: there is a hand image in the upper body human image corresponding to the video frame, wherein, during the execution of step S3, it is determined whether the video frame meets the fourth preset condition; Fifth preset condition: after performing the second-level gesture classification on the hand detection information corresponding to the video frame, the second-level gesture classification result obtained is included in the second preset gesture category set, wherein, during the execution of step S4, it is determined whether the video frame meets the fifth preset condition; Sixth preset condition: processing the hand image corresponding to the video frame can obtain the position information of multiple set human hand key points, wherein, during the execution of step S5, it is determined whether the video frame meets the sixth preset condition.

3. The gesture recognition method according to claim 2, characterized in that: In response to the video frame not satisfying its corresponding preset condition during the execution of the corresponding step, the processing of the current frame in the video stream is terminated, and the next frame in the video stream is processed starting from step S1.

4. The gesture recognition method according to claim 1, characterized in that: The step S1 comprises: Performing human body detection processing on corresponding video frames in a video stream including the target to be detected to obtain human body position information; The video frame is cropped based on the human body position information to obtain a human body image corresponding to the video frame; The human body image corresponding to the video frame is subjected to human body key point detection processing to obtain position information of a plurality of set human body key points corresponding to the video frame.

5. The gesture recognition method according to claim 1 or 4, characterized in that: The multiple set human body key points include: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle.

6. The gesture recognition method according to claim 1, characterized in that: The step S2 comprises: Filtering the position information of the key points of the human body belonging to the upper body from the position information of the plurality of set key points of the human body; By performing first-level gesture classification on the position information of the key points of the upper body, a first-level gesture classification result is obtained.

7. The gesture recognition method according to claim 1, characterized in that: The step S3 comprises: Based on the position information of multiple set human key points, a human body image corresponding to the video frame is cropped to obtain an upper body human body image corresponding to the video frame; Perform hand detection processing on the upper body human body image corresponding to the video frame to obtain hand detection information corresponding to the video frame.

8. The gesture recognition method according to claim 7, characterized in that: The human hand detection information includes: the position of the human hand, the direction of the human hand and the left and right hand judgment results.

9. The gesture recognition method according to claim 1, characterized in that: The step S5 comprises: Based on the hand detection information, the upper body image corresponding to the video frame is cropped to obtain the hand image corresponding to the video frame; The hand image corresponding to the video frame is subjected to a human hand key point detection process to obtain position information of a plurality of set human hand key points corresponding to the video frame.

10. A gesture recognition system, characterized in that: The gesture recognition method according to any one of claims 1 to 9 is used to perform gesture recognition, and the system comprises: A human body information acquisition module, used for processing a corresponding video frame in a video stream including a target to be detected to obtain a human body image corresponding to the video frame and position information of a plurality of set human body key points corresponding to the video frame; A first-level gesture classification module, used to perform first-level gesture classification based on position information of a plurality of set human body key points corresponding to the video frame, and obtain a first-level gesture classification result; A human hand detection module is used to process the human body image corresponding to the video frame to obtain an upper body human body image corresponding to the video frame and human hand detection information corresponding to the video frame; A second-level gesture classification module, used to perform second-level gesture classification based on human hand detection information to obtain a second-level gesture classification result; A human hand key point detection module is used to process the upper body human body image corresponding to the video frame to obtain the hand image corresponding to the video frame and the position information of multiple set human hand key points corresponding to the video frame; The third-level gesture classification module is used to perform third-level gesture classification based on the first-level gesture classification results, the second-level gesture classification results and the position information of multiple set human hand key points, obtain the third-level gesture classification results, and use the third-level gesture classification results as gesture categories.

Images