Dynamic gesture recognition method, device, apparatus and storage medium

Abstract

The application provides a dynamic gesture recognition method, device and equipment and a storage medium. The method comprises the following steps: determining a target video frame to be recognized; obtaining a joint sequence comprising joint angles of a human body in the target video frame and N previous video frames of the target video frame; determining arrangement order information corresponding to the joint angles of each video frame in the joint sequence according to a shooting time sequence of the target video frame and the N previous video frames; N is an integer greater than 0; and performing dynamic gesture recognition by using the joint sequence fused with the arrangement order information to determine a human body posture in the target video frame. This is beneficial to improving the recognition accuracy of dynamic gestures.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a dynamic gesture recognition method and device, equipment and a storage medium. BACKGROUND

[0002] Human gesture recognition has been an important focus in the field of computer vision, and has been widely applied in various fields. In the related technology of recognizing gestures of body parts, a gesture model is used to extract features of human gestures in a single frame of static image, and then a classifier is used for classification. However, the recognition rate of dynamic gestures is low by this method. SUMMARY

[0003] Therefore, the present application provides a dynamic gesture recognition method, device, equipment and storage medium.

[0004] Specifically, the present application is realized by the following technical solutions.

[0005] In a first aspect, the present application provides a dynamic gesture recognition method, comprising:

[0006] determining a target video frame to be recognized;

[0007] obtaining a joint sequence containing joint angles of a human body in the target video frame and N previous video frames of the target video frame; and determining arrangement order information corresponding to the joint angles of each video frame in the joint sequence according to a shooting time sequence of the target video frame and the N previous video frames; N is an integer greater than 0;

[0008] performing dynamic gesture recognition by using the joint sequence fused with the arrangement order information to determine a human gesture in the target video frame.

[0009] In a second aspect, the present application provides a dynamic gesture recognition device, comprising:

[0010] a video frame determination module configured to determine a target video frame to be recognized;

[0011] a joint information acquisition module configured to obtain a joint sequence containing joint angles of a human body in the target video frame and N previous video frames of the target video frame; and determine arrangement order information corresponding to the joint angles of each video frame in the joint sequence according to a shooting time sequence of the target video frame and the N previous video frames; N is an integer greater than 0;

[0012] a dynamic gesture recognition module configured to perform dynamic gesture recognition by using the joint sequence fused with the arrangement order information to determine a human gesture in the target video frame.

[0013] In a third aspect, an electronic device is provided, comprising a memory, a processor, and executable instructions stored on the memory and executable on the processor, wherein the processor executes the executable instructions to implement the steps in the method of the first aspect.

[0014] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, wherein the program is executed by a processor to implement the steps in the method of the first aspect.

[0015] The embodiment provides a dynamic gesture recognition method. After a target video frame to be recognized is determined, a joint sequence can be acquired based on joint angles of a human body in the target video frame and joint angles of the human body in N previous video frames of the target video frame, and arrangement order information corresponding to the joint angles in each video frame in the joint sequence can be determined according to shooting time sequences of the target video frame and the N previous video frames. Finally, dynamic gesture recognition is performed by using the joint sequence fused with the arrangement order information. The embodiment uses multiple video frames to perform dynamic gesture recognition. The joint sequence fused with the arrangement order information not only saves time sequence information of multiple video frames, but also considers correlation between dynamic gestures, so that the continuous change of the dynamic gestures can be accurately perceived in the recognition process, and the recognition accuracy of the dynamic gestures is improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a schematic diagram of a live broadcast architecture according to an example embodiment of the present application.

[0017] Figure 2A is a schematic diagram of an interactive interface according to an example embodiment of the present application.

[0018] Figure 2B is a schematic diagram of a live broadcast video frame with special effects added according to an example embodiment of the present application.

[0019] Figure 3 is a schematic diagram of a dynamic gesture recognition method according to an example embodiment of the present application.

[0020] Figure 4 is a structural schematic diagram of a gesture classifier according to an example embodiment of the present application.

[0021] Figure 5 is a flowchart of dynamic gesture output according to an example embodiment of the present application.

[0022] Figure 6 is a structural schematic diagram of a dynamic gesture recognition device according to an example embodiment of the present application.

[0023] Figure 7FIG. 1 is a structural schematic diagram of an electronic device according to an example embodiment of the present application. DETAILED DESCRIPTION

[0024] The example embodiments will be described in detail herein with reference to the attached drawings. The following description is made with reference to the accompanying drawings in which like reference numerals refer to like elements, unless the context clearly shows otherwise. The following description of example embodiments is not representative of all embodiments consistent with the present application. Rather, it is merely an example of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0025] The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the present application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0026] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used solely to distinguish one from another only. For example, a first information can be termed a second information, and similarly, a second information can also be termed a first information, without departing from the scope of the present application. The word "if' as used herein means "when" or "upon" or "in response to the determination" depending on the context.

[0027] In the related art of recognizing the posture of a body part, a posture model is used to extract features of a human body posture in a single frame of static image, and then a classifier is used to classify it. The inventors have found that, due to the complex and variable nature of each joint of the human body, the multi-degree of freedom and other characteristics, the dynamic posture cannot be perceived based on a single frame of static image, and the recognition rate of dynamic posture is low.

[0028] Based on the problems in the related art, the embodiment of the present application provides a dynamic gesture recognition method. After determining a target video frame to be recognized, the joint sequence can be obtained based on the joint angles of the human body in the target video frame and the joint angles of the human body in the previous N video frames of the target video frame, and the arrangement order information corresponding to the joint angles of each video frame in the joint sequence can be determined according to the shooting time sequence of the target video frame and the previous N video frames. Finally, the dynamic gesture recognition is performed by using the joint sequence fused with the arrangement order information. In the embodiment, the dynamic gesture recognition is performed by using multiple video frames. The joint sequence fused with the arrangement order information not only saves the time sequence information of multiple video frames, but also considers the correlation between dynamic gestures, so that the continuous change of the dynamic gesture can be accurately perceived in the recognition process, and the recognition accuracy of the dynamic gesture is improved.

[0029] In some embodiments, the dynamic gesture recognition method provided by the embodiment of the present application can be executed by an electronic device, which includes but is not limited to a computer, a laptop, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a server, and other devices with data processing capability. For example, the electronic device includes a memory and a processor, and the memory stores executable instructions of the display method, so that the processor can implement the dynamic gesture recognition method when executing the executable instructions. For example, the display method can be packaged into a computer program product and integrated into the electronic device.

[0030] In an exemplary application scenario, the dynamic gesture recognition method can be applied to a live broadcast scenario, for example, special effects can be added to a live broadcast video in real time based on the dynamic gestures of an anchor in a live broadcast process. In order to better understand the application of the dynamic gesture recognition method disclosed in the present application in the live broadcast scenario, the live broadcast network structure is first exemplarily described as follows. Figure 1 Figure 1 is a live broadcast network architecture diagram according to an exemplary embodiment of the present application. The live broadcast network architecture can include a server 10 and a plurality of terminals 20. The server 10 can be referred to as a background server, a component server, etc., and is used to provide background services for network live broadcast. The server 10 can include a server, a server cluster, or a cloud platform, or can be a program for executing services. The terminal 20 can be a smart terminal with a network live broadcast function, for example, the smart terminal can be a smart phone, a tablet computer, a PDA (Personal Digital Assistant), a multimedia player, a wearable device, etc.

[0031] ​In the live network architecture, the terminal 20 can be divided into a host terminal 21 and a viewer terminal 22. The host terminal 21 is installed with a host client, and the viewer terminal 22 is installed with a viewer client. The host client and the viewer client can be the same video live application, that is, the video live application has both a live mode and a viewer mode, for example, “Huya Live”; or the host client and the viewer client can be different video live applications. For the case that the host client and the viewer client are the same video live application, when the video live application enters the host mode, the video live application can be referred to as the host client (hereinafter referred to as “host end”); when the video live application enters the viewer mode, the video live application can be referred to as the viewer client (hereinafter referred to as “viewer end”). The viewer terminal installed with the viewer client can watch the live video uploaded by the host client. The host terminal and the viewer terminal can be connected with the server through a wired network, a wireless network or a data transmission line.

[0032] In the live network architecture, the viewer can log in to the server of the live network architecture through the viewer client on the viewer terminal 21, and the host can log in to the server of the live network architecture through the host client on the host terminal 22, and the viewer and the host enter the same live channel. The host client uploads the online live content to the server, and the server sends the online live content to the viewer client that logs in to the online live channel, for the viewer of the viewer client to watch. The viewer of the viewer client can not only watch the live content uploaded by the host client, but also interact with the host or other viewers of the live channel through the server.

[0033] In order to make the display effect of the live video more rich and further improve the interactivity between the host and the viewer, in the live process, the live user can add special effects to the live video. As shown in FIG. 1, Figure 2A For example, a special effect adding control can be set in the live interface, and the host end can display a special effect adding interface in response to the triggering operation of the host user on the special effect adding control. The special effect adding interface displays a plurality of special effects and dynamic gestures corresponding to each special effect. For example, the dynamic gesture corresponding to the “bottle” special effect is the “hand grabbing” gesture, the dynamic gesture corresponding to the “love” special effect is the “heart” gesture, and the dynamic gesture corresponding to the “dizziness” special effect is the “body rotating” gesture. The host user can specify the special effect to be added according to actual needs, and after determining that there is no error, the host user can trigger the “confirm” control, and then the host end can obtain the special effect generation instruction according to the special effect specified by the host user in the special effect adding interface.

[0034] The anchor end determines the live broadcast video frame acquired in real time as a target video frame to be recognized in response to the special effect generation instruction, executes the dynamic gesture recognition method provided in the embodiment, determines the human body gesture in the target video frame, and adds the special effect specified in the special effect generation instruction in the live broadcast video frame if the human body gesture in the target video frame is the same as the dynamic gesture specified in the special effect generation instruction. For example, refer to Figure 2B After it is determined that the user makes the "heart" dynamic gesture, the "love heart" special effect can be added in the live broadcast video frame. In addition, the anchor user can also trigger the "cancel" control in the case of not wanting to add the special effect, and the anchor end can return to the live broadcast interface.

[0035] Of course, the dynamic gesture recognition method can also be executed by the server. For example, the anchor end sends the generated special effect generation instruction and the live broadcast video frame to the server, the server executes the dynamic gesture recognition method provided in the embodiment in response to the special effect generation instruction, thereby obtaining the video frame to which the special effect is added, and sends the video frame to which the special effect is added to the anchor end and the audience end, so as to display the video frame to which the special effect is added in the anchor end and the audience end respectively. In this way, the anchor end executes the dynamic gesture recognition method provided in the embodiment, which reduces the transmission delay of sending relevant information to the server, and can meet the real-time display requirement of the anchor user for the video frame to which the special effect is added.

[0036] Next, the display method provided in the embodiment is described. Refer to Figure 3 , Figure 3 The flowchart of the dynamic gesture recognition method provided in the embodiment can be executed by an electronic device. The method comprises the following steps.

[0037] In step S101, a target video frame to be recognized is determined.

[0038] In step S102, a joint sequence containing the joint angles of a human body in the target video frame and N previous video frames of the target video frame is acquired, and the arrangement order information corresponding to the joint angles of each video frame in the joint sequence is determined according to the shooting time sequence of the target video frame and the N previous video frames; N is an integer greater than 0.

[0039] In step S103, the joint sequence fused with the arrangement order information is used for dynamic gesture recognition to determine the human body gesture in the target video frame.

[0040] It can be understood that the dynamic gesture mentioned in the embodiment includes a gesture related to at least one part of the human body. Exemplarily, the dynamic gesture includes but is not limited to a hand dynamic gesture (such as a rubbing, grabbing or throwing action), a foot dynamic gesture (such as a jumping, kicking, lifting foot, walking or running action), a head dynamic gesture (such as a shaking or turning head action), a body dynamic gesture (such as a bending, turning or rotating action) or a dynamic gesture made in combination with at least two parts of the body (such as a rotating handstand, a specified dance action or the like).

[0041] For step S101, in the dynamic gesture recognition process, the electronic device first determines a target video frame to be recognized. For example, in a live broadcast scenario, the target video frame can be a live video frame acquired in real time, and the electronic device can perform dynamic gesture recognition on each live video frame acquired in real time at each time. For another example, in a video that has been completed shooting, the electronic device can perform dynamic gesture recognition on each frame in sequence starting from a specified frame (such as the 10th frame) according to a shooting time sequence, and the target video frame is the 10th frame and the video frames after the 10th frame.

[0042] In some embodiments, after determining the target video frame to be recognized, the electronic device can perform human key point detection on the target device to obtain human two-dimensional key point data. The human two-dimensional key point data includes key points corresponding to each joint of the human body and position information (such as two-dimensional coordinate information) of the key points corresponding to each joint of the human body in the target video frame. Exemplarily, the key points corresponding to each joint of the human body include but are not limited to a pelvis, a left hip, a right hip, a spine, a left knee, a right knee, a left ankle, a right ankle, a left foot, a right foot, a neck, a left clavicle, a right clavicle, a head, a left shoulder, a right shoulder, a left elbow, a right elbow, a left wrist, a right wrist, a left hand, a right hand, a left finger joint and a right finger joint, and the like. Of course, the number and positions of the above key points are only illustrative, and other positions can also be included, and the embodiment does not make any limitation in this regard.

[0043] Exemplarily, in the scene of adding special effects according to dynamic postures, key point detection can be performed on specified parts of the human body in the target video frame according to the dynamic posture corresponding to the target special effect selected by the user. For parts of the human body in the target video frame that are irrelevant to the target special effect, no detection is performed, thereby facilitating reduction of detection workload and improvement of detection efficiency. Exemplarily, in the live broadcast scene described above, if the anchor user selects the special effect “bottle”, the dynamic posture corresponding to the “bottle” special effect is the “hand grabbing” posture, and then key point detection can be performed only on the hand in the target video frame to obtain hand two-dimensional key point data, such as the hand two-dimensional key point data including key points of each joint of the fingers and position information (such as two-dimensional coordinates) of the key points in the target video frame. If the anchor user selects the special effect “love”, the dynamic posture corresponding to the “love” special effect is the “waving” posture, and the “waving” posture is generally related to the upper half of the body, and then key point detection can be performed only on the upper half of the human body in the target video frame, thereby reducing the detection workload. If the anchor user selects the special effect “dizziness”, the dynamic posture corresponding to the “dizziness” special effect is the “rotating body” posture, and generally the whole body needs to be involved, and then key point detection needs to be performed on each part of the human body in the target video frame.

[0044] In some embodiments, after obtaining the human two-dimensional key point data, the electronic device inputs the human two-dimensional key point data into a pre-established joint angle prediction model, performs joint angle prediction on the human two-dimensional key point data through the joint angle prediction model, and obtains the joint angle of the human body in the target video frame. The joint angle is the rotation angle of a joint. The human two-dimensional key point data includes information of one or more two-dimensional key points, and the joint angle of the human body in the target video frame predicted by the joint angle prediction model also includes the joint angle corresponding to each two-dimensional key point, that is, the information of one two-dimensional key point can predict one joint angle, such as the left hand two-dimensional key point information can predict the left hand rotation angle.

[0045] It is considered that there is an ambiguity problem in predicting joint angles through human two-dimensional key point data, that is, one two-dimensional key point data can correspond to two sets of three-dimensional joint angles. For example, the front and back of the hand, although the depth is different, the key points projected to the two-dimensional plane can be the same. Therefore, the embodiment of the present application introduces the actually collected data in the process of training the joint angle prediction model to solve the ambiguity problem. The joint angle prediction model is trained based on a plurality of two-dimensional key point sample data carrying angle labels; wherein at least part of the two-dimensional key point sample data can be obtained in the following manner: obtaining three-dimensional key point data and its corresponding joint angles collected by an inertial motion capture system when a user performs a specified dynamic gesture; the three-dimensional key point data includes three-dimensional key points corresponding to each joint of the human body, and position information (such as three-dimensional coordinate information) of the three-dimensional key points corresponding to each joint of the human body in the world coordinate system; then according to the position relationship between the inertial motion capture system and the camera, the three-dimensional key point data is projected to the camera plane to obtain two-dimensional key point data; and then based on the two-dimensional key point data and the joint angles corresponding to the three-dimensional key point data, two-dimensional key point sample data carrying angle labels are obtained. In the embodiment, the two-dimensional key point sample data carrying angle labels is obtained by collecting real three-dimensional key point data and its corresponding joint angles, different three-dimensional key point data can be projected to obtain different two-dimensional key point data, and a one-to-one correspondence between the two-dimensional key point data and the joint angles is realized, and based on such sample data, model training is beneficial to improve the accuracy of model training, and avoids or reduces the ambiguity problem.

[0046] Among them, the inertial motion capture system can be worn on the user, contact each joint of the user, and capture the displacement data and rotation data of the joint when the user performs a certain gesture, and then collect the three-dimensional key point data related to the joint of the user and its corresponding joint angle; it can be understood that the embodiment does not make any limitation on the specific type of the inertial motion capture system and the dynamic gesture performed by the user, and can be specifically selected according to the actual application scene, such as the MVN motion capture system of Xsens; the dynamic gesture includes but is not limited to the gesture performed by at least one part of the body, such as hand dynamic gesture, foot dynamic gesture, head dynamic gesture or dynamic gesture participated by multiple joints of the whole body, etc.

[0047] Exemplarily, the positional relationship between the inertial motion capture system and the camera can be obtained by the following manner: obtaining images of the user taken by the camera in the process that the user wears the inertial motion capture system to perform a specified dynamic pose; and obtaining two-dimensional key point labeling data in the images, the two-dimensional key point labeling data including two-dimensional key points corresponding to each joint of the human body and two-dimensional coordinates of the two-dimensional key points in the images, which can be manually labeled by the user; and then determining the positional relationship between the inertial motion capture system and the camera (i.e., the camera extrinsic parameter) by using three-dimensional key point data collected by the inertial motion capture system, the two-dimensional key point labeling data and the camera intrinsic parameter.

[0048] In the training process of the joint angle prediction model, after obtaining a plurality of two-dimensional key point sample data carrying angle labels, the electronic device can input the two-dimensional key point sample data into a preset model, obtain a predicted joint angle output by the preset model, adjust parameters of the preset model according to a difference between the predicted joint angle and the angle label, and obtain a trained joint angle prediction model; and then in the actual application process, after obtaining two-dimensional key point data of a human body in a target video frame, the trained joint angle prediction model can be used for joint angle prediction to obtain joint angles of the human body in the target video frame. Exemplarily, the model structure of the joint angle prediction model can be a recurrent neural network (RNN), a long short-term memory neural network (LSTM) or a fully connected neural network (MLP), etc., and the present embodiment does not make any limitation thereto.

[0049] In some embodiments, for step S102, after obtaining the joint angles of the human body in the target video frame, a joint sequence can be obtained based on the joint angles of the human body in the target video frame and the joint angles of the human body in the previous N video frames of the target video frame. The joint angles of the human body in the previous N video frames of the target video frame are obtained in the same manner as the target video frame, which will not be described herein. N is an integer greater than 1, and the larger N is, the better the dynamic pose recognition effect is, but the processing delay is also higher, so the value of N can be determined according to the actual application scenario. Exemplarily, in a live streaming scenario, the value of N can be determined according to the running resources of the electronic device and the real-time requirement of the live streaming scenario. The running resources and N are in a positive correlation, the real-time requirement and N are in a negative correlation, the larger the running resources are, the larger the value of N can be, and the higher the real-time requirement is, the smaller the value of N is, so as to balance between improving the recognition accuracy and reducing the processing delay.

[0050] The joint sequence includes joint angles of N+1 video frames, assuming that the joint angle of each video frame is v, a joint sequence [v0,…,v n+1 ]T wherein the subscripts are used to indicate different video frames in the N+1 video frames.

[0051] To improve the recognition accuracy of the dynamic gesture, the electronic device further determines, according to the shooting time sequence of the target video frame and its previous N video frames, arrangement order information corresponding to the joint angles of each video frame in the joint sequence, and then fuses the arrangement order information with the joint angles of each video frame in the joint sequence to obtain the joint sequence with time sequence information. In one example, assuming that the arrangement order information is a sequential number, the sequential number is determined based on the shooting time sequence of the target video frame and its previous 3 video frames, the later the shooting time sequence is, the larger the sequential number is, and vice versa; then it can be determined that the sequential number corresponding to the joint angles of the target video frame in the joint sequence is 4, the sequential number corresponding to the joint angles of the first previous video frame of the target video frame is 3, the sequential number corresponding to the joint angles of the second previous video frame of the target video frame is 2, and the sequential number corresponding to the joint angles of the third previous video frame of the target video frame is 1.

[0052] Further in step S103, the electronic device can perform dynamic gesture recognition on the joint sequence fused with the arrangement order information to determine the human body gesture in the target video frame. The joint sequence fused with the arrangement order information not only saves the time sequence information of the multiple video frames, but also takes into account the correlation between the dynamic gestures, and thus can accurately perceive the continuous change of the dynamic gesture in the recognition process, thereby improving the recognition accuracy of the dynamic gesture.

[0053] In some possible implementation manners, after obtaining the joint sequence and the arrangement order information corresponding to the joint angles of each video frame in the joint sequence, the joint angles of each video frame in the joint sequence and the arrangement order information corresponding thereto can be spliced, and then the spliced sequence can be input into a pre-established gesture classifier to perform dynamic gesture recognition on the spliced sequence by the gesture classifier to obtain the gesture of the human body in the target video frame. This embodiment splices the joint angles and the arrangement order information thereof, can obtain a sequence with time sequence information, and thus improves the recognition accuracy of the dynamic gesture.

[0054] The posture classifier is trained based on a plurality of samples carrying posture labels, and the samples are joint sequences fused with arrangement sequence information. The inventors find that in the case where the arrangement sequence information and the scale (or value range) of the joint angles in the joint sequence are different, more attention will be paid to the data with larger values during the training process of the posture classifier, and the data with smaller values will be ignored, resulting in low training accuracy. In order to improve the training accuracy of the posture classifier, the embodiments of the present application keep the value range of the joint angles and the value range of the arrangement sequence information the same, so that the posture classifier can pay equal attention to the data of the two parts during the training process, and the classification accuracy of the posture classifier is improved.

[0055] Therefore, in the process of performing dynamic posture recognition on the target video frame, when obtaining the arrangement sequence information corresponding to the joint angles of each video frame in the joint sequence, the electronic device can determine the arrangement sequence information corresponding to the joint angles of each video frame in the joint sequence according to the shooting time sequence of the target video frame and its previous N video frames, and the value range of the joint angles; wherein the value range of the arrangement sequence information is the same as the value range of the joint angles, thereby improving the prediction accuracy of the posture classifier.

[0056] For example, assuming that the joint angles are represented in radians, and the value range of the radian is 0-π, the value range of the arrangement sequence information corresponding to the joint angles of each video frame is also 0-π. For example, assuming that the joint angles are represented in degrees, and the value range of the angle is 0-360°, the value range of the arrangement sequence information corresponding to the joint angles of each video frame is also 0-360°. The specific value of the arrangement sequence information can be determined according to the shooting time sequence corresponding to the video frame, such as a positive correlation between the arrangement sequence information and the shooting time sequence. The later the shooting time sequence, the larger the value of the arrangement sequence information, and the earlier the shooting time sequence, the smaller the value of the arrangement sequence information.

[0057] In one example, such as the aforementioned joint sequence [v0,…,v n+1 ] T The value range of the joint angles is 0-π, and the arrangement sequence information corresponding to the joint angles of each video frame is determined as The joint angles and the arrangement sequence information corresponding to each video frame in the joint sequence are spliced, and the joint sequence carrying the arrangement sequence information can be obtained as Moreover, the inventors also find in practice that the same joint sequence is spliced with the arrangement sequence information after being disturbed, and the human posture recognized based on the disturbed and spliced sequence is different from the human posture recognized based on the original sequence, because the disturbed arrangement sequence information cannot reflect the time sequence information of the N+1 video frames.

[0058] In some embodiments, the pose classification is trained based on a plurality of samples carrying pose labels, the samples being joint sequences fused with arrangement order information, the joint sequences being obtained by using M frames of video frames for human joint detection and joint angle prediction, M being an integer greater than 1, the M frames of video frames corresponding to pose labels.

[0059] For example, the training samples of the pose classifier can be obtained in the following manner: a plurality of videos corresponding to pose labels are obtained; for each video frame in the videos, human key point detection is performed on the video frame to obtain two-dimensional human key point data, the two-dimensional human key point data is input into a pre-established joint angle prediction model, joint angle prediction is performed on the two-dimensional human key point data by using the joint angle prediction model to obtain the joint angles of the human body in the video frame; the videos are cut into a plurality of video frame sequences with a length of M frames, the video frames included in adjacent video frame sequences can be completely different, or adjacent video frame sequences include some same video frames, the pose labels corresponding to the videos are the pose labels corresponding to the video frame sequences; the joint sequences corresponding to each video frame sequence are obtained, and the arrangement order information corresponding to the joint angles in the joint sequence is determined according to the shooting time sequence of each video frame in the video frame sequence and the value range of the joint angles; then the joint angles of each video frame in the joint sequence and the arrangement order information thereof are spliced to obtain the joint sequence fused with the arrangement order information, and the pose labels corresponding to each video frame sequence are added to obtain the samples carrying the pose labels.

[0060] For example, at least one video corresponding to a pose label can be obtained in the following manner: when a user wearing an inertial motion capture system performs a certain pose, the user is photographed by a camera to obtain a video corresponding to a pose label.

[0061] After obtaining a plurality of samples carrying pose labels, the electronic device can input the samples into a preset model to obtain a predicted pose output by the preset model, adjust the parameters of the preset model according to the difference between the predicted pose and the pose label, and obtain a trained pose classifier; then in the actual application process, after obtaining the joint sequence fused with the arrangement order information, the trained pose classifier can be used for dynamic pose recognition to obtain the pose of the human body in the target video frame (i.e., the dynamic pose recognition result output by the pose classifier). For example, the model structure of the pose classifier can be a recurrent neural network (RNN) or a long short-term memory neural network (LSTM), and the present embodiment does not make any limitation thereto.

[0062] In some example embodiments, in order to meet the real-time requirement in some scenarios (such as live streaming scenarios), the present embodiment provides a lightweight pose classifier. As shown in Figure 4 The pose classifier includes a first neural network layer 11, a second neural network layer 12, and a pose prediction network 13. The first neural network layer 11 is configured to extract low-dimensional features from the joint sequence with arrangement order information. The low-dimensional features are data that can be directly extracted from the joint sequence with arrangement order information, such as joint angles carrying arrangement order information. The second neural network layer 12 is configured to extract high-dimensional features from the low-dimensional features. The high-dimensional features are data that need to be further processed, such as analyzing the differences, correlations, or motion trends between joint angles carrying arrangement order information. The pose parameter prediction network 13 is configured to identify dynamic poses according to the high-dimensional features and obtain dynamic pose recognition results. The pose classifier has a simple structure, less computational complexity than using a recurrent neural network (RNN), and low processing delay, thereby meeting the real-time requirement in some scenarios such as live streaming scenarios.

[0063] For example, the first neural network layer 11 and the second neural network layer 12 can be fully connected layers or convolutional layers. The shape parameter prediction network 13 includes at least one fully connected layer and / or convolutional layer, and the pose parameter prediction network 14 includes at least one fully connected layer and / or convolutional layer.

[0064] In some embodiments, after determining the human pose in the target video frame, the electronic device can directly output the human pose in the target video frame. For example, in a live streaming scenario, the electronic device can add special effects to the target video frame based on the output human pose in the target video frame.

[0065] In other embodiments, the inventors have found in practice that the dynamic pose recognition results output by the pose classifier can have mutation conditions, i.e., the results of several consecutive frames are inconsistent. However, because the dynamic pose is a continuous change process, it usually does not change every frame, i.e., the dynamic pose recognition results of several consecutive frames are not inconsistent. The reasons for the mutation condition can include but are not limited to: (1) In a live streaming scenario, due to camera acquisition limitations, there are continuous frame loss conditions; (2) Detection errors occur in the human key point detection stage, such as identifying the left hand as the right hand; (3) Pose classifier recognition error. The occurrence of the above reasons will cause the final result to have jitter and mutation conditions.

[0066] In order to avoid or reduce the mutation case, the electronic device does not output the human posture in the target video frame after determining the human posture in the target video frame based on the posture classifier, but compares the human posture in the target video frame with the human posture output by the previous video frame of the target video frame. If the human posture in the target video frame is the same as the human posture output by the previous video frame of the target video frame, the human posture in the target video frame is directly output. If the human posture in the target video frame is different from the human posture output by the previous video frame of the target video frame, the output result of the previous video frame of the target video frame is output. That is, in the same case, the output result of the target video frame is the dynamic posture recognition result obtained by the posture classifier; in the different case, the output result of the target video frame is the output result of the previous video frame, so that the mutation case can be effectively avoided or reduced.

[0067] It is considered that a special case is that if the output result of a frame is wrong, and the dynamic posture recognition results obtained based on the posture classifier of the subsequent frames are all different from the output result of the previous frame, the continuous wrong case can be caused. For example, the output result of the first frame is wrong, the dynamic posture recognition result of the second frame is different from the output result of the first frame, the output result of the second frame is the output result of the first frame, the dynamic posture recognition result of the third frame is different from the output result of the second frame, the output result of the third frame is the output result of the second frame (actually, the output result of the first frame), and so on. If the dynamic posture recognition results of the subsequent video frames are also different from the output result of the previous frame, the output result of the video frame will be continuously wrong.

[0068] In view of the above problems, please refer to Figure 5 The embodiment of the present application provides a post-processing process.

[0069] In step S201, the human posture in the target video frame determined by the posture classifier is acquired.

[0070] In step S202, it is judged whether the human posture in the target video frame is the same as the human posture output by the previous video frame of the target video frame. If yes, step S203 is executed; if no, step S204 is executed.

[0071] In step S203, the buffer is emptied, and the human posture in the target video frame is output.

[0072] In step S204, the target frame is stored in the buffer.

[0073] In step S205, it is judged whether the buffer is full and the postures of the human bodies in all the video frames in the buffer are the same. If yes, step S203 is executed; if no, step S206 is executed.

[0074] In step S206, the output result of the previous video frame of the target video frame is outputted.

[0075] In the embodiment, in the case that the posture of the human body in the target video frame is different from the posture of the human body outputted by the previous video frame of the target video frame, the target frame can be stored in the buffer, and it is further judged whether the buffer is full and the postures of the human bodies in all the video frames in the buffer are the same. If yes, it is possible that the error occurs in the previous frame, and the posture of the human body in the target video frame can be directly outputted, so that the continuous error of the output result of the video frame is effectively avoided or reduced.

[0076] For example, the maximum storage number of the buffer is 3 frames. For example, the output result of the first frame is wrong, the dynamic posture recognition result of the second frame is different from the output result of the first frame, the second frame is stored in the buffer, and the output result of the second frame is the output result of the first frame. At this time, the buffer is not full. The dynamic posture recognition result of the third frame is different from the output result of the second frame, the third frame is stored in the buffer, and the output result of the third frame is the output result of the second frame (actually, the output result of the first frame). At this time, the buffer is not full. The dynamic posture recognition result of the fourth frame is different from the output result of the third frame, the fourth frame is stored in the buffer, at this time, the buffer is full, and the dynamic posture recognition results of the second, third and fourth frames are the same. Therefore, the output result of the fourth frame is the dynamic posture recognition result of the fourth frame, so that the continuous error of the output result of the video frame is effectively avoided or reduced.

[0077] For example, the maximum storage number of the buffer and the number of video frames corresponding to the joint sequence are in a positive correlation. The more the number of video frames corresponding to the joint sequence, the larger the maximum storage number of the buffer, and vice versa. In an example, for example, in the process of determining the posture of the human body in the target video frame, the number of video frames corresponding to the joint sequence is N+1. In the case of meeting the above positive correlation, the maximum storage number of the buffer can be set to be less than or equal to N+1.

[0078] It should be understood that the schemes described in the above embodiments can be combined without conflict.

[0079] Correspondingly, please refer to Figure 6 The embodiment of the present application also provides a dynamic posture recognition device, which comprises:

[0080] The video frame determination module 31 is configured to determine a target video frame to be recognized.

[0081] The joint information acquisition module 32 is configured to acquire a joint sequence containing joint angles of a human body in the target video frame and N previous video frames of the target video frame, and determine arrangement order information corresponding to the joint angles of each video frame in the joint sequence according to a shooting time sequence of the target video frame and the N previous video frames. N is an integer greater than 0.

[0082] The dynamic posture recognition module 33 is configured to perform dynamic posture recognition on the joint sequence fused with the arrangement order information, and determine a human posture in the target video frame.

[0083] In some embodiments, the dynamic posture recognition module 33 is specifically configured to splice the joint angles of each video frame in the joint sequence and the arrangement order information thereof, input the spliced sequence into a pre-established posture classifier, perform dynamic posture recognition on the spliced sequence by using the posture classifier, and obtain the human posture in the target video frame.

[0084] In some embodiments, the joint information acquisition module 32 is specifically configured to determine the arrangement order information corresponding to the joint angles of each video frame in the joint sequence according to a shooting time sequence of the target video frame and the N previous video frames and a value range of the joint angles. The value range of the arrangement order information is the same as the value range of the joint angles.

[0085] In some embodiments, the posture classifier is trained based on a plurality of samples carrying posture labels. The samples are joint sequences fused with the arrangement order information. The joint sequences are obtained by using M video frames to detect human joint points and predict joint angles. M is an integer greater than 1. The M video frames correspond to posture labels.

[0086] In some embodiments, the output module is further configured to: if the human posture in the target video frame is the same as a human posture output by a previous video frame of the target video frame, output the human posture in the target video frame; and if the human posture in the target video frame is different from the human posture output by the previous video frame of the target video frame, output an output result of the previous video frame of the target video frame.

[0087] In some embodiments, the output module is further configured to: if the human posture in the target video frame is different from the human posture output by the previous video frame of the target video frame, store the target video frame in a buffer; if the buffer is full and the human postures in all video frames in the buffer are the same, empty the buffer and output the human posture in the target video frame; otherwise, output the output result of the previous video frame of the target video frame; and if the human posture in the target video frame is different from the human posture in the previous video frame of the target video frame, empty the buffer.

[0088] In some embodiments, the maximum number of buffered video frames is positively correlated with the number of video frames corresponding to the joint sequence.

[0089] In some embodiments, the device further comprises a joint angle prediction module configured to: perform human key point detection on the target video frame to obtain two-dimensional human key point data; input the two-dimensional human key point data into a pre-established joint angle prediction model; and perform joint angle prediction on the two-dimensional human key point data by using the joint angle prediction model to obtain the joint angle of the human body in the target video frame.

[0090] In some embodiments, the joint angle prediction model is trained based on a plurality of two-dimensional key point sample data carrying angle labels; and the device further comprises a sample acquisition module configured to: acquire three-dimensional key point data and corresponding joint angles collected by an inertial motion capture system when a user performs a specified dynamic gesture; project the three-dimensional key point data to a camera plane according to a pre-determined positional relationship between the inertial motion capture system and the camera to obtain two-dimensional key point data; and obtain two-dimensional key point sample data carrying angle labels based on the two-dimensional key point data and the corresponding joint angles of the three-dimensional key point data.

[0091] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0092] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can be referred to the part of the method embodiment. The device embodiments described above are only illustrative, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. According to the actual needs, some or all of the modules can be selected to achieve the purpose of the present application. Those skilled in the art can understand and implement without creative labor.

[0093] Correspondingly, please refer toFigure 7 The embodiment of the present application also provides an electronic device 400, comprising a memory 402, a processor 401, and executable instructions stored in the memory 402 and capable of running on the processor 401, wherein the processor 401 executes the executable instructions to implement the steps in the method described above.

[0094] The processor 401 executes the executable instructions included in the memory 402, and the processor 401 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0095] The memory 402 stores the executable instructions of the dynamic gesture recognition method, and the memory 402 can include at least one type of storage medium, including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. Moreover, the device can cooperate with a network storage device performing a storage function of the memory through a network connection. The memory 402 can be an internal storage unit of the device 400, such as a hard disk or a memory of the device 400. The memory 402 can also be an external storage device of the device 400, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the device 400. Further, the memory 402 can include both the internal storage unit and the external storage device of the device 400. The memory 402 is used to store executable instructions and other programs and data required by the device. The memory 402 can also be used to temporarily store data that has been output or will be output.

[0096] The various embodiments described herein can be implemented using computer readable media with, for example, computer software, hardware, or any combination of software and hardware. For a hardware implementation, the embodiments described herein can be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, or a combination thereof. For a software implementation, the embodiments described herein can be implemented with separate software modules, such as procedures or functions, which perform one or more functions or operations. The software codes can be implemented with a software application written in any suitable programming language with the ability to perform at least one function or operation.

[0097] The electronic device 400 can be a computing device such as a desktop computer, a notebook, a palmtop computer, a server, a cloud server, and a mobile phone. The device can include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art can understand that the device can include more or less components, or combine some components, or include different components, for example, the device can also include an input / output device, a network access device, a bus, etc. Figure 7 The electronic device 400 is merely an example and does not constitute a limitation on the electronic device 400, and can include more or less components than those shown, or combine some components, or include different components, for example, the device can also include an input / output device, a network access device, a bus, etc.

[0098] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0099] In an exemplary embodiment, a non-transitory computer readable storage medium including instructions, such as a memory including instructions, is also provided, and the above method can be executed by the processor of the device to complete the above method. For example, the non-transitory computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0100] A non-transitory computer readable storage medium, when the instructions in the storage medium are executed by the processor of the terminal, enables the terminal to perform the above method.

[0101] While this specification contains many specifics, these should not be construed as limitations on the scope of any invention, but rather as descriptions of particular implementations of particular inventions. Certain features that are, for clarity, described above in the context of separate implementations can also be provided in combinations of implementations. Conversely, various features that are, for brevity, described above in the context of a single implementation can also be provided separately or in any suitable subcombination. In addition, while features can be described above as being implemented in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0102] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such order, nor that all illustrated operations be performed, to implement a desired result. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated in a single software product or packaged into multiple software products.

[0103] Accordingly, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous.

[0104] The above description is merely illustrative of the application and does not limit the scope of the application as determined by the appended claims.

Claims

1. A dynamic gesture recognition method, characterized by, The method comprises the following steps: determining a target video frame to be recognized; obtaining a joint sequence containing joint angles of a human body in the target video frame and N previous video frames of the target video frame; and determining arrangement order information corresponding to the joint angles of each video frame in the joint sequence according to a shooting time sequence of the target video frame and the N previous video frames and a value range of the joint angles; N is an integer greater than 0; the value range of the arrangement order information is the same as the value range of the joint angles, and the value of the arrangement order information corresponding to the joint angles of each video frame is in a positive correlation with the shooting time sequence corresponding to the video frame; splicing the joint angles of each video frame in the joint sequence and the arrangement order information corresponding to the joint angles; inputting the spliced sequence into a pre-established gesture classifier, performing dynamic gesture recognition on the spliced sequence by the gesture classifier, and obtaining a human body gesture in the target video frame.

2. The method of claim 1, wherein, The gesture classifier is trained based on a plurality of samples carrying gesture labels, the samples are joint sequences fused with the arrangement order information, and the joint sequences are obtained by using M video frames for human body joint detection and joint angle prediction; M is an integer greater than 1, and the M video frames correspond to a gesture label.

3. The method according to any one of claims 1 to 2, characterized in that, After determining the human body gesture in the target video frame, the method further comprises the following steps: if the human body gesture in the target video frame is the same as a human body gesture output by a previous video frame of the target video frame, outputting the human body gesture in the target video frame; if the human body gesture in the target video frame is different from the human body gesture output by the previous video frame of the target video frame, outputting an output result of the previous video frame of the target video frame.

4. The method of claim 3, wherein, The method further comprises the following steps: if the human body gesture in the target video frame is different from the human body gesture output by the previous video frame of the target video frame, storing the target video frame in a buffer; if the buffer is full and the human body gestures in all video frames in the buffer are the same, emptying the buffer and outputting the human body gesture in the target video frame; otherwise, outputting the output result of the previous video frame of the target video frame; if the human body gesture in the target video frame is different from the human body gesture in the previous video frame of the target video frame, emptying the buffer.

5. The method of claim 4, wherein, The maximum cache number of the buffer is in a positive correlation with the number of video frames corresponding to the joint sequence.

6. The method of claim 1, wherein, After obtaining the target video frame to be recognized, the method further comprises the following steps: performing human body key point detection on the target video frame to obtain human body two-dimensional key point data; inputting the human body two-dimensional key point data into a pre-established joint angle prediction model to perform joint angle prediction on the human body two-dimensional key point data by the joint angle prediction model, and obtaining joint angles of a human body in the target video frame.

7. The method of claim 6, wherein, The joint angle prediction model is trained based on a plurality of two-dimensional key point sample data carrying angle labels; At least part of the two-dimensional key point sample data carrying angle labels is obtained by the following method: obtaining three-dimensional key point data and corresponding joint angles collected by an inertial motion capture system when a user performs a specified dynamic gesture. Project the three-dimensional key point data to a camera plane according to a predetermined positional relationship between an inertial motion capture system and the camera to obtain two-dimensional key point data; Obtain two-dimensional key point sample data carrying angle labels based on the two-dimensional key point data and joint angles corresponding to the three-dimensional key point data.

8. A dynamic gesture recognition apparatus, characterized by, The method comprises the steps of: a video frame determination module configured to determine a target video frame to be recognized; a joint information acquisition module configured to acquire a joint sequence containing joint angles of a human body in the target video frame and N previous video frames of the target video frame, and determine arrangement order information corresponding to the joint angles of each video frame in the joint sequence according to a shooting time sequence of the target video frame and the N previous video frames and a value range of the joint angles; N is an integer greater than 0; the value range of the arrangement order information is the same as the value range of the joint angles, and the value of the arrangement order information corresponding to the joint angles of each video frame is in a positive correlation with the shooting time sequence corresponding to the video frame; a dynamic gesture recognition module configured to splice the joint angles of each video frame in the joint sequence and the arrangement order information corresponding to the joint angles, input the spliced sequence into a pre-established gesture classifier, and perform dynamic gesture recognition on the spliced sequence by using the gesture classifier to obtain a human body gesture in the target video frame.

9. An electronic device, comprising: The method comprises the steps of: a memory, a processor, and executable instructions stored in the memory and executable on the processor, wherein the processor executes the executable instructions to implement the steps in the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps in the method according to any one of claims 1 to 7.

Images