Joint modeling method, application method and device of gesture recognition and pose estimation

Abstract

The application discloses a gesture recognition and posture estimation joint modeling method, application method and device, relates to the technical field of perception interaction, and the modeling method comprises the following steps: performing two-dimensional posture estimation on a historical gesture image region to obtain a two-dimensional key point sequence, and inputting the two-dimensional key point sequence and an initial gesture type label into a three-dimensional posture estimation network to obtain an initial three-dimensional key point sequence; based on a video frame sequence, a first predicted gesture type label is acquired; based on the initial three-dimensional key point sequence, a second predicted gesture type label is acquired; the two predicted gesture type labels are fused to obtain a next round gesture type label; the next round gesture type label is taken as the initial gesture type label, the initial three-dimensional key point sequence is taken as the two-dimensional key point sequence, and the initial three-dimensional key point sequence acquisition step is returned; when a set iteration number is reached or a convergence condition is met, a joint modeling model is obtained. The application can realize more efficient and more accurate gesture recognition and three-dimensional posture estimation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of perceptual interaction, in particular to a joint modeling method of gesture recognition and pose estimation, an application method and device. BACKGROUND

[0002] Perceptual interaction is an important research direction in virtual reality and augmented reality, and its core is to realize the digital modeling and intelligent analysis of human and scene, and then to support the intelligent collaborative interaction between human and machine. As an important means of perceptual interaction, gesture interaction based on three-dimensional hand pose estimation has been a natural interactive interface pursued by the academic and industrial circles, and has important application value in the fields of human-computer interaction such as virtual reality, augmented reality, robots and virtual simulation training. As a more flexible part of the human body, the hand plays an important role in the interaction between human and real physical world.

[0003] In recent years, the computing power of smart terminal devices represented by mobile phones has been continuously improved, and more and more terminal devices have begun to deploy virtual reality and augmented reality applications. Using a virtual simulation training system, not only can the rare and expensive training scene in real life be simulated all-weather and high-fidelity, but also can reduce the potential training risk of dangerous scenes, and provide strong protection for the safety of trainees. The ideal interaction mode in AR and VR scenes is visual perception, and hand pose data is an important means to realize computer understanding of human body language. Through hand and other body language perception, human-computer interaction can connect the physical world and the information space, realize the simulation, virtual control and analysis optimization of the physical world, and realize the accurate perception and intelligent analysis of human-computer interaction in a more natural and real way.

[0004] The interaction mode of three-dimensional gestures has many advantages, but the research on three-dimensional hand pose estimation in interactive scenes is still relatively lacking. A large number of research works focus on estimating the three-dimensional pose information of a single hand from a single image, and through a pre-defined hand model to achieve a better estimation effect, but its application range and performance ability still have great differences with the needs of actual application scenes, and cannot well handle the problem of three-dimensional hand pose estimation under occlusion, especially in dynamic interactive scenes such as virtual simulation training of tactical actions, the hand often performs multiple operation steps with multiple equipment and tools, and there is a need to control unknown types of objects. In such flexible and variable virtual interactive application scenes, high-quality three-dimensional hand motion pose acquisition is still a very challenging task.

[0005] In the research of human-computer interaction, gesture recognition and pose estimation are two highly related problems, but in the current research, they are usually regarded as two independent tasks.

[0006] The main task of gesture recognition is to recognize the type of human gesture action based on input data such as video, image or three-dimensional skeleton data. There are currently two main gesture recognition schemes: one is video-based gesture action recognition, which mainly relies on appearance features in RGB video to realize gesture action recognition by capturing visual characteristics of hands or human bodies, without explicitly recovering three-dimensional pose information. The advantage of this method is that data acquisition is convenient, but its recognition performance is easily affected by changes in lighting, background complexity and occlusion. In addition, using only two-dimensional information of appearance features often makes it difficult to fully represent the motion patterns and spatial structures of gestures, which limits the recognition of complex dynamic gestures. The other is three-dimensional skeleton data-based gesture action recognition, which uses three-dimensional skeleton data collected by motion capture systems or depth sensors (such as Kinect, LeapMotion) to convert human action or gesture recognition tasks into a pattern recognition problem based on skeleton key points. This method can more clearly describe the spatial structure and dynamic changes of gestures by directly obtaining the three-dimensional motion trajectory of hands or human bodies. The difference between the two main schemes is whether to recover the three-dimensional pose in the recognition task. Compared with video-based recognition methods, skeleton data-based methods are generally more competitive in recognition performance, which indicates that there is a strong correlation between the three-dimensional motion trajectory of gestures and their action types.

[0007] The goal of pose estimation task is to recover the two-dimensional or three-dimensional pose of hands or human bodies from input data such as images, videos or point clouds. The typical research method is to predict the two-dimensional position of the hand or human body joint in the image plane from the input image. Three-dimensional pose estimation is to further recover the position or joint angle of the joint in three-dimensional space from two-dimensional data. Although the three-dimensional pose estimation task itself has great challenges, its results can provide rich kinematic information and prior constraints for high-level semantic tasks such as gesture recognition. Therefore, gesture recognition and pose estimation are essentially closely related, and how to achieve more efficient and accurate gesture recognition and three-dimensional pose estimation has become a technical problem that needs to be solved in the field. SUMMARY

[0008] The purpose of the present application is to provide a joint modeling method, application method and device for gesture recognition and pose estimation, which can realize more efficient and accurate gesture recognition and three-dimensional pose estimation.

[0009] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0010] In a first aspect, the present application provides a joint modeling method for gesture recognition and pose estimation, which comprises:

[0011] obtaining a historical gesture image region; the historical gesture image region is a region obtained based on a historical video frame sequence.

[0012] performing two-dimensional pose estimation on the historical gesture image region to obtain a two-dimensional key point sequence.

[0013] extracting appearance features by using an appearance flow classifier based on the video frame sequence to obtain a first predicted gesture type label.

[0014] inputting the two-dimensional key point sequence and an initial gesture type label into a three-dimensional pose estimation network to obtain an initial three-dimensional key point sequence; the initial gesture type label is a label obtained by random initialization or inference from appearance features of the video frame.

[0015] extracting kinematic features by using a pose flow classifier based on the initial three-dimensional key point sequence to obtain a second predicted gesture type label.

[0016] fusing the first predicted gesture type label and the second predicted gesture type label to obtain a next round gesture type label.

[0017] taking the next round gesture type label as the initial gesture type label, taking the initial three-dimensional key point sequence as the two-dimensional key point sequence, and returning to the step of inputting the two-dimensional key point sequence and the initial gesture type label into the three-dimensional pose estimation network to obtain the initial three-dimensional key point sequence.

[0018] when a set number of iterations is reached or a convergence condition is met, obtaining a joint modeling model.

[0019] In a second aspect, the present application provides a joint application method of gesture recognition and pose estimation, which comprises:

[0020] obtaining a to-be-predicted gesture image region; the to-be-predicted gesture image region is a region obtained based on a to-be-predicted video frame sequence.

[0021] inputting the to-be-predicted gesture image region into a joint modeling model to obtain a three-dimensional hand pose sequence and a gesture type label; the joint modeling model is a model obtained by training the joint modeling method of gesture recognition and pose estimation.

[0022] In a third aspect, the present application provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the joint modeling method of gesture recognition and pose estimation or implement the joint application method of gesture recognition and pose estimation.

[0023] In a fourth aspect, the present application provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the joint modeling method of gesture recognition and pose estimation or the joint application method of gesture recognition and pose estimation described above.

[0024] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the joint modeling method of gesture recognition and pose estimation or the joint application method of gesture recognition and pose estimation described above.

[0025] According to the specific embodiments provided by the present application, the following technical effects are disclosed:

[0026] The present application provides a joint modeling method and application method of gesture recognition and pose estimation. By performing two-dimensional pose estimation on a historical gesture image region, a two-dimensional key point sequence is obtained. Based on a video frame sequence, an appearance flow classifier is used to extract appearance features to obtain a first predicted gesture type label. The two-dimensional key point sequence and the initial gesture type label are input into a three-dimensional pose estimation network to obtain an initial three-dimensional key point sequence. Based on the initial three-dimensional key point sequence, a pose flow classifier is used to extract kinematic features to obtain a second predicted gesture type label. The first predicted gesture type label and the second predicted gesture type label are fused to obtain a next round gesture type label. The next round gesture type label is taken as the initial gesture type label, and the initial three-dimensional key point sequence is taken as the two-dimensional key point sequence, and the process of “inputting the two-dimensional key point sequence and the initial gesture type label into the three-dimensional pose estimation network to obtain the initial three-dimensional key point sequence” is returned. When a set number of iterations is reached or a convergence condition is met, a joint modeling model can be obtained. The present application proposes a gesture recognition and three-dimensional hand pose estimation joint modeling method based on a recurrent architecture, aiming to realize more efficient and accurate gesture classification and three-dimensional pose prediction through the interaction and feedback mechanism of the two. This method uses two-dimensional hand key point sequences and appearance information of video frames to construct a bidirectional collaborative optimization process of gesture recognition and three-dimensional pose estimation through a deep learning network, and finally outputs accurate gesture types and three-dimensional hand poses in an end-to-end form. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0028] Figure 1An application environment diagram of a joint modeling method of gesture recognition and pose estimation provided by an embodiment of the present application.

[0029] Figure 2 A flowchart of a joint modeling method of gesture recognition and pose estimation provided by an embodiment of the present application.

[0030] Figure 3 A framework diagram of a joint modeling method of gesture recognition and pose estimation provided by an embodiment of the present application.

[0031] Figure 4 A flowchart of a joint modeling method of gesture recognition and pose estimation provided by an embodiment of the present application.

[0032] Figure 5 A flowchart of a joint modeling method of gesture recognition and pose estimation provided by an embodiment of the present application.

[0033] Figure 6 A structural diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0035] In order to make the above objectives, features and advantages of the present application more apparent, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0036] In the aspect of action recognition, some methods attempt to establish the correlation between action recognition and pose estimation through a multi-task learning framework, and integrate the 3D pose estimation module into a branch of the action recognition network. Due to the diversity and ambiguity of human action definition, although the constraint relationship between human pose and action semantic information under the multi-task learning framework is not fully utilized, it can be shown to some extent that there is a high correlation between the two tasks. In the method based on template matching, action prior information can help reduce the search space of the sample database, making it easier to find the best matching result in the sample database, so the accuracy and reasonableness of the estimation result will be improved to some extent. In some methods, a multi-modal framework is designed around the human pose, which integrates different modalities together to build a more rich semantic, visual and 3D perceptual human pose embedding space, and then through a multi-modal alignment retrieval model, it can regress the parameterized human pose model from the image with optional text prompts. Although the object and content studied in this application are different from this type of work, the application of these semantic prior information can still indirectly indicate that the use of type prior information can improve the accuracy and reasonableness of the estimation result, so the gesture recognition and pose estimation joint modeling method, application method and device proposed in this application have innovative and feasible technology.

[0037] There is a potential synergy between gesture recognition and pose estimation, and this synergy can be fully exploited through joint modeling. By analyzing the hand self-occlusion and two-hand interaction scenes of typical gestures such as fist, grabbing, and clapping, it is not difficult to find that these gesture types with occlusion often implicitly provide some hand semantic information, express some instruction information through the difference between gestures, and these instruction information is highly related to the dynamic gesture recognition task. Obviously, we can use this potential semantic instruction information and structural clues to make reasonable inferences about non-visible areas, and at the same time, combined with the closely coupled gesture recognition task, realize the joint learning and mutual promotion of gesture recognition and pose estimation. Intuitively, more accurate poses can predict more accurate gesture types; similarly, if there is more accurate gesture type labeling, it can also be beneficial to more accurately estimate the three-dimensional hand pose. The core idea of this application is to establish a close coupling relationship between gesture action semantic information and three-dimensional pose structural information based on a cycle mechanism, realize the joint learning and mutual promotion of gesture recognition and pose estimation, not only improve the accuracy of gesture recognition, but also optimize the reasonableness of three-dimensional pose estimation, and provide a more comprehensive and efficient solution for complex dynamic gesture understanding and modeling.

[0038] The gesture recognition and pose estimation joint modeling method provided by the embodiments of the application can be applied to, for example, Figure 1The terminal 102 communicates with the server 104 through a network. The data storage system can store data required to be processed by the server 104. The data storage system can be separately arranged, integrated on the server 104, or placed on a cloud or other server. The terminal 102 can send the obtained historical gesture image region to the server 104, where the historical gesture image region is a region obtained based on a historical video frame sequence; after receiving the historical gesture image region, the server 104 performs two-dimensional pose estimation on the historical gesture image region to obtain a two-dimensional key point sequence; based on the video frame sequence, an appearance flow classifier is used to extract appearance features to obtain a first predicted gesture type label; the two-dimensional key point sequence and an initial gesture type label are input into a three-dimensional pose estimation network to obtain an initial three-dimensional key point sequence; the initial gesture type label is a label obtained by random initialization or inferred from the appearance features of the video frame; based on the initial three-dimensional key point sequence, a pose flow classifier is used to extract kinematic features to obtain a second predicted gesture type label; the first predicted gesture type label and the second predicted gesture type label are fused to obtain a next round gesture type label; the next round gesture type label is taken as the initial gesture type label, the initial three-dimensional key point sequence is taken as the two-dimensional key point sequence, and the operation of inputting the two-dimensional key point sequence and the initial gesture type label into the three-dimensional pose estimation network to obtain the initial three-dimensional key point sequence is returned; when a set number of iterations is reached or a convergence condition is met, a joint modeling model is obtained. The server 104 can feed back the obtained joint modeling model to the terminal 102. In addition, in some embodiments, the joint modeling method of gesture recognition and pose estimation can also be implemented by the server 104 or the terminal 102 alone, for example, the terminal 102 can directly perform joint modeling on the historical gesture image region, or the server 104 can obtain the historical gesture image region from the data storage system and perform joint modeling on the historical gesture image region.

[0039] The terminal 102 can be, but is not limited to, various desktop computers, notebook computers, smart phones, and tablet computers. The server 104 can be implemented by a single server or a server cluster composed of multiple servers, and can also be a cloud server.

[0040] In an exemplary embodiment, as Figure 2 shown, a joint modeling method of gesture recognition and pose estimation is provided, which is executed by a computer device, specifically, can be executed by a terminal or a server, or both, in the embodiment of the present application, the method is applied to the server 104 in Figure 1 The method includes the following steps A1 to A8.

[0041] wherein:

[0042] A1: obtaining a historical gesture image region; the historical gesture image region is a region obtained based on a historical video frame sequence.

[0043] A2: performing two-dimensional pose estimation on the historical gesture image region to obtain a two-dimensional key point sequence.

[0044] A3: based on the video frame sequence, extracting appearance features by using an appearance flow classifier to obtain a first predicted gesture type label.

[0045] A4: inputting the two-dimensional key point sequence and an initial gesture type label into a three-dimensional pose estimation network to obtain an initial three-dimensional key point sequence; the initial gesture type label is a label obtained by random initialization or inferred from the appearance features of the video frame.

[0046] A5: based on the initial three-dimensional key point sequence, extracting kinematic features by using a pose flow classifier to obtain a second predicted gesture type label.

[0047] A6: fusing the first predicted gesture type label and the second predicted gesture type label to obtain a next round gesture type label.

[0048] A7: taking the next round gesture type label as the initial gesture type label, taking the initial three-dimensional key point sequence as the two-dimensional key point sequence, and returning to "inputting the two-dimensional key point sequence and an initial gesture type label into a three-dimensional pose estimation network to obtain an initial three-dimensional key point sequence".

[0049] A8: when a set number of iterations is reached or a convergence condition is met, obtaining a joint modeling model.

[0050] Implementing steps A1 to A8 above, through the interaction and feedback mechanism of gesture recognition and pose estimation, higher efficiency and more accurate gesture recognition and three-dimensional pose estimation are achieved.

[0051] The main processing block diagram of the present application is shown in Figure 3 It mainly includes two core modules of gesture action recognition and 3D pose estimation, and a loop processing mechanism established between the 3D pose estimation and gesture action recognition modules.

[0052] First, the existing hand detector is used to locate the hand region from the original video, and further use the two-dimensional pose estimation network to extract the two-dimensional coordinate information of the hand key point. The two-dimensional pose as the initial input not only provides the basic features for the gesture recognition module, but also lays the geometric foundation for the subsequent three-dimensional pose recovery.

[0053] In the gesture action recognition module, the extracted two-dimensional key point information is utilized to classify the gesture action through a deep learning model (such as a convolutional neural network or a graph neural network), and the corresponding gesture type label is output. The gesture action recognition module considers the spatial distribution and topological relationship of the key points in the classification process to capture the semantic features of the gesture. The output result of the gesture action recognition module not only serves as one of the final gesture classification results, but also is input to the three-dimensional pose estimation module to guide the recovery of the three-dimensional pose. The gesture type label serves as high-level semantic information here, helping the three-dimensional pose estimation module to narrow down the possible pose search space, thereby improving the accuracy and efficiency of three-dimensional pose estimation.

[0054] The three-dimensional pose estimation module combines two-dimensional key points and gesture recognition results to predict the positions of hand key points in three-dimensional space using deep learning or optimization methods. The result of three-dimensional pose estimation not only serves as the final output, but also is fed back to the gesture action recognition module for further optimization of gesture classification. By introducing three-dimensional pose information, the gesture recognition module can more accurately distinguish complex gesture types, especially in cases where two-dimensional information is insufficient to distinguish, such as in complex backgrounds or partially occluded scenes. The global structural information of the three-dimensional pose can significantly improve the robustness of gesture recognition.

[0055] The present application finally outputs two types of results: gesture action type and three-dimensional hand pose. Gesture action type includes static gestures (such as "OK" or "fist") and dynamic gestures (such as "wave" or "click"), while three-dimensional hand pose is composed of three-dimensional coordinates of hand skeleton key points. Both have higher accuracy and consistency under the optimization of the loop architecture. The present application can be widely applied in human-computer interaction, virtual reality, augmented reality, motion capture, intelligent monitoring, and medical rehabilitation fields, providing an efficient and robust solution for understanding and modeling complex dynamic gestures.

[0056] The core innovation of the present application lies in the design of a loop mechanism that tightly integrates gesture action recognition and three-dimensional pose estimation tasks. The results of gesture action recognition are fed back to the three-dimensional pose estimation, while the results of three-dimensional pose estimation are used to optimize gesture action recognition. Through multiple iterations, both performance is continuously improved. On the one hand, the semantic information provided by gesture action recognition can provide prior constraints for three-dimensional pose estimation, such as limiting the possible shape of three-dimensional pose according to different gesture types. On the other hand, the skeleton information generated by three-dimensional pose estimation can provide structural guidance for gesture recognition, enhancing the stability of classification. Through this two-way optimization loop mechanism, the process of gesture recognition and three-dimensional pose estimation gradually reaches a state of higher consistency.

[0057] The cycle architecture of the present application not only improves the accuracy of gesture recognition and three-dimensional pose estimation, but also significantly enhances the adaptability to complex scenes. For example, in the case of uneven lighting, background interference or rapid movement of gestures, the cycle mechanism can correct potential errors through multiple iterations, thereby outputting more reliable results. In addition, the present application also has advantages in computational efficiency. By taking two-dimensional poses as the initial input, the high computational cost of directly recovering three-dimensional poses from videos is avoided, and the search process of three-dimensional poses is further optimized by combining gesture semantic constraints.

[0058] In an exemplary embodiment, in step A1, specifically comprising:

[0059] A11: Obtain a sequence of historical video frames containing hand regions.

[0060] A12: According to the sequence of historical video frames, use a hand detector to locate the hand region in each frame to obtain the located first hand region.

[0061] A13: According to the located first hand region, crop out the image region of single hand or double hands to obtain the historical gesture image region.

[0062] In an exemplary embodiment, in step A2, specifically comprising:

[0063] According to the historical gesture image region, use a two-dimensional pose estimation model to extract the two-dimensional coordinate information of the hand key points to obtain a sequence of two-dimensional key points; the two-dimensional pose estimation model is OpenPose or HRNet.

[0064] The present application proposes a gesture recognition and three-dimensional hand pose estimation joint modeling method based on a cycle architecture, aiming to realize more efficient and accurate gesture classification and three-dimensional pose prediction through the interaction and feedback mechanism of the two. The core of this method is to use the two-dimensional hand key point sequence and the appearance information of the video frame to construct a two-way collaborative optimization process of gesture recognition and three-dimensional pose estimation through a deep learning network, finally output accurate gesture types and three-dimensional hand poses in an end-to-end form, and its specific processing process is as shown in Figure 4

[0065] 1. Data preprocessing.

[0066] Given an input video frame sequence V = {v t ​| t = 1, 2,..., T}, where T represents the total number of video frames, and t is the t-th video frame. The hand region in each frame is located by an existing hand detector (such as YOLO or FasterR-CNN), and the single-hand or double-hand image region is cropped. Then a two-dimensional pose estimation model (such as OpenPose or HRNet) is used to extract the two-dimensional coordinate information of the hand key points, and the key point sequence P = {p t | t = 1, 2,..., T}, where N represents the number of hand joint points, and p t is the two-dimensional coordinate information of the hand key points of each frame.

[0067] 2. Initial three-dimensional pose estimation.

[0068] In the initial three-dimensional pose estimation stage, the two-dimensional key point sequence P and the initial gesture type label (the initial gesture type label can be obtained by random initialization or inferred from the appearance features of the video frames) are input into the initial three-dimensional pose estimation network f init , to learn a mapping function:

[0069] f init : (P, c (0) ) → Q (1) (1);

[0070] The initial three-dimensional key point sequence Q is output by the mapping function:

[0071]

[0072] where q represents the three-dimensional position of the hand key points in each frame.

[0073] The result of the initial three-dimensional pose estimation will be input into the subsequent gesture action recognition module.

[0074] 3. Gesture recognition.

[0075] In the gesture recognition stage, this application draws on the processing idea of the dual-stream network architecture, and designs an appearance stream classifier and a pose stream classifier to extract appearance information and kinematic information from the input video, respectively.

[0076] The appearance stream classifier g app extracts appearance features based on the video frame sequence V, and predicts the gesture type label c app , aiming to establish a mapping relationship between the video frame and the gesture type:

[0077] c app = g app (V) (3);

[0078] pose flow classifier g pose based on the initial 3D keypoint sequence Q (1) extract kinematic features and predict gesture type label c pose , aiming to establish a mapping relationship between 3D poses and gesture types:

[0079] c pose = g pose (Q (1) ) (4);

[0080] The classification results of appearance flow and pose flow are combined through a fusion module to calculate the final gesture type label c (1) :

[0081] c (1) = Fuse(c app ,c pose ) (5);

[0082] where Fuse() is the fusion module. The fusion method can use weighted average, attention mechanism or other strategies.

[0083] The fused gesture type label c (1) is fed back to the 3D pose estimation module for the next round of pose optimization.

[0084] 4. Loop optimization and joint loss.

[0085] In the loop optimization phase, the 3D pose estimation module and the gesture recognition module are optimized alternately.

[0086] (1) 3D pose update.

[0087] For the kth iteration (k > 1), the 3D pose estimation module further optimizes the 3D keypoint sequence based on the 3D keypoint sequence Q (k -1) and the gesture type label c (k-1) of the previous iteration:

[0088] f update :(Q (k-1) ,c (k-1) )→Q (k) (6);

[0089] where f update is the updated 3D keypoint sequence.

[0090] (2) Gesture label update.

[0091] The gesture recognition module calculates a new gesture type label based on the updated 3D pose sequence Q (k) and the video frame sequence V through appearance flow and pose flow:

[0092] c (k) = Fuse(g app (V),g pose (Q (k) )) (7);

[0093] where c (k) is the gesture type label after k iterations.

[0094] (3) Joint loss optimization.

[0095] To realize end-to-end training, the application adopts a joint loss function combining three-dimensional pose estimation error and gesture recognition error, and the total loss function is defined as:

[0096]

[0097] where L is the joint loss function, L is the loss of three-dimensional pose estimation, L is the loss of gesture recognition, λ pose is the weight of three-dimensional pose estimation, λ gesture is the weight of gesture recognition, and k is the kth iteration, and K is the total number of iterations.

[0098] Loss function of three-dimensional pose estimation is defined as the L2 loss between the estimated three-dimensional key point sequence of each frame and the true value:

[0099]

[0100] where T is the total number of video frames, t is the tth video frame, is the three-dimensional key point estimated in the kth iteration, is the true value of the three-dimensional key point estimated in the kth iteration.

[0101] Loss function of gesture recognition is defined as the cross-entropy loss between the predicted type and the true value:

[0102]

[0103] where y c is the true value distribution of the gesture type, is the predicted distribution of the gesture type, c is the gesture type label, and C is the total number of gesture type labels.

[0104] Based on the set cycle mechanism and the joint loss function, the model is trained, and the above cycle iteration process is continuously executed until the set number of iterations K is reached or the convergence condition is met.

[0105] (4) Result output.

[0106] In the inference phase, given a test video V, the trained recurrent architecture directly outputs a sequence of three-dimensional hand poses Q = {q t | t = 1, 2,..., T} and a gesture type label c.

[0107] The present application realizes the collaborative optimization of gesture recognition and three-dimensional pose estimation under the recurrent optimization mechanism, has high prediction accuracy and consistency, and provides an efficient and robust technical solution for human-computer interaction, virtual reality and motion capture fields.

[0108] The present application adopts the joint modeling idea, jointly models gesture recognition and three-dimensional pose estimation, utilizes the collaborative relationship between the two (for example, the gesture type will affect the prediction of the three-dimensional pose), realizes a more efficient modeling process, instead of independently processing the two tasks. At the same time, a recurrent optimization architecture is adopted, through the interaction and feedback mechanism of gesture recognition and three-dimensional hand pose estimation, the results of the two are gradually optimized. Through multiple iterations, the accuracy and consistency of gesture classification and three-dimensional pose prediction are improved.

[0109] The present application innovatively proposes a joint modeling method of correcting pose estimation with gesture semantic information, and designs a recurrent mechanism of end-to-end joint optimization to alternately solve the problems of gesture recognition and pose estimation, realizes the close coupling and mutual promotion of the two tasks through cross iteration, and enhances the immersion of human-computer interaction process with high fidelity three-dimensional hand pose data.

[0110] The application also provides an application scenario of the joint modeling method of gesture recognition and pose estimation. Specifically, the joint modeling method of gesture recognition and pose estimation provided by the embodiment can be applied in a perception interaction scenario. The perception interaction scenario includes a two-dimensional pose estimation link, an appearance feature extraction link, an initial three-dimensional key point sequence acquisition link, a kinematic feature extraction link, a fusion link, a loop link, and an interaction link. First, a historical gesture image region is acquired. The historical gesture image region is a region obtained based on a historical video frame sequence. Two-dimensional pose estimation is performed on the historical gesture image region to obtain a two-dimensional key point sequence. Second, based on the video frame sequence, an appearance flow classifier is used to extract appearance features to obtain a first predicted gesture type label. Third, the two-dimensional key point sequence and an initial gesture type label are input into a three-dimensional pose estimation network to obtain an initial three-dimensional key point sequence. The initial gesture type label is a label obtained by random initialization or inferred from the appearance features of the video frame. Fourth, based on the initial three-dimensional key point sequence, a pose flow classifier is used to extract kinematic features to obtain a second predicted gesture type label. Fifth, the first predicted gesture type label and the second predicted gesture type label are fused to obtain a next round gesture type label. Finally, the next round gesture type label is taken as the initial gesture type label, the initial three-dimensional key point sequence is taken as the two-dimensional key point sequence, and the operation of inputting the two-dimensional key point sequence and the initial gesture type label into the three-dimensional pose estimation network to obtain the initial three-dimensional key point sequence is returned. When a set number of iterations is reached or a convergence condition is met, a joint modeling model is obtained. A to-be-predicted gesture image region is acquired. The to-be-predicted gesture image region is a region obtained based on a to-be-predicted video frame sequence. The to-be-predicted gesture image region is input into the joint modeling model to obtain a three-dimensional hand pose sequence and a gesture type label.

[0111] In one exemplary embodiment, as shown in Figure 5 a joint application method of gesture recognition and pose estimation is provided, which includes:

[0112] B1: acquiring a to-be-predicted gesture image region; the to-be-predicted gesture image region is a region obtained based on a to-be-predicted video frame sequence.

[0113] B2: inputting the to-be-predicted gesture image region into a joint modeling model to obtain a three-dimensional hand pose sequence and a gesture type label; the joint modeling model is a model obtained by training the joint modeling method of gesture recognition and pose estimation according to any one of the above.

[0114] As an optional implementation, in step B1, specifically includes:

[0115] B11: acquire a sequence of video frames to be predicted containing hand regions.

[0116] B12: according to the sequence of video frames to be predicted, locate a hand region in each frame using a hand detector to obtain a second hand region after positioning.

[0117] B13: according to the second hand region after positioning, crop out a single-hand or double-hand image region to obtain a gesture image region to be predicted.

[0118] In an exemplary embodiment, a computer device is provided, which can be a server or a terminal, and an internal structure diagram thereof can be as shown in Figure 6 The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store historical gesture image regions and gesture image regions to be predicted. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement a joint modeling method of gesture recognition and pose estimation or a joint application method of gesture recognition and pose estimation.

[0119] Those skilled in the art can understand that Figure 6 the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0120] In an exemplary embodiment, a computer device is provided, which includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned method embodiments.

[0121] In an exemplary embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the above-mentioned method embodiments.

[0122] In an example embodiment, a computer program product is provided, including a computer program which, when executed by a processor, implements the above-mentioned method embodiments.

[0123] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0124] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0125] The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0126] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, it should be understood that the application encompasses all possible combinations of the technical features unless such a combination is not technically possible.

[0127] The principles and implementations of the present application have been described in specific examples, and the above descriptions of the embodiments are only used to help understand the method and its core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation and application range can be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for joint modeling of gesture recognition and pose estimation, characterized in that, The gesture recognition and pose estimation joint modeling method comprises: acquiring a historical gesture image region; the historical gesture image region is a region obtained based on a historical video frame sequence; performing two-dimensional pose estimation on the historical gesture image region to obtain a two-dimensional key point sequence; extracting appearance features using an appearance flow classifier based on the video frame sequence to obtain a first predicted gesture type label; inputting the two-dimensional key point sequence and an initial gesture type label into a three-dimensional pose estimation network to obtain an initial three-dimensional key point sequence; the initial gesture type label is a label obtained by random initialization or inferred from the appearance features of the video frame; extracting kinematic features using a pose flow classifier based on the initial three-dimensional key point sequence to obtain a second predicted gesture type label; fusing the first predicted gesture type label and the second predicted gesture type label to obtain a next round gesture type label; taking the next round gesture type label as the initial gesture type label, taking the initial three-dimensional key point sequence as the two-dimensional key point sequence, and returning to "inputting the two-dimensional key point sequence and an initial gesture type label into a three-dimensional pose estimation network to obtain an initial three-dimensional key point sequence"; when a set number of iterations is reached or a convergence condition is met, obtaining a joint modeling model.

2. The method of claim 1, wherein, The historical gesture image region is acquired, specifically comprising: acquiring a historical video frame sequence containing a hand region; positioning the hand region in each frame using a hand detector based on the historical video frame sequence to obtain a first positioned hand region; cropping a single-hand or double-hand image region based on the first positioned hand region to obtain a historical gesture image region. 3.The method of claim 1, wherein, The two-dimensional pose estimation is performed on the historical gesture image region to obtain a two-dimensional key point sequence, specifically comprising: extracting two-dimensional coordinate information of hand key points using a two-dimensional pose estimation model based on the historical gesture image region to obtain a two-dimensional key point sequence; the two-dimensional pose estimation model is OpenPose or HRNet. 4.The method of claim 1, wherein, The gesture recognition and pose estimation joint modeling method uses a joint loss function combining three-dimensional pose estimation error and gesture recognition error.

5. The method of claim 1, wherein, The expression of the joint loss function is: wherein, Ljointis a joint loss function, Lposeis a loss for three-dimensional pose estimation, Lgestureis a loss for gesture recognition, λ pose Lposeis a weight for three-dimensional pose estimation, λ gesture Lgestureis a weight for gesture recognition, k is the kth iteration, K is the total number of iterations, T is the total number of video frames, t is the tth video frame, ykis the kth iteration estimated three-dimensional keypoints, ykis the ground truth of the kth iteration estimated three-dimensional keypoints, y c yis the ground truth distribution of the gesture type, is the predicted distribution of the gesture type, c is the gesture type label, C is the total number of gesture type labels.

6. A method for joint application of gesture recognition and pose estimation, characterized in that, The gesture recognition and pose estimation joint application method comprises: acquiring a to-be-predicted gesture image region; the to-be-predicted gesture image region is a region obtained based on a to-be-predicted video frame sequence; inputting the to-be-predicted gesture image region into a joint modeling model to obtain a three-dimensional hand pose sequence and a gesture type label; the joint modeling model is a model trained based on the gesture recognition and pose estimation joint modeling method according to any one of claims 1-5. 7.The method of claim 6, wherein, The to-be-predicted gesture image region is acquired, specifically comprising: acquiring a to-be-predicted video frame sequence containing a hand region; positioning the hand region in each frame using a hand detector based on the to-be-predicted video frame sequence to obtain a second positioned hand region; cropping a single-hand or double-hand image region based on the second positioned hand region to obtain a to-be-predicted gesture image region.

8. A computer device comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the joint modeling method of gesture recognition and pose estimation of any one of claims 1-5 or implement the joint application method of gesture recognition and pose estimation of claim 6 or 7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the joint modeling method of gesture recognition and pose estimation of any one of claims 1-5 or implement the joint application method of gesture recognition and pose estimation of claim 6 or 7.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the joint modeling method of gesture recognition and pose estimation of any one of claims 1-5 or implement the joint application method of gesture recognition and pose estimation of claim 6 or 7.

Images