Hand skeleton posture estimation and action recognition model training method, recognition method and device

Abstract

The present application provides a hand skeleton posture estimation and action recognition model training method, recognition method and equipment, and relates to the field of image processing technology. The training method includes: training a graph attention network into a two-dimensional hand skeleton posture estimation model based on hand image data and a two-dimensional coordinate set of key points; using the estimation result of the two-dimensional coordinate set of key points of the hand image data, the three-dimensional coordinate set of key points and the hand action type label to train a medical image segmentation model and an action recognition network to obtain a three-dimensional hand skeleton posture estimation model and a hand action recognition model. By adopting a graph attention network, the present application can effectively utilize the structural information of the hand image data, and can improve the flexibility and reliability of the use of the structural information of the hand image data, can improve the performance and generalization ability of the trained hand skeleton posture estimation and action recognition model, and thus can improve the accuracy of hand skeleton posture estimation and action recognition using the model.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a hand skeleton posture estimation and action recognition model training method, recognition method and equipment. Background Art

[0002] With the rapid development of fields such as virtual reality and augmented reality, users have an increasing demand for gesture recognition and hand skeleton posture estimation. Existing hand skeleton posture estimation methods are mainly divided into two categories: methods based on graph convolutional neural networks (GCN) and methods based on convolutional neural networks (CNN). Among them, in the method based on graph convolutional neural networks, the initial two-dimensional coordinates are extracted, the two-dimensional hand skeleton posture is estimated through the graph convolutional neural network, and finally the two-dimensional coordinates are converted into three-dimensional coordinates. In the method based on convolutional neural networks, a three-dimensional volume representation of the hand is generated from the input three-dimensional point cloud using projection, and a three-dimensional convolutional neural network is trained in an end-to-end manner to map the three-dimensional volume representation to the three-dimensional hand joint relative position in the three-dimensional volume. The DeepHand system is a depth map-based method that uses regression to find the position of the key points of the hand and trains a convolutional neural network for each finger.

[0003] However, existing convolutional neural networks are not suitable for processing structured data such as hand bones. Although existing methods for hand skeleton posture estimation and action recognition based on graph neural networks can use graph convolutional neural networks suitable for processing structured data to estimate the coordinates of key points of hand bones, the representation of node features in graph convolutional neural networks is relatively fixed and not flexible enough, and the structure of the graph needs to be defined in advance.

[0004] Therefore, there is an urgent need to design a training method for hand skeleton posture estimation and action recognition models that can effectively utilize the structural information of graph data. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a hand skeleton posture estimation and action recognition model training method, recognition method and device to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present application provides a hand skeleton posture estimation and action recognition model training method, comprising:

[0007] Respectively obtain a plurality of continuous hand image data corresponding to a plurality of hand motion types, and a hand motion type label, a key point two-dimensional coordinate set, and a key point three-dimensional coordinate set corresponding to each of the hand image data;

[0008] Based on each of the hand image data and the corresponding two-dimensional coordinate sets of key points, training the graph attention network into a two-dimensional hand skeleton posture estimation model for outputting an estimation result of the two-dimensional coordinate set of key points of the hand image data;

[0009] A medical image segmentation model is trained using the estimation results of the two-dimensional coordinate sets of key points and the three-dimensional coordinate sets of key points corresponding to each of the hand image data, and a motion recognition network is trained using the estimation results of the two-dimensional coordinate sets of key points and the hand motion type labels corresponding to each of the hand image data, so as to train the medical image segmentation model into a three-dimensional hand skeleton posture estimation model for outputting the estimation results of the three-dimensional coordinate sets of key points of hand image data, and to train the motion recognition network into a hand motion recognition model for outputting the hand motion type recognition results of the hand image data.

[0010] In some embodiments of the present application, based on each of the hand image data and the corresponding two-dimensional coordinate sets of key points, the graph attention network is trained as a two-dimensional hand skeleton posture estimation model for outputting the estimation result of the two-dimensional coordinate set of key points of the hand image data, including:

[0011] Input each of the hand image data into a preset residual neural network, so that the residual neural network extracts features of each key point of each of the hand image data, so as to obtain an initial two-dimensional coordinate set of key points and initial feature data corresponding to each of the hand image data, and then splice the initial two-dimensional coordinate set of key points and the initial feature data corresponding to each of the hand image data, so as to output a feature vector corresponding to each of the hand image data;

[0012] Based on the feature vectors and the two-dimensional coordinate sets of key points corresponding to each of the hand image data, a preset graph attention network is trained with the mean square error loss of the two-dimensional coordinates of the key points as a loss function, so as to train the graph attention network into a two-dimensional hand skeleton posture estimation model for outputting the estimation results of the two-dimensional coordinate sets of key points of the hand image data.

[0013] In some embodiments of the present application, the graph attention network includes:

[0014] An attention coefficient learning layer, used for acquiring and correcting the attention coefficient matrix corresponding to the hand image data according to the feature vector corresponding to the currently input hand image data, and performing weighted summation processing on the feature vector according to the corrected attention coefficient matrix to obtain the feature vector corresponding to the hand image data after being processed by the attention mechanism;

[0015] A multi-head attention layer, used for obtaining, based on a multi-head attention mechanism, recognition results of multiple attention heads corresponding to the feature vector processed by the attention mechanism corresponding to the hand image data, and splicing the recognition results of each of the attention heads to obtain a spliced ​​feature vector corresponding to each of the key points in the hand image data;

[0016] The graph attention output layer is used to generate and output the estimation result of the two-dimensional coordinate set of the key points corresponding to the hand image data according to the spliced ​​feature vectors corresponding to each of the key points in the hand image data.

[0017] In some embodiments of the present application, the attention coefficient learning layer includes:

[0018] A similarity coefficient calculation unit, used to calculate the attention coefficient between each key point in the hand image data and its adjacent key points according to the feature vector corresponding to the hand image data currently input;

[0019] A rectified linear unit, used for normalizing the attention coefficient between each of the key points in the hand image data and the respective adjacent key points, so as to obtain an attention coefficient matrix corresponding to the hand image data;

[0020] An attention coefficient correction unit, used for correcting the attention coefficient matrix corresponding to the hand image data according to the current importance matrix between the key points of the hand image data, so as to obtain a corrected attention coefficient matrix corresponding to the hand image data;

[0021] An attention mechanism processing unit is used to perform weighted summation of the feature vectors of each of the key points in the hand image data and their adjacent key points based on the modified attention coefficient matrix, so as to obtain the feature vector corresponding to the hand image data after being processed by the attention mechanism.

[0022] In some embodiments of the present application, the method of using the two-dimensional coordinate set estimation results of the key points corresponding to each of the hand image data and the three-dimensional coordinate set of the key points to train a medical image segmentation model includes:

[0023] Based on the estimation results of the two-dimensional coordinate sets of key points corresponding to each of the hand image data and the three-dimensional coordinate sets of key points, a preset medical image segmentation model is trained with a preset combined loss function;

[0024] The combined loss function is composed of the mean square error loss of the two-dimensional coordinates of the key points and the three-dimensional coordinates of the key points and the cross entropy loss of the hand movement type recognition result.

[0025] In some embodiments of the present application, the action recognition network includes:

[0026] A spatiotemporal graph convolutional network, used for performing feature extraction on a three-dimensional coordinate set estimation result of key points of the hand image data currently input, so as to obtain spatiotemporal features corresponding to the hand image data;

[0027] A soft attention layer, used to calculate an attention coefficient of the hand image data, and weight the spatiotemporal features based on the attention coefficient to obtain weighted features corresponding to the hand image data;

[0028] A two-dimensional average pooling layer, used for performing two-dimensional average pooling processing on the weighted features corresponding to the hand image data to obtain pooling features corresponding to the hand image data;

[0029] A two-dimensional convolutional layer, used to obtain a category probability feature corresponding to the hand image data according to the pooled feature recognition corresponding to the hand image data;

[0030] The normalization layer is used to obtain the probability of each hand movement corresponding to the hand image data by normalizing the category probability features corresponding to the hand image data, so as to output the hand movement type recognition result of the hand image data.

[0031] In some embodiments of the present application, the step of respectively acquiring a plurality of continuous hand image data corresponding to each of the plurality of hand motion types, and a hand motion type label, a two-dimensional coordinate set of key points, and a three-dimensional coordinate set of key points corresponding to each of the hand image data, comprises:

[0032] Acquire multiple continuous real hand images corresponding to multiple types of hand movements;

[0033] Performing dimension preprocessing on each of the real hand images respectively to obtain hand image data with the same dimension;

[0034] Receive the hand action type label, key point two-dimensional coordinate set and key point three-dimensional coordinate set corresponding to each of the hand image data, wherein each key point in the key point two-dimensional coordinate set and the key point three-dimensional coordinate set includes multiple hand key points and multiple object corner points.

[0035] Another aspect of the present application provides a hand skeleton posture estimation and action recognition method, comprising:

[0036] Acquire target hand image data;

[0037] Inputting the target hand image data into a preset two-dimensional hand skeleton posture estimation model, so that the two-dimensional hand skeleton posture estimation model outputs an estimation result of a two-dimensional coordinate set of key points corresponding to the target hand image data;

[0038] Inputting the estimation results of the two-dimensional coordinate set of key points corresponding to the target hand image data into a preset three-dimensional hand skeleton posture estimation model and a hand action recognition model respectively, so that the three-dimensional hand skeleton posture estimation model outputs the estimation results of the three-dimensional coordinate set of key points corresponding to the target hand image data, and the hand action recognition model outputs the hand action type recognition results corresponding to the target hand image data;

[0039] Wherein, the two-dimensional hand skeleton posture estimation model, the three-dimensional hand skeleton posture estimation model and the hand action recognition model are trained in advance based on the hand skeleton posture estimation and action recognition model training method.

[0040] The third aspect of the present application provides a hand skeleton posture estimation and action recognition model training device, comprising:

[0041] A training data acquisition module, used to respectively acquire a plurality of continuous hand image data corresponding to a plurality of hand motion types, and a hand motion type label, a key point two-dimensional coordinate set and a key point three-dimensional coordinate set corresponding to each of the hand image data;

[0042] A two-dimensional model training module, for training the graph attention network into a two-dimensional hand skeleton posture estimation model for outputting an estimation result of the two-dimensional coordinate set of the key points of the hand image data based on each of the hand image data and the two-dimensional coordinate sets of the key points corresponding to each of the hand image data;

[0043] A three-dimensional and recognition model training module is used to train a medical image segmentation model using the estimation results of the two-dimensional coordinate sets of key points corresponding to each of the hand image data and the three-dimensional coordinate sets of key points, and to train a motion recognition network using the estimation results of the two-dimensional coordinate sets of key points corresponding to each of the hand image data and the hand motion type labels, so as to train the medical image segmentation model into a three-dimensional hand skeleton posture estimation model for outputting the estimation results of the three-dimensional coordinate sets of key points of hand image data, and to train the motion recognition network into a hand motion recognition model for outputting the hand motion type recognition results of hand image data.

[0044] A fourth aspect of the present application provides a hand skeleton posture estimation and action recognition device, comprising:

[0045] A target data acquisition module, used to acquire target hand image data;

[0046] A two-dimensional hand skeleton posture estimation module, used for inputting the target hand image data into a preset two-dimensional hand skeleton posture estimation model, so that the two-dimensional hand skeleton posture estimation model outputs an estimation result of a two-dimensional coordinate set of key points corresponding to the target hand image data;

[0047] A three-dimensional hand skeleton posture estimation and action recognition module, used to input the estimation results of the two-dimensional coordinate set of key points corresponding to the target hand image data into a preset three-dimensional hand skeleton posture estimation model and a hand action recognition model, respectively, so that the three-dimensional hand skeleton posture estimation model outputs the estimation results of the three-dimensional coordinate set of key points corresponding to the target hand image data, and the hand action recognition model outputs the hand action type recognition results corresponding to the target hand image data;

[0048] Wherein, the two-dimensional hand skeleton posture estimation model, the three-dimensional hand skeleton posture estimation model and the hand action recognition model are trained in advance based on the hand skeleton posture estimation and action recognition model training method.

[0049] The fifth aspect of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the hand skeleton posture estimation and action recognition model training method is implemented, and / or the hand skeleton posture estimation and action recognition method is implemented.

[0050] The sixth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hand skeleton posture estimation and action recognition model training method, and / or implements the hand skeleton posture estimation and action recognition method.

[0051] The seventh aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the hand skeleton posture estimation and action recognition model training method, and / or implements the hand skeleton posture estimation and action recognition method.

[0052] The hand skeleton posture estimation and action recognition model training method provided in the present application respectively obtains a plurality of continuous hand image data corresponding to a plurality of hand action types, as well as a hand action type label, a two-dimensional coordinate set of key points and a three-dimensional coordinate set of key points corresponding to each of the hand image data; based on each of the hand image data and the corresponding two-dimensional coordinate set of key points, trains a graph attention network into a two-dimensional hand skeleton posture estimation model for outputting the estimation result of the two-dimensional coordinate set of key points of the hand image data; uses the two-dimensional coordinate set estimation result of the key point corresponding to each of the hand image data and the three-dimensional coordinate set of key points to train a medical image segmentation model, and uses the two-dimensional coordinate set estimation result of the key point corresponding to each of the hand image data and the hand action type label to train an action recognition network, so as to train the medical image segmentation model into a three-dimensional hand skeleton posture estimation model for outputting the estimation result of the three-dimensional coordinate set of key points of the hand image data, and The action recognition network is trained as a hand action recognition model for outputting hand action type recognition results of hand image data. By adopting the graph attention network GAT as the two-dimensional hand skeleton posture estimation model, the hand image data is converted into a suitable input representation. The structural information of the hand image data can be effectively utilized based on the attention mechanism, and the flexibility and reliability of utilizing the structural information of the hand image data can be effectively improved. As a result, the trained two-dimensional hand skeleton posture estimation model, as well as the three-dimensional hand skeleton posture estimation model and the hand action recognition model trained according to the output of the two-dimensional hand skeleton posture estimation model can have better performance and generalization ability, and can effectively improve the accuracy of estimating the two-dimensional coordinates of the key points of the hand image using the two-dimensional hand skeleton posture estimation model, can effectively improve the accuracy of estimating the three-dimensional coordinates of the key points of the hand image using the three-dimensional hand skeleton posture estimation model, and can effectively improve the accuracy of identifying the hand action type using the hand action recognition model.

[0053] Additional advantages, purposes, and features of the present application will be partially described in the following description, and will become partially apparent to those skilled in the art after studying the following, or may be learned from the practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the specification and the drawings.

[0054] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0056] Figure 1 This is a first flow chart of a method for training a hand skeleton posture estimation and motion recognition model in one embodiment of the present application.

[0057] Figure 2 This is a second flow chart of the hand skeleton posture estimation and action recognition model training method in one embodiment of the present application.

[0058] Figure 3 Schematic diagram of the architecture of a graph attention network in one embodiment of the present application.

[0059] Figure 4 Schematic diagram of the architecture of a motion recognition network in one embodiment of the present application.

[0060] Figure 5 Schematic diagram of the flow of a hand skeleton posture estimation and action recognition method in one embodiment of the present application.

[0061] Figure 6 Schematic diagram of the structure of a hand skeleton posture estimation and action recognition model training device in one embodiment of the present application.

[0062] Figure 7 Schematic diagram of the structure of a hand skeleton posture estimation and action recognition device in one embodiment of the present application.

[0063] Figure 8 This is a flow chart of a hand skeleton posture estimation and action recognition model training method in an application example of the present application.

[0064] Fig. 9 This is a schematic diagram of the execution logic of the hand skeleton posture estimation and action recognition model training method in an application example of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the implementation modes and the accompanying drawings. Here, the illustrative implementation modes and descriptions of the present application are used to explain the present application, but are not intended to limit the present application.

[0066] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the scheme according to the present application are shown in the accompanying drawings, while other details that are not very relevant to the present application are omitted.

[0067] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0068] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0069] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0070] Hand 3D posture estimation and motion recognition can provide a good human-computer interaction method for virtual reality interaction, sign language translation, smart home and other fields. Therefore, this technology has been widely used in somatosensory games, AR / VR and 3D modeling. Especially in the operation of smart driving cars, hand gesture estimation and motion recognition technology provide feasibility for contactless operation.

[0071] Traditional hand skeletal pose estimation methods, such as the HOPE-Net method, first use the ResNet model to extract the initial two-dimensional coordinates of the hand joints and object corners, and then use two cascaded adaptive graph convolutional neural networks to estimate the hand skeletal pose, one of which is used to optimize the two-dimensional coordinates of the hand joints and object corners, followed by another graph convolutional network (Adaptive Graph U-Net) to convert the two-dimensional coordinates into three-dimensional coordinates. Hand action recognition methods include the end-to-end interactive prototype learning (IPL) framework and the symbiotic attention framework with privileged information. Both methods use target detection models to identify the objects touched by the hand and recognize actions based on this. However, the representation of node features in the graph convolutional neural network GCN is relatively fixed and not flexible enough, and the structure of the graph needs to be defined in advance. Therefore, the structural information of the hand image data cannot be effectively utilized, which in turn affects the performance and generalization ability of the trained hand skeletal pose estimation and action recognition models.

[0072] Based on this, in order to solve the above problems, the embodiments of the present application respectively provide a hand skeleton posture estimation and motion recognition model training method, a hand skeleton posture estimation and motion recognition model training device for executing the hand skeleton posture estimation and motion recognition model training method, a hand skeleton posture estimation and motion recognition method, a hand skeleton posture estimation and motion recognition device for executing the hand skeleton posture estimation and motion recognition method, and an electronic device (i.e., a computer device), a computer-readable storage medium, and a computer program product, which can effectively utilize the structural information of hand image data and improve the performance and generalization ability of the hand skeleton posture estimation and motion recognition model.

[0073] The details are described in detail through the following examples.

[0074] Based on this, the present application embodiment provides a hand skeleton posture estimation and action recognition model training method that can be implemented by a hand skeleton posture estimation and action recognition model training device, see Figure 1 The hand skeleton posture estimation and action recognition model training method specifically includes the following contents:

[0075] Step 100: respectively obtain a plurality of continuous hand image data corresponding to a plurality of hand motion types, and a hand motion type label, a two-dimensional coordinate set of key points, and a three-dimensional coordinate set of key points corresponding to each of the hand image data.

[0076] In one or more embodiments of the present application, hand skeleton posture estimation refers to estimating the position information of the key points of the hand from a single RGB hand image data or multiple continuous hand image data (such as video frames), and the position information may include two-dimensional skeleton posture (two-dimensional coordinates of key points) and three-dimensional skeleton posture (three-dimensional coordinates of key points).

[0077] In step 100, the key point two-dimensional coordinate set includes two-dimensional coordinate data corresponding to each key point in the hand image data, and the key point three-dimensional coordinate set includes three-dimensional coordinate data corresponding to each key point in the hand image data.

[0078] It should be noted that the key points of the hand image data mentioned in this application may include at least a plurality of skeletal key points. If the application scenario of this application is the action interaction between the hand and the object, then in order to further improve the interaction between the hand and the object, the key points of the hand image data mentioned in this application may also include a plurality of object corner points.

[0079] That is to say, in one embodiment of the present application, the key points of the hand image data may include 21 hand key points and 8 object corner points. The 21 hand key points refer to 21 main bone nodes for locating the hand, including fingertips, joints of phalanges, etc.

[0080] In a specific example, the numerical representations and meanings of the 21 main bone nodes include: the value 0 is used to represent the wrist point, located at the base of the pisiform bone; the value 1 is used to represent the thumb, located at the base of the first trapezium; the value 2 is used to represent the thumb, located at the base of the second thumb metacarpal bone; the value 3 is used to represent the thumb, located at the base of the third index finger; the value 4 is used to represent the tip of the thumb, located at the thumb phalange; the value 5 is used to represent the first section of the index finger, located at the base of the proximal phalanx of the index finger; the value 6 is used to represent the second section of the index finger, located at the base of the middle phalanx of the index finger; the value 7 is used to represent the third section of the index finger, located at the base of the distal phalanx of the index finger; the value 8 is used to represent the tip of the index finger, located at the distal phalanx of the index finger; the value 9 is used to represent the first section of the middle finger, located at the base of the proximal phalanx of the middle finger; the value 10 is used to represent the third section of the middle finger The second knuckle is located at the base of the middle phalanx of the middle finger; the value 11 is used to represent the third knuckle of the middle finger, located at the base of the distal phalanx of the middle finger; the value 12 is used to represent the tip of the middle finger, located at the head of the distal phalanx of the middle finger; the value 13 is used to represent the first knuckle of the ring finger, located at the base of the proximal phalanx of the ring finger; the value 14 is used to represent the second knuckle of the ring finger, located at the base of the middle phalanx of the ring finger; the value 15 is used to represent the third knuckle of the ring finger, located at the base of the distal phalanx of the ring finger; the value 16 is used to represent the tip of the ring finger, located at the head of the distal phalanx of the ring finger; the value 17 is used to represent the first knuckle of the little finger, located at the base of the proximal phalanx of the little finger; the value 18 is used to represent the second knuckle of the little finger, located at the base of the middle phalanx of the little finger; the value 19 is used to represent the third knuckle of the little finger, located at the base of the distal phalanx of the little finger; the value 20 is used to represent the tip of the little finger, located at the head of the distal phalanx of the little finger.

[0081] In a specific example, the eight object corner points may specifically include: eight vertices of a cuboid approximated by the object contour.

[0082] Step 200: Based on each of the hand image data and the corresponding two-dimensional coordinate sets of key points, the graph attention network is trained to be a two-dimensional hand skeleton posture estimation model for outputting an estimation result of the two-dimensional coordinate set of key points of the hand image data.

[0083] In step 200, the Graph Attention Network (GAT) is a deep learning graph network model that introduces an attention mechanism. With each key point as each node in the hand image, the Graph Attention Network (GAT) can adaptively learn the relationship between each node. Therefore, the embodiment of the present application can use the Graph Attention Network (GAT) to solve the graph modeling task of the interaction between human hands and objects. By designing a specific attention mechanism and network structure, the image data is converted into a suitable input representation, and the feature representation of the node is learned through training. At the same time, the attention mechanism is added on the basis of the learnable graph structure, and the attention coefficient representing the importance between nodes is learned, so as to effectively utilize the structural information of the graph data and obtain better performance and generalization ability.

[0084] Step 300: Use the two-dimensional coordinate set estimation results of the key points corresponding to each of the hand image data and the three-dimensional coordinate set of the key points to train a medical image segmentation model, and use the two-dimensional coordinate set estimation results of the key points corresponding to each of the hand image data and the hand motion type labels to train a motion recognition network, so as to train the medical image segmentation model into a three-dimensional hand skeleton posture estimation model for outputting the three-dimensional coordinate set estimation results of the key points of the hand image data, and train the motion recognition network into a hand motion recognition model for outputting the hand motion type recognition results of the hand image data.

[0085] It is understandable that in terms of action recognition, both the interactive prototype learning (IPL) framework and the symbiotic attention framework with privileged information use a detection model to detect objects in the image, and since multiple objects may exist in the image at the same time, such technologies also need to accurately identify objects that interact with the hand among multiple objects. In contrast, step 300 provided in the embodiment of the present application uses the estimation result of the two-dimensional coordinate set of key points of the hand image data output by the two-dimensional hand skeleton posture estimation model as the input of the action recognition network, without the need to detect objects in the image, making the model more lightweight.

[0086] In one or more embodiments of the present application, the two-dimensional hand skeleton posture estimation model can also be abbreviated as a two-dimensional coordinate estimation module, the three-dimensional hand skeleton posture estimation model can also be abbreviated as a three-dimensional coordinate estimation module, and the hand motion recognition model can also be written as a motion recognition module.

[0087] In a specific example of step 300, the action recognition network may adopt an Attention Graph U-Net model.

[0088] From the above description, it can be seen that the hand skeleton posture estimation and action recognition model training method provided in the embodiment of the present application, by adopting the graph attention network GAT as the two-dimensional hand skeleton posture estimation model, converts the hand image data into a suitable input representation, can effectively utilize the structural information of the hand image data based on the attention mechanism, and can effectively improve the flexibility and reliability of utilizing the structural information of the hand image data, thereby enabling the trained two-dimensional hand skeleton posture estimation model, and the three-dimensional hand skeleton posture estimation model and hand action recognition model trained according to the output of the two-dimensional hand skeleton posture estimation model to have better performance and generalization ability, can effectively improve the accuracy of estimating the two-dimensional coordinates of the key points of the hand image using the two-dimensional hand skeleton posture estimation model, can effectively improve the accuracy of estimating the three-dimensional coordinates of the key points of the hand image using the three-dimensional hand skeleton posture estimation model, and can effectively improve the accuracy of identifying the hand action type using the hand action recognition model.

[0089] In order to further improve the effectiveness and reliability of training a two-dimensional hand skeleton posture estimation model, in a hand skeleton posture estimation and action recognition model training method provided in an embodiment of the present application, see Figure 2 , step 200 in the hand skeleton posture estimation and action recognition model training method may specifically include the following contents:

[0090] Step 210: Input each of the hand image data into a preset residual neural network, so that the residual neural network extracts features of each key point of each of the hand image data, so as to obtain an initial two-dimensional coordinate set of key points and initial feature data corresponding to each of the hand image data, and then splice the initial two-dimensional coordinate set of key points and initial feature data corresponding to each of the hand image data, so as to output a feature vector corresponding to each of the hand image data.

[0091] It can be understood that the residual neural network (Residual network, ResNet) is a deep learning network model with a residual connection module, which can be abbreviated as a residual network.

[0092] Specifically, the hand image data p in step 100 are respectively f The residual neural network in the input graph attention network is used to extract features through the residual neural network to obtain the initial two-dimensional coordinates (29×2) and corresponding initial features (29×2048) of 29 key points (21 hand key points and 8 object corner points). The two are then concatenated into a feature vector h (29×2050).

[0093] Step 220: Based on the feature vectors and the two-dimensional coordinate sets of key points corresponding to each of the hand image data, a preset graph attention network is trained with the mean square error loss of the two-dimensional coordinates of the key points as a loss function, so as to train the graph attention network into a two-dimensional hand skeleton posture estimation model for outputting the estimation results of the two-dimensional coordinate sets of key points of the hand image data.

[0094] To train the ResNet and graph attention network layers, the mean square error loss of the two-dimensional coordinates of the key points is used as the loss function. The goal of this stage is to enable the graph attention network to have the ability to detect the two-dimensional coordinates of key points in the hand image in order to improve the training effect of the subsequent stages. The loss function L of the graph attention network total1 The calculation formula is as follows:

[0095] L totoal1 =L 2D

[0096]

[0097] Among them, L 2D represents the mean square error loss of the two-dimensional coordinates of the key points, n is the total number of key points, Represents the two-dimensional coordinates of the i-th key point in the two-dimensional coordinate set of key points corresponding to the hand image data, Represents the estimation result of the two-dimensional coordinate set of the key points of the i-th key point in the hand image data output by the graph attention network.

[0098] In order to further improve the training effectiveness and reliability of the graph attention network, in a hand skeleton posture estimation and action recognition model training method provided in an embodiment of the present application, see Figure 3 The graph attention network in the hand skeleton posture estimation and action recognition model training method may specifically include the following contents:

[0099] The attention coefficient learning layer 31 is used to obtain and correct the attention coefficient matrix corresponding to the hand image data according to the feature vector corresponding to the currently input hand image data, and perform weighted summation processing on the feature vector according to the corrected attention coefficient matrix to obtain the feature vector corresponding to the hand image data after being processed by the attention mechanism.

[0100] The multi-head attention layer 32 is used to obtain the recognition results of multiple attention heads corresponding to the feature vector of the hand image data processed by the attention mechanism based on the multi-head attention mechanism, and splice the recognition results of each attention head to obtain the spliced ​​feature vector corresponding to each key point in the hand image data.

[0101] The graph attention output layer 33 is used to generate and output the estimation result of the two-dimensional coordinate set of the key points corresponding to the hand image data according to the spliced ​​feature vectors corresponding to each of the key points in the hand image data.

[0102] Among them, in one embodiment of the present application, see Figure 3 In order to further improve the application reliability and effectiveness of the attention coefficient learning layer 31, the attention coefficient learning layer 31 may specifically include the following contents:

[0103] The similarity coefficient calculation unit 311 is used to calculate the attention coefficient between each key point in the hand image data and its adjacent key points according to the feature vector corresponding to the currently input hand image data.

[0104] A corrected linear unit 312 is used to normalize the attention coefficients between each of the key points in the hand image data and the respective adjacent key points to obtain an attention coefficient matrix corresponding to the hand image data;

[0105] The attention coefficient correction unit 313 is used to correct the attention coefficient matrix corresponding to the hand image data according to the current importance matrix between the key points of the hand image data to obtain a corrected attention coefficient matrix corresponding to the hand image data.

[0106] The attention mechanism processing unit 314 is used to perform weighted summation of the feature vectors of each key point in the hand image data and its adjacent key points based on the modified attention coefficient matrix to obtain the feature vector corresponding to the hand image data after being processed by the attention mechanism.

[0107] Based on the architecture of the above graph attention network, the functions of each unit are described as follows:

[0108] (1) The similarity coefficient calculation unit 311 performs the following:

[0109] The feature vector h output by the residual neural network in step 210 is input into the similarity coefficient calculation unit 311, so that the similarity coefficient calculation unit 311 estimates the two-dimensional coordinates, and the 29 key points are respectively used as nodes. For each node i, the attention coefficient e is calculated one by one on its neighboring node j. ij :

[0110] e ij =([Wh i ||Wh j ]) Among them, || represents the connection operation, h i and h jThey represent the feature vectors of node i and its neighbor node j respectively, and W represents the parameter matrix.

[0111] (2) The corrected linear unit 312 performs the following:

[0112] The attention coefficient e output by the similarity coefficient calculation unit 311 is ij Input the corrected linear unit 312, so that the corrected linear unit 312 normalizes the attention coefficient of each node i and its neighbor node j using softmax to obtain the attention coefficient matrix a ij :

[0113]

[0114] LeakyRelu is a modified linear unit activation function. Compared with the traditional ReLU, it allows negative input to have a small output. k refers to the neighbor node of node i, e ik refers to the attention coefficient of node i and node k, exp() refers to the exponential function, N i Represents the set of neighbor nodes of node i.

[0115] (3) The attention coefficient correction unit 313 performs the following:

[0116] The attention coefficient matrix a corresponding to the hand image data output by the rectified linear unit 312 ij Input the attention coefficient correction unit 313, so that the attention coefficient correction unit 313 can learn the importance matrix A between each node through training, and use the importance matrix A and the attention coefficient matrix a ij Perform matrix multiplication to obtain the corrected attention coefficient matrix a′ ij :

[0117] a′ ij =a ij A

[0118] (4) The attention mechanism processing unit 314 performs the following:

[0119] The corrected attention coefficient matrix a′ corresponding to the hand image data output by the attention coefficient correction unit 313 ij Input the attention mechanism processing unit 314 so that the attention mechanism processing unit 314 compares each node i with the feature vector h of the neighboring node j. j According to the corrected attention coefficient matrix a′ ij Perform weighted summation to obtain the feature vector h after attention mechanism processing i ′:

[0120]

[0121] Among them, σ represents the activation function, N i Represents the set of neighbor nodes of node i.

[0122] (5) The execution content of the multi-head attention layer 32 is as follows:

[0123] The feature vector h output by the attention mechanism processing unit 314 after being processed by the attention mechanism i ′ is input into the multi-head attention layer 32 including multiple attention heads, so that the multi-head attention layer 32 applies the multi-head attention mechanism (Multi-Head Attention) to obtain the recognition results obtained by N attention heads and concatenate them to obtain the concatenated feature vector h corresponding to each node i τ(N):

[0124]

[0125] in, represents the concatenation operation, m represents the number of attention heads, represents the attention coefficient matrix after correction of the mth attention head, W m Represents the parameter matrix of the mth attention head.

[0126] (6) The execution content of the graph attention output layer 33 is as follows:

[0127] The concatenated feature vector h′ corresponding to each key point in the hand image data output by the multi-head attention layer 32 i (N) is input into the image attention output layer 33 so that the image attention output layer 33 performs an average operation to generate a single hand image data p f Final estimation result of the key point two-dimensional coordinate set The estimation result of the two-dimensional coordinate set of the key points of each hand image data Together they constitute the total set of estimation results of the two-dimensional coordinate set of key points Among them, R 29×2 It refers to the 39×2 dimensional space.

[0128] In order to further improve the effectiveness and reliability of training medical image segmentation models, in a hand skeleton posture estimation and action recognition model training method provided in an embodiment of the present application, see Figure 2 , step 300 in the hand skeleton posture estimation and action recognition model training method may specifically include the following contents:

[0129] Step 310: Based on the estimation results of the two-dimensional coordinate sets of key points and the three-dimensional coordinate sets of key points corresponding to each of the hand image data, a preset medical image segmentation model is trained with a preset combined loss function to train the medical image segmentation model into a three-dimensional hand skeleton posture estimation model for outputting the estimation results of the three-dimensional coordinate sets of key points of the hand image data; wherein the combined loss function is composed of the mean square error loss of the two-dimensional coordinates of the key points and the three-dimensional coordinates of the key points and the cross entropy loss of the hand action type recognition results.

[0130] In one example, a medical image segmentation model (which can use an Attention Graph U-Net model) can include five graph pooling layers and five graph upsampling layers connected in sequence, each graph pooling layer and each graph upsampling layer can be connected to a graph attention network, and two fully connected layers are provided between the graph pooling layer and the graph upsampling layer, both of which are used for global feature extraction.

[0131] In step 310, the two-dimensional coordinate set estimation result of the key point output by the graph attention output layer 33 in step 200 is As the input of the medical image segmentation model, the three-dimensional coordinates are estimated from the two-dimensional coordinates. In each graph pooling layer, the number of nodes is halved, and in each graph upsampling layer, the number of nodes in the hand image data is doubled. The graph pooling layer and the graph upsampling layer are followed by a graph attention network to optimize the node features according to the attention coefficient. This operation obtains the final three-dimensional coordinate set estimation result of the key points

[0132] In step 310, the combined loss function is composed of the mean square error loss of the key point two-dimensional coordinates and the key point three-dimensional coordinates and the cross entropy loss of the hand action type recognition result. The goal of the combined loss function is to train the Attention Graph U-Net model while ensuring the robustness of the graph attention network effect, so that the Attention Graph U-Net model can map the key point two-dimensional coordinates to three-dimensional space, thereby further improving the accuracy and accurately identifying hand actions.

[0133] Combined loss function L total2 The calculation formula is as follows:

[0134] L total2 =λ1L 2D +λ2L 3D +L R

[0135]

[0136]

[0137] Among them, L 3D represents the mean square error loss of the three-dimensional coordinates of the key points, L R represents the cross entropy loss of the hand action type recognition result, λ1 and λ2 are hyperparameters, M is the number of hand action types, and p c is the probability that the hand image data belongs to the cth hand action type, y c is the hand action type label, p c is the model prediction result, where if the action belongs to the cth hand action type, then y c =1, if the action does not belong to the cth hand action type, then y c =0, represents the three-dimensional coordinates of the i-th key point in the three-dimensional coordinate set of key points corresponding to the hand image data, Represents the estimated result of the three-dimensional coordinate set of the key points of the i-th key point in the hand image data output by the graph attention network.

[0138] Step 320: Use the estimation results of the two-dimensional coordinate sets of key points corresponding to each of the hand image data and the hand motion type labels to train the action recognition network to train the action recognition network into a hand motion recognition model for outputting hand motion type recognition results of the hand image data.

[0139] Step 330: Input the test hand image data in the preset test set into the trained two-dimensional hand skeleton posture estimation model, the three-dimensional hand skeleton posture estimation model and the hand motion recognition model, and obtain the test results corresponding to each of the test hand image data, including the key point two-dimensional coordinate set estimation result, the key point three-dimensional coordinate set estimation result and the hand motion type recognition result.

[0140] If the test result meets the preset accuracy threshold or required data, step 340 is executed.

[0141] Step 340: Output the two-dimensional hand skeleton posture estimation model, the three-dimensional hand skeleton posture estimation model and the hand action recognition model to form a corresponding hand skeleton posture estimation and action recognition model.

[0142] In order to further improve the effectiveness and reliability of training medical image segmentation models, in a hand skeleton posture estimation and action recognition model training method provided in an embodiment of the present application, see Figure 4 The action recognition network in the hand skeleton posture estimation and action recognition model training method may specifically include the following contents:

[0143] The spatiotemporal graph convolutional network 41 is used to extract features from the estimation results of the three-dimensional coordinate set of key points of the currently input hand image data to obtain the spatiotemporal features corresponding to the hand image data.

[0144] A soft attention layer 42 is used to calculate the attention coefficient of the hand image data, and weight the spatiotemporal features based on the attention coefficient to obtain weighted features corresponding to the hand image data.

[0145] The two-dimensional average pooling layer 43 is used to perform two-dimensional average pooling processing on the weighted features corresponding to the hand image data to obtain pooling features corresponding to the hand image data.

[0146] The two-dimensional convolutional layer 44 is used to obtain the category probability features corresponding to the hand image data according to the pooled features corresponding to the hand image data.

[0147] The normalization (Softmax) layer 45 is used to obtain the probability of each hand movement corresponding to the hand image data according to the normalization of the category probability characteristics corresponding to the hand image data, so as to output the hand movement type recognition result of the hand image data, which can be abbreviated as the recognition result.

[0148] Specifically, the estimation results of the two-dimensional coordinate sets of each key point output by the graph attention output layer 33 in step 200 are As the input of the action recognition network, the spatiotemporal features are extracted by the spatiotemporal graph convolutional network (ST-GCN), and the attention coefficient is calculated through the soft attention mechanism. The spatiotemporal features are weighted according to the attention coefficient to obtain weighted features.

[0149] Among them, the construction details of the spatiotemporal graph convolutional network are as follows:

[0150] For node v it , the nodes used in the spatiotemporal graph convolution can be expressed as:

[0151] N(v it )={v jq |d(v it , v jq )<K,|qt|≤L}

[0152] The operation d(·) represents the distance between nodes, K represents the distance threshold, L represents the time distance threshold, and v jq represents node j at time q, q represents a certain time, and t represents v it The time, N(v it ) represents node v itThe set of nodes used for spatiotemporal graph convolution.

[0153] The process of spatiotemporal graph convolution can be expressed as:

[0154]

[0155] Where w(·,·) is the weight function, that is, the attention coefficient, f in (·) represents the spatiotemporal features corresponding to the node, f out (v it ) represents node v it The weighted features.

[0156] Then, a two-dimensional average pooling operation is performed to obtain the pooling features, and the pooling features are input into the two-dimensional convolutional layer to obtain the category probability features, followed by softmax normalization to obtain the probability p of each hand movement, and the hand movement with the maximum probability is taken as the hand movement type recognition result of the hand image data.

[0157] In order to further improve the application effectiveness and reliability of hand image data, in a hand skeleton posture estimation and action recognition model training method provided in an embodiment of the present application, see Figure 2 , step 100 in the hand skeleton posture estimation and action recognition model training method may specifically include the following contents:

[0158] Step 110: Acquire a plurality of continuous real hand images corresponding to a plurality of hand motion types.

[0159] Step 120: perform dimension preprocessing on each of the real hand images to obtain hand image data with the same dimension.

[0160] Step 130: Receive the hand action type label, key point two-dimensional coordinate set and key point three-dimensional coordinate set corresponding to each of the hand image data, wherein each key point in the key point two-dimensional coordinate set and the key point three-dimensional coordinate set includes multiple hand key points and multiple object corner points.

[0161] Specifically, each real hand image in the training set is resized to obtain F consecutive images P = {P1, P2, ..., P F}, after preprocessing, the real hand image is converted into F pieces of hand image data p={p1, p2, ..., p F}, as the input of the subsequent two-dimensional hand skeleton posture estimation model, and obtain the hand action type label y corresponding to the F hand image data and the real two-dimensional coordinate set of key points And the three-dimensional coordinate set of key points Used to calculate losses.

[0162] Based on the hand skeleton posture estimation and action recognition model training method provided in the above embodiment, the present application also provides an embodiment of a hand skeleton posture estimation and action recognition method, see Figure 5 The hand skeleton posture estimation and action recognition method specifically includes the following contents:

[0163] Step 400: Acquire target hand image data;

[0164] Step 500: inputting the target hand image data into a preset two-dimensional hand skeleton posture estimation model, so that the two-dimensional hand skeleton posture estimation model outputs an estimation result of a two-dimensional coordinate set of key points corresponding to the target hand image data;

[0165] Step 600: Input the estimation results of the two-dimensional coordinate set of key points corresponding to the target hand image data into the preset three-dimensional hand skeleton posture estimation model and hand motion recognition model respectively, so that the three-dimensional hand skeleton posture estimation model outputs the estimation results of the three-dimensional coordinate set of key points corresponding to the target hand image data, and the hand motion recognition model outputs the hand motion type recognition results corresponding to the target hand image data; wherein the two-dimensional hand skeleton posture estimation model, the three-dimensional hand skeleton posture estimation model and the hand motion recognition model are pre-trained based on the hand skeleton posture estimation and motion recognition model training method.

[0166] It can be understood that the hand skeleton posture estimation and action recognition model training method in step 600 can be specifically executed with reference to the above-mentioned embodiment of the hand skeleton posture estimation and action recognition model training method, which will not be repeated here.

[0167] From the above description, it can be seen that the hand skeleton posture estimation and action recognition method provided in the embodiment of the present application, by adopting a graph attention network, can effectively utilize the structural information of the hand image data, and can improve the flexibility and reliability of utilizing the structural information of the hand image data, and can improve the performance and generalization ability of the trained hand skeleton posture estimation and action recognition model, thereby improving the accuracy of hand skeleton posture estimation and action recognition using the model.

[0168] On this basis, the present application also provides a hand skeleton posture estimation and action recognition model training device for executing all or part of the hand skeleton posture estimation and action recognition model training method, see Figure 6 The hand skeleton posture estimation and action recognition model training device specifically includes the following contents:

[0169] The training data acquisition module 01 is used to respectively acquire a plurality of continuous hand image data corresponding to a plurality of hand motion types, and a hand motion type label, a key point two-dimensional coordinate set and a key point three-dimensional coordinate set corresponding to each of the hand image data;

[0170] A two-dimensional model training module 02 is used to train the graph attention network into a two-dimensional hand skeleton posture estimation model for outputting the estimation result of the two-dimensional coordinate set of the key points of the hand image data based on each of the hand image data and the two-dimensional coordinate set of the key points corresponding to each of the hand image data;

[0171] The three-dimensional and recognition model training module 03 is used to train a medical image segmentation model using the estimation results of the two-dimensional coordinate sets of the key points corresponding to each of the hand image data and the three-dimensional coordinate sets of the key points, and to train a motion recognition network using the estimation results of the two-dimensional coordinate sets of the key points corresponding to each of the hand image data and the hand motion type labels, so as to train the medical image segmentation model into a three-dimensional hand skeleton posture estimation model for outputting the estimation results of the three-dimensional coordinate sets of the key points of the hand image data, and to train the motion recognition network into a hand motion recognition model for outputting the hand motion type recognition results of the hand image data.

[0172] The embodiment of the hand skeleton posture estimation and action recognition model training device provided in the present application can be specifically used to execute the processing flow of the embodiment of the hand skeleton posture estimation and action recognition model training method in the above-mentioned embodiment. Its functions will not be repeated here, and reference can be made to the detailed description of the above-mentioned hand skeleton posture estimation and action recognition model training method embodiment.

[0173] The part that the hand skeleton posture estimation and action recognition model training device carries out hand skeleton posture estimation and action recognition model training can be completed in a server, and can also be completed in a client device. Specifically, it can be selected according to the processing power of the client device, and the limitation of the user's use scene, etc. This application is not limited to this. If all operations are completed in the client device, the client device can also include a processor, which is used for the specific processing of hand skeleton posture estimation and action recognition model training.

[0174] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0175] The server and the client device may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0176] From the above description, it can be seen that the hand skeleton posture estimation and action recognition model training device provided in the embodiment of the present application, by adopting the graph attention network GAT as the two-dimensional hand skeleton posture estimation model, converts the hand image data into a suitable input representation, can effectively utilize the structural information of the hand image data based on the attention mechanism, and can effectively improve the flexibility and reliability of utilizing the structural information of the hand image data, thereby enabling the trained two-dimensional hand skeleton posture estimation model, and the three-dimensional hand skeleton posture estimation model and hand action recognition model trained according to the output of the two-dimensional hand skeleton posture estimation model to have better performance and generalization ability, can effectively improve the accuracy of estimating the two-dimensional coordinates of the key points of the hand image using the two-dimensional hand skeleton posture estimation model, can effectively improve the accuracy of estimating the three-dimensional coordinates of the key points of the hand image using the three-dimensional hand skeleton posture estimation model, and can effectively improve the accuracy of identifying the hand action type using the hand action recognition model.

[0177] On this basis, the present application also provides a hand skeleton posture estimation and action recognition device for executing all or part of the hand skeleton posture estimation and action recognition method, see Figure 7 The hand skeleton posture estimation and action recognition device specifically includes the following contents:

[0178] The target data acquisition module 04 is used to acquire the target hand image data.

[0179] The two-dimensional hand skeleton posture estimation module 05 is used to input the target hand image data into a preset two-dimensional hand skeleton posture estimation model so that the two-dimensional hand skeleton posture estimation model outputs the estimation result of the two-dimensional coordinate set of key points corresponding to the target hand image data.

[0180] The three-dimensional hand skeleton posture estimation and action recognition module 06 is used to input the estimation results of the two-dimensional coordinate set of key points corresponding to the target hand image data into the preset three-dimensional hand skeleton posture estimation model and hand action recognition model respectively, so that the three-dimensional hand skeleton posture estimation model outputs the estimation results of the three-dimensional coordinate set of key points corresponding to the target hand image data, and the hand action recognition model outputs the hand action type recognition results corresponding to the target hand image data; wherein the two-dimensional hand skeleton posture estimation model, the three-dimensional hand skeleton posture estimation model and the hand action recognition model are pre-trained based on the hand skeleton posture estimation and action recognition model training method.

[0181] The embodiment of the hand skeleton posture estimation and action recognition device provided in the present application can be specifically used to execute the processing flow of the embodiment of the hand skeleton posture estimation and action recognition method in the above-mentioned embodiment. Its functions will not be repeated here, and reference can be made to the detailed description of the above-mentioned hand skeleton posture estimation and action recognition method embodiment.

[0182] The part of hand skeleton posture estimation and action recognition performed by the hand skeleton posture estimation and action recognition device can be completed in a server or in a client device.

[0183] From the above description, it can be seen that the hand skeleton posture estimation and action recognition device provided in the embodiment of the present application, by adopting a graph attention network, can effectively utilize the structural information of the hand image data, and can improve the flexibility and reliability of utilizing the structural information of the hand image data, and can improve the performance and generalization ability of the trained hand skeleton posture estimation and action recognition model, thereby improving the accuracy of hand skeleton posture estimation and action recognition using the model.

[0184] To further illustrate the above scheme, the present application also provides a specific application example of a hand skeleton posture estimation and action recognition model training method, which uses a graph attention network (GAT) to learn the relationship between nodes with different weights instead of using fixed neighbor weights, so as to more flexibly capture the complex relationship information in the graph and improve the accuracy of the model. The coordinate estimation module is used to obtain the two-dimensional coordinates of the hand, and then the action recognition module based on the spatial temporal graph convolutional network (ST-GCN) combined with soft attention generates action recognition results from these estimated two-dimensional hand coordinates. Compared with the method of directly recognizing actions from video frames, this method has higher action recognition accuracy.

[0185] See also Figure 8 and Fig. 9The hand skeleton posture estimation and action recognition model training method uses a graph attention network to estimate the hand skeleton posture, and based on the two-dimensional hand posture key points output by the posture estimation network, uses a spatiotemporal graph convolutional network to recognize hand actions. This application example specifically includes the following contents:

[0186] S1. Obtain training set data and labels.

[0187] Each real hand image in the training set is resized to obtain F consecutive images P = {P1, P2, ..., P F}, after preprocessing, the real hand image is converted into F pieces of hand image data p={p1, p2, ..., p F}, as the input of the subsequent two-dimensional hand skeleton posture estimation model, and obtain the hand action type label y corresponding to the F hand image data and the real two-dimensional coordinate set of key points And the three-dimensional coordinate set of key points Used to calculate losses.

[0188] S2. Construct a two-dimensional coordinate estimation network, a three-dimensional coordinate estimation network and an action recognition network.

[0189] (1) Construct a network model for two-dimensional coordinate estimation, which includes a feature extraction module and a decoder module. The feature extraction part includes a residual neural network (ResNet), followed by a graph attention network (GAT) as a decoder. The graph attention network (GAT) consists of n attention heads and an attention output layer. The attention head consists of an attention calculation method, a softmax calculation to normalize the attention weights, and a matrix multiplication.

[0190] (2) Construct a network model for 3D coordinate estimation (Attention Graph U-Net), which consists of five graph pooling layers, each followed by a graph attention network, five graph upsampling layers, each followed by a graph attention network, and two fully connected layers for global feature extraction.

[0191] (3) Construct an action recognition network model, which includes a batch normalization layer, a 10-layer spatiotemporal graph convolutional network (ST-GCN), a soft attention layer, a two-dimensional average pooling operation, and a two-dimensional convolutional layer followed by a softmax operation.

[0192] S3. Input the image feature map into the two-dimensional coordinate estimation network and output the two-dimensional coordinates of the objects and hands in the image.

[0193] The hand image data p in step S1 f The residual neural network in the input graph attention network is used to extract features through the residual neural network to obtain the initial two-dimensional coordinates (29×2) and corresponding initial features (29×2048) of 29 key points (21 hand key points and 8 object corner points). The two are then concatenated into a feature vector h (29×2050).

[0194] The two-dimensional coordinates are estimated based on the feature vector h output by the residual neural network. The 29 key points are taken as nodes. For each node i, the attention coefficient e is calculated one by one on its neighboring node j. ij :

[0195] e ij =f([Wh i ||Wh j ])

[0196] Among them, || represents the connection operation, h i and h j They represent the feature vectors of adjacent nodes i and j respectively, and W represents the parameter matrix.

[0197] Afterwards, the attention coefficients of each node i and its neighbor node j are normalized using softmax to obtain the attention coefficient matrix a ij :

[0198]

[0199] Among them, LeakyRelu is a rectified linear unit activation function. Compared with the traditional ReLU, it allows negative input to have a small output.

[0200] Different from the traditional Mask graph attention method, the adjacency matrix used in this model is a learnable adjacency matrix. Through training, the importance matrix A between each node is learned, and the importance matrix A and the attention coefficient matrix a are used to calculate the relationship between the nodes. ij Perform matrix multiplication to obtain the corrected attention coefficient matrix a′ ij :

[0201] a′ ij =a ij A

[0202] For each node i, compare it with the feature vector h of its neighbor node j j According to the corrected attention coefficient matrix a′ ij Perform weighted summation, and then obtain the corrected feature vector h′ after processing by the attention mechanism i :

[0203]

[0204] Among them, σ represents the activation function, N i Represents the set of neighbor nodes of node i.

[0205] Finally, h′ i Apply the multi-head attention mechanism to obtain the recognition results obtained by N attention heads and concatenate them to obtain the concatenated feature vector h′ corresponding to each node. i (N):

[0206]

[0207] in, represents the concatenation operation, m represents the number of attention heads, Indicates that W m express.

[0208] Finally h′ i (N) Take an average operation through a single-layer graph attention layer (i.e., graph attention output layer) to generate a single hand image data p f Final estimation result of the key point two-dimensional coordinate set

[0209] S4. Input the two-dimensional coordinate estimation result into the three-dimensional coordinate estimation network and output the three-dimensional coordinate estimation result.

[0210] The estimated result of the two-dimensional coordinate set of key points As the input of the medical image segmentation model, the three-dimensional coordinates are estimated from the two-dimensional coordinates. In each graph pooling layer, the number of nodes is halved, and in each graph upsampling layer, the number of nodes in the hand image data is doubled. The graph pooling layer and the graph upsampling layer are followed by a graph attention network to optimize the node features according to the attention coefficient. This operation obtains the final three-dimensional coordinate set estimation result of the key points

[0211] S5. Identify the action based on the estimated two-dimensional coordinates.

[0212] The two-dimensional coordinate set estimation results of each key point output in step S3 As the input of the action recognition network, the spatiotemporal features are extracted by the spatiotemporal graph convolutional network (ST-GCN), and the attention coefficient is calculated through the soft attention mechanism. The spatiotemporal features are weighted according to the attention coefficient to obtain weighted features.

[0213] Among them, the construction details of the spatiotemporal graph convolutional network are as follows:

[0214] For node v it , the nodes used in the spatiotemporal graph convolution can be expressed as:

[0215] N(v it )={v jq |d(v it , v jq )<K,|qt|≤L}

[0216] The operation d(·) represents the distance between nodes, K represents the distance threshold, and L represents the time distance threshold.

[0217] The process of spatiotemporal graph convolution can be expressed as:

[0218]

[0219] Where w(·,·) is the weight function, that is, the attention coefficient, f in (·) represents the spatiotemporal features corresponding to the node, f out (v it ) represents node v it The weighted features.

[0220] Then, a two-dimensional average pooling operation is performed to obtain the pooling features, and the pooling features are input into the two-dimensional convolutional layer to obtain the category probability features, followed by softmax normalization to obtain the probability p of each hand movement, and the hand movement with the maximum probability is taken as the hand movement type recognition result of the hand image data.

[0221] S6. Train the model to obtain a trained model.

[0222] Different loss functions are used to train different model stages to achieve the best results.

[0223] In the first stage, in order to train the ResNet and graph attention network layers, the mean square error loss of the two-dimensional coordinates of the key points is used as the loss function. The goal of this stage is to enable the graph attention network to have the ability to detect the two-dimensional coordinates of the key points in the hand image, so as to improve the training effect of the subsequent stages. The loss function L of the graph attention network total1 The calculation formula is as follows:

[0224] L total1 =L 2D

[0225]

[0226] Among them, L 2D represents the mean square error loss of the two-dimensional coordinates of the key points, n is the total number of key points, Represents the two-dimensional coordinates of the i-th key point in the two-dimensional coordinate set of key points corresponding to the hand image data, Represents the estimation result of the two-dimensional coordinate set of the key points of the i-th key point in the hand image data output by the graph attention network.

[0227] In the second stage, a combined loss function is used, which consists of the mean squared error loss of the 2D and 3D key points and the cross entropy loss of the action recognition results. The goal of this loss function is to train the AttentionGraph U-Net while ensuring the robustness of the graph attention network, so that the model can map the 2D key points to the 3D space, thereby further improving the accuracy and accurately identifying the hand movements. Combined loss function L total2 The calculation formula is as follows:

[0228] L total2 =λ1L 2D +λ2L 3D +L R

[0229]

[0230]

[0231] Among them, L 3D represents the mean square error loss of the three-dimensional coordinates of the key points, L R represents the cross entropy loss of the hand action type recognition result, λ1 and λ2 are hyperparameters, M is the number of hand action types, and p c is the probability that the hand image data belongs to the cth hand action type, y c is the hand action type label, p c is the model prediction result, where if the action belongs to the cth hand action type, then y c =1, if the action does not belong to the cth hand action type, then y c =0, represents the three-dimensional coordinates of the i-th key point in the three-dimensional coordinate set of key points corresponding to the hand image data, Represents the estimated result of the three-dimensional coordinate set of the key points of the i-th key point in the hand image data output by the graph attention network.

[0232] S7. Input the data in the test set into the trained model to obtain the estimated two-dimensional and three-dimensional coordinates and action recognition results.

[0233] The test image is resized according to step S1, and then the processed feature map is input into the trained model to obtain the estimated two-dimensional and three-dimensional coordinates and action recognition results.

[0234] Based on this, the hand skeleton posture estimation and action recognition model training method provided by the application example of this application applies the graph attention network (GAT) to the Attention Graph U-Net in the two-dimensional coordinate estimation and three-dimensional coordinate estimation process, and applies soft attention combined with ST-GCN to action recognition; (2) a multi-task learning framework that can be used for hand skeleton posture estimation and action recognition at the same time is proposed, and lightweight action recognition is realized by using the two-dimensional coordinates output by the hand skeleton posture estimation module as input.

[0235] Compared with the prior art, the application example of this application has the following beneficial effects:

[0236] (1) The graph attention network (GAT) is used to automatically learn the attention coefficient between the hand and object nodes through the attention mechanism, which represents the importance of the nodes. Compared with the method of using graph neural network, it can extract the interactive information of each skeletal posture of the hand more flexibly and efficiently, and effectively improve the accuracy of the model.

[0237] (2) The coordinate estimation module is used to obtain the hand coordinates, and then the hand action recognition module is used to recognize the hand action based on the result of the hand two-dimensional coordinate estimation. Compared with the method of directly performing action recognition, the application example of this application introduces Soft attention to make the model focus on the bone points that contribute more to action recognition, thereby effectively improving the accuracy of action recognition.

[0238] The embodiment of the present application also provides an electronic device, which may include a processor, a memory, a receiver and a transmitter, wherein the processor is used to execute the hand skeleton posture estimation and action recognition model training method and / or the hand skeleton posture estimation and action recognition method mentioned in the above embodiment, wherein the processor and the memory may be connected via a bus or other means, with the bus connection being taken as an example. The receiver may be connected to the processor and the memory via a wired or wireless manner.

[0239] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0240] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the hand skeleton posture estimation and action recognition model training method and / or the hand skeleton posture estimation and action recognition method in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, implementing the hand skeleton posture estimation and action recognition model training method and / or the hand skeleton posture estimation and action recognition method in the above method embodiment.

[0241] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0242] The one or more modules are stored in the memory, and when executed by the processor, perform the hand skeleton posture estimation and action recognition model training method and / or the hand skeleton posture estimation and action recognition method in the embodiment.

[0243] In some embodiments of the present application, the user equipment may include a processor, a memory, and a transceiver unit, which may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0244] As an implementation method, the functions of the receiver and the transmitter in the present application can be considered to be implemented through a transceiver circuit or a dedicated chip for transceiver, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general chip.

[0245] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiment of the present application, that is, to store the program code for implementing the functions of the processor, receiver, and transmitter in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0246] The present application embodiment also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned hand skeleton posture estimation and action recognition model training method and / or the hand skeleton posture estimation and action recognition method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0247] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0248] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0249] In the present application, features described and / or illustrated for one embodiment may be used in the same manner or in a similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0250] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A hand skeleton posture estimation and action recognition model training method, characterized in that: include: Respectively obtain a plurality of continuous hand image data corresponding to a plurality of hand motion types, and a hand motion type label, a key point two-dimensional coordinate set, and a key point three-dimensional coordinate set corresponding to each of the hand image data; Based on each of the hand image data and the corresponding two-dimensional coordinate sets of key points, training the graph attention network into a two-dimensional hand skeleton posture estimation model for outputting an estimation result of the two-dimensional coordinate set of key points of the hand image data; The medical image segmentation model is trained using the estimation results of the two-dimensional coordinate sets of the key points and the three-dimensional coordinate sets of the key points corresponding to each of the hand image data, and the action recognition network is trained using the estimation results of the two-dimensional coordinate sets of the key points and the hand action type labels corresponding to each of the hand image data, so as to train the medical image segmentation model into a three-dimensional hand skeleton posture estimation model for outputting the estimation results of the three-dimensional coordinate sets of the key points of the hand image data, and train the action recognition network into a hand action recognition model for outputting the hand action type recognition results of the hand image data; The method of training the graph attention network to a two-dimensional hand skeleton posture estimation model for outputting an estimation result of the two-dimensional coordinate set of the key points of the hand image data based on each of the hand image data and the two-dimensional coordinate set of the key points corresponding to each of the hand image data comprises: Input each of the hand image data into a preset residual neural network, so that the residual neural network extracts features of each key point of each of the hand image data, so as to obtain an initial two-dimensional coordinate set of key points and initial feature data corresponding to each of the hand image data, and then splice the initial two-dimensional coordinate set of key points and the initial feature data corresponding to each of the hand image data, so as to output a feature vector corresponding to each of the hand image data; Based on the feature vectors and the two-dimensional coordinate sets of the key points corresponding to each of the hand image data, a preset graph attention network is trained with the mean square error loss of the two-dimensional coordinates of the key points as a loss function, so as to train the graph attention network into a two-dimensional hand skeleton posture estimation model for outputting an estimation result of the two-dimensional coordinate set of the key points of the hand image data; The graph attention network includes: An attention coefficient learning layer, used for acquiring and correcting the attention coefficient matrix corresponding to the hand image data according to the feature vector corresponding to the currently input hand image data, and performing weighted summation processing on the feature vector according to the corrected attention coefficient matrix to obtain the feature vector corresponding to the hand image data after being processed by the attention mechanism; A multi-head attention layer, used for obtaining, based on a multi-head attention mechanism, recognition results of multiple attention heads corresponding to the feature vector processed by the attention mechanism corresponding to the hand image data, and splicing the recognition results of each of the attention heads to obtain a spliced ​​feature vector corresponding to each of the key points in the hand image data; The graph attention output layer is used to generate and output the estimation result of the two-dimensional coordinate set of the key points corresponding to the hand image data according to the spliced ​​feature vectors corresponding to each of the key points in the hand image data.

2. The hand skeleton posture estimation and action recognition model training method according to claim 1, characterized in that: The attention coefficient learning layer includes: A similarity coefficient calculation unit, used to calculate the attention coefficient between each key point in the hand image data and its adjacent key points according to the feature vector corresponding to the hand image data currently input; A rectified linear unit, used for normalizing the attention coefficient between each of the key points in the hand image data and the respective adjacent key points, so as to obtain an attention coefficient matrix corresponding to the hand image data; An attention coefficient correction unit, used for correcting the attention coefficient matrix corresponding to the hand image data according to the current importance matrix between the key points of the hand image data, so as to obtain a corrected attention coefficient matrix corresponding to the hand image data; An attention mechanism processing unit is used to perform weighted summation of the feature vectors of each of the key points in the hand image data and their adjacent key points based on the modified attention coefficient matrix, so as to obtain the feature vector corresponding to the hand image data after being processed by the attention mechanism.

3. The hand skeleton posture estimation and action recognition model training method according to claim 1, characterized in that: The method of using the two-dimensional coordinate set estimation results of the key points corresponding to each of the hand image data and the three-dimensional coordinate set of the key points to train the medical image segmentation model includes: Based on the estimation results of the two-dimensional coordinate sets of key points corresponding to each of the hand image data and the three-dimensional coordinate sets of key points, a preset medical image segmentation model is trained with a preset combined loss function; The combined loss function is composed of the mean square error loss of the two-dimensional coordinates of the key points and the three-dimensional coordinates of the key points and the cross entropy loss of the hand movement type recognition result.

4. The hand skeleton posture estimation and action recognition model training method according to claim 1, characterized in that: The action recognition network includes: A spatiotemporal graph convolutional network, used for performing feature extraction on a three-dimensional coordinate set estimation result of key points of the hand image data currently input, so as to obtain spatiotemporal features corresponding to the hand image data; A soft attention layer, used to calculate an attention coefficient of the hand image data, and weight the spatiotemporal features based on the attention coefficient to obtain weighted features corresponding to the hand image data; A two-dimensional average pooling layer, used for performing two-dimensional average pooling processing on the weighted features corresponding to the hand image data to obtain pooling features corresponding to the hand image data; A two-dimensional convolutional layer, used to obtain a category probability feature corresponding to the hand image data according to the pooled feature recognition corresponding to the hand image data; The normalization layer is used to obtain the probability of each hand movement corresponding to the hand image data by normalizing the category probability features corresponding to the hand image data, so as to output the hand movement type recognition result of the hand image data.

5. The hand skeleton posture estimation and action recognition model training method according to any one of claims 1 to 4, characterized in that: The method of respectively acquiring a plurality of continuous hand image data corresponding to a plurality of hand motion types, and a hand motion type label, a key point two-dimensional coordinate set, and a key point three-dimensional coordinate set corresponding to each of the hand image data, comprises: Acquire multiple continuous real hand images corresponding to multiple types of hand movements; Performing dimension preprocessing on each of the real hand images respectively to obtain hand image data with the same dimension; Receive the hand action type label, key point two-dimensional coordinate set and key point three-dimensional coordinate set corresponding to each of the hand image data, wherein each key point in the key point two-dimensional coordinate set and the key point three-dimensional coordinate set includes multiple hand key points and multiple object corner points.

6. A hand skeleton posture estimation and action recognition method, characterized in that: include: Acquire target hand image data; Inputting the target hand image data into a preset two-dimensional hand skeleton posture estimation model, so that the two-dimensional hand skeleton posture estimation model outputs an estimation result of a two-dimensional coordinate set of key points corresponding to the target hand image data; Inputting the estimation results of the two-dimensional coordinate set of key points corresponding to the target hand image data into a preset three-dimensional hand skeleton posture estimation model and a hand action recognition model respectively, so that the three-dimensional hand skeleton posture estimation model outputs the estimation results of the three-dimensional coordinate set of key points corresponding to the target hand image data, and the hand action recognition model outputs the hand action type recognition results corresponding to the target hand image data; The two-dimensional hand skeleton posture estimation model, the three-dimensional hand skeleton posture estimation model and the hand motion recognition model are pre-trained based on the hand skeleton posture estimation and motion recognition model training method according to any one of claims 1 to 5.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the hand skeleton posture estimation and action recognition model training method as described in any one of claims 1 to 5, and / or implements the hand skeleton posture estimation and action recognition method as described in claim 6.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the hand skeleton posture estimation and action recognition model training method as described in any one of claims 1 to 5, and / or implements the hand skeleton posture estimation and action recognition method as described in claim 6.

Images