Human motion feature extraction method for motion quality evaluation

Abstract

The invention relates to a human motion feature extraction method for motion quality evaluation, and belongs to the field of image processing. The method comprises the following steps: acquiring nodes for human body action quality evaluation, and extracting preliminary features for each node; in a self-attention layer in the heterogeneous graph neural network model based on the double-layer attention mechanism, projecting the preliminarily extracted node features of different node types to the same space through matrix transformation; aggregating the feature information of neighbor nodes by adopting a self-attention aggregation method based on a meta-path learning node neighbor weight to obtain node embedding with specific semantic information; the importance of different meta-paths is distinguished by applying semantic-level attention, an optimal weighted combination of specific semantic node embedding for specific tasks is obtained, and final embedding of nodes is obtained. According to the method, the complexity of the heterogeneous graph can be effectively processed, and the structure and semantic information in the heterogeneous graph can be better captured.

Description

Technical Field

[0001] The invention belongs to the field of image processing and relates to a method for extracting human motion features for motion quality assessment. Background Art

[0002] As an extension of human behavior recognition, action quality assessment (AQA) has also become a new hot topic in human behavior understanding research and has been widely used in the industry, such as rehabilitation treatment for patients, automatic scoring of sports, skill determination of robot automation services, and level assessment of specific skills. Action quality assessment (or action evaluation) refers to the evaluation of the action quality of human objects in the video, such as calculating action quality scores, levels, or the pros and cons of different human performances. It is an important direction in video understanding and computer vision research.

[0003] Graph data analysis has attracted widespread attention from academic and industrial researchers in recent years. In the field of motion quality assessment, more and more studies tend to use graph data to analyze human motion characteristics. In order to better analyze, most works model graph data into homogeneous graphs, that is, graphs only contain objects and links of the same type, such as author collaboration networks and friend circles. Homogeneous graph modeling methods often only model part of the information in the actual interactive system, or do not distinguish the heterogeneity of objects and their relationships, resulting in irreversible information loss. In recent years, researchers have modeled multi-type and interconnected graph data into heterogeneous graphs to achieve a more complete and natural abstraction of the real world. For example, literature data contains different types of objects such as authors, papers, and conferences. There are many types of relationships between these objects: the writing / being written relationship between authors and papers, the publishing / being published relationship between conferences and papers, and the joints in the human skeleton that carry different types of information. Using heterogeneous graphs to model this type of rich and complex interactive data can retain more comprehensive semantic and structural information.

[0004] A heterogeneous graph, denoted as G = (V, E), consists of an object set V and a link set E. Heterogeneous graphs are also associated with node-type mapping functions Φ:V→A and link-type mapping functions ψ:E→R. A and R represent sets of predefined object types and link types, where |A|+|R|>2. The complexity and richness of heterogeneous graphs make them perform well in many data mining tasks, such as node classification, link prediction, and community discovery. However, how to fully utilize and mine this heterogeneous information remains a challenge. To this end, researchers have proposed a variety of methods to capture and utilize the rich semantics in heterogeneous graphs.

[0005] A commonly used method is the mining technology based on meta-path. Meta-path defines the connection pattern between different types of nodes in a heterogeneous graph. In this way, meta-path can not only help identify direct connections between objects, but also capture more complex indirect relationships. PathSim proposed a similarity calculation method based on meta-path. It evaluates the similarity between objects by considering the semantics contained in different meta-paths, thereby mining more meaningful relationships in heterogeneous graphs. PathPredict proposed another method that uses meta-paths to extract features and trains a logistic regression model for prediction. This method can predict possible future relationships based on existing heterogeneous graph data, which has been widely used in academia and industry.

[0006] In addition to meta-paths, Graph Neural Networks (GNNs) have also made significant progress in heterogeneous graph modeling. Heterogeneous Graph Neural Networks (HGNNs) can learn the feature representations of different types of nodes and edges in the graph, thereby capturing more structural and semantic information. Such methods include HAN (Heterogeneous Attention Network) and RGCN (Relational Graph Convolutional Network), which can more effectively aggregate heterogeneous information and perform tasks such as node classification and link prediction by introducing attention mechanisms and relational convolutions.

[0007] Meta-paths is a conceptual framework for describing the relationships between nodes in a heterogeneous graph. Specifically, a meta-path consists of a series of node types connected in a specific order. Figure 1 Take the literature network graph data shown in the figure as an example, where Figure 1 (a) represents a heterogeneous graph, Figure 1 (b) represents the meta-path, which contains three types of nodes: paper, author, and conference. The literature network graph data contains different types of objects such as authors, papers, and conferences. There are many types of relationships between these objects. The meta-path author-paper-author (APA) reflects the cooperation between two authors, and the meta-path paper-conference-paper (PCP) indicates that two papers are included in the conference together. It can be seen that depending on the different meta-paths, the relationship between nodes in the heterogeneous graph can have different semantics. Due to the complexity of heterogeneous graphs, traditional graph neural networks cannot be directly applied to heterogeneous graphs. Summary of the invention

[0008] In view of this, the purpose of the present invention is to provide a method for extracting human motion features for motion quality assessment, aiming to solve the challenges brought by the diversity of node and link types in heterogeneous graphs, and propose a novel feature extraction method for human skeleton heterogeneous graphs suitable for human motion quality assessment. Effectively handle the complexity of heterogeneous graphs and better capture the structural and semantic information in heterogeneous graphs. Provide a graph neural network architecture that can be directly applied to heterogeneous graphs and achieve superior performance in heterogeneous graph analysis tasks. Provide a new idea and method for research and application in the field of graph neural networks, and provide a brand-new solution for processing heterogeneous graph data, which is expected to promote the development and progress of the field of heterogeneous graph analysis.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] A method for extracting human motion features for motion quality assessment comprises the following steps:

[0011] S1, obtain nodes for human motion quality assessment and extract preliminary features for each node;

[0012] S2. In the self-attention layer of the heterogeneous graph neural network model based on the double-layer attention mechanism, the node features of different node types initially extracted are projected into the same space through matrix transformation;

[0013] S3, using the self-attention aggregation method based on meta-path learning node neighbor weights to aggregate the feature information of neighbor nodes and obtain node embedding with specific semantic information;

[0014] S4. Apply semantic-level attention to distinguish the importance of different meta-paths and obtain the optimal weighted combination of specific semantic node embeddings for a specific task to obtain the final embedding of the node.

[0015] Furthermore, in step S1, the process of obtaining the preliminary features is as follows:

[0016] Obtaining original node data of the skeleton sequence through the posture estimation model, the original nodes at least include: head, shoulder, elbow, hip, knee, ankle;

[0017] A multi-stage reasoning structure is used to predict the confidence of the skeleton joints and the partial affinity field in the image respectively, where each stage of the multi-stage reasoning structure contains two parallel convolutional network branches;

[0018] After obtaining the skeleton points, calculate the spatial coordinates, angle, velocity and acceleration characteristics of the joint points;

[0019] Record the three-dimensional coordinate data of each joint point in space, calculate the distance between adjacent joint points, and obtain the overall position information of the action; at the same time, construct the joint point vector and calculate the angle characteristics of the key joints.

[0020] Further, in step S2, the designed heterogeneous graph neural network model based on the dual-layer attention mechanism includes at least a self-attention layer and a semantic-level attention layer, and uses self-attention and semantic-level attention for hierarchical aggregation respectively;

[0021] When projecting different types of nodes into the same dimension, the query vector q, key vector k and value vector v of the self-attention layer are defined, where the query vector q and key vector k are used to calculate the attention weights, and the value vector v represents the feature information of the neighboring nodes. q , W k , W v The initial features of each node are projected into the same dimension. The process is expressed as:

[0022] q i =W Q h i ,k i =W K h i ,v i =W V h i ,i∈V

[0023] In the formula, h i represents the initial features of node i, W q , W k , W v A trainable transformation matrix shared by all neighboring nodes.

[0024] Further, in step S3, the self-attention aggregation method for learning node neighbor weights based on meta-path includes the following steps:

[0025] S31, using the dot product method to calculate the similarity between the query vector q of the central node and the key vector k of each adjacent node;

[0026] S32, obtaining the attention weight distribution of the nodes based on the meta-path neighbors according to the similarity between the center point and the adjacent nodes;

[0027] S33, based on the value vector v of the neighbor node j of node i j The corresponding attention weight score The weighted sum of is used to get the node embedding of node i.

[0028] Further, in step S31, given a node pair (i, j) connected based on the meta-path Φ, by calculating the self-attention layer coefficient To indicate the importance of node j to node i, the self-attention layer coefficient The calculation method is:

[0029]

[0030] In the formula, q i represents the query vector of node i, k j represents the key vector of the adjacent node j, represents the dot product function; among them, the self-attention layer coefficient The larger the value, the stronger the correlation between the two nodes.

[0031] Further, in step S32, according to the self-attention layer coefficient Calculate the self-attention weight score of node pair (i, j) by Represents the neighbor set of node i including itself based on meta-path Φ, and the importance score of node pair (i, j) through softmax function Normalize and get the self-attention layer weight score of node pair (i, j)

[0032]

[0033] In the formula, represents the attention weight ratio of node j among all neighbors of node i, and σ represents the activation function.

[0034] Further, in step S33, according to the value vector v of the neighbor node j of node i j The corresponding attention weight score The learned embedding of node i based on the meta-path Φ is calculated by the weighted sum of

[0035] First, retain the original feature information h of the central node i i :

[0036]

[0037] In the formula, represents the embedding learned for node i based on the meta-path Φ;

[0038] The self-attention layer is expanded to multi-head attention. The self-attention layer is replicated k times and the learned embeddings are connected to obtain node embeddings containing specific semantic information:

[0039]

[0040] Π represents concatenation, β is a trainable parameter; given a set of meta-paths {Φ0,Φ1...,Φp}, then the node embedding containing p specific semantics is expressed as

[0041] Further, in step S4, the nodes are embedded into Aggregate together to get the final embedding Z of node i i :Suppose there is an m-th element path Φ m Generated semantically specific embeddings The polymerization process includes:

[0042] S41, yes Perform a nonlinear transformation, calculate the similarity between the transformed embedding and the semantic-level attention vector Q, and then average all semantic embeddings to obtain the meta-path Φ m Importance The process is expressed as:

[0043]

[0044] Where V represents the set of target nodes, W and b are both learnable parameters, Q represents the semantic-level attention vector, and the parameters are shared among all meta-paths and semantic embeddings;

[0045] S42, using softmax function Normalize and generate weights

[0046]

[0047] Meta-path weight β Φm represents the mth element path Φ m Importance to the target node;

[0048] S43. Calculate the weighted sum of all semantic level embeddings to obtain the final embedding Z of node i i :

[0049]

[0050] The final embedding Z of node i is obtained i as node features and used for action quality assessment.

[0051] The beneficial effects of the present invention are:

[0052] In the method for extracting human motion features for action quality assessment proposed in the present invention, a semi-supervised heterogeneous graph neural network (DA-HGNN) based on a heterogeneous graph element path method and a double-layer attention mechanism is designed. The DA-HGNN of the present invention can accurately capture the importance of adjacent nodes and different element paths through a self-attention layer and a semantic-level attention layer, thereby generating more accurate and meaningful node embeddings. This feature enables more detailed capture and analysis of human motion features in action quality assessment, thereby improving the accuracy and reliability of the assessment.

[0053] The method of the present invention not only improves the expressiveness of node representation, but also reduces information loss and retains rich semantic information. This means that in the process of motion quality assessment, the human body motion state can be more comprehensively understood and described, providing more powerful support for subsequent evaluation and analysis.

[0054] The DA-HGNN of the present invention can be trained with a small amount of labeled data and a large amount of unlabeled data under the semi-supervised learning framework, which significantly improves the generalization ability and accuracy of the model. This makes the method more adaptable and robust when dealing with different scenes and different objects in action quality assessment.

[0055] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0057] Figure 1 is an example of an existing heterogeneous graph, where Figure 1 (a) represents a heterogeneous graph, Figure 1 (b) represents the meta-path;

[0058] Figure 2 Schematic diagram of the structure of the self-attention layer of the present invention;

[0059] Figure 3 Schematic diagram of the structure of the semantic level attention layer of the present invention. DETAILED DESCRIPTION

[0060] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0061] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0062] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0063] See also Figure 2-3 , which is a human motion feature extraction method for action quality assessment.

[0064] Graph Attention Network (GAT) is a graph neural network model based on the attention mechanism, which is designed to effectively handle the complex relationships between nodes in graph data. GAT dynamically aggregates the information of neighboring nodes by introducing attention weights, thereby learning and updating the node representation. However, in heterogeneous graphs, there are multiple relationships between different types of nodes, and the ordinary GAT model cannot effectively handle this diversity. Due to the complexity and diversity of heterogeneous graph data, traditional homogeneous graph models cannot fully mine the structural and semantic information in heterogeneous graphs, so it is necessary to design a customized model based on the characteristics of heterogeneous graph data.

[0065] To address the above problems, the present invention proposes a self-attention aggregation method based on meta-path learning of node neighbor weights, which aggregates the feature information of neighbor nodes to obtain node embeddings with specific semantic information. Then, semantic-level attention is applied to distinguish the importance of different meta-paths to obtain the optimal weighted combination of specific semantic node embeddings tailored for specific tasks.

[0066] Example

[0067] This embodiment proposes a method for extracting human motion features for motion quality assessment, which includes the following steps:

[0068] S1, obtain nodes for human motion quality assessment and extract preliminary features for each node;

[0069] S2. In the self-attention layer of the heterogeneous graph neural network model based on the double-layer attention mechanism, the node features of different node types initially extracted are projected into the same space through matrix transformation;

[0070] S3, using the self-attention aggregation method based on meta-path learning node neighbor weights to aggregate the feature information of neighbor nodes and obtain node embedding with specific semantic information;

[0071] S4. Apply semantic-level attention to distinguish the importance of different meta-paths and obtain the optimal weighted combination of specific semantic node embeddings for a specific task to obtain the final embedding of the node.

[0072] In step S1 of this embodiment, for preliminary features, the original node data of the skeleton sequence is obtained based on the OpenPose posture estimation model, including but not limited to the head, shoulders, elbows, hips, knees, ankles and other parts, which can accurately reflect the posture and limb movement of the human body. It mainly adopts a multi-stage reasoning structure, each stage contains two parallel convolutional network branches, which respectively predict the confidence of the skeleton joint points and the partial affinity domain field in the image. After obtaining the skeleton points, the spatial coordinates, angles, speeds and acceleration features of the joint points are calculated.

[0073] Then, the three-dimensional coordinate data of each joint point in space is recorded, and the distance between adjacent joint points is calculated to obtain the overall position information of the action. At the same time, the joint point vector is constructed to calculate the angle characteristics of key joints (such as shoulders, elbows, knees, etc.).

[0074] In step S2 of this embodiment, a heterogeneous graph neural network model (DA-HGNN) based on a dual-layer attention mechanism is designed using the meta-path method. The model includes at least a self-attention layer and a semantic-level attention layer, and uses self-attention and semantic-level attention for hierarchical aggregation respectively. Specifically, it first projects the given node features of different node types into the same space through matrix transformation for subsequent neighbor feature aggregation. Self-attention aggregation calculates the importance weights of different adjacent nodes according to the similarity of node features, and performs weighted aggregation to better capture the local relationships between nodes. Semantic-level attention helps the model adaptively aggregate different semantically related information between nodes, thereby more effectively capturing the global relationships between nodes. Through hierarchical aggregation, the model generates node embeddings based on meta-paths and can be applied to heterogeneous graph data.

[0075] When generating node embeddings based on a specific meta-path, it can be observed that the importance of each neighbor is different, and their importance in node embeddings varies in different tasks. Therefore, a self-attention layer is introduced to learn the importance of different neighbors of a node under the same meta-path by generating self-attention coefficients. These coefficients are used to combine the features of different neighbors according to their respective weights to further generate node embeddings.

[0076] Specifically, in step S2, since different types of nodes in the heterogeneous graph belong to different feature spaces, it is necessary to perform feature transformation first to project different types of nodes into the same dimension. In the self-attention layer, the query vector, key vector, and value vector are first defined, denoted as q, k, and v, respectively. Here, q and k are usually used to calculate the attention weights, while v represents the feature information of neighboring nodes. These three types of vectors are obtained by projecting the transformed node features. The projection process is performed using three transformation matrices to project the features of each node into the same dimensions of q, k, and v, as shown below:

[0077] q i =W Q h i ,k i =W K h i ,v i =W V h i ,i∈V

[0078] In the above formula, h i represents the initial features of node i, W q , W k , W v are trainable parameters shared by all neighboring nodes. Through the projection transformation, these parameters can handle any type of node in the dataset.

[0079] In step S3, the self-attention aggregation method for learning node neighbor weights based on meta-path includes the following steps:

[0080] S31, using the dot product method to calculate the similarity between the q-value of the central node and the k-value of each adjacent node;

[0081] S32, obtaining the attention weight distribution of the nodes based on the meta-path neighbors according to the similarity between the center point and the adjacent nodes;

[0082] S33, based on the value vector v of the neighbor node j of node i j The corresponding attention weight score The weighted sum of is used to get the node embedding of node i.

[0083] In step S31, given a node pair (i, j) connected based on the meta-path Φ, the self-attention layer coefficient can be calculated by To obtain the importance of node j to node i. Self-attention layer coefficient The calculation method is as follows:

[0084]

[0085] where q i represents the query vector of node i, k j represents the key vector of the adjacent node j, represents the dot product function, and calculates q i With k j The similarity between them generates the attention coefficient The importance of the node between nodes i and j is measured by using the dot product of the projected features, where a larger dot product indicates a stronger correlation between the two nodes. Therefore, according to the above formula, the importance of a node pair (i, j) based on the meta-path Φ depends on their feature similarity.

[0086] In step S32, by using The self-attention weight score of the node pair (i, j) can be obtained as Represents the attention weight ratio of node j among all neighbors of node i. For example, Represents the neighbor set of node i based on the meta-path Φ (including itself), in obtaining the importance score of the node pair (i, j) After that, it is normalized by the softmax function to obtain the self-attention layer weight score of the node pair (i, j)

[0087]

[0088] where σ represents the activation function and · represents the dot product function. It can be seen from the formula that the weight score generated for the node pair (i, j) also depends on their feature similarity.

[0089] In step S33, the learned embedding of node i based on the meta-path Φ can be calculated as the value vector v of neighbor node j j The corresponding attention weight score In order to achieve better aggregation effect, the original feature information h of the central node i is retained. i The calculation method is as follows:

[0090]

[0091] in Denotes the embedding learned for node i based on the meta-path Φ. Each node embedding is generated by aggregation with its neighbors and itself. In order to solve the problem of high variance in graph data, the self-attention layer is expanded to multi-head attention, making the training process more stable. Specifically, the self-attention layer is replicated k times and the learned embeddings are concatenated to form embeddings specific to certain semantics.

[0092]

[0093] The above formula shows the obtained value on a specific meta-path Φ based on the multi-head attention mechanism. The splicing process of , where || represents splicing and β represents a trainable parameter. are generated for a specific meta-path Φ, so they represent specific semantic information and can only capture one type of semantic information. p}, the embedding of node i under specific semantics is first learned through the self-attention layer to obtain node embedding with p specific semantics Subsequently, multiple semantic information is aggregated through a semantic-level attention layer.

[0094] In step S4, the semantic level attention layer is a method commonly used to aggregate the semantic information of different meta-paths in heterogeneous graphs. Heterogeneous graphs contain complex semantic information. In order to enrich the learned semantics of the target node, the learned embedding Aggregate together to get the final embedding Z of node i i . Take the mth element path Φ m Generated semantically specific embeddings For example, the specific process includes:

[0095] S41. First of all Perform a nonlinear transformation, calculate the similarity between the transformed embedding and the semantic-level attention vector Q, and then average all semantic embeddings to obtain the meta-path Φ m Importance The process is expressed as:

[0096]

[0097] Where V represents the set of target nodes, W and b are learnable parameters, and Q is the semantic-level attention vector. The above parameters are shared in all meta-paths and semantic embeddings.

[0098] S42, apply the softmax function to Normalize and generate weights The calculation formula is as follows:

[0099]

[0100] Meta-path weight β Φm represents the mth element path Φ m Importance of the target node.

[0101] S43. Calculate the weighted sum of all semantic level embeddings to obtain the final embedding Z i .

[0102]

[0103] Z, which contains all semantic information i The information contained in the heterogeneous graph is applied to different tasks to test the performance of the model, such as node classification, node clustering, etc. Different loss functions are designed for different graph tasks. For example, in the semi-supervised node classification task, the cross entropy loss function is applied to minimize the difference between the true value and the predicted value of all labeled nodes.

[0104] The semi-supervised heterogeneous graph neural network (DA-HGNN) based on the heterogeneous graph meta-path method and the double-layer attention mechanism proposed in the present invention has the following beneficial effects: through the self-attention layer and the semantic-level attention layer, DA-HGNN can accurately capture the importance of adjacent nodes and different meta-paths, thereby generating more accurate and meaningful node embeddings. This not only improves the expressiveness of node representation, but also reduces information loss and retains rich semantic information. In addition, under the semi-supervised learning framework, DA-HGNN is trained with a small amount of labeled data and a large amount of unlabeled data, which significantly improves the generalization ability and accuracy of the model. The method performs well in tasks such as node classification, link prediction, and community discovery, showing strong robustness and adaptability, and is suitable for a wide range of application scenarios, such as academic networks, social networks, and recommendation systems.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for extracting human motion features for action quality assessment, characterized in that: The method comprises the following steps: S1, obtain nodes for human motion quality assessment and extract preliminary features for each node; S2. In the self-attention layer of the heterogeneous graph neural network model based on the double-layer attention mechanism, the node features of different node types initially extracted are projected into the same space through matrix transformation; S3, using the self-attention aggregation method based on meta-path learning node neighbor weights to aggregate the feature information of neighbor nodes and obtain node embedding with specific semantic information; S4. Apply semantic-level attention to distinguish the importance of different meta-paths and obtain the optimal weighted combination of specific semantic node embeddings for a specific task to obtain the final embedding of the node.

2. The method for extracting human motion features for action quality assessment according to claim 1, characterized in that: In step S1, the process of obtaining preliminary features is as follows: Obtaining original node data of the skeleton sequence through the posture estimation model, the original nodes at least include: head, shoulder, elbow, hip, knee, ankle; A multi-stage reasoning structure is used to predict the confidence of the skeleton joints and the partial affinity field in the image respectively, where each stage of the multi-stage reasoning structure contains two parallel convolutional network branches; After obtaining the skeleton points, calculate the spatial coordinates, angle, velocity and acceleration characteristics of the joint points; Record the three-dimensional coordinate data of each joint point in space, calculate the distance between adjacent joint points, and obtain the overall position information of the action; at the same time, construct the joint point vector and calculate the angle characteristics of the key joints.

3. The method for extracting human motion features for action quality assessment according to claim 2, characterized in that: In step S2, the designed heterogeneous graph neural network model based on the dual-layer attention mechanism includes at least a self-attention layer and a semantic-level attention layer, and uses self-attention and semantic-level attention for hierarchical aggregation respectively; When projecting different types of nodes into the same dimension, the query vector q, key vector k and value vector v of the self-attention layer are defined, where the query vector q and key vector k are used to calculate the attention weights, and the value vector v represents the feature information of the neighboring nodes. q , W k , W v The initial features of each node are projected into the same dimension. The process is expressed as: q i =W Q h i ,k i =W K h i ,v i =W V h i ,i∈V In the formula, h i represents the initial features of node i, W q , W k , W v A trainable transformation matrix shared by all neighboring nodes.

4. The method for extracting human motion features for action quality assessment according to claim 3, characterized in that: In step S3, the self-attention aggregation method for learning node neighbor weights based on meta-path includes the following steps: S31, using the dot product method to calculate the similarity between the query vector q of the central node and the key vector k of each adjacent node; S32, obtaining the attention weight distribution of the nodes based on the meta-path neighbors according to the similarity between the center point and the adjacent nodes; S33, based on the value vector v of the neighbor node j of node i. j The corresponding attention weight score The weighted sum of is used to get the node embedding of node i.

5. The method for extracting human motion features for action quality assessment according to claim 4, characterized in that: In step S31, given a node pair (i, j) connected based on the meta-path Φ, the self-attention layer coefficient is calculated To indicate the importance of node j to node i, the self-attention layer coefficient The calculation method is: In the formula, q i represents the query vector of node i, k j represents the key vector of the adjacent node j, represents the dot product function; among them, the self-attention layer coefficient The larger the value, the stronger the correlation between the two nodes.

6. A method for extracting human motion features for action quality assessment according to claim 5, characterized in that: In step S32, according to the self-attention layer coefficient Calculate the self-attention weight score of node pair (i, j) by Represents the neighbor set of node i including itself based on meta-path Φ, and the importance score of node pair (i, j) through softmax function Normalize and get the self-attention layer weight score of node pair (i, j) In the formula, represents the attention weight ratio of node j among all neighbors of node i, and σ represents the activation function.

7. A method for extracting human motion features for action quality assessment according to claim 6, characterized in that: In step S33, according to the value vector v of the neighbor node j of node i j The corresponding attention weight score The learned embedding of node i based on the meta-path Φ is calculated by the weighted sum of First, retain the original feature information h of the central node i i : In the formula, represents the embedding learned for node i based on the meta-path Φ; The self-attention layer is expanded to multi-head attention. The self-attention layer is replicated k times and the learned embeddings are connected to obtain node embeddings containing specific semantic information: Π represents concatenation, β is a trainable parameter; given a set of meta-paths {Φ0,Φ1...,Φ p }, then the node embedding containing p specific semantics is expressed as 8. A method for extracting human motion features for motion quality assessment according to claim 7, characterized in that: In step S4, the nodes are embedded into Aggregate together to get the final embedding Z of node i i :Suppose there is an m-th element path Φ m Generated semantically specific embeddings The polymerization process includes: S41, yes Perform a nonlinear transformation, calculate the similarity between the transformed embedding and the semantic-level attention vector Q, and then average all semantic embeddings to obtain the meta-path Φ m Importance The process is expressed as: Where V represents the set of target nodes, W and b are both learnable parameters, Q represents the semantic-level attention vector, and the parameters are shared among all meta-paths and semantic embeddings; S42, using softmax function Normalize and generate weights Meta-path weight β Φm represents the mth element path Φ m Importance to the target node; S43. Calculate the weighted sum of all semantic level embeddings to obtain the final embedding Z of node i i : The final embedding Z of node i is obtained i as node features and used for action quality assessment.

Images