A method for constructing a sports behavior knowledge graph using PAMS sports coding

Abstract

The present invention provides a method for constructing a motion behavior knowledge graph using PAMS motion coding, which is characterized by: (1) using the sample actions of the PAMs behavior sample open platform (www.pams.cn) as a template, a hierarchical construction model is adopted for the motion behaviors occurring in real scenes, and the key points of the human skeleton based on deep learning are converted into action units (Pose Unit, PU) of corresponding different levels of the behavior sample open platform; (2) a deep neural coding network is further used to learn and simultaneously output the action types of action units of different levels in a hierarchical manner, and the knowledge graph attribute relationships of action units of different levels are constructed from top to bottom.

Description

1. Technical Field

[0001] Computer vision, artificial intelligence, knowledge graph, sports action recognition 2. Background Technology

[0002] 2.1 Introduction to general technical methods

[0003] Long Short-Term Memory (LSTM) is a time recurrent neural network that is specially designed to solve the long-term dependency problem of general recurrent neural networks. All recurrent neural networks have a chain form of repeated neural network modules.

[0004] Convolution operation is an important module of deep convolutional neural network. Its essence is a weight-sharing node similar to the output function of a neuron, which is used to extract features. Among them, one-dimensional convolution is a convolution operation that only performs convolution on one dimension of the feature.

[0005] Knowledge Graph is an important branch technology of artificial intelligence. It is a structured semantic knowledge base used to describe concepts and their relationships in the physical world in symbolic form. Its basic components are "entity-relationship-entity" triples, as well as entities and their related attribute-value pairs. Entities are connected to each other through relationships to form a network knowledge structure.

[0006] Human skeleton key point detection is a research task in the field of computer vision. It usually uses deep learning-based methods to predict the positions of pre-defined human key points. The method of human skeleton key point detection belongs to regression model.

[0007] 2.2 Introduction to Similar Methods

[0008] A generalized application system and method based on the hierarchical model of individual motion behavior (application number: 202110178598.3)[1] constructed a multi-level human motion knowledge graph.

[0009] Compared with the above, the differences of the present application are as follows: (1) in the human action definition stage, the present application first uses a human skeleton key point detection method based on deep learning to extract the human skeleton key points, and then converts them into posture units (PUs) of similar methods based on the spatial position relationship between the key points; (2) the present method can realize multi-level recognition of actions based on a series of continuous inputs, while the document [1] first recognizes posture based on a single image and then recognizes actions using a rule-based method; (3) the definition of action labels of the present method adopts the deep neural network model proposed in the present application (such as Figure 3 (4) This method uses visual methods, while the literature [1] uses motion capture methods such as sensors. III. Summary of the invention

[0010] The present invention provides a method for constructing a motion behavior knowledge graph using PAMS motion coding, which is characterized by: (1) using the sample actions of the PAMs behavior sample open platform (www..pams.cn) as a template, a hierarchical construction model is adopted for the motion behaviors occurring in real scenes, and the key points of the human skeleton based on deep learning are converted into action units (Pose Unit, PU) of corresponding different levels of the behavior sample open platform; (2) a deep neural coding network is further used to learn and simultaneously output the action types of action units of different levels in a hierarchical manner, and the knowledge graph attribute relationships of action units of different levels are constructed from top to bottom. IV. Description of the drawings

[0011] Figure 1 This is an example of the serial numbers of the key points of the human skeleton identified by this patent and the display examples of the key point positions on a real person.

[0012] Figure 2 It is the sequence number of the key points of the gesture recognized by this patent.

[0013] Figure 3 This is the architecture diagram of the action label recognition process of this patent. In the figure, the PU1-PU5 encoding vectors are respectively constructed using the spatial position relationship between the key points of the human skeleton. The definition is shown in Table 1 of this document; the one-dimensional convolution is used to extract the features of the PU1-PU5 encoding vectors, and the LSTM module is used to extract the timing information of the feature vectors, that is, the action features of the video clips; the "feature merging" operation of "+" in the figure is used to merge the PU features of the previous level with the PU features of this level to realize the process of coarse-to-fine recognition. The multiple levels of labels finally output by the model are classification models.

[0014] Figure 4 exist Figure 3 Based on the results, an example of using knowledge graph to construct multi-level action labels is shown. PU1-PU5 represents the category code of each level of action labels. V. Specific implementation methods

[0015] This application achieves this through 4 steps.

[0016] Step 1: 3D human posture recognition

[0017] Using the method in reference [3], we extract Figure 1 , Figure 2 The spatial coordinates of the human skeleton key points, gesture joints, and big toe joints are shown.

[0018] Step 2: PU coding vector extraction

[0019] According to the correspondence between PU classification and skeleton key points shown in Table 1, the spatial vector of the connection relationship between the skeleton key points corresponding to the action posture (PPU) of different levels in each frame of the video clip is calculated, and all connections are merged into feature vectors according to the order of the third column of Table 1. The serial numbers in the third column of Table 1 correspond to Figure 1 The sequence number of the key points in . When calculating 4-right hand joint and 5-left hand joint, Figure 1 Middle joints 4 and 5 are respectively Figure 2 The 0 point of the hand posture is further calculated to obtain the feature vectors of the adjacent connected hand postures.

[0020] Table 1 Correspondence between PU classification and skeleton key points

[0021]

[0022] Step 3: Multi-level action label generation

[0023] like Figure 3 As shown in the figure, a deep neural network is used to extract the features of the PU coding vectors at all levels generated by the video clips in the order of the video frames through blindly rolling up and then input into the single-layer LSTM module. The "feature merging" operation of "+" in the figure is used to merge the PU features of the previous level with the PU features of the current level to achieve the process of coarse-to-fine recognition. The model finally outputs multiple levels of labels.

[0024] Step 4: Knowledge graph construction

[0025] According to the top-down hierarchical structure from PU1 to PU5, a tree-like action hierarchical knowledge graph label is established for each action. The effect is as follows: Figure 4 As shown in Figure 2, based on the classification system of experts and literature [2], non-subordinate connections are made between similar PU1-level actions.

[0026] References:

[0027] [1] China invention patent: Generalized application system and method based on individual movement behavior hierarchical model. Application number: 202110178598.3

[0028] [2] Behavior Sample Open Platform www.pams.cn

[0029] [3] Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, Cewu Lu: HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation. CVPR 2021: 3383-3393

Claims

1. A method for constructing a sports behavior knowledge graph using PAMS sports coding, characterized in that: (1) Using the sample actions of the pams behavior sample open platform as a template, a hierarchical construction model is adopted for the movement behaviors occurring in real scenes. The PU coding vectors of the human skeleton key points based on deep learning are extracted and converted into the corresponding action units PU of different levels of the behavior sample open platform; the PU1-PU5 coding vectors are respectively constructed by the spatial position relationship between the key points of the human skeleton; (2) A deep neural network is further used to output the action types of action units of different levels in a hierarchical manner, and the knowledge graph attribute relationship of action units of different levels is constructed from top to bottom; The method for constructing a sports behavior knowledge graph using PAMS sports coding includes: an LSTM module, and the operation steps are: Step 1: 3D human posture recognition; extracting the spatial coordinates of human skeleton key points, gesture key points and big toe joint points; Step 2: Extract PU coding vectors; According to the correspondence between PU classification and skeleton key points, calculate the spatial vectors of the skeleton key point connection relationship corresponding to the action postures at different levels of each frame in the video clip of the real scene, and merge all the connections into a feature vector in the order of the skeleton key point connection relationship; Step 3: Multi-level action label generation: Using a deep neural network, the PU coding vectors of each level generated by the video clips of the real scene are extracted by one-dimensional convolution according to the input order of the video frames, and then input into a single-layer LSTM module to obtain the hierarchical labels of the corresponding levels; the LSTM module is used to extract the time sequence information of the feature vector; Step 4: Knowledge graph construction; according to the top-down hierarchical structure of the category codes PU1-PU5 of the action labels at each level corresponding to the PU coding vector, a tree-like action hierarchical knowledge graph label is established for each action, and subordinate hierarchical connections or non-subordinate connections are used between the action hierarchical knowledge graph labels.

Images