Human body action recognition method and device based on heterogeneous space-time diagram convolution

Abstract

The invention relates to a human body action recognition method and device based on heterogeneous space-time diagram convolution, and belongs to the technical field of computers, and the method comprises the steps: obtaining motion video data, and extracting a frame sequence from the motion video data based on a fixed time interval; extracting a joint target point and a target object of each video frame in the frame sequence, and constructing a first heterogeneous space-time diagram based on the joint target point and the target object; generating feature vectors for replacing shielded nodes according to the trained diffusion model and the shielded nodes in the first heterogeneous space-time diagram, and obtaining a second heterogeneous space-time diagram; and inputting the second heterogeneous space-time diagram into the graph neural network model to obtain a target semantic feature, and determining an action category based on the target semantic feature. According to the method, the accuracy and robustness of human body action recognition in a complex real scene (especially in the presence of shielding) are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computers, and particularly relates to a human action recognition method and device based on heterogeneous spatio-temporal graph convolution. BACKGROUND

[0002] In related technologies, current human action recognition methods based on deep learning mainly rely on the combination of convolutional neural networks (CNN) and recurrent neural networks (RNN). CNN is good at extracting spatial features from video frames, while RNN is used to model dynamic information in the time dimension. In recent years, three-dimensional convolution (3D CNN) and two-stream network (Two-Stream Network) have further improved the performance of action recognition. The former directly captures motion information in videos through spatio-temporal convolution kernels, and the latter realizes more comprehensive feature expression by fusing RGB and optical flow modalities. However, existing methods still have obvious limitations. Existing methods have high computational cost, require a large amount of training data, and are sensitive to dynamic backgrounds, which cannot handle problems such as changes in viewing angle and occlusion. Therefore, how to better realize human action recognition has become a problem to be solved. SUMMARY

[0003] In view of the deficiencies of the above prior art, the purpose of the application is to provide a human action recognition method and device based on heterogeneous spatio-temporal graph convolution.

[0004] In a first aspect of the application, a human action recognition method based on heterogeneous spatio-temporal graph convolution is proposed, comprising: S1, acquiring motion video data and extracting a frame sequence from the motion video data based on a fixed time interval; S2, extracting joint target points and target objects of each video frame in the frame sequence, and constructing a first heterogeneous spatio-temporal graph based on the joint target points and the target objects; S3, generating a feature vector that replaces an occluded node in the first heterogeneous spatio-temporal graph according to a trained diffusion model and the occluded node, and obtaining a second heterogeneous spatio-temporal graph; S4, inputting the second heterogeneous spatio-temporal graph into a graph neural network model to obtain target semantic features, and determining an action category based on the target semantic features.

[0005] Further, determining the graph nodes and edges of the first heterogeneous spatio-temporal graph comprises: determining the skeletal connection relationship of the joint target points; determining skeletal points within a preset threshold distance from the target object, and determining the connection relationship between the target object and the skeletal points; taking the joint target points and the target objects as the graph nodes of the first heterogeneous spatio-temporal graph; and taking the skeletal connection relationship and the connection relationship between the target object and the skeletal points as the edges of the first heterogeneous spatio-temporal graph.

[0006] Further, the diffusion model is trained by: obtaining a training sample, the training sample including a hidden node and a context of the hidden node; determining a noisy feature of the training sample at time t; inputting the noisy feature, the time t and conditional information into a diffusion decoder to predict noise added in the noisy feature; calculating a mean square error between the predicted noise added in the noisy feature and real added noise, and obtaining the diffusion model when the mean square error is less than a target threshold.

[0007] Further, according to the trained diffusion model and the hidden node in the first heterogeneous spatio-temporal graph, a feature vector replacing the hidden node is generated, and a second heterogeneous spatio-temporal graph is obtained, including: for the hidden node, extracting a spatio-temporal neighbor node of the hidden node from the first heterogeneous spatio-temporal graph; inputting features of the spatio-temporal neighbor node into the diffusion model as a condition to obtain a generated feature vector replacing the hidden node; adding the generated feature vector replacing the hidden node to the first heterogeneous spatio-temporal graph, and taking the repaired first heterogeneous spatio-temporal graph as the second heterogeneous spatio-temporal graph.

[0008] Further, the convolution layer in the graph neural network model includes a spatial graph convolution layer and a temporal graph convolution layer, wherein the second heterogeneous spatio-temporal graph is input into the graph neural network model to obtain a target semantic feature, including: inputting the second heterogeneous spatio-temporal graph into the spatial graph convolution layer to obtain a spatial convolution feature of a node in the second heterogeneous spatio-temporal graph; inputting the second heterogeneous spatio-temporal graph into the temporal graph convolution layer to obtain a temporal convolution feature of the node in the second heterogeneous spatio-temporal graph; and combining the spatial convolution feature of the node in the second heterogeneous spatio-temporal graph and the temporal convolution feature of the node in the second heterogeneous spatio-temporal graph to obtain the target semantic feature.

[0009] Further, inputting the second heterogeneous spatio-temporal graph into the spatial graph convolution layer to obtain a spatial convolution feature of a node in the second heterogeneous spatio-temporal graph includes: for a target node of the second heterogeneous spatio-temporal graph, generating linear transformation information for each neighbor node of the target node; aggregating information of linear transformation information of all the neighbor nodes based on a weighted summation calculation method to obtain aggregated information; combining the aggregated information with a feature of the target node, and inputting the combined information into a fully connected layer to obtain a spatial convolution feature of the target node; wherein inputting the second heterogeneous spatio-temporal graph into the temporal graph convolution layer to obtain a temporal convolution feature of a node in the second heterogeneous spatio-temporal graph includes: for a feature of the target node at time i of the second heterogeneous spatio-temporal graph, determining a feature of the target node; and aggregating information of the features of the target node based on a weighted summation calculation method to obtain a feature of the target node at time i. ​aggregating information of the target node in the second heterogeneous spatio-temporal graph, obtaining aggregated information; combining the aggregated information with the feature of the target node at time i, and inputting the combined information into a full connection layer to obtain a time convolution feature of the target node; wherein the target semantic feature is obtained by combining the spatial convolution feature of the node in the second heterogeneous spatio-temporal graph and the time convolution feature of the node in the second heterogeneous spatio-temporal graph, comprising: inputting the spatial convolution feature of the node in the second heterogeneous spatio-temporal graph and the time convolution feature of the node in the second heterogeneous spatio-temporal graph into a pooling layer in the graph neural network model to obtain the target semantic feature.

[0010] Further, determining the action category based on the target semantic feature comprises: classifying the target semantic feature based on a classifier to obtain a classification result; and determining the action category based on the classification result; wherein the method further comprises: positioning a time point of the action category, and determining a fixed time period based on the time point, and taking actions in the fixed time period as a target segment.

[0011] In a second aspect of the present application, a human action recognition device based on a heterogeneous spatio-temporal graph convolution is provided, comprising: an acquisition module configured to acquire motion video data and extract a frame sequence from the motion video data based on a fixed time interval; a construction module configured to extract joint target points and target objects of each video frame in the frame sequence, and construct a first heterogeneous spatio-temporal graph based on the joint target points and the target objects; an obtaining module configured to generate a feature vector replacing an occluded node in the first heterogeneous spatio-temporal graph according to a trained diffusion model and the occluded node, and obtain a second heterogeneous spatio-temporal graph; and a determination module configured to input the second heterogeneous spatio-temporal graph into a graph neural network model to obtain a target semantic feature, and determine an action category based on the target semantic feature.

[0012] In a third aspect of the present application, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of the first aspect of the present application.

[0013] In a fourth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method of any one of the first aspect of the present application.

[0014] The present application has the following advantages: The human action recognition method and device based on heterogeneous spatio-temporal graph convolution provided by the present application obtain motion video data, and extract a frame sequence from the motion video data based on a fixed time interval; joint target points and target objects of each video frame in the frame sequence are extracted, and a first heterogeneous spatio-temporal graph is constructed based on the joint target points and the target objects; a feature vector replacing an occluded node is generated according to a trained diffusion model and the occluded node in the first heterogeneous spatio-temporal graph, and a second heterogeneous spatio-temporal graph is obtained; the second heterogeneous spatio-temporal graph is input into a graph neural network model to obtain a target semantic feature, and an action category is determined based on the target semantic feature. The method significantly improves the accuracy and robustness of human action recognition in complex real scenes, especially when there is occlusion. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0016] Figure 1 is a flowchart of a human action recognition method based on heterogeneous spatio-temporal graph convolution according to an embodiment of the present application; Figure 2 is a flowchart of a human action recognition method based on heterogeneous spatio-temporal graph convolution according to an embodiment of the present application; Figure 3 is a schematic diagram of a human action recognition device based on heterogeneous spatio-temporal graph convolution according to an embodiment of the present application; Figure 4 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to enable persons skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. It should be understood that these descriptions are only exemplary, and are not used to limit the scope of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of the present application.

[0018] In addition, in the following description, the description of well-known structures and techniques is omitted to avoid unnecessary confusion of the concepts disclosed in the present application.

[0019] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" are only for descriptive purposes and cannot be understood as indicating or implying relative importance. The terms "mounting", "connection", "connection" should be broadly understood, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or the communication between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0020] The exemplary embodiments will be described in detail hereinbelow, examples of which are shown in the drawings. When the following description refers to the drawings, identical numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments are not meant to represent all implementations consistent with the present application. Rather, they are merely examples of methods and systems consistent with some aspects of the present application as detailed in the appended claims.

[0021] Current deep learning-based human action recognition methods mainly rely on the combination of convolutional neural networks (CNN) and recurrent neural networks (RNN). CNN is good at extracting spatial features from video frames, while RNN is used to model dynamic information in the time dimension. In recent years, three-dimensional convolution (3D CNN) and two-stream network (Two-Stream Network) have further improved the performance of action recognition. The former directly captures the spatio-temporal features of video frames through a spatio-temporal convolution kernel, while the latter achieves more comprehensive feature expression by fusing RGB and optical flow modalities. However, existing methods still have obvious limitations. Existing methods have high computational cost, require a large amount of training data, and are sensitive to dynamic backgrounds, making it difficult to handle changes in viewing angle and occlusion problems. Most of these methods treat the human body as a whole, making it difficult to capture the structured relationship between human joints and key objects, ignoring the key characteristic that the essence of action is generated by the spatio-temporal interaction between joints and key objects. At the same time, when there is occlusion, rapid motion or low-quality frames in the video, the key joint data is severely missing, and existing methods handle missing data by interpolation or mean filling, resulting in distortion of action features. The former captures motion information in the video, and the latter achieves more comprehensive feature expression by fusing RGB and optical flow modalities.

[0022] To this end, the application provides a human motion recognition method and device based on heterogeneous spatio-temporal graph convolution and related equipment, and specifically, the human motion recognition method and device based on heterogeneous spatio-temporal graph convolution of the embodiments of the application are described below with reference to the drawings.

[0023] Figure 1 is a flowchart of a human motion recognition method based on heterogeneous spatio-temporal graph convolution according to an embodiment of the application. It should be noted that the human motion recognition method based on heterogeneous spatio-temporal graph convolution of the embodiments of the application can be applied to the human motion recognition device based on heterogeneous spatio-temporal graph convolution of the embodiments of the application. The human motion recognition device based on heterogeneous spatio-temporal graph convolution can be configured on an electronic device or in a server. The embodiments of the present application do not limit this.

[0024] As shown in Figure 1 , the human motion recognition method based on heterogeneous spatio-temporal graph convolution comprises: S110, acquiring motion video data and extracting a frame sequence from the motion video data based on a fixed time interval.

[0025] In the embodiments of the application, the motion video data can be acquired through a terminal device, and in the case where the motion video data is acquired, a frame sequence can be extracted from the motion video data based on a fixed time interval. Extracting the frame sequence based on the fixed time interval avoids high calculation cost due to too dense time and possible loss of key motion information due to too sparse time.

[0026] S120, extracting joint target points and target objects of each video frame in the frame sequence and constructing a first heterogeneous spatio-temporal graph based on the joint target points and the target objects.

[0027] In the embodiments of the application, in the case where the frame sequence is extracted, irrelevant background, color, texture and other information in each video frame in the frame sequence can be filtered, joint target points and target objects of each video frame are extracted, and then a first heterogeneous spatio-temporal graph is constructed based on the joint target points and the target objects.

[0028] For example, the joint target points include but are not limited to shoulders, elbows, wrists, knees, ankles, etc.

[0029] For example, the target objects include but are not limited to basketballs, bats, etc.

[0030] In the embodiments of the application, in the case where the joint target points and the target objects are extracted, the skeletal connection relationship of the joint target points is determined, the skeletal points within a preset threshold distance from the target objects are determined, and the connection relationship between the target objects and the skeletal points is determined; the joint target points and the target objects are taken as graph nodes of the first heterogeneous spatio-temporal graph; and the skeletal connection relationship and the connection relationship between the target objects and the skeletal points are taken as edges of the first heterogeneous spatio-temporal graph.

[0031] The determining of the bone point within the preset threshold distance from the target object can be understood as determining a bone point closest to the target object.

[0032] The determining of the connection relationship between the target object and the bone point simulates the interaction relationship between a person and an object.

[0033] S130, generating a feature vector replacing the occluded node according to the trained diffusion model and the occluded node in the first heterogeneous spatio-temporal graph, and obtaining a second heterogeneous spatio-temporal graph.

[0034] In the embodiment of the present application, in the case of obtaining the first heterogeneous spatio-temporal graph, the occluded node in the first heterogeneous spatio-temporal graph can be determined, and then a feature vector replacing the occluded node is generated according to the trained diffusion model and the occluded node, the feature vector replacing the occluded node is added to the first heterogeneous spatio-temporal graph, and the repaired first heterogeneous spatio-temporal graph is taken as the second heterogeneous spatio-temporal graph. The specific implementation mode can be referred to in the subsequent embodiments.

[0035] S140, inputting the second heterogeneous spatio-temporal graph into the graph neural network model to obtain a target semantic feature, and determining an action category based on the target semantic feature.

[0036] In the embodiment of the present application, the convolution layer in the graph neural network model includes a spatial graph convolution layer and a temporal graph convolution layer.

[0037] In the embodiment of the present application, in the case of obtaining the second heterogeneous spatio-temporal graph, the second heterogeneous spatio-temporal graph can be input into the spatial graph convolution layer to obtain a spatial convolution feature of a node in the second heterogeneous spatio-temporal graph; the second heterogeneous spatio-temporal graph is input into the temporal graph convolution layer to obtain a temporal convolution feature of the node in the second heterogeneous spatio-temporal graph; and the spatial convolution feature of the node in the second heterogeneous spatio-temporal graph and the temporal convolution feature of the node in the second heterogeneous spatio-temporal graph are combined to obtain a target semantic feature. That is, the graph neural network model performs complex operations such as node type perception, edge type perception, causal convolution and multi-scale convolution on the second heterogeneous spatio-temporal graph. Because the input second heterogeneous spatio-temporal graph is complete, the message passing mechanism can be smoothly performed, and the context information of each part of the human body, the object and across time can be effectively aggregated, so as to extract robust high-level semantic features (i.e., target semantic features). The specific implementation mode can be referred to in the subsequent embodiments.

[0038] According to the human action recognition method based on heterogeneous spatio-temporal graph convolutional network, motion video data is acquired, and frame sequences are extracted from the motion video data based on fixed time intervals; joint target points and target objects of each video frame in the frame sequences are extracted, and a first heterogeneous spatio-temporal graph is constructed based on the joint target points and the target objects; a feature vector replacing an occluded node is generated according to a trained diffusion model and the occluded node in the first heterogeneous spatio-temporal graph, and a second heterogeneous spatio-temporal graph is obtained; the second heterogeneous spatio-temporal graph is input into a graph neural network model, target semantic features are obtained, and an action category is determined based on the target semantic features. The method significantly improves the accuracy and robustness of human action recognition in complex real scenes, especially when there is occlusion.

[0039] In order for those skilled in the art to more easily understand the present application, Figure 2 is a human action recognition method based on heterogeneous spatio-temporal graph convolutional network according to one specific embodiment of the present application, as Figure 2 shown, the human action recognition method based on heterogeneous spatio-temporal graph convolutional network comprises: S210, acquiring motion video data, and extracting frame sequences from the motion video data based on fixed time intervals.

[0040] S220, extracting joint target points and target objects of each video frame in the frame sequences, and constructing a first heterogeneous spatio-temporal graph based on the joint target points and the target objects.

[0041] In the embodiments of the present application, the implementation modes of steps S210-S220 can refer to the implementation modes of steps S110-S120 described above. The present application will not repeat them here.

[0042] S230, generating a feature vector replacing an occluded node according to a trained diffusion model and the occluded node in the first heterogeneous spatio-temporal graph, and obtaining a second heterogeneous spatio-temporal graph.

[0043] In the embodiments of the present application, the diffusion model is trained in the following way: training samples are acquired, the training samples including occluded nodes and contexts of the occluded nodes; a noisy feature of the training samples at time t is determined; the noisy feature, the time t, and the conditional information are input into a diffusion decoder to predict the added noise in the noisy feature; the mean square error between the added noise in the predicted noisy feature and the real added noise is calculated, and the diffusion model is obtained when the mean square error is less than a target threshold.

[0044] In the embodiment of the present application, for the occluded node, the spatio-temporal neighbor nodes of the occluded node are extracted from the first heterogeneous spatio-temporal graph; the features of the spatio-temporal neighbor nodes are input into the diffusion model as conditions to obtain a feature vector generated to replace the occluded node; the feature vector generated to replace the occluded node is added to the first heterogeneous spatio-temporal graph, and the repaired first heterogeneous spatio-temporal graph is taken as the second heterogeneous spatio-temporal graph.

[0045] The occluded node, that is, the information of the occluded node itself is empty or noisy, and it cannot provide any valid information to its neighbor nodes. For example, an occluded right hand node cannot tell the basketball node that "I am holding you", and cannot obtain meaningful updates from neighbor nodes because the aggregation operation is based on its own invalid features and neighbor features.

[0046] That is, firstly, it is judged whether there is a node with missing features (occluded node) in the first heterogeneous spatio-temporal graph, if there is, the trained diffusion model is started. The trained diffusion model takes the spatio-temporal neighbor nodes of the occluded node as conditions to generate a reasonable feature vector conforming to the current scene logic to fill the occluded node. For example: in the current scene, most players are looking at the basket, and the position of the basketball is rising in the air. Then even if the hand of a player is occluded, the trained diffusion model generates a "shooting" hand position and speed feature. Then the generated "shooting" hand position and speed feature replaces the occluded node in the first heterogeneous spatio-temporal graph, and the repaired first heterogeneous spatio-temporal graph is taken as the second heterogeneous spatio-temporal graph. By generating a reasonable feature vector conforming to the current scene logic to fill the occluded node, the feature breakage problem caused by node missing in the traditional method is solved, and the recognition accuracy is effectively improved in complex scenes such as occlusion, view change and low light.

[0047] S240, inputting the second heterogeneous spatio-temporal graph into a spatial graph convolution layer to obtain spatial convolution features of nodes in the second heterogeneous spatio-temporal graph.

[0048] In the embodiment of the present application, for the target node of the second heterogeneous spatio-temporal graph, each neighbor node of the target node generates linear transformation information; the linear transformation information of all neighbor nodes is aggregated based on a weighted summation calculation method to obtain aggregated information; the aggregated information is combined with the features of the target node, and the combined information is input into a fully connected layer to obtain spatial convolution features of the target node. That is, the second heterogeneous spatio-temporal graph is input into the spatial graph convolution layer, and the spatial graph convolution layer performs message generation, message aggregation and node update.

[0049] For example, message generation: for a target node w (e.g. left wrist) in the second heterogeneous spatio-temporal graph, each of its spatial neighbor nodes j (e.g. left elbow, left shoulder, basketball) will generate a message. This message is usually a linear transformation of the feature vector of neighbor node j.

[0050] Message aggregation: the target node w (left wrist node) will aggregate messages from all spatial neighbor nodes j, i.e. the aggregated information is m_i. The aggregation function is usually weighted summation. For example, weight source one: edge type. The edge connecting "wrist-elbow" (skeletal edge) and the edge connecting "wrist-basketball" (interaction edge) can have different weight parameters, which means the model will learn the different importance of object interaction and human skeleton relationship for information transmission. Weight source two: attention mechanism. More advanced models will use graph attention network (GAT) to let node i automatically learn which neighbor node j it should pay more "attention" to. For example, in the "shooting" action, the wrist node will give the "basketball" node a higher attention weight, while in the "running" action, it may pay more attention to the elbow and shoulder nodes.

[0051] Node update: the aggregated information m_i will be combined with the feature h_i of the target node i itself, usually through a concatenation operation or addition operation, and then input into a fully connected layer and a nonlinear activation function (e.g. ReLU) to obtain the updated spatial feature h_i' of the target node w (i.e. the spatial convolution feature of the target node).

[0052] In embodiments of the present application, after passing through the spatial graph convolution layer, the feature of each node is no longer just its coordinates, but contains information about its local body structure and interactive objects. For example, the feature of the left wrist node now contains the information that "I am connected to the elbow, the elbow is connected to the shoulder, and there is a basketball near me".

[0053] S250, inputting the second heterogeneous spatio-temporal graph into a temporal graph convolution layer to obtain a temporal convolution feature of a node in the second heterogeneous spatio-temporal graph.

[0054] In embodiments of the present application, for the feature of a target node in the second heterogeneous spatio-temporal graph at time i, the feature of the target node is determined The feature of the target node is aggregated based on a weighted summation calculation method, and the aggregated information is obtained. The aggregated information is combined with the feature of the target node at time i, and the combined information is input into a fully connected layer to obtain a temporal convolution feature of the target node.

[0055] Among them, the temporal graph convolution layer is used to track the motion trajectory of the same node between different frames and capture the dynamic changes of the motion.

[0056] ​where the operation is exactly the same as spatial convolution, but the definition of neighbor nodes is changed. That is, for the feature of target node w in the i-th frame, its "temporal neighbor nodes" are the same node in the previous frames (connected by temporal edges).

[0057] In embodiments of the present application, after time graph convolution, the feature of each node further contains the information of its motion trajectory. For example, the feature of the left wrist node now increases the dynamic information of "I have been moving upward for several frames, with a high speed, and the trajectory is a smooth arc". This is crucial for distinguishing "shooting" and "passing" (both are the interaction of hand and ball, but the trajectories are different).

[0058] S260, combine the spatial convolution feature of the node in the second heterogeneous spatio-temporal graph and the time convolution feature of the node in the second heterogeneous spatio-temporal graph to obtain the target semantic feature.

[0059] In embodiments of the present application, the spatial convolution feature of the node in the second heterogeneous spatio-temporal graph and the time convolution feature of the node in the second heterogeneous spatio-temporal graph are input into the pooling layer in the graph neural network model to obtain the target semantic feature.

[0060] In embodiments of the present application, by fusing the spatial convolution feature and the time convolution feature, not only the instantaneous pose can be understood, but also the collaborative mode between spatial joints and the dynamic trajectory rule over time can be learned, so that the complete evolution process and context of complex actions can be mastered. And by generating the target semantic feature, the core mode feature of the whole motion is highly condensed, which has strong discrimination ability and can represent a specific action type, providing a simple and information-rich input basis for subsequent efficient and accurate highlight action recognition.

[0061] S270, determine the action category based on the target semantic feature.

[0062] In embodiments of the present application, the target semantic feature is classified based on the classifier to obtain a classification result; and the action category is determined based on the classification result.

[0063] For example, the target semantic feature is input into the classifier, and the classifier outputs a probability distribution (i.e. the classification result) after one or more layers of nonlinear transformation. For example: [0.02, 0.05, 0.8, 0.13], this distribution corresponds to the pre-defined categories, such as: category 0: background / ordinary action (corresponding probability 0.02), category 1: walking / running (corresponding probability 0.05), category 2: dunking / shot / scoring (corresponding probability 0.8), and category 3: passing (corresponding probability 0.13).

[0064] ​In an embodiment of the present application, the class with the highest probability is selected from the classification result as the action class of the current video clip. For example, the class with the highest probability (class 2: dunk / shot / score) is selected from [0.02, 0.05, 0.8, 0.13] as the action class of the current video clip.

[0065] S280, determining the target clip based on the action class.

[0066] In an embodiment of the present application, the time point of the action class can be located, and a fixed time period is determined based on the time point, and the action in the fixed time period is taken as the target clip.

[0067] That is, in the case of determining the action class, the highlight clip (i.e., the target clip) can be obtained by locating the time point of the action class (i.e., the highlight action) and clipping the fixed time period before and after it. Without the need for manual repeated viewing and marking, intelligent clipping is realized, and the content production efficiency is greatly improved, saving a lot of manpower and time cost.

[0068] According to the human action recognition method based on heterogeneous spatio-temporal graph convolutional network of the embodiment of the present application, the frame sequence is extracted based on the fixed time interval, which avoids the huge calculation overhead caused by frame-by-frame processing and prevents the loss of key action frames (such as the moment of shooting a basketball) due to oversampling. By constructing the first heterogeneous spatio-temporal graph, instead of processing redundant pixel information, the most critical elements (joints, objects) and their relationships are abstracted, greatly reducing the data volume and focusing on high-level semantic information. The graph contains human structure information (skeleton edges) and human-object interaction information (interaction edges). This means that the model can understand both "human posture" and "how humans interact with objects", i.e., identify the key of many sports actions (such as basketball, tennis). Different types of edges (such as "hand- elbow" and "hand-ball") can have different weights and importance, and the model can learn the contribution of the interaction relationship to the identification of a specific action. By using the diffusion model to repair the occluded nodes, the repaired graph no longer has nodes with missing information, so that the subsequent graph convolutional network can effectively transmit and aggregate information, avoiding performance degradation due to missing node features. Through spatial graph convolution, the static posture and interaction context are captured. Through time graph convolution, the dynamic motion trajectory and timing pattern are captured. Through feature fusion and classification, the final judgment is based on the fusion information of spatial form and temporal dynamics, which makes the classification decision more comprehensive and accurate. This method not only identifies the action class, but also accurately locates the time point of the action, making it suitable for automatic highlight moment detection and video summary, without the need for manual editing, improving efficiency.

[0069] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software on a necessary general hardware platform, and of course can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a part that contributes to the prior art, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disc) and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device) to execute the method of each embodiment of the present application.

[0070] According to an aspect of an embodiment of the present application, a human action recognition device based on heterogeneous spatio-temporal graph convolution is further provided, Figure 3 is a structural block diagram of the human action recognition device based on heterogeneous spatio-temporal graph convolution according to the embodiment of the present application; as Figure 3 shown, comprising: The acquisition module 310 is configured to acquire motion video data and extract a frame sequence from the motion video data based on a fixed time interval. The construction module 320 is configured to extract joint target points and target objects of each video frame in the frame sequence and construct a first heterogeneous spatio-temporal graph based on the joint target points and the target objects. The obtaining module 330 is configured to generate a feature vector replacing an occluded node in the first heterogeneous spatio-temporal graph according to the trained diffusion model and the occluded node, and obtain a second heterogeneous spatio-temporal graph. The determination module 340 is configured to input the second heterogeneous spatio-temporal graph into a graph neural network model to obtain a target semantic feature and determine an action category based on the target semantic feature.

[0071] According to the human action recognition device based on heterogeneous spatio-temporal graph convolution of the embodiment of the present application, motion video data is acquired and a frame sequence is extracted from the motion video data based on a fixed time interval. Joint target points and target objects of each video frame in the frame sequence are extracted, and a first heterogeneous spatio-temporal graph is constructed based on the joint target points and the target objects. A feature vector replacing an occluded node in the first heterogeneous spatio-temporal graph is generated according to a trained diffusion model and the occluded node, and a second heterogeneous spatio-temporal graph is obtained. The second heterogeneous spatio-temporal graph is input into a graph neural network model to obtain a target semantic feature, and an action category is determined based on the target semantic feature. Thus, the accuracy and robustness of human action recognition in a complex real scene (especially when there is occlusion) are significantly improved.

[0072] Optionally, the determining the graph nodes and edges of the first heterogeneous spatio-temporal graph comprises: determining a skeleton connection relationship of the joint target point; determining a skeleton point within a preset threshold distance from the target object, and determining a connection relationship of the target object and the skeleton point; taking the joint target point and the target object as the graph nodes of the first heterogeneous spatio-temporal graph; and taking the skeleton connection relationship and the connection relationship of the target object and the skeleton point as the edges of the first heterogeneous spatio-temporal graph.

[0073] Optionally, the diffusion model is trained in the following manner: a training sample is obtained, the training sample comprising a masked node and a context of the masked node; a noisy feature of the training sample at time t is determined; the noisy feature, the time t, and conditional information are input into a diffusion decoder to predict noise added in the noisy feature; a mean square error between the predicted noise added in the noisy feature and real added noise is calculated, and the diffusion model is obtained in a case where the mean square error is less than a target threshold.

[0074] Optionally, the obtaining module 330 is specifically configured to: for the occluded node, extract a spatio-temporal neighbor node of the occluded node from the first heterogeneous spatio-temporal graph; input a feature of the spatio-temporal neighbor node into the diffusion model as a condition to obtain a generated feature vector replacing the occluded node; add the generated feature vector replacing the occluded node into the first heterogeneous spatio-temporal graph, and take the repaired first heterogeneous spatio-temporal graph as the second heterogeneous spatio-temporal graph.

[0075] Optionally, the determining module 340 is specifically configured to: input the second heterogeneous spatio-temporal graph into the spatial graph convolution layer to obtain a spatial convolution feature of a node in the second heterogeneous spatio-temporal graph; input the second heterogeneous spatio-temporal graph into the temporal graph convolution layer to obtain a temporal convolution feature of the node in the second heterogeneous spatio-temporal graph; and combine the spatial convolution feature of the node in the second heterogeneous spatio-temporal graph and the temporal convolution feature of the node in the second heterogeneous spatio-temporal graph to obtain the target semantic feature.

[0076] Optionally, the determining module 340 is specifically configured to: for a target node of the second heterogeneous spatio-temporal graph, each neighbor node of the target node generates linear transformation information; aggregate information of the linear transformation information of all the neighbor nodes based on a weighted summation calculation method to obtain aggregated information; combine the aggregated information with a feature of the target node, and input the combined information into a fully connected layer to obtain a spatial convolution feature of the target node; and wherein the inputting the second heterogeneous spatio-temporal graph into the temporal graph convolution layer to obtain a temporal convolution feature of a node in the second heterogeneous spatio-temporal graph comprises: for a feature of the target node at time i of the second heterogeneous spatio-temporal graph, determining The characteristics of the target node; the calculation method based on weighted summation. The features of the target node are aggregated to obtain aggregated information; the aggregated information is combined with the features of the target node at time i, and the combined information is input into a fully connected layer to obtain the temporal convolutional features of the target node; wherein, the spatial convolutional features of the nodes in the second heterogeneous spatiotemporal graph and the temporal convolutional features of the nodes in the second heterogeneous spatiotemporal graph are combined to obtain the target semantic features, including: inputting the spatial convolutional features of the nodes in the second heterogeneous spatiotemporal graph and the temporal convolutional features of the nodes in the second heterogeneous spatiotemporal graph into the pooling layer in the graph neural network model to obtain the target semantic features.

[0077] Optionally, the determining module 340 is specifically used to classify the target semantic features based on a classifier to obtain a classification result; and to determine the action category based on the classification result. Optionally, the device further includes a target segment determination module, used to locate the time point of the action category, and determine a fixed time period based on the time point, and use the action of the fixed time period as the target segment.

[0078] According to one aspect of the present invention, an electronic device is provided.

[0079] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Figure 4 As shown, an electronic device may include one or more ( Figure 4 Only one is shown in the image. A processor 102 (which may include, but is not limited to, a microprocessor unit (MPU) or a programmable logic device (PLD)) and a memory 104 for storing data are also shown. In one exemplary embodiment, the electronic device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 4 The structure shown is for illustrative purposes only and does not limit the structure of the terminal device described above. For example, the terminal device may also include components that are more... Figure 4 The more or fewer components shown, or having the same Figure 4 Equivalent functions or ratios shown Figure 4 The functions shown have more different configurations.

[0080] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as a computer program corresponding to the human action recognition method based on heterogeneous spatio-temporal graph convolution in the embodiments of the present application. The processor 102 can execute various functional applications and data processing, i.e., implement the above method, by running the computer program stored in the memory 104. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, which can be connected to the terminal device through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0081] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network can include a wireless network provided by a communication provider of a switching device. In one example, the transmission device 106 includes a network adapter (NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet in a wireless manner.

[0082] The present application provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the human action recognition method based on heterogeneous spatio-temporal graph convolution.

[0083] The applicant of the present application has made a detailed description and explanation of the embodiments of the present application in combination with the drawings of the specification. However, those skilled in the art should understand that the above embodiments are only preferred embodiments of the present application, and the detailed description is only to help the reader better understand the spirit of the present application, and is not a limitation on the protection scope of the present application. On the contrary, any improvement or modification made on the basis of the spirit of the present application should fall within the protection scope of the present application.

[0084] In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The above integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0085] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present application, and are not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed by the present application should be covered within the protection scope of the present application.

Claims

1. A method for human action recognition based on heterogeneous spatiotemporal graph convolution, characterized in that, include: S1, acquire motion video data, and extract frame sequences from the motion video data based on fixed time intervals; S2, extract the joint target points and target objects of each video frame in the frame sequence, and construct a first heterogeneous spatiotemporal graph based on the joint target points and target objects; S3. Based on the trained diffusion model and the occluded nodes in the first heterogeneous spatiotemporal graph, generate feature vectors to replace the occluded nodes and obtain the second heterogeneous spatiotemporal graph. S4, input the second heterogeneous spatiotemporal graph into the graph neural network model to obtain the target semantic features, and determine the action category based on the target semantic features.

2. The human action recognition method based on heterogeneous spatiotemporal graph convolution according to claim 1, characterized in that, Determining the graph nodes and edges of the first heterogeneous spatiotemporal graph includes: Determine the skeletal connection relationships of the joint target points; Determine the skeletal points within a preset threshold distance from the target object, and determine the connection relationship between the target object and the skeletal points; The joint target points and the target object are used as graph nodes of the first heterogeneous spatiotemporal graph; The connection relationships between the skeleton, the target object, and the skeleton points are used as edges of the first heterogeneous spatiotemporal graph.

3. The human action recognition method based on heterogeneous spatiotemporal graph convolution according to claim 1, characterized in that, The diffusion model is trained in the following way: Obtain training samples, which include the occluded node and the context of the occluded node; Determine the noisy features of the training samples at time t; The noisy feature, the time t, and the conditional information are input into the diffusion decoder to predict the noise added to the noisy feature; The mean square error between the noise added to the predicted noisy feature and the actual added noise is calculated. If the mean square error is less than the target threshold, the diffusion model is obtained.

4. The human motion recognition method based on heterogeneous spatiotemporal graph convolution according to claim 3, characterized in that, Based on the trained diffusion model and the occluded nodes in the first heterogeneous spatiotemporal graph, a feature vector is generated to replace the occluded nodes, resulting in a second heterogeneous spatiotemporal graph, including: For the occluded node, extract the spatiotemporal neighbor nodes of the occluded node from the first heterogeneous spatiotemporal graph; The features of the spatiotemporal neighbor nodes are input into the diffusion model as conditions to generate the feature vector that replaces the occluded node; The generated feature vector replacing the occluded node is added to the first heterogeneous spatiotemporal graph, and the repaired first heterogeneous spatiotemporal graph is used as the second heterogeneous spatiotemporal graph.

5. The human action recognition method based on heterogeneous spatiotemporal graph convolution according to claim 1, characterized in that, The convolutional layers in the graph neural network model include spatial graph convolutional layers and temporal graph convolutional layers. The second heterogeneous spatiotemporal graph is input into the graph neural network model to obtain target semantic features, including: The second heterogeneous spatiotemporal graph is input into the spatial graph convolutional layer to obtain the spatial convolutional features of the nodes in the second heterogeneous spatiotemporal graph; The second heterogeneous spatiotemporal graph is input into the temporal graph convolutional layer to obtain the temporal convolutional features of the nodes in the second heterogeneous spatiotemporal graph. The target semantic features are obtained by combining the spatial convolution features of the nodes in the second heterogeneous spatiotemporal graph with the temporal convolution features of the nodes in the second heterogeneous spatiotemporal graph.

6. The human action recognition method based on heterogeneous spatiotemporal graph convolution according to claim 5, characterized in that, The second heterogeneous spatiotemporal graph is input into the spatial graph convolutional layer to obtain the spatial convolutional features of the nodes in the second heterogeneous spatiotemporal graph, including: For the target node in the second heterogeneous spatiotemporal graph, each of the target node's neighboring nodes generates linear transformation information; The linear transformation information of all the neighboring nodes is aggregated based on a weighted summation calculation method to obtain the aggregated information; The aggregated information is combined with the features of the target node, and the combined information is input into a fully connected layer to obtain the spatial convolutional features of the target node. Specifically, the second heterogeneous spatiotemporal graph is input into the temporal graph convolutional layer to obtain the temporal convolutional features of the nodes in the second heterogeneous spatiotemporal graph, including: For the characteristics of the target node in the second heterogeneous spatiotemporal graph at time i, determine the characteristics of the target node at time i-1, i-2, ... iN; The features of the target nodes i-1, i-2, ... iN are aggregated using a weighted summation method to obtain aggregated information. The aggregated information is combined with the features of the target node at time i, and the combined information is input into a fully connected layer to obtain the temporal convolutional features of the target node. Specifically, the target semantic features are obtained by combining the spatial convolutional features of nodes in the second heterogeneous spatiotemporal graph and the temporal convolutional features of nodes in the second heterogeneous spatiotemporal graph, including: The spatial convolutional features and temporal convolutional features of nodes in the second heterogeneous spatiotemporal graph are input into the pooling layer of the graph neural network model to obtain the target semantic features.

7. The human motion recognition method based on heterogeneous spatiotemporal graph convolution according to claim 1, characterized in that, Determining the action category based on the target semantic features includes: The target semantic features are classified based on the classifier to obtain the classification result; The action category is determined based on the classification results; The method further includes: locating the time point of the action category, determining a fixed time period based on the time point, and using the action of the fixed time period as the target segment.

8. A human motion recognition device based on heterogeneous spatiotemporal graph convolution, characterized in that, include: An acquisition module is used to acquire motion video data and extract frame sequences from the motion video data based on fixed time intervals; A construction module is used to extract the joint target points and target objects of each video frame in the frame sequence, and construct a first heterogeneous spatiotemporal graph based on the joint target points and the target objects; The module is used to generate a feature vector to replace the occluded node based on the trained diffusion model and the occluded node in the first heterogeneous spatiotemporal graph, and to obtain a second heterogeneous spatiotemporal graph. The determination module is used to input the second heterogeneous spatiotemporal graph into the graph neural network model to obtain target semantic features, and determine the action category based on the target semantic features.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 7.

Images