Behavior recognition model training method, recognition method, device and storage medium

Abstract

The present invention discloses a behavior recognition model training method, a recognition method, a device, and a storage medium. The behavior recognition model training method includes: generating a semantic graph of a skeletal sequence based on a skeletal sequence in a training sample set, the semantic graph comprising at least a structural semantic graph, wherein the neighboring nodes corresponding to a root node in the structural semantic graph have a range of L hops, where L is a natural number greater than 1; performing a graph convolutional network (GCN) operation on the semantic graph to extract features corresponding to the skeletal sequence; performing behavior prediction on the skeletal sequence based on the extracted features; and adjusting model parameters of the behavior recognition model based on recognition errors of the behavior prediction to obtain a trained behavior recognition model.

Description

Technical Field

[0001] The present invention relates to the field of behavior recognition, and in particular to a behavior recognition model training method, recognition method, device and storage medium. Background Art

[0002] Behavior recognition, as an important branch of computer vision, aims to enable computers to perceive what objects in visual scenes are doing through acquisition devices such as cameras. It can be applied to many fields, including security monitoring, unmanned supermarkets, education and entertainment. It is one of the important means of modern care and monitoring, and has greatly improved the intelligence level of equipment.

[0003] Action recognition can be categorized as either image- and video-based or skeleton-based. Images and videos not only contain complex backgrounds but also face uncertainties such as lighting variations, perspective changes, and changes in appearance. This makes image- and video-based action recognition limited. Compared to image- and video-based action recognition, skeleton-based action recognition can effectively overcome these uncertainties.

[0004] In related technologies, skeleton topology graphs are often modeled based on graph convolutional networks (GCNs) to express the natural connection relationship between bones. However, the skeleton behavior recognition method based on graph convolutional networks has the following shortcomings: the behavior features are only extracted locally and cannot fully and accurately express the skeleton behavior, which affects the accuracy of skeleton behavior recognition. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a behavior recognition model training method, recognition method, device and storage medium, aiming to improve the accuracy of skeletal behavior recognition.

[0006] The technical solution of the embodiment of the present invention is achieved as follows:

[0007] An embodiment of the present invention provides a behavior recognition model training method, comprising:

[0008] Generate a semantic graph of the skeleton sequence based on the skeleton sequence in the training sample set, the semantic graph at least including: a structural semantic graph, wherein the neighbor node range corresponding to the root node in the structural semantic graph is L hops, where L is a natural number greater than 1;

[0009] Performing a graph convolutional network (GCN) operation on the semantic graph to extract features corresponding to the skeleton sequence;

[0010] Performing behavior prediction on the skeleton sequence based on the extracted features;

[0011] The model parameters of the behavior recognition model are adjusted based on the recognition error of the behavior prediction to obtain a trained behavior recognition model.

[0012] An embodiment of the present invention further provides a behavior recognition method, comprising:

[0013] Obtain the bone sequence to be identified;

[0014] Generate a semantic graph for the skeleton sequence to be identified using the behavior recognition model trained by the method described in the above embodiment;

[0015] The behavior recognition model extracts features from the semantic graph based on GCN;

[0016] The behavior recognition model performs behavior prediction on the skeleton sequence to be identified based on the extracted features, and determines the behavior corresponding to the skeleton sequence to be identified.

[0017] An embodiment of the present invention further provides a behavior recognition model training device, comprising:

[0018] A first semantic graph generation module is configured to generate a semantic graph of the skeleton sequence based on the skeleton sequence in the training sample set, wherein the semantic graph at least includes: a structural semantic graph, wherein the neighbor node range corresponding to the root node in the structural semantic graph is L hops, where L is a natural number greater than 1;

[0019] A first feature extraction module is used to perform GCN operation on the semantic graph to extract features of the skeleton sequence;

[0020] A first behavior prediction module, configured to perform behavior prediction on the skeleton sequence based on the extracted features;

[0021] The model training module is used to adjust the model parameters of the behavior recognition model based on the recognition error of the behavior prediction to obtain a trained behavior recognition model.

[0022] An embodiment of the present invention further provides a behavior recognition device, comprising:

[0023] An acquisition module, used to obtain the skeleton sequence to be identified;

[0024] A second semantic graph generating module is used to generate a semantic graph for the skeleton sequence to be identified;

[0025] A second feature extraction module is used to extract features from the semantic graph based on GCN;

[0026] A second behavior prediction module is used to perform behavior prediction on the skeleton sequence to be identified based on the extracted features, and determine the behavior corresponding to the skeleton sequence to be identified;

[0027] Among them, the semantic graph generation module and the behavior prediction module are trained based on the behavior recognition model training device described in the above embodiment.

[0028] An embodiment of the present invention also provides a behavior recognition model training device, comprising: a processor and a memory for storing a computer program that can be run on the processor, wherein the processor is used to execute the steps of the behavior recognition model training method described in the embodiment of the present invention when running the computer program.

[0029] An embodiment of the present invention further provides a behavior recognition device, comprising: a processor and a memory for storing a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the behavior recognition method described in the embodiment of the present invention when running the computer program.

[0030] An embodiment of the present invention also provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the behavior recognition model training method described in the embodiment of the present invention or the steps of the behavior recognition method described in the embodiment of the present invention are implemented.

[0031] The technical solution provided by the embodiment of the present invention generates a semantic graph that at least includes a structural semantic graph for the skeleton sequence in the training sample set, and the range of neighbor nodes corresponding to the root node in the structural semantic graph is greater than the range of 1 hop, so that the receptive field of GCN is larger, and more comprehensive behavioral features corresponding to the skeleton sequence can be extracted to train the behavior recognition model, which can improve the accuracy of behavior recognition based on the behavior recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the process of training a behavior recognition model according to an embodiment of the present invention;

[0033] Figure 2 This is a flow chart of a behavior recognition model training method according to an embodiment of the present invention;

[0034] Figures 3A to 3C Schematic diagram of the structural semantic graph, implicit relationship semantic graph, and attention enhancement semantic graph corresponding to the skeleton sequence in the application embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram of the structure of the GCN operation in the application embodiment of the present invention;

[0036] Figure 5 Schematic diagram of the process of behavior recognition method according to an embodiment of the present invention;

[0037] Figure 6 This is a schematic diagram of the structure of a behavior recognition model training device according to an embodiment of the present invention;

[0038] Figure 7 This is a schematic diagram of the structure of a behavior recognition device according to an embodiment of the present invention;

[0039] Figure 8 This is a schematic diagram of the structure of a behavior recognition model training device according to an embodiment of the present invention;

[0040] Figure 9 Schematic diagram of the structure of a behavior recognition device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0043] In related technologies, the skeleton topology is modeled based on GCN. The skeleton topology is fixed, and often only the topological relationship of adjacent nodes is extracted, resulting in the behavioral features being extracted only locally. It is impossible to fully and accurately express the skeleton behavior, affecting the accuracy of skeleton behavior recognition.

[0044] Based on this, in various embodiments of the present invention, a semantic graph that at least includes a structural semantic graph is generated for the skeleton sequence in the training sample set, and the range of neighbor nodes corresponding to the root node in the structural semantic graph is greater than the range of 1 hop, so that the receptive field of GCN is larger, and more comprehensive behavioral features corresponding to the skeleton sequence can be extracted to train the behavior recognition model, which can improve the accuracy of behavior recognition based on the behavior recognition model.

[0045] The embodiment of the present invention provides a behavior recognition model training method, such as Figure 1 As shown, the behavior recognition model training method includes:

[0046] Step 101, generating a semantic graph of the skeleton sequence based on the skeleton sequence in the training sample set;

[0047] Here, the training sample set includes at least one skeleton sequence of a known action, which can be a skeleton sequence generated based on a skeleton sequence detection algorithm (such as OpenPose) for a video image frame. For example, the skeleton sequence can be C in ×T×N, where C in Represents the three-dimensional coordinates of each bone node, T represents the number of bone nodes, and N represents the number of image frames.

[0048] Here, the semantic graph at least includes: a structural semantic graph, in which the range of neighbor nodes corresponding to a root node in the structural semantic graph is L hops, where L is a natural number greater than 1.

[0049] Step 102, performing GCN operation on the semantic graph to extract features corresponding to the skeleton sequence;

[0050] Here, GCN is used to extract features from irregular semantic maps. For example, a skeleton sequence is a topological sequence of the human skeleton, with each bone node as a node and the bones between the nodes as edges. GCN can extract features from this topological sequence of the skeleton.

[0051] In some embodiments, performing a GCN operation on the semantic graph to extract features corresponding to the skeleton sequence includes:

[0052] A GCN operation is performed on the structural semantic graph based on a neighbor node range to extract a first feature corresponding to the structural semantic graph of the skeleton sequence.

[0053] Since the neighbor node range of the root node in the structural semantic graph is expanded from a single hop to a multi-hop range, the receptive field range of the convolution kernel of the GCN can be improved, thereby expanding the feature extraction range from local to global, thereby improving the accuracy of subsequent behavior recognition.

[0054] In some embodiments, the neighbor nodes in the structural semantic graph are further divided into subsets based on the distance between the neighbor nodes and the root node, and the performing of the GCN operation on the structural semantic graph based on the neighbor node range includes:

[0055] A GCN operation is performed on the structural semantic graph based on the neighbor node range and the neighbor node subset in which the neighbor node is located.

[0056] In an application example, the GCN operation corresponding to the structural semantic graph is defined as follows:

[0057]

[0058] Among them, f stc is the extracted feature corresponding to the structural semantic graph (i.e., the first feature), L is the number of hops corresponding to the neighbor node range, l is the number of hops from the neighbor node to the root node, p is the neighbor node subset where the neighbor node is located, p∈P, is the weight function corresponding to the structural semantic graph, is the adjacency matrix corresponding to the structural semantic graph, f in A sequence of bones.

[0059] Here, L can be a natural number greater than 1, so that the neighbor nodes of the root node in the natural connection graph of the skeleton can be expanded from 1 hop to multiple hops, so as to improve the receptive field range of the graph convolution kernel of the skeleton sequence, and the feature extraction range can be expanded from local to global, thereby improving the accuracy of subsequent behavior recognition.

[0060] In one application example, P = (0, 1, 2), which means that the neighbor nodes are divided into three subsets. Specifically, based on the first distance between each neighbor node and the center of gravity of the human skeleton and the second distance between the root node and the center of gravity of the human skeleton, the three neighbor node subsets can be divided into: subset 0, subset 1, and subset 2. Subset 0 represents the root node itself, that is, the first distance is equal to the second distance; subset 1 represents the centripetal subset, that is, the first distance is less than the second distance; subset 2 represents the eccentric subset, that is, the first distance is greater than the second distance.

[0061] In practical applications, many behavioral actions can be simply divided into two categories: centripetal movement and centrifugal movement. By dividing neighbor nodes into corresponding neighbor node subsets and using differentiated segmentation methods, different neighbor nodes can be distinguished, thereby improving the efficiency of subsequent behavior recognition.

[0062] Step 103, performing behavior prediction on the skeleton sequence based on the extracted features;

[0063] In some embodiments, performing behavior prediction on the skeleton sequence based on the extracted features includes:

[0064] Behavior prediction is performed based on the first feature to obtain a prediction result.

[0065] For example, a first feature representing the temporal and spatial characteristics of the skeleton sequence can be extracted from the structural semantic graph of the skeleton sequence. Based on the first feature, behavior prediction is performed on the skeleton sequence to obtain the probability of the predicted behavior.

[0066] Step 104 : Adjust the model parameters of the behavior recognition model based on the recognition error of the behavior prediction to obtain a trained behavior recognition model.

[0067] Here, the adjustment of the model parameters of the behavior recognition model based on the recognition error of the behavior prediction includes:

[0068] determining a recognition error of the behavior prediction based on a prediction result of the behavior prediction;

[0069] Based on the recognition error of behavior prediction, the back propagation algorithm is used to adjust the model parameters of the behavior recognition model.

[0070] For example, the probability of the predicted behavior can be obtained based on the first feature. According to the probability of the predicted behavior of the skeleton sequence and the actual behavior corresponding to the skeleton sequence, the recognition error of the behavior prediction can be calculated. Then, based on the recognition error of the behavior prediction, the model parameters of the behavior recognition model can be adjusted using the back propagation algorithm.

[0071] Preferably, based on the adjusted behavior recognition model and the skeleton sequence in the training sample set, the model parameters are continuously adjusted until the set conditions are met to obtain a trained behavior recognition model.

[0072] Here, continuing to adjust the model parameters means repeatedly performing steps 101 to 104 based on the adjusted behavior recognition model and the skeleton sequence in the training sample set until the set conditions are met to obtain a trained behavior recognition model. Here, the skeleton sequence in the training sample set can be a skeleton sequence randomly selected from the training sample set, or a skeleton sequence selected from the training sample set that has not participated in training.

[0073] Here, the satisfying of the set conditions includes:

[0074] Determine that the number of adjustments to the model parameters of the behavior recognition model reaches a set number; or,

[0075] Determine whether the recognition error of the behavior prediction is less than or equal to a preset threshold.

[0076] Here, the number of model training times can be pre-set to determine that the number of adjustments to the model parameters of the behavior recognition model (i.e., the number of training times) reaches the set number, then the iteration is terminated to obtain a trained behavior recognition model; or it can be determined that the recognition error of the behavior prediction is less than or equal to the preset threshold, indicating that the behavior recognition model has been trained, and a trained behavior recognition model is obtained.

[0077] In practical applications, the topology graph used for graph convolution is fixed and predefined based on physical connection points, ignoring the implicit node structure relationship that is highly related to behavior. Based on this, in some embodiments, the semantic graph also includes: an implicit relationship semantic graph, and generating the implicit relationship semantic graph of the skeleton sequence includes:

[0078] Extracting behavior-related features corresponding to each skeletal node of the skeletal sequence, and determining a first relationship value between each skeletal node itself and the behavior and a second relationship value between the behavior-related features corresponding to any two skeletal nodes;

[0079] The adjacency matrix of the implicit relationship semantic graph is determined based on the first relationship value and the second relationship value. The elements on the diagonal of the adjacency matrix of the implicit relationship semantic graph are the first relationship values corresponding to each bone node, and the remaining elements are the second relationship values corresponding to any two bone nodes, so as to characterize the implicit connection relationship related to behavior between different bone nodes in the bone sequence.

[0080] In one application example, a multi-layer perceptron function can be used to extract the behavior-related features corresponding to each skeletal node, and a normalization function can be used to measure the second relationship value between the features of any two skeletal nodes, as well as the first relationship value between the features of the skeletal node itself and the behavior. A larger first relationship value indicates a higher degree of association between the skeletal node and the behavior, and a larger second relationship value indicates a higher degree of association between the two skeletal nodes and the behavior. In this way, the implicit connection relationship related to behavior between different skeletal nodes in a skeletal sequence can be represented through an implicit relationship semantic graph.

[0081] Step 102 performs GCN operation on the semantic graph to extract features corresponding to the skeleton sequence, and also includes:

[0082] Perform a GCN operation on the implicit relationship semantic graph to extract the second feature corresponding to the implicit relationship semantic graph of the skeleton sequence.

[0083] Here, since the implicit relationship semantic graph can represent the implicit connection relationship related to the behavior between different bone nodes in the bone sequence, the higher the correlation with the behavior, the greater the relationship value of the two bone nodes. In this way, GCN operation is performed on the implicit relationship semantic graph to extract the second feature corresponding to the implicit semantic relationship graph of the bone sequence. This can mine the implicit structural relationship that is highly related to the action behavior and flexibly model the key, non-physical connection structure, thereby improving the accuracy and efficiency of subsequent behavior recognition.

[0084] In some embodiments, the neighbor nodes in the implicit relationship semantic graph are further divided into subsets based on the distance between the neighbor nodes and the root node, and the performing of the GCN operation on the implicit relationship semantic graph includes:

[0085] A GCN operation is performed on the implicit relationship semantic graph based on a neighbor node subset where the neighbor node is located.

[0086] In this way, by differentially segmenting the neighbor nodes in the implicit relationship semantic graph, different neighbor nodes can be distinguished, thereby improving the efficiency of subsequent behavior recognition.

[0087] In an application example, the implicit relationship semantic graph extracts the features of each skeleton node and uses a multi-layer perceptron to model the relationship between different skeleton nodes to obtain the implicit connection relationship between skeleton nodes related to specific behaviors. The elements in the corresponding graph adjacency matrix are defined as:

[0088]

[0089] Among them, θ (p) (v i ) is about node v i The feature extraction function, f in (v i ) is v i Input, is θ (p) The weight parameter of the function, p is the subset of neighbor nodes where the neighbor node is located, is about node v j The feature extraction function, f in (v j ) is v j Input, for The weight parameter of the function.

[0090] The GCN operation corresponding to the implicit relationship semantic graph is defined as follows:

[0091]

[0092] Among them, f act is the extracted feature corresponding to the implicit relationship semantic graph (i.e., the second feature), is the weight function corresponding to the implicit relationship semantic graph, is the adjacency matrix corresponding to the implicit relationship semantic graph, f in is a skeleton sequence, p is the subset of neighbor nodes where the neighbor node is located, p∈P.

[0093] In actual applications, all skeleton node information is used for learning calculations without distinction, without considering the importance of key information for behavior discrimination, which often affects the efficiency of behavior recognition. Based on this, in some embodiments, the semantic graph also includes: an attention-enhanced semantic graph, and generating the attention-enhanced semantic graph of the skeleton sequence includes:

[0094] Determine the adjacency matrix of the attention semantic graph, the elements on the diagonal of the adjacency matrix of the attention enhanced semantic graph are the attention weight values of each bone node in the bone sequence, and the remaining elements are the attention weight values between any two bone nodes, so as to characterize the attention weights of each bone node in the bone sequence and the connection relationship between any two bone nodes.

[0095] Step 102 performs GCN operation on the semantic graph to extract features corresponding to the skeleton sequence, and further includes:

[0096] Perform a GCN operation on the attention-enhanced semantic map to extract the third feature corresponding to the attention-enhanced semantic map of the skeleton sequence.

[0097] Here, the behavior recognition model can also generate an attention-enhanced semantic map for the skeleton sequence, perform GCN operations on the attention-enhanced semantic map, and extract the corresponding third feature, which can adaptively enhance key information, thereby improving the accuracy and efficiency of subsequent behavior recognition.

[0098] In some embodiments, the neighbor nodes in the attention-enhanced semantic graph are further divided into subsets based on the distance between the neighbor nodes and the root node, and the performing GCN operation on the attention-enhanced semantic graph includes:

[0099] The attention-enhanced semantic graph is subjected to GCN operation based on the neighbor node subset where the neighbor node is located.

[0100] In this way, by differentially segmenting the neighbor nodes in the attention-enhanced semantic graph, different neighbor nodes can be distinguished, thereby improving the efficiency of subsequent behavior recognition.

[0101] In one application example, an attention-enhanced semantic graph can be designed for each skeletal node and the connection relationship between each pair of skeletal nodes. Different attention weights can be assigned to different nodes and different connection relationships to improve the computational efficiency of key information. For example, the attention-enhanced semantic graph can be constructed using the natural connection graph of the skeleton as the initial value. However, the parameters in the attention-enhanced semantic graph can be arbitrary and optimized and iterated along with the entire semantic graph convolutional network. Its adjacency matrix can be defined as:

[0102]

[0103] Where N is the number of bone nodes, is the attention weight value of skeleton node i, is the attention weight of the feature of skeleton node j to skeleton node i.

[0104] The GCN operation corresponding to the attention-enhanced semantic graph is defined as follows:

[0105]

[0106] Among them, f att The extracted features corresponding to the attention-enhanced semantic graph (i.e., the third feature) are is the weight function corresponding to the attention-enhanced semantic graph, is the adjacency matrix corresponding to the attention-enhanced semantic graph, f in is a skeleton sequence, p is the subset of neighbor nodes where the neighbor node is located, p∈P.

[0107] The present invention will be described in further detail below with reference to application examples.

[0108] In the behavior recognition model training method of this application embodiment, the semantic graphs constructed for the skeleton sequence include: a structural semantic graph, an implicit relationship semantic graph, and an attention-enhanced semantic graph. Feature extraction is performed on the three semantic graphs corresponding to the skeleton sequence based on GCN. Behavior prediction is performed on the skeleton sequence based on the extracted features. The model parameters of the behavior recognition model are adjusted based on the recognition error of the behavior prediction. Based on the adjusted behavior recognition model and the skeleton sequences in the training sample set, the model parameters are further adjusted until the set conditions are met, thereby obtaining a trained behavior recognition model.

[0109] like Figure 2 As shown, the behavior recognition model training method of this application embodiment includes:

[0110] Step 201: Perform semantic graph modeling on the skeleton sequences in the sample training set.

[0111] Here, semantic graph modeling of the skeleton sequence includes: constructing a structural semantic graph, an implicit relationship semantic graph, and an attention enhancement semantic graph of the skeleton sequence.

[0112] like Figure 3A As shown in the figure, the range of neighbor nodes of the root node in the structural semantic graph is expanded from a 1-hop range to a multi-hop range, for example, to an L-hop range, where L is a natural number greater than 1. Neighbor nodes are also divided into subsets based on the distance from the root node. For example, p is the neighbor node subset in which the neighbor node is located, p∈P, P=(0,1,2), that is, the neighbor nodes are divided into three subsets. The division can be based on the first distance between each neighbor node and the center of gravity of the human skeleton and the second distance between the root node and the center of gravity of the human skeleton. For example, the three neighbor node subsets are: subset 0, subset 1, and subset 2, where subset 0 represents the root node itself, that is, the first distance is equal to the second distance; subset 1 represents the centripetal subset, that is, the first distance is less than the second distance; subset 2 represents the eccentric subset, that is, the first distance is greater than the second distance.

[0113] like Figure 3B As shown in Figure 1, the implicit relational semantic graph is used to represent the implicit connection relationships between different skeletal nodes in the skeletal sequence related to behavior. The features of each skeletal node can be extracted, and the relationship between different skeletal nodes can be modeled using a multi-layer perceptron to obtain the implicit connection relationships between different skeletal nodes related to specific behaviors. The elements in the corresponding adjacency matrix are defined as:

[0114]

[0115] Among them, θ (p) (v i ) is about node v i The feature extraction function, f in (v i ) is v i Input, is θ (p) The weight parameter of the function, p is the subset of neighbor nodes where the neighbor node is located, is about node v j The feature extraction function, f in (v j ) is v j Input, for The weight parameter of the function.

[0116] like Figure 3C As shown in the figure, the attention-enhanced semantic graph is used to characterize the attention weights of each skeletal node and the connection relationship between any two skeletal nodes in the skeletal sequence. An attention-enhanced semantic graph can be designed for each skeletal node and the connection relationship between each skeletal node, giving different attention weights to different nodes and different connection relationships to improve the computational efficiency of key information. For example, the skeletal natural connection graph can be used as the initial value to construct an attention-enhanced semantic graph, but the parameters in the attention-enhanced semantic graph can be arbitrary values and optimized and iterated along with the entire semantic graph convolutional network. Its adjacency matrix can be defined as:

[0117]

[0118] Where N is the number of bone nodes, is the attention weight value of skeleton node i, is the attention weight of the feature of skeleton node j to skeleton node i.

[0119] Step 202: extract features from the structural semantic graph, implicit relationship semantic graph, and attention enhancement semantic graph of the skeleton sequence based on GCN.

[0120] Here, the GCN operation corresponding to the structural semantic graph is defined as follows:

[0121]

[0122] Among them, f stc is the extracted feature corresponding to the structural semantic graph (i.e., the first feature), L is the number of hops corresponding to the neighbor node range, l is the number of hops from the neighbor node to the root node, p is the neighbor node subset where the neighbor node is located, p∈P, is the weight function corresponding to the structural semantic graph, is the adjacency matrix corresponding to the structural semantic graph, f in A sequence of bones.

[0123] Here, the GCN operation corresponding to the implicit relationship semantic graph is defined as follows:

[0124]

[0125] Among them, f act is the extracted feature corresponding to the implicit relationship semantic graph (i.e., the second feature), is the weight function corresponding to the implicit relationship semantic graph, is the adjacency matrix corresponding to the implicit relationship semantic graph, f in is a skeleton sequence, p is the subset of neighbor nodes where the neighbor node is located, p∈P.

[0126] Here, the GCN operation corresponding to the attention-enhanced semantic graph is defined as follows:

[0127]

[0128] Among them, f att The extracted features corresponding to the attention-enhanced semantic graph (i.e., the third feature) are is the weight function corresponding to the attention-enhanced semantic graph, is the adjacency matrix corresponding to the attention-enhanced semantic graph, f in is a skeleton sequence, p is the subset of neighbor nodes where the neighbor node is located, p∈P.

[0129] like Figure 4 As shown in Figure 1, the GCN operation can include multiple stacked semantic graph convolution modules, each of which includes: a spatial dimension semantic graph convolution layer, a temporal dimension graph convolution layer, a normalization layer, and an activation layer. In this way, the input skeleton sequence C in ×T×N obtains the output features through multiple graph convolution operations.

[0130] Here, the output features can be defined as follows:

[0131] f out =σ(f stc ,f act ,f att )

[0132] Among them, f outis the output feature, i.e., the feature that can characterize the discriminative temporal and spatial characteristics of the skeletal sequence, and σ is the aggregation function. The aggregation function can be a maximum function, a minimum function, or a concatenation function. The maximum or minimum function is simple to process and has a small amount of data, but it is easy to strengthen some features and ignore others, which is not comprehensive enough. The concatenation function can aggregate all features, but it will increase the amount of data.

[0133] In practical applications, residual networks can also be introduced for skip connections to alleviate the problem of gradient disappearance in GCN and optimize GCN operations.

[0134] Step 203: Perform behavior prediction on the skeleton sequence based on the extracted features to obtain the probability of the predicted behavior.

[0135] The output features of the GCN operation are normalized and activated to obtain the probability of the predicted behavior.

[0136] Step 204 : determining the recognition error of the behavior prediction based on the probability of the predicted behavior and the corresponding actual behavior.

[0137] Here, the recognition error of behavior prediction can be determined based on the loss function. For example, the loss function can be:

[0138]

[0139] in, is the corresponding real behavior, k is the total number of different behaviors used for training, To predict the probability of the current input skeleton sequence as the i-th behavior, L loss is the recognition error of behavior prediction.

[0140] Step 205 : Adjust the model parameters of the behavior recognition model based on the recognition error of the behavior prediction.

[0141] Here, the back propagation algorithm can be used to adjust the model parameters of the behavior recognition model according to the recognition error of the behavior prediction, for example, the back propagation algorithm can be used to adjust at least one of the weight function corresponding to the structural semantic graph, the weight function corresponding to the implicit relationship semantic graph, the weight function corresponding to the attention enhancement semantic graph, the adjacency matrix corresponding to the implicit relationship semantic graph, and the adjacency matrix corresponding to the attention enhancement semantic graph.

[0142] Step 206, repeat steps 201 to 205 until a trained behavior recognition model is obtained.

[0143] Repeat steps 101 to 103 based on the adjusted behavior recognition model and the skeleton sequence in the training sample set until the set conditions are met to obtain a trained behavior recognition model. Here, the skeleton sequence in the training sample set can be a skeleton sequence randomly selected from the training sample set. The number of model training times can be pre-set to determine that the number of adjustments to the model parameters of the behavior recognition model (i.e., the number of training times) reaches the set number, then the iteration is terminated to obtain a trained behavior recognition model; or determine that the recognition error of the behavior prediction is less than or equal to a preset threshold, indicating that the behavior recognition model has been trained, to obtain a trained behavior recognition model.

[0144] The behavior recognition model training method of this application embodiment uses a structural semantic graph to expand the receptive field of GCN, extract global skeletal structure features, and obtain a more complete and rich representation; uses an implicit relationship semantic graph to mine the implicit relationship of skeletal nodes related to actions, which can provide a more flexible and accurate structural feature representation compared to a fixed, pre-defined, physically connected skeletal graph; uses attention enhancement semantic graphs to adaptively select important information, strengthen key skeletal nodes and relationships between nodes, and obtain a more robust representation, so that rich and comprehensive features can be used to train the behavior recognition model, which is beneficial to improving the classification efficiency of the behavior recognition model and the accuracy of behavior prediction.

[0145] The embodiment of the present invention also provides a behavior recognition method, such as Figure 5 As shown, the method includes:

[0146] Step 501, obtaining a skeleton sequence to be identified;

[0147] Step 502: The behavior recognition model generates a semantic graph for the skeleton sequence to be recognized;

[0148] Step 503: The behavior recognition model extracts features from the semantic graph based on GCN.

[0149] Step 504: the behavior recognition model performs behavior prediction on the skeleton sequence to be recognized based on the extracted features, and determines the behavior corresponding to the skeleton sequence to be recognized.

[0150] Here, the skeleton sequence to be identified may be a pre-collected skeleton sequence, or a skeleton sequence corresponding to human activities detected in real time by monitoring equipment, etc., which is not specifically limited in this embodiment of the present invention.

[0151] Here, the behavior recognition model is trained based on the method described in any of the aforementioned embodiments. Wherein, the semantic graph generated by the behavior recognition model for the skeletal sequence to be identified includes at least the aforementioned structural semantic graph. It can be understood that the semantic graph generated for the skeletal sequence to be identified is consistent with the type of semantic graph generated in the behavior recognition model training method, that is, the behavior recognition model can be trained using a structural semantic graph or a structural semantic graph and at least one of the aforementioned implicit relationship semantic graph and the attention enhancement semantic graph. The trained behavior recognition model then performs behavior prediction for the skeletal sequence to be identified based on structural semantics or a structural semantic graph and at least one of the aforementioned implicit relationship semantic graph and the attention enhancement semantic graph.

[0152] Here, the behavior recognition model generates a semantic graph for the skeleton sequence to be recognized and performs feature extraction on the semantic graph based on GCN. You can refer to the above-mentioned behavior recognition model training method embodiment, which will not be repeated here.

[0153] The behavior recognition model predicts behavior based on the extracted features and selects the predicted behavior with the highest probability as the behavior of the skeletal sequence to be identified, thus achieving behavior recognition based on skeletal sequences. Because the behavior recognition model uses a semantic graph convolutional network, it can efficiently extract discriminative semantic spatial structural features and temporal dynamic features. The structural semantic graph is used to expand the convolution receptive field range, extract global skeletal structural features, and obtain a more complete and rich representation; the implicit relationship semantic graph is used to mine the implicit relationships between skeletal nodes related to the action, which is a more flexible and accurate representation compared to the fixed, pre-defined, physically connected skeletal graph; the attention-enhanced semantic graph is used to adaptively select important information, strengthen key skeletal nodes and the relationships between nodes, and obtain a more robust representation, thereby improving the prediction accuracy of the entire behavior category.

[0154] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides a behavior recognition model training device, which corresponds to the above-mentioned behavior recognition model training method, and each step in the above-mentioned behavior recognition model training method embodiment is also fully applicable to the embodiment of this behavior recognition model training device.

[0155] like Figure 6As shown, the behavior recognition model training device includes: a first semantic graph generation module 601, a first feature extraction module 602, a first behavior prediction module 603 and a model training module 604. Among them, the first semantic graph construction module 601 is used to generate a semantic graph of the skeleton sequence based on the skeleton sequence in the training sample set; the first feature extraction module 602 is used to perform GCN operation on the semantic graph to extract features corresponding to the skeleton sequence; the first behavior prediction module 603 is used to perform behavior prediction on the skeleton sequence based on the extracted features; the model training module 604 is used to adjust the model parameters of the behavior recognition model based on the recognition error of the behavior prediction to obtain a trained behavior recognition model.

[0156] Here, the semantic graph at least includes: a structural semantic graph, wherein the neighbor node range corresponding to the root node in the structural semantic graph is L hops, where L is a natural number greater than 1, and the first feature extraction module 602 is specifically configured to:

[0157] A GCN operation is performed on the structural semantic graph based on a neighbor node range to extract a first feature corresponding to the skeleton sequence.

[0158] In some embodiments, the neighbor nodes in the structural semantic graph are further divided into subsets based on the distance between the neighbor nodes and the root node, and the first feature extraction module 602 is specifically configured to:

[0159] A GCN operation is performed on the structural semantic graph based on the neighbor node range and the neighbor node subset in which the neighbor node is located.

[0160] In some embodiments, the semantic graph further includes an implicit relationship semantic graph, and the first semantic graph generation module 601 is further configured to:

[0161] Extracting behavior-related features corresponding to each skeletal node of the skeletal sequence, and determining a first relationship value between each skeletal node itself and the behavior and a second relationship value between the behavior-related features corresponding to any two skeletal nodes;

[0162] The adjacency matrix of the implicit relationship semantic graph is determined based on the first relationship value and the second relationship value. The elements on the diagonal of the adjacency matrix of the implicit relationship semantic graph are the first relationship values corresponding to each bone node, and the remaining elements are the second relationship values between any two bone nodes, so as to characterize the implicit connection relationship related to behavior between different bone nodes in the bone sequence.

[0163] The first feature extraction module 602 is specifically used for:

[0164] Perform a GCN operation on the implicit relationship semantic graph to extract the second feature corresponding to the skeleton sequence.

[0165] In some embodiments, the neighbor nodes in the implicit relationship semantic graph are further divided into subsets based on the distance between the neighbor nodes and the root node, and the first feature extraction module 602 is specifically configured to:

[0166] A GCN operation is performed on the implicit relationship semantic graph based on a neighbor node subset where the neighbor node is located.

[0167] In some embodiments, the semantic graph further includes an attention-enhanced semantic graph, and the first semantic graph generation module 601 is further configured to:

[0168] Determine the adjacency matrix of the attention semantic graph, the elements on the diagonal of the adjacency matrix of the attention enhanced semantic graph are the attention weight values of each bone node in the bone sequence, and the remaining elements are the attention weight values between any two bone nodes, so as to characterize the attention weights of each bone node in the bone sequence and the connection relationship between any two bone nodes.

[0169] The first feature extraction module 602 is specifically used for:

[0170] Perform a GCN operation on the attention-enhanced semantic graph to extract the third feature corresponding to the skeleton sequence.

[0171] In some embodiments, the neighbor nodes in the attention-enhanced semantic graph are further divided into subsets based on the distance between the neighbor nodes and the root node, and the first feature extraction module 602 is specifically used to:

[0172] The attention-enhanced semantic graph is subjected to GCN operation based on the neighbor node subset where the neighbor node is located.

[0173] In some embodiments, the model training module 604 is specifically configured to:

[0174] determining a recognition error of the behavior prediction based on a prediction result of the behavior prediction;

[0175] Based on the recognition error of behavior prediction, the back propagation algorithm is used to adjust the model parameters of the behavior recognition model.

[0176] In some embodiments, the model training module 604 is further configured to:

[0177] Continuing to adjust model parameters based on the adjusted behavior recognition model and the skeleton sequence in the training sample set;

[0178] Determining whether the number of adjustments to the model parameters of the behavior recognition model reaches a set number, or determining whether the recognition error of the behavior prediction is less than or equal to a preset threshold;

[0179] The adjustment of the model parameters of the behavior recognition model is completed, and a trained behavior recognition model is obtained.

[0180] In actual application, the first semantic graph generation module 601, the first feature extraction module 602, the first behavior prediction module 603, and the model training module 604 can be implemented by a processor in the behavior recognition model training device. Of course, the processor needs to run the computer program in the memory to realize its functions.

[0181] It should be noted that the behavior recognition model training device provided in the above embodiment only uses the division of the above program modules as an example when performing behavior recognition model training. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the behavior recognition model training device provided in the above embodiment and the behavior recognition model training method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0182] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides a behavior recognition device, which corresponds to the above-mentioned behavior recognition method. The various steps in the above-mentioned behavior recognition method embodiment are also fully applicable to the embodiment of the present behavior recognition device.

[0183] like Figure 7 As shown, the behavior recognition device includes: an acquisition module 701, a second semantic graph generation module 702, a second feature extraction module 703 and a second behavior prediction module 704. Among them, the acquisition module 701 is used to obtain a skeleton sequence to be identified; the second semantic graph generation module 702 is used to generate a semantic graph for the skeleton sequence to be identified; the second behavior prediction module 703 is used to extract features from the semantic graph based on GCN, and the second behavior prediction module 704 is used to perform behavior prediction on the skeleton sequence to be identified based on the extracted features, and determine the behavior corresponding to the skeleton sequence to be identified.

[0184] Here, the second semantic graph generation module 702, the second feature extraction module 703, and the second behavior prediction module 704 are trained based on the behavior recognition model training device of the aforementioned embodiment. The second semantic graph generation module 702 corresponds to the trained first semantic graph generation module 601, the second feature extraction module 703 corresponds to the trained first feature extraction module 602, and the second behavior prediction module 704 corresponds to the trained first behavior prediction module 603.

[0185] In actual application, the acquisition module 701, the second semantic graph generation module 702, the second feature extraction module 703 and the second behavior prediction module 704 can be implemented by a processor in the behavior recognition device. Of course, the processor needs to run the computer program in the memory to implement its functions.

[0186] It should be noted that the behavior recognition device provided in the above embodiment only uses the division of the above-mentioned program modules as an example to illustrate behavior recognition. In actual applications, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the above-mentioned processing. In addition, the behavior recognition device provided in the above embodiment and the behavior recognition method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0187] Based on the hardware implementation of the above program modules, and in order to implement the behavior recognition model training method of the embodiment of the present invention, the embodiment of the present invention also provides a behavior recognition model training device. Figure 8 Only the exemplary structure of the behavior recognition model training device is shown, not the entire structure, and can be implemented as needed. Figure 8 Partial or complete structure shown.

[0188] like Figure 8 As shown, the behavior recognition model training device 800 provided by the embodiment of the present invention includes: at least one processor 801, a memory 802, a user interface 803 and at least one network interface 804. The various components in the behavior recognition model training device 800 are coupled together through a bus system 805. It can be understood that the bus system 805 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 805 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 8 Various buses are labeled as bus system 805.

[0189] The user interface 803 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.

[0190] The memory 802 in the embodiment of the present invention is used to store various types of data to support the operation of the behavior recognition model training device. Examples of such data include: any computer program used to operate on the behavior recognition model training device.

[0191] The behavior recognition model training method disclosed in the embodiments of the present invention can be applied to or implemented by the processor 801. The processor 801 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the behavior recognition model training method can be completed by the hardware integrated logic circuit in the processor 801 or by instructions in the form of software. The above-mentioned processor 801 can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 801 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 802. The processor 801 reads the information in the memory 802 and, in combination with its hardware, completes the steps of the behavior recognition model training method provided in the embodiments of the present invention.

[0192] In an exemplary embodiment, the behavior recognition model training device can be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to execute the aforementioned method.

[0193] Based on the hardware implementation of the above program modules, and in order to implement the behavior recognition method of the embodiment of the present invention, the embodiment of the present invention further provides a behavior recognition device. Figure 9 Only the exemplary structure of the behavior recognition device is shown, not the entire structure. Figure 9 Partial or complete structure shown.

[0194] like Figure 9As shown, the behavior recognition device 900 provided by the embodiment of the present invention includes: at least one processor 901, a memory 902, a user interface 903 and at least one network interface 904. The various components in the behavior recognition device 900 are coupled together through a bus system 905. It can be understood that the bus system 905 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 905 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 9 Various buses are labeled as bus system 905.

[0195] The user interface 903 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.

[0196] The memory 902 in the embodiment of the present invention is used to store various types of data to support the operation of the behavior recognition device. Examples of such data include: any computer program used to operate on the behavior recognition device.

[0197] The behavior recognition model training method disclosed in the embodiments of the present invention can be applied to or implemented by the processor 901. The processor 901 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the behavior recognition model training method can be completed by the hardware integrated logic circuit in the processor 901 or by instructions in the form of software. The above-mentioned processor 901 can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 901 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 902. The processor 901 reads the information in the memory 902 and, in combination with its hardware, completes the steps of the behavior recognition model training method provided in the embodiments of the present invention.

[0198] In an exemplary embodiment, the behavior recognition device may be implemented by one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, general-purpose processors, controllers, MCUs, Microprocessors, or other electronic components to perform the aforementioned method.

[0199] It is understood that memories 802 and 902 can be volatile memory or non-volatile memory, or can include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM); magnetic surface memory can be magnetic disk or tape memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memories.

[0200] In an exemplary embodiment, the embodiment of the present invention further provides a storage medium, namely a computer storage medium, which can be a computer-readable storage medium, for example, including a memory 802 storing a computer program, and the above-mentioned computer program can be executed by the processor 801 of the behavior recognition model training device to complete the steps described in the behavior recognition model training method of the embodiment of the present invention. For another example, including a memory 902 storing a computer program, the above-mentioned computer program can be executed by the processor 901 of the behavior recognition device to complete the steps described in the behavior recognition method of the embodiment of the present invention. The computer-readable storage medium can be a memory such as ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0201] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0202] In addition, the technical solutions described in the embodiments of the present invention can be arbitrarily combined without conflict.

[0203] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A behavior recognition model training method, characterized in that: include: Generate a semantic graph of the skeleton sequence based on the skeleton sequence in the training sample set, the semantic graph at least including: a structural semantic graph, wherein the neighbor node range corresponding to the root node in the structural semantic graph is L hops, where L is a natural number greater than 1; Performing a graph convolutional network (GCN) operation on the semantic graph to extract features corresponding to the skeleton sequence; Performing behavior prediction on the skeleton sequence based on the extracted features; Adjust the model parameters of the behavior recognition model based on the recognition error of the behavior prediction to obtain a trained behavior recognition model; The semantic graph further includes: an implicit relationship semantic graph, and generating the implicit relationship semantic graph of the skeleton sequence includes: Extracting behavior-related features corresponding to each skeletal node of the skeletal sequence, and determining a first relationship value between each skeletal node itself and the behavior and a second relationship value between the behavior-related features corresponding to any two skeletal nodes; Determining an adjacency matrix of the implicit relationship semantic graph based on the first relationship value and the second relationship value, wherein the elements on the diagonal of the adjacency matrix of the implicit relationship semantic graph are the first relationship values corresponding to each skeletal node, and the remaining elements are the second relationship values between the features of any two skeletal nodes, so as to characterize the implicit connection relationship related to the behavior between different skeletal nodes in the skeletal sequence; The performing of GCN operation on the semantic graph to extract features corresponding to the skeleton sequence further includes: Perform a GCN operation on the implicit relationship semantic graph to extract the second feature corresponding to the implicit relationship semantic graph of the skeleton sequence.

2. The method according to claim 1, characterized in that The neighbor nodes in the structural semantic graph are further divided into subsets based on the distance between the neighbor nodes and the root node, and the GCN operation is performed on the semantic graph to extract features corresponding to the skeleton sequence, including: A GCN operation is performed on the structural semantic graph based on the neighbor node range and the neighbor node subset in which the neighbor node is located, so as to extract the first feature corresponding to the structural semantic graph of the skeleton sequence.

3. The method according to claim 1, characterized in that The neighbor nodes in the implicit relationship semantic graph are further divided into subsets based on the distance between the neighbor nodes and the root node, and the GCN operation on the implicit relationship semantic graph includes: A GCN operation is performed on the implicit relationship semantic graph based on a neighbor node subset where the neighbor node is located.

4. The method according to any one of claims 1 to 3, characterized in that The semantic graph further includes an attention-enhanced semantic graph, and generating the attention-enhanced semantic graph of the skeleton sequence includes: Determine an adjacency matrix of the attention-enhanced semantic graph, wherein the elements on the diagonal of the adjacency matrix of the attention-enhanced semantic graph are the attention weight values of each bone node in the bone sequence, and the remaining elements are the attention weight values between any two bone nodes, so as to characterize the attention weights of each bone node in the bone sequence and the connection relationship between any two bone nodes; The performing of GCN operation on the semantic graph to extract features corresponding to the skeleton sequence further includes: Perform a GCN operation on the attention-enhanced semantic map to extract the third feature corresponding to the attention-enhanced semantic map of the skeleton sequence.

5. The method according to claim 4, characterized in that The neighbor nodes in the attention-enhanced semantic graph are further divided into subsets based on the distance between the neighbor nodes and the root node, and the GCN operation on the attention-enhanced semantic graph includes: The attention-enhanced semantic graph is subjected to GCN operation based on the neighbor node subset where the neighbor node is located.

6. The method according to claim 1, characterized in that The step of adjusting the model parameters of the behavior recognition model based on the recognition error of the behavior prediction includes: determining a recognition error of the behavior prediction based on a prediction result of the behavior prediction; Based on the recognition error of behavior prediction, the back propagation algorithm is used to adjust the model parameters of the behavior recognition model.

7. The method according to claim 6, characterized in that The step of adjusting the model parameters of the behavior recognition model based on the recognition error of the behavior prediction further includes: Continuing to adjust model parameters based on the adjusted behavior recognition model and the skeleton sequence in the training sample set; Determining whether the number of adjustments to the model parameters of the behavior recognition model reaches a set number, or determining whether the recognition error of the behavior prediction is less than or equal to a preset threshold; The adjustment of the model parameters of the behavior recognition model is completed, and a trained behavior recognition model is obtained.

8. A behavior recognition method, characterized in that: include: Obtain the bone sequence to be identified; Generate a semantic graph for the skeleton sequence to be identified using a behavior recognition model trained by the method according to any one of claims 1 to 7; The behavior recognition model extracts features from the semantic graph based on GCN; The behavior recognition model performs behavior prediction on the skeleton sequence to be identified based on the extracted features, and determines the behavior corresponding to the skeleton sequence to be identified.

9. A behavior recognition model training device, characterized in that: include: A first semantic graph generation module is configured to generate a semantic graph of the skeleton sequence based on the skeleton sequence in the training sample set, wherein the semantic graph at least includes: a structural semantic graph, wherein the neighbor node range corresponding to the root node in the structural semantic graph is L hops, where L is a natural number greater than 1; A first feature extraction module is used to perform GCN operation on the semantic graph to extract features of the skeleton sequence; A first behavior prediction module, configured to perform behavior prediction on the skeleton sequence based on the extracted features; The model training module is used to adjust the model parameters of the behavior recognition model based on the recognition error of the behavior prediction to obtain a trained behavior recognition model; The semantic graph further includes an implicit relationship semantic graph, and the first semantic graph generation module is further configured to: Extracting behavior-related features corresponding to each skeletal node of the skeletal sequence, and determining a first relationship value between each skeletal node itself and the behavior and a second relationship value between the behavior-related features corresponding to any two skeletal nodes; Determining an adjacency matrix of the implicit relationship semantic graph based on the first relationship value and the second relationship value, wherein the elements on the diagonal of the adjacency matrix of the implicit relationship semantic graph are the first relationship values corresponding to each skeletal node, and the remaining elements are the second relationship values between the features of any two skeletal nodes, so as to characterize the implicit connection relationship related to the behavior between different skeletal nodes in the skeletal sequence; The first feature extraction module is further configured to: Perform a GCN operation on the implicit relationship semantic graph to extract the second feature corresponding to the implicit relationship semantic graph of the skeleton sequence.

10. A behavior recognition device, characterized in that: include: An acquisition module, used to obtain the skeleton sequence to be identified; A second semantic graph generating module is used to generate a semantic graph for the skeleton sequence to be identified; A second feature extraction module is used to extract features from the semantic graph based on GCN; A second behavior prediction module is used to perform behavior prediction on the skeleton sequence to be identified based on the extracted features, and determine the behavior corresponding to the skeleton sequence to be identified; The second semantic graph generation module, the second feature extraction module and the second behavior prediction module are trained based on the behavior recognition model training device according to claim 9.

11. A behavior recognition model training device, characterized in that: include: A processor and a memory for storing a computer program capable of being executed on the processor, wherein The processor is configured to execute the steps of the method according to any one of claims 1 to 7 when running a computer program.

12. A behavior recognition device, characterized in that: include: A processor and a memory for storing a computer program capable of being executed on the processor, wherein The processor is configured to execute the steps of the method according to claim 8 when running a computer program.

13. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented, or the steps of the method according to claim 8 are implemented.

Images