Human body behavior recognition method and system

Abstract

The invention provides a human body behavior recognition method and system, and belongs to the field of behavior recognition. The method comprises the following steps: preprocessing a human motion video, and extracting position coordinates and motion information of key skeleton nodes; obtaining static specific similarities and dynamic specific similarities among different vertexes by utilizing the decoupled dynamic co-occurrence graph convolution; utilizing a context sensing module to obtain a sensing feature map; obtaining a dense knowledge aggregation graph by using time convolution of hierarchical dense connection; and flattening the knowledge aggregation graph to obtain a human behavior recognition result. The method solves the problem that an existing method is insufficient in characterization capacity for long-distance space-time dependency relationships and complex action features.

Description

Technical Field

[0001] The present invention belongs to the field of behavior recognition, and in particular relates to a human behavior recognition method and system. Background Art

[0002] Graph Convolutional Networks (GCNNs) have demonstrated remarkable capabilities in modeling graph-structured data dependencies, making them an effective tool for action recognition. This is demonstrated by their successful deployment in various action recognition challenges. Nevertheless, existing GCNN-driven approaches are not without their shortcomings. A major limitation is their limited focus on encoding the structural topology of graph convolution operations, which usually remain constant or commonly shared intrinsic topology across different action categories. Considering that joint interactions are motion-dependent, pre-defined graph structures may fail to capture subtle and unique relationships. In other words, to truly capture the joint dependencies of skeleton sequences, the graph topology and associated weights should be dynamically adjusted and refined throughout the end-to-end learning process. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method and system for human behavior recognition, which solves the problem that the prior methods are insufficient in characterizing long-distance spatiotemporal dependencies and complex action features.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is: a human behavior recognition method, comprising the following steps:

[0005] S1. Preprocess the human motion video to extract the position coordinates and motion information of key skeleton nodes;

[0006] S2. Based on the extracted results, the static specific similarity and dynamic specific similarity between different vertices are obtained by using the decoupled dynamic co-occurrence graph convolution;

[0007] S3, obtaining a perceptual feature map using a context-aware module according to static specific similarities and dynamic specific similarities between different vertices;

[0008] S4, based on the perceptual feature graph, a dense knowledge aggregation graph is obtained using layered densely connected temporal convolution;

[0009] S5. Flatten the knowledge aggregation graph to obtain the recognition result of human behavior.

[0010] The beneficial effects of the present invention are as follows: the present invention proposes a decoupled static-dynamic co-occurrence graph convolution, which adaptively aggregates the global correlation of inter-frame and intra-frame discriminators in a decomposed manner. The present invention also proposes a multi-level context-aware module to comprehensively model the potential significance of multiple domains in a skeleton sequence. The module refines the spatial nodes, temporal dynamics, channel characteristics and motion correlation within the graph convolution block. At the same time, the present invention also proposes a hierarchical densely connected temporal convolution, which enhances the representation of local features through partial dense connections and enriches the temporal information in the convolution process. The present invention facilitates the adaptive integration and comprehensive modeling of intra-frame and inter-frame skeleton sequence correlations, thereby effectively fine-tuning the importance of vertex and vertex connections during training iterations.

[0011] Furthermore, the S1 is specifically: preprocessing the human body motion video, and extracting the position coordinates and motion information of key skeleton nodes using the following formula:

[0012] M J[C,t,N] ={J [C,t+1,N] -J [C,t,N] |t∈[1,T-1]}

[0013] M B[C,t,N] = {B [C,t+1,N] -B [C,t,N] |t∈[1,T-1]}

[0014] Among them, M J[C,t,N] and M B[C,t,N] represents the motion features of joints and bones respectively, C, T and N represent channels, frames and joints respectively, J and B represent the three-dimensional positions of joints and bones respectively, t represents the tth frame, J [C,t+1,N] and J [C,t,N] Respectively represent the joint point coordinate information at the t+1th and tth frames, B [C,t+1,N] and B [C,t,N] Represents the bone coordinate information at the t+1th and tth frames respectively.

[0015] The beneficial effect of the above further scheme is that the present invention comprehensively considers the information in the spatial and temporal domains to more comprehensively capture the high-order features in the human skeleton sequence. By extracting the joint point coordinate information, joint point motion information, skeleton coordinate information and skeleton motion information, more abundant input information is provided for the human behavior recognition model.

[0016] Furthermore, the S2 comprises the following steps:

[0017] S201, for the time feature, the input feature map is represented as X in , where the input feature map is the extracted result;

[0018] S202, using feature conversion, the input feature map Xin Convert to high-level representation and compress the channel dimension;

[0019] S203, the feature map X after channel compression in Embed the enhanced information u1 and u2 respectively to obtain a first vector embedding and a second vector embedding;

[0020] S204, performing difference and activation operations on the first vector embedding and the second vector embedding along the channel dimension, and obtaining static feature similarity between different vertices by obtaining a first distance;

[0021] S205. For motion features, given a pair of input feature maps X i and X j , where the input feature map is the extracted result;

[0022] S206, according to the input feature map X i and X j , using the transformation function to globally compress information along the channel dimension, and obtaining the transformed third vector embedding and fourth vector embedding;

[0023] S207, performing feature rescaling and calibration processing through nonlinear projection functions θ1 and θ2 according to the third vector embedding and the fourth vector embedding;

[0024] S208. After the processing of step S207, the third vector embedding and the fourth vector embedding are subjected to difference and activation operations along the channel dimension, and the dynamic specific similarity between different vertices is obtained by calculating the second distance between the third vector embedding and the fourth vector embedding.

[0025] The beneficial effect of the above further scheme is that the present invention can effectively extract the temporal and spatial features of the human skeleton sequence by decoupling the dynamic co-occurrence graph convolution.

[0026] Furthermore, the S3 comprises the following steps:

[0027] S301, using time and space average pooling to perform dimensionality reduction processing on static specific similarity and dynamic specific similarity;

[0028] S302, after the feature map is processed by dimensionality reduction, the spatial and temporal descriptors are concatenated along the channel, and two independent fully connected layers are used to provide nonlinear interaction processing between frames;

[0029] S303, the feature map processed by the fully connected layer uses the sigmoid activation function to perceive the attention map of the spatial domain and the temporal domain;

[0030] S304, generating a spatiotemporal attention perception map according to the attention maps in the spatial domain and the temporal domain;

[0031] S305, generating weighted distinguishing features according to the temporal attention perception map and the original human motion feature map;

[0032] S306, according to the weighted distinguishing features, using channel excitation to aggregate spatiotemporal features along the spatial and temporal dimensions to generate a channel attention map;

[0033] S307, according to the generated channel attention map, compress the number of channels in proportion through a convolution layer, and model the continuous frames to construct motion dependency features;

[0034] S308, splicing the motion dependency features along the time dimension to obtain the final output motion dependency features, wherein during the splicing process, the last element is filled with 0;

[0035] S309, performing spatial average pooling and convolution operations on the motion dependency features outputted finally to generate a motion excitation map;

[0036] S3010, according to the motion excitation map and the input feature map X in , and obtain the perceptual feature map.

[0037] The beneficial effect of the above further scheme is that the present invention is based on the static specific similarity and dynamic specific similarity between different vertices and utilizes the context-aware module to effectively extract the channel features and dynamic motion features of the human skeleton sequence.

[0038] Furthermore, the S4 comprises the following steps:

[0039] S401, dividing the perception feature map into three sub-feature maps;

[0040] S402, inputting the three sub-feature graphs into three temporal convolutional layers respectively, wherein the last sub-feature graph is processed by all the third temporal convolutional layers, and the first sub-feature graph and the second sub-feature graph are hierarchically processed by the first temporal convolutional layer and the second temporal convolutional layer respectively;

[0041] S403: Obtain a dense knowledge aggregation graph based on the processing results of the three temporal convolutional layers.

[0042] The beneficial effect of the above further scheme is that the present invention gradually transfers features from early layers while maintaining local and global temporal information. Although partial cascading may not propagate all early layer features to later layers, this design reduces computational redundancy and promotes selective feature aggregation.

[0043] Furthermore, the S5 comprises the following steps:

[0044] S501, flattening the knowledge aggregation graph;

[0045] S502: Based on the flattened knowledge aggregation graph, global average pooling and Softmax classifier processing are performed to obtain the recognition result of human behavior.

[0046] The beneficial effect of the above further scheme is: the present invention flattens the knowledge aggregation graph, and cooperates with global average pooling and Softmax classifier to simplify model calculation, improve compatibility and computational efficiency, and effectively capture global features, reduce parameters and overfitting risks, provide intuitive probability output for human behavior recognition, and enhance generalization ability.

[0047] The present invention also provides a human behavior recognition system, wherein the human behavior recognition system performs a human behavior recognition method, including:

[0048] The first processing module is used to pre-process the human body motion video and extract the position coordinates and motion information of key skeleton nodes;

[0049] The second processing module is used to obtain a dense knowledge aggregation graph based on the position coordinates and motion information using a human behavior recognition model;

[0050] The third processing module is used to flatten the knowledge aggregation graph, and based on the flattened knowledge aggregation graph, perform global average pooling and Softmax classifier processing to obtain the recognition result of human behavior.

[0051] The beneficial effects of the present invention are as follows: the present invention proposes a decoupled static-dynamic co-occurrence graph convolution, which adaptively aggregates the global correlation of inter-frame and intra-frame discriminators in a decomposed manner. The present invention also proposes a multi-level context-aware module to comprehensively model the potential significance of multiple domains in a skeleton sequence. The module refines the spatial nodes, temporal dynamics, channel characteristics and motion correlation within the graph convolution block. At the same time, the present invention also proposes a hierarchical densely connected temporal convolution, which enhances the representation of local features through partial dense connections and enriches the temporal information in the convolution process. The present invention facilitates the adaptive integration and comprehensive modeling of intra-frame and inter-frame skeleton sequence correlations, thereby effectively fine-tuning the importance of vertex and vertex connections during training iterations.

[0052] Furthermore, the human behavior recognition model includes:

[0053] A decoupled dynamic co-occurrence graph convolution module is used to obtain static specific similarities and dynamic specific similarities between different vertices using decoupled dynamic co-occurrence graph convolution based on the extracted position coordinates and motion information;

[0054] A context-aware module, used to obtain a perceptual feature map according to static specific similarities and dynamic specific similarities between different vertices;

[0055] The layered densely connected temporal convolution module is used to obtain a dense knowledge aggregation graph based on the perceptual feature map using layered densely connected temporal convolution.

[0056] The beneficial effect of the above further scheme is that the human behavior recognition model provided by the present invention can achieve excellent recognition accuracy on multiple public data sets by deeply mining the spatiotemporal dependencies in the skeleton sequence, and exhibits strong versatility and robustness. The present invention is not only of great significance to the theoretical development of human behavior recognition technology, but also provides new ideas and solutions for improving the accuracy of behavior recognition in complex scenarios in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 The present invention is a flow chart of the method.

[0058] Figure 2 It is a human skeleton sequence diagram.

[0059] Figure 3 It is a decoupling diagram of time and space relationship.

[0060] Figure 4 It is a multi-level context-aware module diagram.

[0061] Figure 5 is a densely connected temporal convolutional graph.

[0062] Figure 6 The overall framework of the behavior recognition classification network.

[0063] Figure 7 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0064] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0065] Example 1

[0066] In response to the limitations described in the underlying technology, the present invention proposes a new behavior recognition method based on human skeleton nodes. This method facilitates the adaptive integration and comprehensive modeling of intra- and inter-frame skeleton sequence correlations, thereby effectively fine-tuning the importance of node and vertex connections during training iterations. The present invention proposes a decoupled static-dynamic co-occurrence graph convolution, which is specifically designed to adaptively aggregate the global correlations of inter- and intra-frame discriminators in a decomposed manner. In addition, a multi-level context-aware module is proposed to comprehensively model the potential significance of multiple domains in the skeleton sequence. This module refines the spatial nodes, temporal dynamics, channel characteristics, and motion correlations within the graph convolution block. In addition, a hierarchical densely connected temporal convolution is proposed to enhance the representation of local features through partial dense connections and enrich the temporal information in the convolution process. As Figure 1 As shown, the present invention provides a method for human behavior recognition, and its implementation method is as follows:

[0067] S1. Preprocess the human motion video to extract the position coordinates and motion information of key skeleton nodes;

[0068] In this embodiment, the human motion video is preprocessed to extract the position coordinates and motion information of key skeleton nodes. Specifically, the inter-frame motion information of the two is obtained by calculating the difference between the position coordinates of the joint points and bones between two adjacent frames. The coordinate information and motion information of the joint points and bones are used as input features of the behavior recognition classification model, such as Figure 2 As shown below:

[0069] M J[C,t,N] ={J [C,t+1,N] -J [C,t,N] |t∈[1,T-1]}

[0070] M B[C,t,N] = {B [C,t+1,N] -B [C,t,N] |t∈[1,T-1]}

[0071] Among them, M J[C,t,N] and M B[C,t,N] represents the motion features of joints and bones respectively, C, T and N represent channels, frames and joints respectively, J and B represent the three-dimensional positions of joints and bones respectively, t represents the tth frame, J [C,t+1,N] and J [C,t,N] Respectively represent the joint point coordinate information at the t+1th and tth frames, B [C,t+1,N] and B [C,t,N] Represents the bone coordinate information at the t+1th and tth frames respectively.

[0072] S2. Based on the extracted results, the static specific similarity and dynamic specific similarity between different vertices are obtained by using the decoupled dynamic co-occurrence graph convolution. The implementation method is as follows:

[0073] S201, for the time feature, the input feature map is represented as X in , where the input feature map is the extracted result;

[0074] S202, using feature conversion, the input feature map X in Convert to high-level representation and compress the channel dimension;

[0075] S203, the feature map X after channel compression in Embed the enhanced information u1 and u2 respectively to obtain a first vector embedding and a second vector embedding;

[0076] S204, performing difference and activation operations on the first vector embedding and the second vector embedding along the channel dimension, and obtaining static feature similarity between different vertices by obtaining a first distance;

[0077] S205. For motion features, given a pair of input feature maps X i and X j , where the input feature map is the extracted result;

[0078] S206, according to the input feature map X i and X j , using the transformation function to globally compress information along the channel dimension, and obtaining the transformed third vector embedding and fourth vector embedding;

[0079] S207, performing feature rescaling and calibration processing through nonlinear projection functions θ1 and θ2 according to the third vector embedding and the fourth vector embedding;

[0080] S208. After the processing of step S207, the third vector embedding and the fourth vector embedding are subjected to difference and activation operations along the channel dimension, and the dynamic specific similarity between different vertices is obtained by calculating the second distance between the third vector embedding and the fourth vector embedding.

[0081] In this embodiment, the present invention proposes a new decoupled dynamic co-occurrence graph convolution, which can decouple the entire learning goal into two subtasks, namely static correlation learning and dynamic correlation learning, while enriching the representativeness. At the same time, in order to facilitate the refinement of learning to static and dynamic representations, a static-dynamic contrast learning strategy is also proposed, which aims to maximize the mutual information between the learned semantics and guide the framework to explore unique information; human skeleton sequences show two types of correlations: explicit and implicit. Explicit correlations are inherent in the invariant physical graph structure and can be directly represented by an adjacency matrix. In contrast, implicit associations usually occur between non-physically connected joints and can change with different movements. The present invention hopes to make in-depth use of the correlation within the node and the skeleton sequence within the frame. To solve this problem, the present invention proposes a decoupled dynamic co-occurrence graph convolution, which aims to study specific contrast relationships. This convolution operation involves decomposing the overall learning goal into static correlation learning and dynamic correlation learning.

[0082] like Figure 3 As shown, for the time feature, the input feature map is represented as X in , where C represents the number of channels, N represents the number of vertices, T represents the time length, and V represents the number of nodes. Feature transformation is first applied to transform the input features into high-level representations, compressing the channel dimension from C to C1. Where f1 represents the feature transformation operation. Afterwards, the transformed feature map is embedded into the enhanced information μ1 and μ2 to obtain a vector embedding. Through the above process, the number of output channels is compressed to reduce feature redundancy and reduce computational complexity. Difference and activation operations are performed on these two vector embeddings along the channel dimension to calculate the mutual distance and obtain the static specific similarity between different vertices.

[0083] In this embodiment, for motion features, given a pair of input feature maps X i and X j , use the transformation function f2 to globally compress the information along the channel dimension and obtain the transformed vector embedding X in The dynamic representation is modeled by taking the difference of these two obtained transformation features along the time axis. Then, the two vector embeddings are subjected to difference and activation operations along the channel dimension through two independent nonlinear projection functions θ1 and θ2 for feature rescaling and calibration to calculate the mutual distance and obtain the dynamic specific similarity between different vertices. Finally, the static specific similarity is summed with the dynamic specific similarity to obtain the final output.

[0084] S3. According to the static specific similarity and dynamic specific similarity between different vertices, a perceptual feature map is obtained using a context-aware module, and the implementation method is as follows:

[0085] S301, using time and space average pooling to perform dimensionality reduction processing on static specific similarity and dynamic specific similarity;

[0086] S302, after the feature map is processed by dimensionality reduction, the spatial and temporal descriptors are concatenated along the channel, and two independent fully connected layers are used to provide nonlinear interaction processing between frames;

[0087] S303, the feature map processed by the fully connected layer uses the sigmoid activation function to perceive the attention map of the spatial domain and the temporal domain;

[0088] S304, generating a spatiotemporal attention perception map according to the attention maps in the spatial domain and the temporal domain;

[0089] S305, generating weighted distinguishing features according to the temporal attention perception map and the original human motion feature map;

[0090] S306, according to the weighted distinguishing features, using channel excitation to aggregate spatiotemporal features along the spatial and temporal dimensions to generate a channel attention map;

[0091] S307, according to the generated channel attention map, compress the number of channels in proportion through a convolution layer, and model the continuous frames to construct motion dependency features;

[0092] S308, splicing the motion dependency features along the time dimension to obtain the final output motion dependency features, wherein during the splicing process, the last element is filled with 0;

[0093] S309, performing spatial average pooling and convolution operations on the motion dependency features outputted finally to generate a motion excitation map;

[0094] S3010, input feature map X according to motion excitation map in , and obtain the perceptual feature map.

[0095] In this embodiment, the present invention proposes a new efficient multi-level context-aware module, which improves the modeling ability of spatial nodes, temporal dynamics, channel frames and skeletal sequence motion features by enabling the network to adaptively learn important features and discard irrelevant features. The context-aware module assigns different weights to respective positions and explicitly utilizes holistic representations across multiple domains, providing a solution integrated in the graph convolution block. Spatiotemporal information, channel features, and motion patterns all play a vital and mutually reinforcing role in identifying action categories, and contribute differently to classification performance. Previous studies tended to focus only on specific exploration locations and failed to fully capture the discriminative potential of skeletal data. In response to this, the present invention solves this problem by proposing a learnable structure, which includes spatiotemporal channel excitation and motion excitation components.

[0096] In this embodiment, Figure 4As shown, given an input feature f in ∈R C×N×T×V First, the input feature map is reduced in dimensionality using temporal and spatial average pooling. After the pooling operation, the spatial and temporal descriptors are concatenated along the channel axis and then fed into two independent fully connected (FC) layers to provide nonlinear interactions between frames, followed by a sigmoid activation function to perceive the attention maps in the spatial and temporal domains. Afterwards, the attentions in the spatial and temporal domains are multiplied to generate a spatiotemporal attention perception map, and finally, the spatiotemporal attention perception map is multiplied and added with the original human motion map to generate weighted discriminative features.

[0097] In this embodiment, in order to exploit global channel dependencies, a channel excitation is designed to generate a channel attention map. Specifically, two separate average pooling operations are used to aggregate spatiotemporal features along the spatial and temporal dimensions and generate complementary information for each other, compressing the feature tensor into a channel descriptor.

[0098] In this embodiment, the excitation module has sequentially included representative features of spatial, temporal and channel dimensions. However, the present invention believes that it is still insufficient in modeling explicit motion dependencies. Inspired by this idea, motion excitation is used to capture global motion representation. Specifically, given an input feature map f in ∈R C×N×T×V First, by using a 1×1 convolutional layer to compress the number of channels by a ratio r, the motion dependency f is constructed by modeling the saliency within consecutive frames. m , the perceived motion dependence f m Then concatenate along the time dimension and fill the last element with 0 to obtain the final output motion dependency feature f M By using the motion-dependent features of the final output f M Perform spatial average pooling and 1×1 convolution operations to generate a motion activation map f o ∈R C×T×1×V Finally, the motion excitation graph f o With the input feature map f in Multiply and add them sequentially to obtain the perceptual feature map of the final output.

[0099] S4. Based on the perceptual feature graph, a dense knowledge aggregation graph is obtained by using layered densely connected temporal convolution. The implementation method is as follows:

[0100] S401, dividing the perceptual feature map into three sub-feature maps;

[0101] S402, inputting the three sub-feature graphs into three temporal convolutional layers respectively, wherein the last sub-feature graph is processed by all the third temporal convolutional layers, and the first sub-feature graph and the second sub-feature graph are hierarchically processed by the first temporal convolutional layer and the second temporal convolutional layer respectively;

[0102] S403: Obtain a dense knowledge aggregation graph based on the processing results of the three temporal convolutional layers.

[0103] In this embodiment, the present invention proposes a new densely connected temporal convolution to enhance the encoding of comprehensive global dependencies along the time dimension, while retaining and utilizing more structural features in the temporal convolution operation. The partial connection design can preserve features from different layers and promote dense knowledge aggregation. Traditional temporal modeling methods typically utilize standard local two-dimensional convolutions (K×1Conv) along the time dimension with a fixed kernel size. However, this method cannot fully capture the overall global motion dynamics. To solve this problem, the present invention proposes a hierarchical densely connected temporal convolution. This innovative scheme can seamlessly transfer feature maps from previous layers to subsequent layers while minimizing repeated gradients and replacing traditional temporal convolution operations. The structure diagram is shown in Figure 5 shown.

[0104] In this embodiment, Figure 5 As shown, motion dependency is constructed by modeling saliency within consecutive frames. Specifically, given a perceptual feature map, first, it is segmented into three sub-parts f1, f2 and f3. "Temporal convolution" includes temporal convolution combined with batch normalization and ReLU activation function. Instead of directly inputting the entire feature into the block from the beginning, the present invention does it in a partially connected manner. More specifically, the three truncated feature maps are partially and hierarchically input into the first, second and third temporal convolutional layers, respectively. In the proposed method, only the f3 sub-feature map is thoroughly processed by all three temporal convolutional layers, while the other two features f1 and f2 are input into the layer midway. Therefore, thanks to the partial connection design, the segmented features f1 and f2 are not processed through the entire dense layer, and the computational cost is relatively low, compared with the fully densely connected paradigm.

[0105] In this embodiment, the hierarchical densely connected temporal convolution HDC-TConv proposed in the present invention allows information to be passed from lower-level layers to higher-level layers, which helps the network capture more global temporal representation during the feature learning process.

[0106] S5. Flatten the knowledge aggregation graph to obtain the recognition result of human behavior. The implementation method is as follows:

[0107] S501, flattening the knowledge aggregation graph;

[0108] S502: Based on the flattened knowledge aggregation graph, global average pooling and Softmax classifier processing are performed to obtain the recognition result of human behavior.

[0109] In this embodiment, the generated knowledge aggregation graph is flattened, processed by a global average pooling layer, and then the recognition result of human behavior is obtained by a Softmax classifier. The detailed structure is as follows: Figure 6 As shown in the figure, DSDC-GCN represents a graph convolutional network that decouples static-dynamic co-occurrence features.

[0110] The present invention facilitates adaptive integration and comprehensive modeling of intra-frame and inter-frame skeleton sequence correlations, thereby effectively fine-tuning the importance of vertex and vertex connections during training iterations.

[0111] Example 2

[0112] like Figure 7 As shown, the present invention provides a human behavior recognition system, and the human behavior recognition system is used to execute the method of embodiment 1, including:

[0113] The first processing module is used to pre-process the human body motion video and extract the position coordinates and motion information of key skeleton nodes;

[0114] The second processing module is used to obtain a dense knowledge aggregation graph based on the position coordinates and motion information using a human behavior recognition model;

[0115] The third processing module is used to flatten the knowledge aggregation graph, and based on the flattened knowledge aggregation graph, perform global average pooling and Softmax classifier processing to obtain the recognition result of human behavior.

[0116] The human behavior recognition model comprises:

[0117] A decoupled dynamic co-occurrence graph convolution module is used to obtain static specific similarities and dynamic specific similarities between different vertices using decoupled dynamic co-occurrence graph convolution based on the extracted position coordinates and motion information;

[0118] A context-aware module, used to obtain a perceptual feature map according to static specific similarities and dynamic specific similarities between different vertices;

[0119] The hierarchical densely connected temporal convolution module is used to obtain a dense knowledge aggregation graph based on the perceptual feature graph using hierarchical densely connected temporal convolution.

[0120] like Figure 7The human behavior recognition system provided in the illustrated embodiment can execute the technical solution shown in the human behavior recognition method in the above method embodiment, and its implementation principle and beneficial effects are similar and will not be repeated here.

[0121] In this embodiment, the present application can divide the functional units according to the human behavior recognition method. For example, each function can be divided into each functional unit, or two or more functions can be integrated into one processing unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the present invention is schematic and is only a logical division. There may be other division methods in actual implementation.

[0122] In this embodiment, in order to realize the principles and beneficial effects of the human behavior recognition method, the human behavior recognition system includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the schematic units and algorithm steps described in the embodiments disclosed in the present invention, the present invention can be implemented in the form of hardware and / or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven manner depends on the specific application and design constraints of the technical solution. Different methods can be used for each specific application to implement the described function, but such implementation should not be considered to exceed the scope of this application.

Claims

1. A human behavior recognition method, characterized in that: The following steps are involved: S1. Preprocess the human motion video to extract the position coordinates and motion information of key skeleton nodes; S2. Based on the extracted results, the static specific similarity and dynamic specific similarity between different vertices are obtained by using the decoupled dynamic co-occurrence graph convolution; S3, obtaining a perceptual feature map using a context-aware module according to static specific similarities and dynamic specific similarities between different vertices; S4, based on the perceptual feature graph, a dense knowledge aggregation graph is obtained using layered densely connected temporal convolution; S5. Flatten the knowledge aggregation graph to obtain the recognition result of human behavior.

2. The human behavior recognition method according to claim 1, characterized in that: Specifically, S1 is as follows: preprocessing the human motion video and extracting the position coordinates and motion information of key skeleton nodes using the following formula: M J[C,t,N] ={J [C,t+1,N] -J [C,t,N] |t∈[1,T-1]} M B[C,t,N] ={B [C,t+1,N] -B [C,t,N] |t∈[1,T-1]} Among them, M J[C,t,N] and M B[C,t,N] represents the motion features of joints and bones respectively, C, T and N represent channels, frames and joints respectively, J and B represent the three-dimensional positions of joints and bones respectively, t represents the tth frame, J [C,t+1,N] and J [C,t,N] Respectively represent the joint point coordinate information at the t+1th and tth frames, B [C,t+1,N] and B [C,t,N] Represents the bone coordinate information at the t+1th and tth frames respectively.

3. The human behavior recognition method according to claim 1, characterized in that: The S2 comprises the following steps: S201, for the time feature, the input feature map is represented as X in , where the input feature map is the extracted result; S202, using feature conversion, the input feature map X in Convert to high-level representation and compress the channel dimension; S203, the feature map X after channel compression in Embed the enhanced information u1 and u2 respectively to obtain a first vector embedding and a second vector embedding; S204, performing difference and activation operations on the first vector embedding and the second vector embedding along the channel dimension, and obtaining static feature similarity between different vertices by obtaining a first distance; S205. For motion features, given a pair of input feature maps X i and X j , where the input feature map is the extracted result; S206, according to the input feature map X i and X j , using the transformation function to globally compress information along the channel dimension, and obtaining the transformed third vector embedding and fourth vector embedding; S207, performing feature rescaling and calibration processing through nonlinear projection functions θ1 and θ2 according to the third vector embedding and the fourth vector embedding; S208. After the processing of step S207, the third vector embedding and the fourth vector embedding are subjected to difference and activation operations along the channel dimension, and the dynamic specific similarity between different vertices is obtained by calculating the second distance between the third vector embedding and the fourth vector embedding.

4. The human behavior recognition method according to claim 1, characterized in that: The S3 comprises the following steps: S301, using time and space average pooling to perform dimensionality reduction processing on static specific similarity and dynamic specific similarity; S302, after the feature map is processed by dimensionality reduction, the spatial and temporal descriptors are concatenated along the channel, and two independent fully connected layers are used to provide nonlinear interaction processing between frames; S303, the feature map processed by the fully connected layer uses the sigmoid activation function to perceive the attention map of the spatial domain and the temporal domain; S304, generating a spatiotemporal attention perception map according to the attention maps in the spatial domain and the temporal domain; S305, generating weighted distinguishing features according to the temporal attention perception map and the original human motion feature map; S306, according to the weighted distinguishing features, using channel excitation to aggregate spatiotemporal features along the spatial and temporal dimensions to generate a channel attention map; S307, according to the generated channel attention map, compress the number of channels in proportion through a convolution layer, and model the continuous frames to construct motion dependency features; S308, splicing the motion dependency features along the time dimension to obtain the final output motion dependency features, wherein during the splicing process, the last element is filled with 0; S309, performing spatial average pooling and convolution operations on the motion dependency features outputted finally to generate a motion excitation map; S3010, according to the motion excitation map and the input feature map X in , and obtain the perceptual feature map.

5. The human behavior recognition method according to claim 1, characterized in that: The S4 comprises the following steps: S401, dividing the perception feature map into three sub-feature maps; S402, inputting the three sub-feature graphs into three temporal convolutional layers respectively, wherein the last sub-feature graph is processed by all the third temporal convolutional layers, and the first sub-feature graph and the second sub-feature graph are hierarchically processed by the first temporal convolutional layer and the second temporal convolutional layer respectively; S403: Obtain a dense knowledge aggregation graph based on the processing results of the three temporal convolutional layers.

6. The human behavior recognition method according to claim 1, characterized in that: The S5 comprises the following steps: S501, flattening the knowledge aggregation graph; S502: Based on the flattened knowledge aggregation graph, global average pooling and Softmax classifier processing are performed to obtain the recognition result of human behavior.

7. A human behavior recognition system, characterized in that: The human behavior recognition system performs a human behavior recognition method, including: The first processing module is used to pre-process the human body motion video and extract the position coordinates and motion information of key skeleton nodes; The second processing module is used to obtain a dense knowledge aggregation graph based on the position coordinates and motion information using a human behavior recognition model; The third processing module is used to flatten the knowledge aggregation graph, and based on the flattened knowledge aggregation graph, perform global average pooling and Softmax classifier processing to obtain the recognition result of human behavior.

8. The human behavior recognition system according to claim 7, characterized in that: The human behavior recognition model comprises: A decoupled dynamic co-occurrence graph convolution module is used to obtain static specific similarities and dynamic specific similarities between different vertices using decoupled dynamic co-occurrence graph convolution based on the extracted position coordinates and motion information; A context-aware module, used to obtain a perceptual feature map according to static specific similarities and dynamic specific similarities between different vertices; The hierarchical densely connected temporal convolution module is used to obtain a dense knowledge aggregation graph based on the perceptual feature graph using hierarchical densely connected temporal convolution.

Images