Human behavior recognition method and device based on global-local collaborative graph convolution network and medium

Abstract

The application discloses a human behavior recognition method and device based on a global-local collaborative graph convolution network and a medium, relates to the technical field of human behavior recognition in the computer vision field. The method comprises the following steps: acquiring skeleton data, and preprocessing the skeleton data to obtain input data of four modes; inputting the input data of the four modes into four information flow subnetworks respectively for feature extraction, obtaining corresponding space-time features, performing vector dimension reduction on the space-time features through a global average pooling layer, and performing classification through a full connection layer to obtain a human behavior recognition result. The application can greatly improve the classification accuracy of human behaviors in the skeleton data, and further improve the accuracy of human behavior recognition.

Description

Technical Field

[0001] This invention relates to the field of human behavior recognition technology in the field of computer vision, and more specifically, to a method, apparatus and medium for human behavior recognition based on a global-local collaborative graph convolutional network. Background Technology

[0002] Human behavior recognition refers to the analysis of human movements, postures, and positions to identify and understand the behavior of individuals or groups. In recent years, with the rapid development of computer vision, human behavior recognition has shown enormous application potential in video surveillance, motion analysis, and human-computer interaction. Human behavior recognition can be based on various data formats, such as RGB video, depth images, skeleton data, and infrared data. Among these, skeleton data is particularly suitable for human behavior recognition. Skeleton data is unaffected by the external environment and is robust to changes in lighting and environment. Furthermore, skeleton data naturally preserves the spatiotemporal relationships between human body parts, enabling it to provide more efficient, stable, and reliable motion recognition solutions for various application scenarios.

[0003] Therefore, research on human behavior recognition based on skeleton data is not limited to theory; its applications in real life are also quite valuable. The method mentioned in this paper uses skeleton data to recognize human behavior.

[0004] In skeleton-based action recognition, modeling the positions and trajectories of key points on the human body allows for more accurate capture of the essential features of actions, unaffected by sensors, environment, or external interference. With the rapid development of deep learning, skeleton-based action recognition methods have gained increasing attention. Early approaches typically used convolutional neural networks (CNNs) or recurrent neural networks (RNNs). However, neither of these methods effectively represents skeleton data. To address this non-Euclidean data for human action recognition, spatiotemporal graph convolutional networks (SPCRs) emerged, aggregating spatiotemporal features using independent graph convolutional modules and temporal convolutional modules. Thus, using graph convolutional methods to solve skeleton-based action recognition has gradually become mainstream.

[0005] Existing technologies have proposed spatiotemporal graph convolutional networks (GCNs), pioneering graph convolution-based methods for solving this task. GCNs utilize predefined topologies to extract inter-joint feature information. Subsequently, numerous researchers have innovated upon this approach. However, GCN-based methods still face many challenges. The information of interest differs across actions. When processing spatial information, the skeleton graph used is heuristically predefined, representing only the physical structure of the human body. Therefore, it is difficult to ensure the extraction of useful feature representations. In some actions, dependencies may exist between non-physically connected joints, indicating that focusing solely on physical connections between joints is clearly insufficient. Although some studies have addressed this issue—for example, some techniques enhance the modeling of global contextual information by incorporating prior human knowledge, and others employ structured pooling in graph convolution to extract richer features—they all rely on basic graph convolutional architectures to solve the action recognition problem. Graph convolution primarily depends on convolutional operations within local neighborhoods, excelling at modeling local information while having relatively limited ability to capture global dependencies. Recently, due to the rapid development of Transformer models, many researchers have explored their application in skeleton-based action recognition, aiming to completely replace graph convolution. While Transformer models excel at modeling global information, they tend to overlook local features of complex actions when processing spatial information. However, in skeleton-based action recognition, local and global information are complementary and cannot be ignored. For example, for the actions of "reading" and "writing," the model learns global features such as the position of hand joints and head orientation for initial recognition. However, since the actions are mainly concentrated in the hand joints, to effectively distinguish between the two, the model should be guided to pay more attention to local feature information, especially the subtle movements of the hand joints. For "side kick," the local features of the leg joints reflect the basic form of the action, and the coordination between the leg joints and the torso reflects the overall coordination; local and global information complement each other. Recognizing these actions will significantly test the model's ability to aggregate local and global features. Therefore, there is still room for improvement in simultaneously representing the dependencies between local and global features in skeleton information.

[0006] Furthermore, the mainstream method for capturing temporal features currently is to use multi-scale temporal convolution modules, which employ convolutional kernels of different sizes to capture various temporal dependencies. However, convolution is essentially still a locally linear operation. Compared to models like the Transformer, which can model long-term non-linear dependencies, relying solely on multi-scale temporal convolution to extract temporal features has certain limitations.

[0007] In the research for realizing this invention, the inventors found that the existence of the above-mentioned problems would affect the final classification accuracy to some extent. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides a method, apparatus, and medium for human behavior recognition based on a global-local collaborative graph convolutional network, thereby improving the classification accuracy of human behavior and ultimately enhancing the accuracy of human behavior recognition.

[0009] In a first aspect, the present invention provides a human behavior recognition method based on a global-local collaborative graph convolutional network, the method comprising:

[0010] Acquire skeleton data and preprocess the skeleton data to obtain input data for four modalities;

[0011] The input data of the four modalities are respectively input into four information flow sub-networks for feature extraction to obtain the corresponding spatiotemporal features. Based on the spatiotemporal features, the vector dimensionality is reduced by a global average pooling layer and classified by a fully connected layer to obtain the human behavior recognition result.

[0012] The information flow sub-network includes several basic blocks, each of which contains a spatial feature module and a temporal feature module.

[0013] The spatial feature module takes the input data as input and inputs the input data into a first graph structure and a second graph structure respectively. The first graph structure obtains weight coefficients by calculating the first embedding vector and the second embedding vector in the self-attention mechanism, and obtains a data-driven dynamic perception graph based on the weight coefficients. The second graph structure combines nearest neighbors and minimum thresholds to determine the connection combination related to the action and obtains a node-enhanced spatiotemporal graph. Based on the dynamic perception graph and the node-enhanced spatiotemporal graph, a comprehensive matrix containing global spatial features and local spatial features is determined. Based on the comprehensive matrix, graph convolution is performed on the input data to obtain an output feature vector.

[0014] The time feature module takes the output feature vector as input, processes the output feature vector through convolutional branch and Transformer branch respectively to obtain a first output and a second output, and fuses the first output and the second output to obtain the final output.

[0015] Preferably, the input data for the four modalities includes input data for joint point mode, bone mode, joint motion mode, and bone motion mode. Skeleton data is obtained, and the skeleton data is preprocessed to obtain the input data for the four modalities, specifically including:

[0016] Based on the skeleton data, the joint coordinates are extracted as input data for the joint modality;

[0017] The bone vector is used as the input data for the bone modality, and the formula for calculating the bone vector is as follows;

[0018] b i,j,t =v i，t -v j,t

[0019] In the formula, b i,j,t Let v represent the bone vector at time t. i,t v represents the position of joint i at time t. j,t This indicates the position of joint point j at time t, where joint point i and joint point j are adjacent joint points;

[0020] The positional change of each joint point between adjacent frames is used as the input data for the joint motion mode. The formula for calculating the positional change of each joint point between adjacent frames is as follows:

[0021] Δv i，t =v i，t -v i，t-1

[0022] In the formula, Δv i,t This indicates the positional change of keypoint i between adjacent frames, v i,t-1 This indicates the position of joint i at time t-1;

[0023] The change in bone vectors between adjacent frames is used as the input data for bone motion modality. The formula for calculating the change in bone vectors between adjacent frames is as follows:

[0024] Δb i，j，t =b i，j，t -b i，j，t-1

[0025] In the formula, Δb i,j,t b represents the change in the bone vector at time t between adjacent frames. i,j,t-1 This represents the bone vector at time t-1.

[0026] Preferably, the first graph structure is obtained as a data-driven dynamic perception graph in the following manner:

[0027] The first feature vector is obtained by averaging the input data over the time dimension. in, Let N represent the real number space, N represent the batch size of the data, V represent the number of nodes, and C represent the number of nodes. out Output channel dimension;

[0028] The second feature vector X′ is obtained by adding spatial location information to the first feature vector through an embedding layer. s ;

[0029] Based on the second feature vector, the attention tensor calculated from the first embedding vector and the second embedding vector is used as the dynamic perception map. The calculation process is as follows:

[0030]

[0031] In the formula, A g This represents a dynamic perceptron graph, and softmax represents a normalized exponential function. Let C represent the first learnable parameter and the second learnable parameter, respectively. in Indicates the input channel, C mid This indicates the output channel, and T indicates the matrix transpose.

[0032] Preferably, the second graph structure combines nearest neighbor and minimum threshold to determine the action-related connection combinations using the following formula, resulting in a node-enhanced spatiotemporal graph:

[0033]

[0034] In the formula, A l Represents a node-enhanced spatiotemporal graph. Let λ represent the set of the r largest elements in the i-th row of the adjacency matrix A, and let λ represent a predefined threshold. ij This represents the element in the i-th row and j-th column of the adjacency matrix.

[0035] Preferably, in the spatial feature module, the comprehensive matrix is ​​calculated using the following formula:

[0036] A′=(1-γ)A g +γA l

[0037] In the formula, A' represents the comprehensive matrix, and γ represents the learnable weight coefficients;

[0038] Based on the comprehensive matrix, the process of performing graph convolution on the input data to obtain the output feature vector is as follows:

[0039] X out =∑A′X in W

[0040] In the formula, These represent the input data and the output feature vector, respectively, where T' is the number of frames. This is the weight parameter matrix.

[0041] Preferably, in the time feature module, the first output and the second output are fused to obtain the final output Y using the following formula. out :

[0042] Y out =Y CNN +YTransformer

[0043] In the formula, Y CNN Y represents the output of the convolution branch. Transformer This indicates the output of the Transformer branch.

[0044] Preferably, in the time feature module, the output of the Transformer branch is obtained in the following way:

[0045] Calculate the mean and standard deviation of the input values, and then use the obtained mean and standard deviation to perform z-score standardization on the input values ​​to obtain standardized values;

[0046] Based on the standardized values, the weights are calculated using a weighting function based on a Gaussian distribution;

[0047] The input values ​​are squared and then summed with the weights to obtain the feature weights.

[0048] Feature values ​​are obtained based on the aforementioned feature weights and Top-k feature selection.

[0049] Based on the aforementioned feature values, the recasting input X is calculated using the following formula. kv :

[0050] X kv =LayerNorm(Concat(X) topk ,X avgpool ))

[0051] In the formula, LayerNorm represents the layer normalization operation, Concat represents the concatenation operation, and X... avgpool X represents the feature vector obtained after average pooling. topk Represents eigenvalues;

[0052] Based on recast input X kv The output of the Transformer branch is determined by the following formula:

[0053]

[0054] In the formula, X Trans W represents the input value. q W k W v Let C represent the learnable parameter matrix of the first embedding vector, the second embedding vector, and the third embedding vector. in represents the input channel, softmax represents the normalized exponential function, and T represents the matrix transpose.

[0055] Preferably, the spatiotemporal features obtained after processing multiple basic blocks are in the form of (N×M,C,T',V), where N, M, C, T', and V represent the batch size, number of users, number of channels, number of frames, and number of nodes, respectively. Global average pooling is used to reduce the dimensionality of the spatiotemporal features, resulting in a two-dimensional vector. This two-dimensional vector has two dimensions: the first represents the number of channels, and the second represents the number of categories. The two-dimensional vector is then fed into a fully connected layer for classification, and identification is performed based on the output score. The calculation process is as follows:

[0056] Y = W j ×Y j +W b ×Y b +W jm ×Y jm +Wb m ×Y bm

[0057] In the formula, Y represents the output score, and W... j W b W jm and W bm Y represents the learnable weights of the joint mode, bone mode, joint motion mode, and bone motion mode, respectively. j Y b Y jm and Y bm These are two-dimensional vectors representing the joint mode, bone mode, joint motion mode, and bone motion mode, respectively.

[0058] Secondly, the present invention provides a human behavior recognition device based on a global-local collaborative graph convolutional network, the device comprising:

[0059] The data preprocessing module is configured to acquire skeleton data and preprocess the skeleton data to obtain input data for four modalities.

[0060] The behavior recognition module is configured to input the input data of the four modalities into four information flow sub-networks respectively for feature extraction to obtain the corresponding spatiotemporal features. Based on the spatiotemporal features, the vector dimensionality is reduced through a global average pooling layer and classified through a fully connected layer to obtain the human behavior recognition result.

[0061] The information flow sub-network includes several basic blocks, each of which contains a spatial feature module and a temporal feature module.

[0062] The spatial feature module takes the input data as input and inputs the input data into a first graph structure and a second graph structure respectively. The first graph structure obtains weight coefficients by calculating the first embedding vector and the second embedding vector in the self-attention mechanism, and obtains a data-driven dynamic perception graph based on the weight coefficients. The second graph structure combines nearest neighbors and minimum thresholds to determine the connection combination related to the action and obtains a node-enhanced spatiotemporal graph. Based on the dynamic perception graph and the node-enhanced spatiotemporal graph, a comprehensive matrix containing global spatial features and local spatial features is determined. Based on the comprehensive matrix, graph convolution is performed on the input data to obtain an output feature vector.

[0063] The time feature module takes the output feature vector as input, processes the output feature vector through convolutional branch and Transformer branch respectively to obtain a first output and a second output, and fuses the first output and the second output to obtain the final output.

[0064] Thirdly, the present invention provides a readable storage medium storing one or more programs that can be executed by one or more processors to implement the method described above.

[0065] The present invention has at least the following beneficial effects:

[0066] 1. This invention employs a four-stream parallel structure encompassing joint position, velocity, bone vector, and velocity, enabling the model to simultaneously capture static posture, dynamic motion trends, skeletal geometric relationships, and structural change features. This multi-stream complementary information effectively avoids the limitations of a single feature stream, significantly improving the comprehensiveness of feature representation, especially in complex movements (such as rapid, continuous actions or subtle posture changes).

[0067] 2. This invention stacks multiple spatiotemporal basic blocks in each sub-network, and achieves high-order spatiotemporal feature learning through layer-by-layer abstraction, thereby improving the modeling ability of long-term action sequences (such as dance and sports movements), while enhancing detail perception by combining local and global features.

[0068] 3. The self-attention mechanism enables the spatial graph structure to dynamically adjust according to the input data, breaking through the limitations of the traditional fixed adjacency matrix. For example, in the "waving" action, the association weights between the hand joints and the shoulder are automatically strengthened, while in the "kicking" action, the interaction of the leg joints is focused, achieving motion-sensitive feature aggregation.

[0069] 4. This invention significantly enhances the robustness of the model to real-world scenarios such as viewpoint changes and partial joint occlusion through multi-stream feature fusion, making it suitable for non-ideal data such as surveillance videos.

[0070] 5. This invention reduces the computational complexity of joint modeling by separating the spatiotemporal processing modules (space first, then time), while the sparse connectivity of graph convolution reduces the number of parameters compared to fully connected networks. The multi-stream structure allows for distributed computing while maintaining accuracy, making it possible to accurately identify human behavior in real time.

[0071] 6. The hybrid feature flow design (position + velocity + skeleton vector) of this invention is compatible with different skeleton annotation specifications. The calculation of velocity features alleviates the data differences caused by different acquisition frame rates and enhances the model's ability to transfer across datasets. Attached Figure Description

[0072] Figure 1 A general structure diagram of a network model for implementing a human behavior recognition method according to an embodiment of the present invention is shown;

[0073] Figure 2 A flowchart illustrating an overall process for human behavior recognition based on a global-local collaborative graph convolutional network according to an embodiment of the present invention is shown.

[0074] Figure 3 A structural diagram of a spatial feature module according to an embodiment of the present invention is shown;

[0075] Figure 4 A structural diagram of a time feature module according to an embodiment of the present invention is shown;

[0076] Figure 5 A structural diagram of a human behavior recognition device based on a global-local collaborative graph convolutional network according to an embodiment of the present invention is shown. Detailed Implementation

[0077] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and specific examples, but this is not intended to limit the present invention. If there is no necessary sequential relationship between the various steps described herein, the order in which they are described as examples should not be considered a limitation. Those skilled in the art should understand that the order can be adjusted, as long as it does not disrupt the logical consistency between them and render the entire process impossible.

[0078] This invention provides a human behavior recognition method based on a global-local collaborative graph convolutional network. This method can be implemented using a network model comprising four information flow sub-networks: joint position flow, joint velocity flow, skeleton vector flow, and skeleton velocity flow. Each sub-network consists of ten basic blocks, each containing spatial and temporal feature processing blocks. The model takes a skeleton sequence as input and extracts its spatiotemporal features through the ten basic modules. Then, a global average pooling layer is used to reduce the dimensionality of the vectors, and finally, a linear transformation is applied to map the channel data to the number of recognized action categories. (This is just an example.) Figure 1 The diagram shows the overall structure of the network model used to implement the human behavior recognition method. X: [N×3×T'×V] represents the collected skeleton data. Part 1 is used to preprocess the collected skeleton data to obtain input data for joint point modality, bone modality, joint motion modality, and bone motion modality. The input data for these four modalities are respectively input into the corresponding information flow sub-networks for processing. Each information flow sub-network's input data processing flow includes at least Part 2 and Part 3. Part 2 processes the data through a constructed spatial feature module, and Part 3 processes it through a constructed temporal feature module. Finally, the processed features are classified using GAP+FC (classification layer) to output the recognition result. For example, Figure 1 The skeleton data in the example shows that the human behavior identified is "waving".

[0079] Specifically, such as Figure 2 A flowchart illustrating an overall process for human behavior recognition based on a global-local collaborative graph convolutional network according to an embodiment of the present invention is shown. This human behavior recognition method based on a global-local collaborative graph convolutional network includes the following steps S10 and S20.

[0080] S10: Acquire skeleton data and preprocess the skeleton data to obtain input data for four modalities.

[0081] For example, the skeleton data in this embodiment is obtained by using the NTU-RGB+D 60 and NTU-RGB+D 120 datasets. These datasets contain 60 categories of everyday actions, such as waving, standing up, and sitting down, covering multiple daily life scenarios and exhibiting high diversity in action types and postures. 3D skeleton data is extracted from the NTU-RGB+D 60 and NTU-RGB+D 120 datasets. Each frame of data includes the 3D coordinates of 25 joints, and the data is encapsulated in a 4D matrix format.

[0082] Based on the extracted skeleton data, the preprocessing method is as follows: First, standardization is performed, including subtracting the center point coordinates and normalizing the positions of each joint. Second, in order to adapt to the input of each information flow sub-network, the obtained skeleton data needs to be computationally processed.

[0083] In some embodiments, the input data for the four modalities includes input data for joint point modality, bone modality, joint motion modality, and bone motion modality. For the preprocessing of the skeleton data, the objective of this invention is to obtain the input forms of the four modalities from the dataset. The specific process and formulas are as follows:

[0084] 1) Joint mode

[0085] Joint data directly represents the position of joints in each frame of the skeleton data. Typically, skeleton data provides joint coordinates, such as 3D coordinates (x, y, z), which can be obtained directly from the dataset itself.

[0086] 2) Bone modality

[0087] Bone modality refers to the bone vector in each frame. The bone vector can be obtained from the positional difference between adjacent joints, expressed by the following formula:

[0088] b i,j,t =v i，t -v j,t

[0089] In the formula, b i,j,t Let v represent the bone vector at time t. i,t v represents the position of joint i at time t. j,t Let represent the position of joint j at time t, where joint i and joint j are adjacent joints.

[0090] 3) Joint motion modes

[0091] Joint motion mode is the positional change of each joint point between adjacent frames, expressed by the following formula:

[0092] Δv i,t =v i，t -v i,t-1

[0093] In the formula, Δv i,t This indicates the positional change of keypoint i between adjacent frames, v i,t-1 This indicates the position of joint point i at time t-1.

[0094] 4) Bone movement mode

[0095] Bone motion modality is the change in bone vectors between adjacent frames, expressed by the following formula:

[0096] Δbi，j,t =b i，j，t -b i，j，t-1

[0097] In the formula, Δb i,j,t b represents the change in the bone vector at time t between adjacent frames. i,j,t-1 This represents the bone vector at time t-1.

[0098] S20: Input the input data of the four modalities into four information flow sub-networks respectively for feature extraction to obtain the corresponding spatiotemporal features. Based on the spatiotemporal features, vector dimensionality reduction is performed through a global average pooling layer, and classification is performed through a fully connected layer to obtain the human behavior recognition result.

[0099] In this embodiment, the information flow sub-network includes several basic blocks, each of which contains a spatial feature module and a temporal feature module.

[0100] like Figure 3 The diagram shows the structure of the spatial feature module. In this module, two graph structures (a first graph structure and a second graph structure) are fused to extract the output feature vector. The first graph structure is used to obtain a dynamic spatiotemporal awareness graph based on a data-driven strategy, obtained through an improved attention mechanism, capturing the connections between different nodes from a global perspective. The second graph structure is a node-enhanced spatiotemporal graph obtained through the prominent node representation module designed in this invention. Starting from the original skeleton graph, this module gives higher attention to nodes directly related to actions using both nearest neighbor and minimum threshold methods. This method only retains nodes in action-related regions. Finally, this embodiment uses matrix addition to fuse the node-enhanced spatiotemporal graph and the dynamic spatiotemporal awareness graph to obtain the final hybrid graph, i.e., the output feature vector.

[0101] In some embodiments, the key to constructing the spatial feature module lies in obtaining two graph structures.

[0102] First, for the data-driven dynamic perception map, the weight coefficients are obtained by calculating the two embedding vectors Q and K in the traditional self-attention mechanism. However, unlike the traditional method, the calculated weight coefficients are not used to perform a weighted summation with the value vector to obtain the final attention output. Instead, they are used to obtain the adjacency matrix A with global spatial feature information. g .

[0103] Specifically, first, input The input data is then increased in dimensionality through convolution to accommodate subsequent feature processing. To better obtain spatial feature information, the input data is averaged over the time dimension. Then, spatial location information is added to X through an embedding layer. s We obtain X′s The Q and K embedding vectors are fed as input into the improved self-attention mechanism. Unlike traditional self-attention mechanisms, the attention tensor calculated from these vectors is used as a dynamic perceptual map carrying global information. This process can be described as follows:

[0104]

[0105] In the formula, A g This represents a dynamic perceptron graph, and softmax represents a normalized exponential function. Let C represent the first learnable parameter and the second learnable parameter, respectively. in Indicates the input channel, C mid This represents the middle channel, and T represents the matrix transpose.

[0106] For node-enhanced spatiotemporal graphs, a salient location selection module is proposed. This module combines nearest neighbor and minimum threshold methods to determine action-related connection combinations. Similarity values ​​greater than a predefined threshold λ are filtered, and the top "r" neighbors with the highest similarity for each node are retained. Simultaneously, the remaining matrix values ​​are set to zero. A new adjacency matrix is ​​then generated. Represented as:

[0107]

[0108] In the formula, A l Represents a node-enhanced spatiotemporal graph. Let λ represent the set of the r largest elements in the i-th row of the adjacency matrix A, and let λ represent a predefined threshold. ij This represents the element in the i-th row and j-th column of the adjacency matrix.

[0109] After obtaining the two graphs, the node-enhanced spatiotemporal graph and the dynamic spatiotemporal-aware graph were weighted and fused using matrix addition. Weight coefficients were set to control the contribution of each feature matrix to the fusion result. This graph fusion strategy yields a comprehensive matrix that includes both global and local spatial features, expressed by the formula:

[0110] A′=(1-γ)A g +γA l

[0111] In the formula, A' represents the comprehensive matrix, and γ represents the learnable weight coefficients.

[0112] Finally, given the input feature vector X in The formula for graph convolution, based on the adjacency matrix A′ obtained after the above processing, is as follows:

[0113] X out =∑A′X in W

[0114] In the formula, These represent the input data and the output feature vector, respectively, where T' is the number of frames. This is the weight parameter matrix.

[0115] like Figure 4 The diagram shows the structure of the temporal feature module. Within this module, a dual-branch structure is designed, incorporating both convolutional and Transformer operations. Similar to the multi-scale temporal convolution module, pooling and 5x1 convolutions are used in parallel on the convolutional branch to capture local temporal dynamics from the time series. The Transformer branch primarily extracts global features from the time series. Inside the Transformer branch, a salient location selection module enhances the focus on keyframes. This module adaptively selects keyframes using a top-k feature selection method. Finally, the feature information extracted from the two branches is added and fused to obtain the final output.

[0116] In some embodiments, the temporal feature module mainly consists of convolution-based branches and Transformer-based branches. The final output is obtained by adding and fusing the outputs from the two branches, as shown in the following formula:

[0117] Y out =Y CNN +Y Transformer

[0118] in Represents the output of the convolution branch. This represents the output of the Transformer branch.

[0119] Specifically, for the convolution branch, 3×1 max pooling and 5×1 convolution operations are introduced, and reasonable convolution padding and stride are set to adapt to the module's output.

[0120] For the Transformer branch, this design aims to leverage the Transformer's ability to process global information, proposing an improved temporal self-attention mechanism for extracting global temporal features. However, due to the high complexity of the Transformer, directly applying this method to process input data often yields poor results. Therefore, a salient location selection module for extracting temporal features is designed to select keyframes in the time series.

[0121] Specifically, this design first addresses the input of the time feature module. Calculate the mean μ and standard deviation σ, and then use the obtained mean and standard deviation to perform z-score normalization on the input tensor:

[0122]

[0123] Here, ε represents a constant used to prevent division by zero errors.

[0124] Subsequently, to help select more representative features without being influenced by outliers, this embodiment calculates the weights using a Gaussian distribution-based weighting function based on the standardized values:

[0125]

[0126] To enhance the impact of prominent features, the input values ​​are squared and then summed with the obtained weights ω to obtain the final feature weights and X′. Trans :

[0127]

[0128] Next, using X′ Trans Top-k feature selection is performed to obtain

[0129] X topk =Topk(X′) Trans ,k)

[0130] Where k is a learnable parameter, set to 20.

[0131] Subsequently, average pooling was introduced to obtain global feature information. The X obtained after the SLS module... topk X obtained after average pooling avgpool Input X for splicing and recasting Trans :

[0132] X kv =LayerNorm(Concat(X) topk ,X avgpool ))

[0133] In the formula, LayerNorm represents the layer normalization operation, Concat represents the concatenation operation, and X... avgpool X represents the feature vector obtained after average pooling. topk Represents the eigenvalue.

[0134] in, The input is used to obtain the embedding vectors Key and Value in the self-attention mechanism. After the above processing, the lengths of K and V are smaller than the input, and K and V contain highly abstract feature information. Therefore, the improved self-attention mechanism is not only more efficient than the traditional self-attention mechanism, but also has a stronger ability to model global context dependencies.

[0135] In summary, the output Y of the Transformer branch can be obtained. Transformer Represented as:

[0136]

[0137] In the formula, X Trans W represents the input value. q W k W v C represents the learnable parameter matrix of the first embedding vector (q), the second embedding vector (k), and the third embedding vector (v) in self-attention. in represents the input channel, softmax represents the normalized exponential function, and T represents the matrix transpose.

[0138] Finally, the data obtained after processing through ten basic blocks is in the form of (N×M,C,T',V), where N, M, C, T', and V represent the batch size, number of users, number of channels, number of frames, and number of nodes, respectively. Global average pooling is then used to reduce the dimensionality of the data, resulting in a two-dimensional vector. The first dimension represents the number of channels, and the second dimension represents the number of categories. Finally, a fully connected layer is used for classification, and the data is identified based on the output score. This process can be described as follows:

[0139] Y = W j ×Y j +W b ×Y b +W jm ×Y jm +W bm ×Y bm

[0140] In the formula, Y represents the output score, and W... j W b W jm and W bm Y represents the learnable weights of the joint mode, bone mode, joint motion mode, and bone motion mode, respectively. j Y b Y jm and Y bm These are two-dimensional vectors representing the joint mode, bone mode, joint motion mode, and bone motion mode, respectively.

[0141] For example, identification based on output scores can be achieved through two methods: single-label classification and multi-label classification. For single-label classification, the category corresponding to the highest score is directly selected as the prediction result. For instance, if the output scores are: walking: A, waving: B, reading: C, sitting: D, where B has the highest score, then the identified human behavior is waving. Single-label classification is suitable for most everyday scenarios. If simultaneous actions are considered, multi-label classification can be used. This method requires setting independent thresholds for each category; categories with scores exceeding the thresholds are activated. For example, if the output scores are: walking: A, waving: B, reading: C, sitting: D, where walking and waving exceed the corresponding thresholds A1 and B1, respectively, while reading and sitting are less than the corresponding thresholds C1 and D1, then the walking and waving categories are activated, and the identified human behavior is "waving + walking".

[0142] This invention also provides a human behavior recognition device based on a global-local collaborative graph convolutional network, such as... Figure 5 As shown, the device includes:

[0143] The data preprocessing module 501 is configured to acquire skeleton data and preprocess the skeleton data to obtain input data for four modalities.

[0144] The behavior recognition module 502 is configured to input the input data of the four modalities into four information flow sub-networks respectively for feature extraction to obtain the corresponding spatiotemporal features. Based on the spatiotemporal features, the vector dimensionality is reduced through a global average pooling layer and classified through a fully connected layer to obtain the human behavior recognition result.

[0145] The information flow sub-network includes several basic blocks, each of which contains a spatial feature module and a temporal feature module.

[0146] The spatial feature module takes the input data as input and inputs the input data into a first graph structure and a second graph structure respectively. The first graph structure obtains weight coefficients by calculating the first embedding vector and the second embedding vector in the self-attention mechanism, and obtains a data-driven dynamic perception graph based on the weight coefficients. The second graph structure combines nearest neighbors and minimum thresholds to determine the connection combination related to the action and obtains a node-enhanced spatiotemporal graph. Based on the dynamic perception graph and the node-enhanced spatiotemporal graph, a comprehensive matrix containing global spatial features and local spatial features is determined. Based on the comprehensive matrix, graph convolution is performed on the input data to obtain an output feature vector.

[0147] The time feature module takes the output feature vector as input, processes the output feature vector through convolutional branch and Transformer branch respectively to obtain a first output and a second output, and fuses the first output and the second output to obtain the final output.

[0148] It should be noted that the structures of the various human behavior recognition devices based on global-local collaborative graph convolutional networks described in this embodiment belong to the same technical concept as the previously described human behavior recognition method based on global-local collaborative graph convolutional networks, and achieve the same beneficial effects through the same principle, which will not be elaborated here.

[0149] This invention also provides a readable storage medium storing one or more programs that can be executed by one or more processors to implement the methods described in any of the above embodiments.

[0150] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A human behavior recognition method based on a global-local collaborative graph convolutional network, characterized in that, The method includes: Acquire skeleton data and preprocess the skeleton data to obtain input data for four modalities; The input data of the four modalities are respectively input into four information flow sub-networks for feature extraction to obtain the corresponding spatiotemporal features. Based on the spatiotemporal features, the vector dimensionality is reduced by a global average pooling layer and classified by a fully connected layer to obtain the human behavior recognition result. The information flow sub-network includes several basic blocks, each of which contains a spatial feature module and a temporal feature module. The spatial feature module takes the input data as input and inputs the input data into a first graph structure and a second graph structure respectively. The first graph structure obtains weight coefficients by calculating the first embedding vector and the second embedding vector in the self-attention mechanism, and obtains a data-driven dynamic perception graph based on the weight coefficients. The second graph structure combines nearest neighbors and minimum thresholds to determine the connection combination related to the action and obtains a node-enhanced spatiotemporal graph. Based on the dynamic perception graph and the node-enhanced spatiotemporal graph, a comprehensive matrix containing global spatial features and local spatial features is determined. Based on the comprehensive matrix, graph convolution is performed on the input data to obtain an output feature vector. The time feature module takes the output feature vector as input, processes the output feature vector through a convolution branch and a Transformer branch respectively to obtain a first output and a second output, and fuses the first output and the second output to obtain the final output. In the time feature module, the output of the Transformer branch is obtained in the following way: Calculate the mean and standard deviation of the input values, and then use the obtained mean and standard deviation to perform z-score standardization on the input values ​​to obtain standardized values; Based on the standardized values, the weights are calculated using a weighting function based on a Gaussian distribution; The input values ​​are squared and then summed with the weights to obtain the feature weights. Feature values ​​are obtained based on the aforementioned feature weights and Top-k feature selection. Based on the aforementioned feature values, the recasting input is calculated using the following formula. X kv : In the formula, LayerNorm Presentation layer normalization operation, Concat This indicates a splicing operation. X avgpool This represents the feature vector obtained after average pooling. X topk Represents eigenvalues; Based on recast input X kv The output of the Transformer branch is determined by the following formula: In the formula, X Trans Indicates the input value. W q , W k , W v Let represent the learnable parameter matrix of the first embedding vector, the second embedding vector, and the third embedding vector. C in Indicates the input channel. soft max represents the normalized exponential function. T This indicates the matrix transpose.

2. The method according to claim 1, characterized in that, The input data for the four modalities includes input data for joint point mode, bone mode, joint motion mode, and bone motion mode. Skeletal data is acquired and preprocessed to obtain the input data for the four modalities, specifically including: Based on the skeleton data, the joint coordinates are extracted as input data for the joint modality; Bone vectors are used as input data for bone modalities, and the formula for calculating the bone vectors is as follows; In the formula, b i,j,t express t Bone vector at time, express t Key Moments i Location, express t Key Moments j Location, key points i and key points j Adjacent joints; The positional change of each joint point between adjacent frames is used as the input data for the joint motion mode. The formula for calculating the positional change of each joint point between adjacent frames is as follows: In the formula, Indicates key points i Positional changes between adjacent frames express t -1 key point i Location; The change in bone vectors between adjacent frames is used as the input data for bone motion modality. The formula for calculating the change in bone vectors between adjacent frames is as follows: In the formula, b i,j,t express t Changes in bone vectors between adjacent frames at any given time. b i,j,t-1 express t Bone vector at time -1.

3. The method according to claim 1, characterized in that, The first graph structure is obtained as a data-driven, dynamically perceptual graph in the following way: The first feature vector is obtained by averaging the input data over the time dimension. ,in, Let N represent the real number space, N represent the batch size of the data, and V represent the number of nodes. C out For output channels; The second feature vector is obtained by adding spatial location information to the first feature vector through an embedding layer. ; Based on the second feature vector, the attention tensor calculated from the first embedding vector and the second embedding vector is used as the dynamic perception map. The calculation process is as follows: In the formula, Represents a dynamic sensing image. soft max represents the normalized exponential function. , These represent the first learnable parameter and the second learnable parameter, respectively. C in Indicates the input channel. C mid Indicates the middle passage. T This indicates the matrix transpose.

4. The method according to claim 3, characterized in that, The second graph structure combines nearest neighbor and minimum threshold to determine action-related connection combinations using the following formula, resulting in a node-enhanced spatiotemporal graph: In the formula, A l Represents a node-enhanced spatiotemporal graph. Represents the adjacency matrix A The Middle i The largest in the row r A collection of elements This represents a predefined threshold. a ij Represents the adjacency matrix of the nth element. i Line number j The elements of the column.

5. The method according to claim 4, characterized in that, In the spatial feature module, the comprehensive matrix is ​​calculated using the following formula: In the formula, Represents the composite matrix, Represents the learnable weight coefficients; Based on the comprehensive matrix, the process of performing graph convolution on the input data to obtain the output feature vector is as follows: In the formula, , These represent the input data and the output feature vector, respectively. T’ For frame number, This is the weight parameter matrix.

6. The method according to claim 1, characterized in that, In the time feature module, the first output and the second output are fused together using the following formula to obtain the final output. Y out : In the formula, Y CNN This represents the output of the convolution branch. Y Transformer This indicates the output of the Transformer branch.

7. The method according to claim 1, characterized in that, The spatiotemporal features obtained after processing through multiple basic blocks are in the form of ( N × M , C , T ', V ),in N,M , C , T ', V These represent the batch size, number of users, number of channels, number of frames, and number of nodes, respectively. Global average pooling is used to reduce the dimensionality of the data based on the spatiotemporal features, resulting in a two-dimensional vector. This vector has two dimensions: the first represents the number of channels, and the second represents the number of categories. This two-dimensional vector is then fed into a fully connected layer for classification. Recognition is based on the output score. The calculation process is as follows: In the formula, Y Indicates the size of the output score. , , and These represent the learnable weights for the articular mode, bone mode, joint motion mode, and bone motion mode, respectively. Y j , Y b , Y jm and Y bm These are two-dimensional vectors representing the joint mode, bone mode, joint motion mode, and bone motion mode, respectively.

8. A human behavior recognition device based on a global-local collaborative graph convolutional network, characterized in that, The device includes: The data preprocessing module is configured to acquire skeleton data and preprocess the skeleton data to obtain input data for four modalities. The behavior recognition module is configured to input the input data of the four modalities into four information flow sub-networks respectively for feature extraction to obtain the corresponding spatiotemporal features. Based on the spatiotemporal features, the vector dimensionality is reduced through a global average pooling layer and classified through a fully connected layer to obtain the human behavior recognition result. The information flow sub-network includes several basic blocks, each of which contains a spatial feature module and a temporal feature module. The spatial feature module takes the input data as input and inputs the input data into a first graph structure and a second graph structure respectively. The first graph structure obtains weight coefficients by calculating the first embedding vector and the second embedding vector in the self-attention mechanism, and obtains a data-driven dynamic perception graph based on the weight coefficients. The second graph structure combines nearest neighbors and minimum thresholds to determine the connection combination related to the action and obtains a node-enhanced spatiotemporal graph. Based on the dynamic perception graph and the node-enhanced spatiotemporal graph, a comprehensive matrix containing global spatial features and local spatial features is determined. Based on the comprehensive matrix, graph convolution is performed on the input data to obtain an output feature vector. The time feature module takes the output feature vector as input, processes the output feature vector through a convolution branch and a Transformer branch respectively to obtain a first output and a second output, and fuses the first output and the second output to obtain the final output. In the time feature module, the output of the Transformer branch is obtained in the following way: Calculate the mean and standard deviation of the input values, and then use the obtained mean and standard deviation to perform z-score standardization on the input values ​​to obtain standardized values; Based on the standardized values, the weights are calculated using a weighting function based on a Gaussian distribution; The input values ​​are squared and then summed with the weights to obtain the feature weights. Feature values ​​are obtained based on the aforementioned feature weights and Top-k feature selection. Based on the aforementioned feature values, the recasting input is calculated using the following formula. X kv : In the formula, LayerNorm Presentation layer normalization operation, Concat This indicates a splicing operation. X avgpool This represents the feature vector obtained after average pooling. X topk Represents eigenvalues; Based on recast input X kv The output of the Transformer branch is determined by the following formula: In the formula, X Trans Indicates the input value. W q , W k , W v Let represent the learnable parameter matrix of the first embedding vector, the second embedding vector, and the third embedding vector. C in Indicates the input channel. soft max represents the normalized exponential function. T This indicates the matrix transpose.

9. A non-transitory computer-readable storage medium storing instructions, characterized in that, When the instructions are executed by the processor, the method according to any one of claims 1 to 7 is performed.

Images