Human behavior recognition method and device based on action link enhanced graph convolution network, computer program product and storage medium

Abstract

This invention discloses a method, apparatus, computer program product, and storage medium for human behavior recognition based on an action-linked enhanced graph convolutional network. The recognition process includes acquiring human skeleton sequence data, which contains the three-dimensional spatial coordinates of multiple frames of human joints; processing the human skeleton sequence data through an action-linked inference module, which dynamically infers the functional coordination relationships between joints based on the input human skeleton sequence data and generates a discretized action-linked probability matrix using a corresponding function; inputting the human skeleton sequence data and the discretized action-linked probability matrix into a graph convolutional network to output the human behavior recognition result. The graph convolutional network includes sequentially connected action-linked graph convolutional blocks, a spatiotemporal attention module, and a temporal convolutional block. This invention avoids interference from redundant skeleton data in the spatiotemporal dimension, effectively improving the behavior recognition accuracy of the proposed model.

Description

Technical Field

[0001] This invention relates to the field of computer vision recognition, specifically to a method, apparatus, computer program product, and storage medium for human behavior recognition based on an action linking augmented graph convolutional network. Background Technology

[0002] Human behavior recognition is an important foundation of human-computer interaction. Its accuracy directly affects the system's analysis and response speed of user behavior. Improving the accuracy of human behavior recognition is of great significance for abnormal behavior early warning.

[0003] In current technologies, behavior analysis based on human skeleton joint data using deep learning models has become the mainstream approach. Human skeleton data is acquired through capture devices (such as RGB-D cameras and inertial sensors) and includes the three-dimensional spatial coordinates and time-series information of joints, offering advantages such as small data volume and strong resistance to occlusion. However, existing deep models have significant shortcomings in feature extraction and redundant data processing. Traditional deep learning models (such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) mainly rely on the dependencies between local joints for modeling, ignoring the global skeleton topological features. However, human behavior is essentially driven by complex functional collaborations (i.e., action links) between joints. Existing models struggle to effectively capture long-range dependencies and dynamic associations in the global skeleton topology, resulting in limited model extraction and learning capabilities for global information and impacting the accuracy of behavior recognition. Summary of the Invention

[0004] To address the shortcomings of the existing technologies, the technical problem to be solved by this invention is: how to provide a human behavior recognition method based on action link augmented graph convolutional networks that improves the ability to extract global skeleton topology features from human skeleton joint data, thereby improving the accuracy of human behavior recognition.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A method for human behavior recognition based on action link augmented graph convolutional networks includes:

[0007] Acquire human skeleton sequence data, which contains the three-dimensional spatial coordinates of multiple frames of human joints;

[0008] The motion link inference module processes the human skeleton sequence data. The motion link inference module is used to dynamically infer the functional coordination relationship between joints based on the input human skeleton sequence data, and uses the Gumbel-Softmax function to generate a discretized motion link probability matrix.

[0009] Human skeleton sequence data and a discretized action link probability matrix are input into a graph convolutional network to output human behavior recognition results. The graph convolutional network includes sequentially connected action link graph convolutional blocks, spatiotemporal attention modules, and temporal convolutional blocks. The action link graph convolutional blocks use the discretized action link probability matrix as a dynamic adjacency matrix to perform graph convolution operations. The spatiotemporal attention module includes a spatial attention module and a temporal attention module. The spatial attention module is used to enhance the spatial features of key joints, and the temporal attention module is used to enhance the temporal features of key frames. The temporal convolutional block is used to aggregate information of the same joint point within a continuous time period through temporal convolution operations.

[0010] As an optimization, the method for generating a discretized action link probability matrix by the action link inference module includes: extracting features from the input human skeleton sequence data to obtain joint features; calculating the degree of association between joints based on the current joint features using a multilayer perceptron; updating the joint features according to the degree of association; repeating the above process K times, where K is a preset number of iterations; and discretizing the association probability obtained after the Kth iteration using the Gumbel-Softmax function to generate the action link probability matrix.

[0011] As an optimization, in the action link graph convolution block, the discretized action link probability matrix is ​​used to replace the preset fixed adjacency matrix as the dynamic adjacency matrix in the graph convolution operation.

[0012] As an optimization, the spatial attention module performs average pooling on the input features, followed by convolution operations and... An activation function generates a spatial attention map to enhance the spatial features of key joints; the temporal attention module performs average pooling on the features enhanced by the spatial attention module, and then passes them through a multilayer perceptron and... The activation function generates a temporal attention map to enhance the temporal features of keyframes.

[0013] As an optimization, the temporal convolution block adopts a temporal convolution operation with a kernel size of 1×Γ, where Γ is a hyperparameter used to control the aggregation range of adjacent time domains of the joint. Each temporal convolution aggregates information for the same joint within a continuous time period.

[0014] The present invention also discloses a human behavior recognition device based on an action link enhanced graph convolutional network, comprising: a data acquisition module, an action link inference module, and a graph convolutional network module;

[0015] The data acquisition module is used to acquire human skeleton sequence data, which contains the three-dimensional spatial coordinates of multiple frames of human joints.

[0016] The action link inference module is used to process human skeleton sequence data, dynamically infer the functional coordination relationship between joints based on the input human skeleton sequence data, and use the Gumbel-Softmax function to generate a discretized action link probability matrix.

[0017] The graph convolutional network module is used to input human skeleton sequence data and a discrete action link probability matrix together, and then process them sequentially through the action link graph convolutional block, the spatiotemporal attention module and the temporal convolutional block to output human behavior recognition results.

[0018] Action link graph convolutional blocks use a discretized action link probability matrix as a dynamic adjacency matrix to perform graph convolution operations;

[0019] The spatiotemporal attention module includes a spatial attention module and a temporal attention module. The spatial attention module is used to enhance the spatial features of key joints, and the temporal attention module is used to enhance the temporal features of key frames.

[0020] Temporal convolution blocks are used to aggregate information from the same key point within a continuous time period through temporal convolution operations.

[0021] As an optimization, the action link inference module extracts features from the input human skeleton sequence data to obtain joint features. Based on the current joint features, it calculates the correlation between joints using a multilayer perceptron and updates the joint features according to the correlation. This process is repeated K times, where K is a preset number of iterations. The correlation probability obtained after the Kth iteration is discretized using the Gumbel-Softmax function to generate an action link probability matrix.

[0022] As an optimization, the spatial attention module is used to perform average pooling on the input features, followed by convolution operations and... An activation function generates a spatial attention map to enhance the spatial features of key joints; the temporal attention module is used to perform average pooling on the features enhanced by the spatial attention module, followed by a multilayer perceptron and... The activation function generates a temporal attention map to enhance the temporal features of keyframes.

[0023] A computer program product, characterized in that it includes a computer program, which, when executed by a computer, implements the method described above.

[0024] A computer-readable storage medium having a computer program stored thereon, characterized in that: when the computer program is executed by a computer, it implements the method described above.

[0025] Compared to existing technologies, this invention uses an action link inference module to dynamically infer the functional collaborative relationships between joints based on the input skeleton data. It employs the Gumbel-Softmax function to generate a discretized action link probability matrix, replacing the preset fixed adjacency matrix. This allows the graph convolutional network to adapt to changes in joint relationships under different actions, effectively extracting global skeleton topological features. The discretized action link probability matrix suppresses weak connection noise from redundant joints, highlighting key collaborative joint pairs and improving the model's ability to focus on key features. Furthermore, spatial attention and temporal attention enhance the features of key joints and keyframes respectively, avoiding interference between spatiotemporal features and further improving recognition accuracy. Attached Figure Description

[0026] Figure 1 This is the spatiotemporal skeleton topology diagram in this invention;

[0027] Figure 2 This is a diagram of the temporal convolution operation in this invention;

[0028] Figure 3 This is the action link enhancement graph convolutional network diagram in this invention;

[0029] Figure 4 This is a diagram showing the human body movement links in this invention;

[0030] Figure 5 This is a diagram of the action link inference module in this invention;

[0031] Figure 6 This is a diagram of the spatiotemporal attention module in this invention;

[0032] Figure 7 This is a diagram showing the number of input / output channels and step size for different network layers in this invention;

[0033] Figure 8 This is a confusion matrix diagram of ALEGCNet in this invention evaluated on the NTU-RGB+D dataset for CV.

[0034] Figure 9 This is a confusion matrix diagram of ALEGCNet in this invention on the MSR-Action 3D dataset;

[0035] Figure 10 This is a 2D skeletal structure sequence diagram of the jumping behavior in this invention;

[0036] Figure 11 This is a 3D spatial skeletal structure sequence diagram of the jumping behavior in this invention;

[0037] Figure 12 This is a 2D skeletal structure sequence diagram of the fall behavior in this invention;

[0038] Figure 13 This is a 3D spatial skeletal structure sequence diagram of the fall behavior in this invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0040] The human behavior recognition method based on action link augmented graph convolutional networks in this specific embodiment includes:

[0041] Acquire human skeleton sequence data, which contains the three-dimensional spatial coordinates of multiple frames of human joints;

[0042] The motion link inference module processes the human skeleton sequence data. The motion link inference module is used to dynamically infer the functional coordination relationship between joints based on the input human skeleton sequence data, and uses the Gumbel-Softmax function to generate a discretized motion link probability matrix.

[0043] Human skeleton sequence data and a discretized action link probability matrix are input into a graph convolutional network to output human behavior recognition results. The graph convolutional network includes sequentially connected action link graph convolutional blocks, spatiotemporal attention modules, and temporal convolutional blocks. The action link graph convolutional blocks use the discretized action link probability matrix as a dynamic adjacency matrix to perform graph convolution operations. The spatiotemporal attention module includes a spatial attention module and a temporal attention module. The spatial attention module is used to enhance the spatial features of key joints, and the temporal attention module is used to enhance the temporal features of key frames. The temporal convolutional block is used to aggregate information of the same joint point within a continuous time period through temporal convolution operations.

[0044] The method for generating a discretized action link probability matrix by the action link inference module includes: extracting features from the input human skeleton sequence data to obtain joint features; calculating the degree of association between joints based on the current joint features using a multilayer perceptron; updating the joint features according to the degree of association; repeating the above process K times, where K is a preset number of iterations; and discretizing the association probability obtained after the Kth iteration using the Gumbel-Softmax function to generate the action link probability matrix.

[0045] In the action link graph convolution block, the discretized action link probability matrix is ​​used to replace the preset fixed adjacency matrix as the dynamic adjacency matrix in the graph convolution operation.

[0046] The spatial attention module performs average pooling on the input features, followed by convolution operations and... An activation function generates a spatial attention map to enhance the spatial features of key joints; the temporal attention module performs average pooling on the features enhanced by the spatial attention module, and then passes them through a multilayer perceptron and... The activation function generates a temporal attention map to enhance the temporal features of keyframes.

[0047] The temporal convolution block employs temporal convolution operations with a kernel size of 1×Γ, where Γ is a hyperparameter used to control the aggregation range of adjacent time domains of the joint. Each temporal convolution aggregates information for the same joint within a continuous time period.

[0048] A human behavior recognition device based on an action link augmented graph convolutional network includes: a data acquisition module, an action link inference module, and a graph convolutional network module;

[0049] The data acquisition module is used to acquire human skeleton sequence data, which contains the three-dimensional spatial coordinates of multiple frames of human joints.

[0050] The action link inference module is used to process human skeleton sequence data, dynamically infer the functional coordination relationship between joints based on the input human skeleton sequence data, and use the Gumbel-Softmax function to generate a discretized action link probability matrix.

[0051] The graph convolutional network module is used to input human skeleton sequence data and a discrete action link probability matrix together, and then process them sequentially through the action link graph convolutional block, the spatiotemporal attention module and the temporal convolutional block to output human behavior recognition results.

[0052] Action link graph convolutional blocks use a discretized action link probability matrix as a dynamic adjacency matrix to perform graph convolution operations;

[0053] The spatiotemporal attention module includes a spatial attention module and a temporal attention module. The spatial attention module is used to enhance the spatial features of key joints, and the temporal attention module is used to enhance the temporal features of key frames.

[0054] Temporal convolution blocks are used to aggregate information from the same key point within a continuous time period through temporal convolution operations.

[0055] The action link inference module extracts features from the input human skeleton sequence data to obtain joint features. Based on the current joint features, it calculates the correlation between joints using a multilayer perceptron and updates the joint features according to the correlation. This process is repeated K times, where K is a preset number of iterations. The correlation probability obtained after the Kth iteration is discretized using the Gumbel-Softmax function to generate an action link probability matrix.

[0056] The spatial attention module is used for average pooling of the input features, followed by convolution operations and... An activation function generates a spatial attention map to enhance the spatial features of key joints; the temporal attention module is used to perform average pooling on the features enhanced by the spatial attention module, followed by a multilayer perceptron and... The activation function generates a temporal attention map to enhance the temporal features of keyframes.

[0057] A computer program product, characterized in that it includes a computer program, which, when executed by a computer, implements the method described above.

[0058] A computer-readable storage medium having a computer program stored thereon, characterized in that: when the computer program is executed by a computer, it implements the method described above.

[0059] In the specific implementation, two datasets were used: NTU-RGB+D and MSR-Action 3D. The NTU-RGB+D dataset contains 56,880 skeleton sample sequences along with their corresponding RGB data, depth map data, and infrared image data. This dataset was obtained by 40 volunteers using Microsoft Kinect V2.0 sensors placed at the same height from three viewpoints: -45°, 0°, and +45°, capturing different behaviors. The NTU-RGB+D dataset contains 60 categories of daily behaviors, including 50 categories of single-person daily behaviors and 10 categories of two-person interactive behaviors. The MSR-Action 3D dataset is a small to medium-sized human behavior analysis dataset based on a Kinect depth camera, containing 20 human behavioral actions and a total of 567 skeleton behavior sample sequences. Each frame of skeleton data includes the spatial 3D coordinates of 20 human joints. The dataset contains 20 behavior categories: high wave (HiW), low wave (HoW), tap with hand (H), reach out and grab (Hch), punch forward (FP), throw high (HT), draw an X (DX), draw a hook (DT), draw a circle (DC), clap (HCp), raise hands and cheer (HW), punch to the side (SB), bend over (B), kick forward (FK), kick to the side (SK), jog (J), swing racket (TSw), serve upwards (TSr), golf swing (GS), pick up and throw (PT).

[0060] Before using the two datasets, the data needs to be preprocessed, including: extracting skeleton data from behavioral samples, normalizing human skeleton dimensions, and aligning the number of frames in the skeleton sequence.

[0061] (1) Extraction of skeleton data of behavioral samples

[0062] Skeleton sequences in different datasets contain some redundant information. However, an action often only requires the motion features of a few key nodes to describe it. Therefore, it is necessary to filter the skeleton data of action samples from different datasets. For example, the skeleton sequences in NTU-RGB+D cover all skeleton data and are usually a high-dimensional tensor with dimensions [M, N, C, T, V]. Here, M is the number of people, N is the number of training batches, C is the number of channels, T is the number of frames in the skeleton sequence, and V is the number of human joints. In the NTU-RGB+D dataset, this high-dimensional tensor is [2, 64, 3, 300, 25], so further extraction of this tensor is required.

[0063] (2) Normalization of human skeleton dimensions

[0064] Due to differences in age, gender, body shape, and height among the subjects in the dataset, the spatial positions of skeletal joints vary, making it difficult to adjust the model's weights. Therefore, it is necessary to normalize the skeletal size. The proposed method employs the following normalization algorithm: First, a reference skeletal frame is selected, and all other skelets are normalized in reference to this frame. For a given human skeleton, a joint is selected as the root node, such as a spinal joint. The spatial coordinates of other nodes in the skeleton are subtracted from the spatial coordinates of the root node to obtain the vector representation between the nodes and the root node. Then, its direction vector is calculated and multiplied with the root node's connection vector to obtain the corrected joint coordinates. This method normalizes the coordinates of each joint in the skeleton, resulting in corrected coordinate values ​​that are proportional to the reference skeletal frame. This process not only scales the skeletal size but also preserves the original directional and angular relationships between the skeletal joints to the greatest extent possible.

[0065] (3) Alignment of skeleton sequence frames

[0066] Different behaviors have varying durations from occurrence to cessation, resulting in inconsistent frame counts in the collected data. Therefore, aligning the frame counts of the skeleton sequence is necessary. If the 25 joints of the human body are processed at 30 frames per second, then a single skeleton frame contains the spatial coordinates (x, y, z) of the 25 joints. A behavior lasting 3 seconds would have 90 frames, and a behavior lasting 5 seconds would have 150 frames. In this embodiment, the length of the skeleton sequence in the NTU-RGBD dataset is uniformly processed to 300 frames. Sequences shorter than 300 frames are padded with zeros. For the skeleton frame length in the MSR-Action 3D dataset, it is processed to 200 frames, with zeros padded if necessary.

[0067] The topological graph of the human skeleton can be viewed as a graph G=(V,E) consisting of nodes and edges, where It is a skeleton diagram A set of key points It is a skeleton diagram A collection of skeletal edges. Skeletal edges are divided into spatial skeletal edges and temporal skeletal edges. A single-frame skeleton is composed of 25 joints of the human body. The arm skeleton is formed by connecting the elbow joint and the hand joint; then, the same joints in consecutive frames are connected to form temporal edges, thereby constructing a spatiotemporal skeleton topology, such as... Figure 1 As shown.

[0068] The spatiotemporal graph convolutional network consists of a series of ST-GCN blocks. Each module includes a spatial graph convolutional block and a temporal convolutional block, which alternately extract complex and high-level spatiotemporal co-occurrence features from the skeleton graph data. The last ST-GCN block is followed by a softmax classifier to classify the skeleton data samples. Its spatial graph convolution can be represented as:

[0069] (1);

[0070] In the formula, , where represents the normalized adjacency matrix. In the skeleton graph, the connection relationships between joints are represented by the adjacency matrix A, which indicates the connections between joints, and the degree matrix D of each joint. The self-connections of nodes are represented by the identity matrix I. and Both are learnable weights, representing the importance of capturing edges and the importance of features, respectively. The parameter p represents the neighbor node partitioning strategy, determining the type of the adjacency matrix A.

[0071] (2);

[0072] Equation (2) defines a neighbor node partitioning function, which is used to divide the neighbor nodes of each node into different subsets according to the spatial distance relationship between nodes in spatial graph convolution, thereby realizing weight sharing and differentiated feature learning. The neighbor nodes are divided into p sets, and a weight vector is assigned to each node. By partitioning, weight sharing can be achieved, and the differentiated features of each neighbor node can be learned. Commonly used neighbor node partitioning strategies include: (1) single label strategy, that is, each neighbor node belongs to an independent category; (2) distance-based partitioning, which partitions according to the distance relationship between the key point and the node itself, that is, 1-neighborhood is one category, 2-neighborhood is another category; (3) partitioning strategy according to human spatial configuration, which divides the neighbor set into the following three subsets: root node set, proximal point set and distal point set. Since the third strategy provides a finer-grained neighbor node classification, it helps to extract the relationship between nodes of different categories, so that the graph convolutional network module can better capture and learn the human motion pattern, this embodiment selects the third partitioning strategy and divides it according to the centripetal and centrifugal motion characteristics of human skeleton behavior.

[0073] Building upon spatial graph convolution, temporal convolution (TCN) employs parameterization. The range of the temporal domain adjacent to its keypoints is controlled. In temporal convolution, a single convolution kernel is used. The column vectors are used for each temporal convolution, which aggregates information from the same key point within a continuous time period. Convolution is performed on the same key point frame by frame, and then the convolution is performed on the next node. The temporal convolution operation is as follows: Figure 2 As shown.

[0074] This application addresses the problem of low accuracy in behavior recognition by deep learning models due to redundant skeleton data, proposing a behavior recognition method based on a graph convolutional network enhanced by human action links. The proposed method enhances the network's ability to extract significant features in the spatiotemporal dimensions by designing a spatiotemporal attention module; simultaneously, it constructs an adaptive directional graph convolutional layer to adjust the connection relationships of skeleton joints based on the differences between global and local skeleton features, thereby achieving accurate behavior recognition of the target person. The overall architecture of the proposed graph convolutional network module is as follows: Figure 3 As shown.

[0075] Human limb behavior is the result of the coordinated action of various joints. In the feature extraction stage of behavior analysis, the same behavior in different people, as well as multiple behaviors in the same person, will lead to changes in the intrinsic correlation information between the joints of the skeleton. ST-GCN predefines a fixed human skeleton structure diagram based on the natural connectivity of the human body to describe various behaviors. As different behaviors occur, the correlation information of the joints of the skeleton also changes continuously, making the behavior analysis task quite difficult. This application, based on a data-driven approach, infers the human action link relationships based on different input skeleton behavior samples, obtaining a flexible and varied skeleton topology structure to adapt to different skeleton behavior samples. Its human action links are as follows: Figure 4 As shown in the figure, a is the skeleton diagram, b is the first action link, and c is the second action link. The dashed lines represent the anatomical connections between human joints, while the solid lines represent the functional related movements between joints that are not anatomically connected during the execution of a specific action.

[0076] The core objective of action link inference is to learn and deduce the dynamic correlations between various joints under different actions through a network model. For example, in action one, "upper limb waving," there is a significant linkage between the wrist and shoulder joints; in action two, "gait movement (walking / running)," there is extensive coordination between the upper and lower limb joints. The solid lines in action link one and action link two represent the behavior-specific dependencies between joints and other joints used to infer these relationships, which can be represented as follows:

[0077] (3);

[0078] In the formula, and This represents a multilayer perceptron used for extracting and transmitting joint information. The computation process requires multiple inference iterations of the network to calculate the degree of association. The joint features have been updated again. Then calculate the degree of correlation. The joint features have been updated again. The key-point relationships can be represented as follows:

[0079] (4);

[0080] In the formula, This represents a vector concatenation operation. This represents the mean operation and element-wise maximization operation for aggregated link features and joint features. After K iterations, the action link inference module obtains the link probabilities between nodes as follows:

[0081] (5);

[0082] In the formula, r is a random vector sampled from the Gumbel(0,1) distribution, and the parameter is... The degree of discretization of the link probability is controlled by the parameter in this embodiment. Set to 0.5. Each possible related connection is scored using joint correlation calculation, then normalized to a probability using equation (5), and then... Constructing the action link probability matrix Then it is used for graph convolution operation in equation (6). Assume It is the first The probability of linking actions of the same type, then adopt Alternate Adjacency Matrix This will give you the action link graph convolutional block:

[0083] (6);

[0084] In the formula, The trainable weight matrix embedded operation representing the importance of link features, and its action link graph convolutional module, are as follows: Figure 5 As shown.

[0085] Human behavioral characteristics are a series of spatiotemporal evolutionary relationships of skeletal joints. Inputting these features into a network for deep learning aims to extract more complex and discriminative high-level features. Analyzing and judging a person's behavior from the perspective of the human visual system is not complex; often, a few simple key postures are sufficient to roughly determine the behavior. For example, for the action of raising an arm, it is only necessary to focus on the movement trajectory of the arm joints; focusing too much on changes in other joints may actually affect the judgment that the arm is raised. Based on this, the proposed method adds an attention mechanism to the graph convolutional network module.

[0086] Attention mechanisms are mechanisms proposed based on the characteristics of human vision, allowing the human eye to intuitively process perceived information. They primarily achieve rapid and accurate feature extraction by focusing on important information and ignoring less important information. Their spatiotemporal attention network structure is as follows: Figure 6 As shown.

[0087] When performing spatial attention operations, the skeleton data is first processed by an average pooling layer (AvgPool), and then... The convolution kernel performs the corresponding convolution operation. Finally, after Activation function to obtain spatial attention module Its mathematical expression is:

[0088] (7);

[0089] The specific operation of the temporal attention module is as follows: Similarly, for the input features... Average pooling is performed, followed by a multi-layer perceptron (MLP) to extract the temporal correlation between features, and finally... Activation function generation time attention module Its mathematical expression is:

[0090] (8);

[0091] Introducing spatial attention and temporal attention modules between the action link graph convolution and temporal convolution modules can effectively improve the network's ability to extract spatial structural features and temporally continuous dynamic features of skeleton behavior, thereby achieving more accurate target personnel behavior analysis.

[0092] To verify the effectiveness of the method in this embodiment, two datasets, NTU-RGB+D and MSR-Action 3D, were used. Ten layers of adaptive spatiotemporal augmentation map convolutional blocks were stacked on the NTU-RGB+D dataset, with the number of input and output channels and stride of each layer as shown below. Figure 7 As shown in (a); four layers of adaptive spatiotemporal augmentation graph convolutional blocks were stacked on the MSR-Action 3D dataset, where the number of input and output channels and stride of each network block are as follows. Figure 7 As shown in (b).

[0093] For the NTU-RGB+D dataset, the proposed graph convolutional network module uses stochastic gradient descent with momentum, where the momentum is 0.9. The loss function is cross-entropy loss, and the model is trained for 120 epochs. An adaptive learning rate is employed, with an initial learning rate of 0.1, which decays to 0.01 at the 30th epoch and to 0.001 at the 60th epoch until training ends. Furthermore, Dropout is set to 0.5, the weight decay coefficient is set to 0.0001, and the number of training and testing batches is 64 each.

[0094] To further verify the performance of the proposed method on the NTU-RGB+D dataset, different methods such as Lie Group, HBRNN, STA-LSTM, and Res-TCN were compared with the proposed graph convolutional network module. To quantify the robustness and correctness of different models under different subjects and viewpoints, various methods were employed. The recognition accuracy of the different methods was quantitatively evaluated, and the final results are shown in Table 1.

[0095]

[0096] Table 1

[0097] As shown in Table 1, the proposed method achieves a cross-subject recognition accuracy of 88.63% and a cross-viewpoint evaluation accuracy of 95.34% on the NTU-RGB+D dataset. In the CV evaluation, compared to the baseline network ST-GCN, the proposed method improves the recognition rate by 6.97%. While the traditional Lie Group model and the HBRNN model alone perform poorly, the graph convolutional network-based model shows good performance, demonstrating that graph convolutional networks can better fit 3D skeleton behavior data. Further improvements in model recognition performance can be achieved by extracting more multi-scale information from the spatiotemporal dimensions of the human skeleton.

[0098] To further verify the effectiveness of the proposed method on the MSR-Action 3D dataset, different methods such as LieGroup, Graph-based, Differential-RNN, and Two-layer AP were compared with the proposed model (graph convolutional network module). The optimization strategy, loss function, and weight decay coefficient were the same as those used on the NTU-RGB+D dataset. Other hyperparameters, such as a training epoch of 100, Dropout of 0.25, and training and testing batch sizes of 16, were also used. The final experimental results are shown in Table 2.

[0099]

[0100] Table 2

[0101] As shown in Table 2, the model in this application achieves the best recognition performance on the NTU-RGB+D dataset; the recognition accuracy is 95.27%, which is 5.79%, 3.07%, 3.24%, and 1.86% higher than the comparison methods, respectively. The reason for this phenomenon is that the model in this application is designed with spatial attention module and temporal attention module, which effectively improves the model's feature extraction capability, thereby improving the model's recognition accuracy.

[0102] To further verify the effectiveness of each module in the model of this application, experiments were conducted on different modules of the model on the MSR-Action 3D and NTU-RGB+D datasets. The results are shown in Table 3. In the table, AGC represents the action link graph convolution module, TS represents the spatiotemporal attention enhancement module, and ST-GCN+AGC+TS represents the model of this application.

[0103]

[0104] Table 3

[0105] Table 3 shows that in the MSR-Action 3D dataset, the baseline network with 4 layers of ST-GCN blocks stacked achieved a recognition accuracy of only 92.27%. Adding the AGC block improved the accuracy by 1.44%, and adding the TS block improved it by 0.92%. Using both modules together resulted in a recognition accuracy of 95.32%. In the cross-view evaluation CV test results on the NTU-RGB+D dataset, the baseline network with 10 layers of ST-GCN blocks stacked achieved a recognition accuracy of 91.34%. Embedding the AGC module increased the accuracy by 2.41%, and adding the TS module increased it by 1.84%. Using both modules together resulted in a recognition accuracy of 94.84%, a 3.50% improvement over the baseline ST-GCN network. Furthermore, the cross-subject evaluation CS recognition results also demonstrate that both the action link inference module and the spatiotemporal attention enhancement module effectively improve performance, and using both modules together can further enhance the model's performance.

[0106] The above experiments demonstrate that the model presented in this application exhibits good recognition accuracy for 3D skeleton behavior data. To further illustrate the recognition performance of the model on the NTU-RGB+D dataset and the MSR-Action 3D dataset, the output results are visualized using a confusion matrix. In the confusion matrix, the color intensity of the diagonal elements represents the probability of correctly classifying that behavior type, while other elements represent the probability of being misclassified as other behaviors. The final output results are shown below. Figure 8 and Figure 9 As shown.

[0107] Depend on Figure 8As can be seen, the model in this application can accurately identify 60 behavior categories in the NTU-RGB+D dataset. Among them, behaviors 12, 29, and 30 are relatively light in color. These behaviors are writing, playing on a mobile phone, and typing, respectively, all of which are related to local movements of the finger joints. That is, the model in this application is not good at recognizing behaviors with subtle joint changes. The model in this application can accurately identify 20 behavior categories in the MSR-Action 3D dataset. However, the recognition accuracy of behaviors such as knocking (H), throwing high (HT), reaching out to grab (HCh), and punching from the side (SB) is poor. These behaviors are very similar in terms of motion amplitude, frequency, and trajectory in the skeleton data, and global joint features interfere with the extraction of effective features.

[0108] To verify the effectiveness of the proposed method in recognizing whole-body movements, the proposed model was used to visualize the skeletal structure of human jumping and falling behaviors. The results are as follows: Figures 10 to 13 As shown.

[0109] As Figures 10 to 13 As shown, the model in this application can accurately represent human jumping and falling behaviors. Both jumping and falling are whole-body movements, involving the coordinated action of various parts of the body, exhibiting strong spatial correlation and temporal variation characteristics. The Action Link Augmentation Graph Convolutional Network, through temporal and spatial attention modules, can effectively extract the skeletal structural features of human jumping and falling behaviors. The spatial structural sequences of the human skeleton in continuous actions can be well extracted and represented, thereby achieving accurate behavior recognition.

[0110] The method described in this application achieves a cross-subject recognition accuracy of 88.6% and a cross-viewpoint evaluation accuracy of 94.8% on the NTU-RGB+D dataset. The model described in this application demonstrates the best recognition performance on the NTU-RGB+D dataset, achieving a recognition accuracy of 95.3%, which is 5.8%, 3.1%, 3.3%, and 1.9% higher than the comparative methods, respectively. The model described in this application incorporates spatial attention and temporal attention modules, effectively improving the model's feature extraction capability and thus enhancing its recognition accuracy.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A method for human behavior recognition based on action link augmented graph convolutional networks, characterized in that: include: Acquire human skeleton sequence data, which contains the three-dimensional spatial coordinates of multiple frames of human joints; The motion link inference module processes the human skeleton sequence data. The motion link inference module is used to dynamically infer the functional coordination relationship between joints based on the input human skeleton sequence data, and uses the Gumbel-Softmax function to generate a discretized motion link probability matrix. Human skeleton sequence data and a discretized action link probability matrix are input into a graph convolutional network to output human behavior recognition results. The graph convolutional network includes sequentially connected action link graph convolutional blocks, spatiotemporal attention modules, and temporal convolutional blocks. The action link graph convolutional blocks use the discretized action link probability matrix as a dynamic adjacency matrix to perform graph convolution operations. The spatiotemporal attention module includes a spatial attention module and a temporal attention module. The spatial attention module is used to enhance the spatial features of key joints, and the temporal attention module is used to enhance the temporal features of key frames. The temporal convolutional block is used to aggregate information of the same joint point within a continuous time period through temporal convolution operations.

2. The human behavior recognition method based on action link augmented graph convolutional network according to claim 1, characterized in that: The method for generating a discretized action link probability matrix by the action link inference module includes: extracting features from the input human skeleton sequence data to obtain joint features; calculating the degree of association between joints based on the current joint features using a multilayer perceptron; updating the joint features according to the degree of association; repeating the above process K times, where K is a preset number of iterations; and discretizing the association probability obtained after the Kth iteration using the Gumbel-Softmax function to generate the action link probability matrix.

3. The human behavior recognition method based on action link augmented graph convolutional network according to claim 1, characterized in that: In the action link graph convolution block, the discretized action link probability matrix is ​​used to replace the preset fixed adjacency matrix as the dynamic adjacency matrix in the graph convolution operation.

4. The human behavior recognition method based on action link augmented graph convolutional network according to claim 1, characterized in that: The spatial attention module performs average pooling on the input features, followed by convolution operations and... Activation functions generate spatial attention maps to enhance the spatial features of key joints; The temporal attention module performs average pooling on the features enhanced by the spatial attention module, followed by a multilayer perceptron and... The activation function generates a temporal attention map to enhance the temporal features of keyframes.

5. The human behavior recognition method based on action link augmented graph convolutional network according to claim 1, characterized in that: The temporal convolution block employs a temporal convolution operation with a kernel size of 1× ,in This is a hyperparameter used to control the aggregation range of adjacent time domains of key points. Each temporal convolution aggregates information for the same key point within a continuous time period.

6. A human behavior recognition device based on an action link augmented graph convolutional network, characterized in that: include: Data acquisition module, action link inference module, and graph convolutional network module; The data acquisition module is used to acquire human skeleton sequence data, which contains the three-dimensional spatial coordinates of multiple frames of human joints. The action link inference module is used to process human skeleton sequence data, dynamically infer the functional coordination relationship between joints based on the input human skeleton sequence data, and use the Gumbel-Softmax function to generate a discretized action link probability matrix. The graph convolutional network module is used to input human skeleton sequence data and a discrete action link probability matrix together, and then process them sequentially through the action link graph convolutional block, the spatiotemporal attention module and the temporal convolutional block to output human behavior recognition results. Action link graph convolutional blocks use a discretized action link probability matrix as a dynamic adjacency matrix to perform graph convolution operations; The spatiotemporal attention module includes a spatial attention module and a temporal attention module. The spatial attention module is used to enhance the spatial features of key joints, and the temporal attention module is used to enhance the temporal features of key frames. Temporal convolution blocks are used to aggregate information from the same key point within a continuous time period through temporal convolution operations.

7. The human behavior recognition device based on action link augmented graph convolutional network according to claim 6, characterized in that: The action link inference module extracts features from the input human skeleton sequence data to obtain joint features. Based on the current joint features, it calculates the correlation between joints using a multilayer perceptron and updates the joint features according to the correlation. This process is repeated K times, where K is a preset number of iterations. The correlation probability obtained after the Kth iteration is discretized using the Gumbel-Softmax function to generate an action link probability matrix.

8. The human behavior recognition device based on action link augmented graph convolutional network according to claim 6, characterized in that: The spatial attention module is used for average pooling of the input features, followed by convolution operations and... Activation functions generate spatial attention maps to enhance the spatial features of key joints; The temporal attention module is used to perform average pooling on the features enhanced by the spatial attention module, followed by a multilayer perceptron and... The activation function generates a temporal attention map to enhance the temporal features of keyframes.

9. A computer program product, characterized in that: It includes a computer program, which, when executed by a computer, implements the method as described in any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer, it implements the method as described in any one of claims 1 to 5.

Images