A motion recognition method, system, device and medium for skeleton scale difference feature extraction

Abstract

A motion recognition method, system, device and medium for skeleton scale difference feature extraction, the method is: according to human behavior category, the human behavior video shot is marked, and human motion sample is obtained;Extract the spatial three-dimensional information of human skeleton joint node, create human joint dataset and transform it, construct human skeleton dataset and human motion speed dataset;Build dynamic multi-scale spatial feature extraction module and multi-scale time feature extraction module, joint-part feature enhancement attention module and space-time domain result bias module;The human joint dataset, human skeleton dataset and human motion speed dataset are respectively subjected to feature learning by the constructed module to obtain the corresponding recognition result, the recognition results are weighted and fused, and the behavior recognition classification is realized;The system, device and medium are used to realize the method;The application effectively models the multi-scale skeleton node and body part structure, and effectively improves the recognition performance of the human behavior recognition model.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to an action recognition method, system, device and medium for extracting skeleton scale difference features. Background Technology

[0002] Human motion recognition is one of the most important research topics in the field of computer vision. Its ultimate goal is to analyze human motion in videos and automatically identify the type of human motion based on human motion tracking and human feature extraction. Video can be regarded as a data structure composed of a set of image frames arranged in chronological order. Motion recognition not only needs to analyze the content of each frame of the video, but also needs to extract clues from the temporal information between video frames. In the early stages, motion recognition technology was mainly based on RGB image sequences, depth image sequences, or a fusion of the two for feature extraction. Although these methods have good recognition accuracy, they still have the following limitations: (1) they can only process specific datasets; (2) they have weak ability to process background noise; (3) they are affected by changes in the speed of the target person's movement; (4) processing all pixels of the image brings great computational costs. Compared with image data containing pixels, human skeleton data only contains relevant information of key human joint data in each frame of the image, reducing the amount of data. At the same time, there is a strong natural correlation between adjacent joints in skeleton data, which preserves rich spatial information. Currently, there are many mature methods to obtain skeleton data from videos, such as OpenPose and DensePose.

[0003] Human skeleton-based motion recognition technology plays a vital role in many fields. In recent years, with advancements in hardware and computer science, motion recognition technology has become increasingly sophisticated and is gradually being applied to various aspects of daily life, such as virtual reality, security monitoring, human-computer interaction, and intelligent video analytics.

[0004] Human skeleton topology graphs contain rich information about human joint connections, making them particularly important in graph convolution-based human behavior recognition research. Currently, many researchers using graph convolutional network models only aggregate information locally when processing physical skeleton topology graphs, sampling only the sampled nodes and their neighbors (nodes with a distance of 1). However, joints with a distance greater than 1 still exhibit correlations, and sampling only nodes with a distance of 1 results in the loss of some information, failing to fully utilize the rich dependencies between joints. Compared to adjacency local sampling, multi-scale k-distance sampling can preserve more skeleton information, leading to a more accurate representation of the results.

[0005] Furthermore, human skeleton nodes are divided into multiple sets based on the structural relationships of their parts. When recognizing a certain action, skeleton nodes in the same set have similar motion trends. Compared to a single skeleton node, the skeleton part structure also contains rich motion feature relationships. However, the skeleton node spatial attention module is usually used to learn the importance of a single skeleton node, making it difficult to model the skeleton part structure.

[0006] In existing research methods, models such as AS-GCN (from the CVPR paper "Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition") use matrix exponentiation to capture the correlation of distant skeleton nodes, representing the connectivity between skeleton nodes k distances apart as the k-th power of the skeleton topology matrix. While this can capture information about distant skeleton nodes, the connection strength approaches zero when k is large, rendering the captured dependencies of distant skeleton nodes ineffective. Later, MS-G3D (from the CVPR paper "Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition") proposed a disentangling method to capture the correlation of distant skeleton nodes. It uses a k-distance adjacency matrix to represent the connectivity between distant skeleton nodes, exhibiting good node connectivity even when k is large. However, it fixes the connection strength between the sampled node and k-distance nodes, neglecting the influence of the distance between the sampled node and nodes k distances away. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, the present invention aims to provide an action recognition method, system, device, and medium for extracting skeleton-scale differential features. Addressing the problems of small sampling distances for nodes and the difficulty in representing the entire motion behavior by individual joint node features in graph convolutional network models, this invention utilizes a dynamic multi-scale spatial feature extraction module to enhance the node sampling range and spatial features, a joint-part feature enhancement attention module to enhance the spatial feature expression of joint sequences based on part and joint features, and a spatiotemporal domain result bias module to improve the recognition accuracy of the network model. This invention focuses on skeleton-scale differential features and effectively improves the recognition performance of human behavior recognition models by effectively modeling multi-scale skeleton nodes and body part structures.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] An action recognition method for extracting skeleton scale difference features includes the following steps:

[0010] Step 1: Record videos of human behavior and label the videos according to the categories of human behavior to obtain human action samples;

[0011] Step 2: Using the Openpose algorithm, extract the spatial three-dimensional information of the human skeletal joints from the human motion samples obtained in Step 1 to create a human joint dataset.

[0012] Step 3: By transforming the human joint dataset (Joint DataSet) from Step 2, construct the human skeleton dataset (Bone DataSet) and the human motion velocity dataset (Motion DataSet);

[0013] Step 4: Construct a dynamic multi-scale spatial feature extraction module and a multi-scale temporal feature extraction module. The dynamic multi-scale spatial feature extraction module is used to extract the spatial correlation between joints within the same frame, and the multi-scale temporal feature extraction module is used to extract the temporal correlation between joints within different frames.

[0014] Step 5: Construct a joint-site feature enhancement attention module for feature representation of strong skeleton sequences, achieving mutual complementarity between joint features and human body site features;

[0015] Step 6: Construct a spatiotemporal domain result bias module to extract temporal and spatial bias features from the skeleton sequence in a global manner, obtain the corresponding temporal and spatial bias recognition results respectively, and then weight and fuse the recognition results to complement each other.

[0016] Step 7: The human joint dataset (Joint DataSet) from Step 2, the human skeleton dataset (Bone DataSet) from Step 3, and the human motion speed dataset (Motion DataSet) are processed through the modules constructed in Steps 4, 5, and 6 to learn features and obtain corresponding recognition results. Finally, the recognition results are weighted and fused to achieve behavior recognition and classification.

[0017] The specific steps of step 2 include:

[0018] The Openpose algorithm was used to process the human behavior video labeled in step 1, and the three-dimensional coordinate information of human skeleton in each frame was extracted to obtain the human joint dataset.

[0019] Each image frame in the human behavior video contains 25 joint nodes, each joint node contains 3-dimensional coordinate points, and each behavior category contains T frames. The skeletons of the corresponding image frames of each behavior video are stacked together in chronological order to form a tensor of size (C0, T0, N0), where C0 = 3 represents the three-dimensional coordinates x, y, z; N0 represents the 25 joint nodes; and T0 represents the number of video frames.

[0020] Using the feature matrix The feature information of the joints is represented by C0, T0, and N0, which represent the number of channels, frames, and joints of the skeleton sequence, respectively. Based on the joint feature matrix V of the human skeleton, the human joint dataset is defined as set J.

[0021] J = {V ti |t=1,…,T0,i=1,…,N0}

[0022] Among them, V ti This indicates that the joint point is the i-th joint point of the t-th frame of the video.

[0023] The specific steps of step 3 include:

[0024] In a diagram of the human skeleton, the joint closest to the center of gravity of the skeleton on each side is defined as the source joint V. i =(x i y i , z i The joint furthest from the center of gravity of the skeleton is the target joint V. j =(x j y j , z j If each bone is represented as a vector B pointing from the source joint to the target joint, then each bone is represented as a vector B. j The formula for calculating this skeleton is as follows:

[0025] B j =(x j -x i y j -y i , z j -z i )

[0026] Expand the human joint dataset (Joint DataSet) from step 2 in the spatial dimension to obtain the human skeleton dataset (Bone DataSet); the skeleton data includes bone length information and bone orientation information, and each bone corresponds to a unique target joint; at the same time, an empty bone with a value of 0 is assigned to the central joint;

[0027] Using the feature matrix The feature information representing the human skeleton is denoted by C0, T0, and N0, which represent the number of channels, frames, and bones in the skeleton sequence, respectively. Based on the skeleton feature matrix P, the human skeleton dataset (Bone DataSet) is defined as set B.

[0028] B = {P} ti |t=1,…,T0,i=1,…,N0}

[0029] Among them, B ti This indicates that the bone is the i-th bone in the t-th frame of the video;

[0030] Extend the human joint dataset from step 2 in the time dimension, defining T as the earlier frame in two adjacent frames of the joint sequence. t-1 Later frames T t The velocity S at the joint is then expressed as T. t Joint information V t =(x t y t , z i ) and T t-1 Joint information V t-1 =(x t-1 y t-1 , z i-1 The difference between the values ​​of the two points, and the specific formula for calculating the velocity at the joints are as follows:

[0031] S i，t =(x i，t -x i，t-1 y i，t -y i，t-1 , z i，t -z i，t-1 )

[0032] Among them, S i，t This represents the velocity information of the i-th node in the t-th frame, while adding 0 elements in the 0th frame to make the human motion velocity dataset (Motion DataSet) and the human joint dataset (Joint DataSet) have the same dimensions.

[0033] Using the feature matrix The velocity information of human joints is represented by C0, T0, and No, which represent the number of channels, frames, and joints in the skeleton sequence, respectively. Based on the joint velocity feature matrix Q, the human motion velocity dataset (MotionDataSet) is defined as a set M.

[0034] M = {Q ti |t=1,…,T0,i=1,…,N0}

[0035] Among them, Qti This represents the velocity information of the i-th joint in the t-th frame of the video.

[0036] The specific steps of step 4 include:

[0037] Step 4.1, Construct a dynamic multi-scale skeleton topology graph:

[0038] A multi-scale skeletal topology relation matrix A is defined based on the topological information of the human skeletal joints. (k) ∈R k×N×N Where k represents the scale and N represents the number of joints; A (k) This indicates that, based on the deentanglement method, the key point vi and the key point v that are k jumps apart are... j The topological relationship and strength between them are defined by the following unentanglement relationship:

[0039]

[0040] Wherein d(v i v j ) represents the key point v i With the joint v j The shortest distance between them, A (0) The self-connection relationship of the joints is represented by the identity matrix I; A (1) The information between a joint node and its adjacent joints at a distance of 1 is represented by an adjacency matrix A; when k > 1, A (k) It represents the high-scale information of the joint node and the joint nodes that are k hops away from it; it generates a relation matrix from 1 to k hops by unentanglement to describe the connection information between the joint nodes;

[0041] When k > 1, A (k) Join A (0) To make corrections, a trainable parameter α is introduced to dynamically adjust the connection strength between the sampling node and nodes k hops away, as shown in the following formula:

[0042]

[0043] Step 4.2, Construct a dynamic multi-scale spatial feature extraction module:

[0044] Let the features of the original skeleton sequence be denoted as V0∈R. C×T×N Where C, T, and N represent the number of channels, frames, and joints of the skeleton sequence features, respectively;

[0045] Step 4.2.1: Compare the original skeleton sequence features V0 with the dynamic multi-scale skeleton topology map from Step 4.1. The fusion process generates a skeleton sequence feature V1 with skeleton topological information.

[0046]

[0047] Where V1∈R C×T×k×N Perform dimensional transformation on V1 to adapt it to the network:

[0048] V1:R C×T×k×N →R C′×T×N

[0049] Where C′=C×k;

[0050] Step 4.2.2: Use a convolutional layer with a kernel size of 1×1 to learn the skeleton sequence feature V1 from Step 4.2.1 to obtain the skeleton sequence feature V2. Then, use a convolutional layer with BatchNorm and ReLU functions to enhance the feature representation ability of the skeleton sequence feature V2 and obtain the output feature X. out The formula is as follows:

[0051] V2 = Conv 2D(1×1) (V1)

[0052] Xo ut =Relu(BN(V2))

[0053] Under multi-scale conditions, the feature information corresponding to different scales k is aggregated by addition, so that the model can still remain effective when the value of k is large. The complete formula of this module is as follows:

[0054]

[0055] in, express The normalized matrix, σ(·) represents the activation function with convolution operation;

[0056] Step 4.3, Construct a multi-scale temporal feature extraction module:

[0057] The multi-scale temporal feature extraction module models the feature sequence based on different temporal dilation rates. This module contains four branches, each containing a 1×1 convolution. The first branch does not contain any additional convolutional layers, indicating that sampling is performed frame-by-frame in the temporal dimension. The second branch contains a temporal convolutional layer with a dilation rate of 1, indicating that sampling is performed on frames with an interval of 1 in the temporal dimension. The third branch contains a temporal convolutional layer with a dilation rate of 2, indicating that sampling is performed on frames with an interval of 2 in the temporal dimension. The fourth branch contains a max pooling layer, which is used to downsample the feature sequence. Finally, the results of the four branches are accumulated according to their weights to obtain the output feature.

[0058] The specific steps of step 5 are as follows:

[0059] The joint-site feature enhancement attention module includes: joint attention branch and site attention branch;

[0060] The input skeleton sequence features are denoted as V0∈R C×T×N Where C, T, and N represent the number of channels, frames, and joints of the skeleton sequence, respectively;

[0061] Step 5.1, Construct joint attention branches

[0062] Step 5.1.1: Perform dimensionality transformation on the skeleton sequence feature V0. The joint attention branch focuses on learning the spatial features of the skeleton sequence. The specific transformation is as follows:

[0063] V0:R C×T×N →R V×T×N

[0064] Step 5.1.2: The skeleton sequence feature V0 is pooled into a tensor of size (N×1×1) through adaptive average pooling operation, and the output size is specified as 1×1. The skeleton sequence feature V0 is compressed in the time dimension and channel dimension.

[0065] Step 5.1.3: Use a convolutional layer with a kernel size of 1×1 to learn features from the skeleton sequence feature V0 in Step 5.1.2. Then, use a convolutional layer with BatchNorm and ReLU functions to enhance the feature representation ability of the skeleton sequence feature V0 and obtain the joint attention feature Att. joint The formula is as follows:

[0066] Att joint =Relu(BN(Conv) 2D(1×1) (V0)));

[0067] Step 5.2, construct the part attention branch:

[0068] Step 5.2.1: Based on the location information of the natural skeleton, the 25 joints of the human body are divided into 5 parts: the torso, left hand, left foot, right hand, and right foot, and all joints of the corresponding parts are included. By manually selecting the corresponding joints, five body part sequences Po to P4 are obtained from the skeleton sequence feature V0. Where, N i This indicates the number of joints contained in the i-th part;

[0069] Step 5.2.2: Perform dimensional transformation on the five body part sequences P0 to P4 from step 5.2.1, transforming them into sequences with dimension (N). i A tensor of ×T×C, where N is a tensor of ×T×C. iThis represents the number of joints contained in the i-th body part; then, each body part set sequence P0 to P4 is pooled into a tensor of size (1×1×1) using adaptive average pooling; finally, P0 to P4 are concatenated to obtain the body part sequence feature V1∈R. 5×1×1 It is represented as follows:

[0070] V1 = Concat({P i |i=0,1,2,3,4})

[0071] Step 5.2.3: Use a convolutional layer with a kernel size of 1×1 to learn features from the body part sequence feature V1 in Step 5.2.2. Then, use a convolutional layer with BatchNorm and ReLU functions to enhance the feature representation ability of the body part sequence feature V1 and obtain the body part attention feature Att. part The formula is as follows:

[0072] Att part =Relu(BN(Conv) 2D(1×1) (V1)))

[0073] Step 5.3: Construct a joint-site feature enhancement attention module

[0074] Using the body part attention feature Att from step 5.2.3 part Enhance the joint attention feature Att in step 5.1.3 joint The feature representation is used to ultimately obtain the attention feature Att enhanced by joint-location features. enhance The formula is as follows:

[0075] Att enhance [i] = Att joint [i]+δAtt part [j]

[0076] Where i represents the i-th joint, j represents the location of the i-th joint, and δ represents Att. par The weight of t;

[0077] The formula for the joint-site feature-enhanced attention module is as follows:

[0078] X out =Att en九ance ×V0

[0079] Where V0 is the input skeleton sequence feature, X out It is the output skeleton sequence feature.

[0080] The specific steps of step 6 are as follows:

[0081] Step 6.1, Construct the time-domain biased branch

[0082] Step 6.1.1: Perform a dimensionality transformation on the skeleton sequence feature V0 to obtain the time-biased skeleton sequence feature V1, as follows:

[0083] V0∈R C×T×N →V1∈R N×C×T

[0084] A convolutional layer with a kernel size of 1×1 is used to perform a convolution operation on the skeleton sequence feature V1 to obtain the sequence V. T Sequence V T The formula representing the time information of the time frame itself is as follows:

[0085] V T =Conv 2D(1×1) (V1)

[0086] Then, the skeleton sequence feature V1 is compressed in the spatial dimension to obtain the skeleton sequence feature V2∈R that retains only the temporal features. C×T Next, a convolutional layer with a kernel size of 1×1 is used to projectively encode the skeleton sequence feature V2, resulting in sequence Q. T and sequence K T , sequence Q T This represents the information that the current time frame hopes to obtain from other time frames, sequence K. T The formula represents the information that the current time frame can provide to other time frames, as shown below:

[0087] K T =Conv 2D(1×1) (V2)

[0088] QT ＝ Conv 2D(1×1) (V2)

[0089] Step 6.1.2, by analyzing sequence K from step 6.1.1 T and the sequence Q T A dot product operation is performed to obtain the similarity matrix between each time frame and other time frames. Then, a softmax operation is performed on the similarity matrix to obtain the weight matrix. Finally, this weight matrix is ​​applied to sequence V. T The time-biased skeleton sequence features X are obtained from the above. t_out The corresponding formula for this process is as follows:

[0090]

[0091] Where C is the sequence K T The number of channels;

[0092] Then for X t_out Dimensional transformation is performed as follows:

[0093] X t_out :R C×T →R N×C×T →R C×T×N

[0094] Step 6.1.3, regarding X in step 6.1.2 t_out Compression is performed on the time dimension T and the spatial dimension N, retaining only the channel dimension C, to obtain the compressed sequence input V3∈R. C A fully connected linear layer with an output channel equal to the number of action categories is used on the sequence input V3 to obtain the temporally biased recognition result Label. T The formula is as follows:

[0095] Label T = softmax(linear(V3));

[0096] Step 6.2, Construct Spatial Domain Biased Branches

[0097] Step 6.2.1: Perform a dimensionality transformation on the skeleton sequence feature V0 to obtain the spatially biased skeleton sequence feature V4, as follows:

[0098] V0∈R C×T×N →V4∈R T×C×N

[0099] A convolutional layer with a kernel size of 1×1 is used to perform a convolution operation on the skeleton sequence feature V4 to obtain the sequence V. S Sequence V s The formula representing the spatial information of the joint node itself is as follows:

[0100] V s =Conv 2D(1×1) (V4)

[0101] Then, the skeleton sequence feature V4 is compressed in the time dimension to obtain the input sequence V5∈R that retains only the spatial features. C×N Next, a convolutional layer with a kernel size of 1×1 is used to projectively encode the skeleton sequence feature V5, resulting in sequence Q. S and sequence K S , sequence Q S The sequence K represents the information that the current joint node wants to obtain from other joint nodes. S The formula represents the information that the current joint node can provide to other joint nodes, as shown below:

[0102] K S =Conv2D(1×1) (V5)

[0103] Q S =Conv 2D(1×1) (V5)

[0104] Step 6.2.2, by analyzing sequence K from step 6.2.1 S and the sequence Q S A dot product operation is performed to obtain the similarity matrix between each joint node and other joint nodes. Then, a softmax operation is performed on the similarity matrix to obtain the weight matrix. Finally, this weight matrix is ​​applied to the sequence V. S Thus, the spatially biased skeleton sequence features X are obtained. s_out The corresponding formula for this process is as follows:

[0105]

[0106] Where C is the sequence K S The number of channels;

[0107] Then for X s_out Dimensional transformation is performed as follows:

[0108] X s_out :R C×N →R T×C×N →R C×T×N

[0109] Step 6.2.3, for X s_out Compression is performed on the time dimension T and the spatial dimension N, retaining only the channel dimension C, to obtain the compressed sequence input V6∈R. C A fully connected linear layer with an output channel equal to the number of action categories is used on the sequence input V6 to obtain the spatial bias recognition result Label. S The formula is as follows:

[0110] Label S =softmax(linear(V6))

[0111] Step 6.3, Construct the spatiotemporal domain result bias module

[0112] A dual-stream fusion architecture is adopted to label the time-biased identification results from step 6.1.3. T Label of spatial bias identification result in step 6.2.3 S Each item is summed according to its weight α to obtain a fusion score, and the action label is predicted using softmax. predict The formula is as follows:

[0113] Labelpredict =softmax(Label) S +aLabel T ).

[0114] The specific steps of step 7 are as follows:

[0115] First, the human joint dataset from step 2 is processed through the modules constructed in steps 4, 5, and 6 to perform feature learning and obtain the corresponding recognition results (Labels). joint Next, the human skeleton dataset (BoneDataSet) and human motion velocity dataset (Motion DataSet) from step 3 are also processed through the modules constructed in steps 4, 5, and 6 to perform feature learning and obtain the corresponding recognition results (Labels). bone and Label motion Ultimately, a three-stream fusion approach was adopted to integrate the Label. joint Label bone and Label motion The results are weighted and fused item by item to obtain the final classification result Label for behavior recognition. final The formula is as follows:

[0116] Label final =softmax(Label) joint +Label bone +Label motion ).

[0117] This invention also provides an action recognition system for extracting skeleton scale difference features, comprising:

[0118] The joint feature extraction module is used to extract the spatial three-dimensional information of human skeletal joints from human motion samples using the Openpose algorithm, and create a human joint dataset.

[0119] The skeletal feature extraction module is used to transform the human joint dataset to construct the human skeleton dataset and the human motion velocity dataset.

[0120] The dynamic multi-scale spatial feature extraction module is used to extract the spatial correlation between joints within the same frame;

[0121] A multi-scale temporal feature extraction module is used to extract the temporal correlation between joints in different frames;

[0122] The joint-site feature enhancement attention module is used for feature representation of strong skeleton sequences, enabling the mutual complementarity of joint features and human body site features;

[0123] The spatiotemporal domain result bias module is used to extract temporal and spatial bias features from the skeleton sequence in a global manner, obtain the corresponding temporal and spatial bias recognition results respectively, and then weight and fuse the recognition results to complement each other.

[0124] The behavior recognition and classification module is used to learn features from the human joint dataset (Joint DataSet), human bone dataset (Bone DataSet), and human motion velocity dataset (Motion DataSet) through a constructed dynamic multi-scale spatial feature extraction module, a multi-scale temporal feature extraction module, a joint-part feature enhancement attention module, and a spatiotemporal domain result bias module, and obtain corresponding recognition results. Finally, the recognition results are weighted and fused to achieve behavior recognition and classification.

[0125] The present invention also provides an action recognition device for extracting skeleton scale difference features, comprising:

[0126] Memory: A computer program that stores the above-mentioned action recognition method for extracting skeleton scale difference features, and is a computer-readable device;

[0127] Processor: Used to implement the action recognition method for extracting skeleton scale difference features when executing the computer program.

[0128] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the action recognition method for extracting skeleton scale difference features.

[0129] Compared with the prior art, the present invention has the following beneficial effects:

[0130] 1. This invention introduces a dynamic multi-scale spatial feature extraction module, which solves the problem of weak connection between high-scale downsampled joints and k-hop neighbor joints by de-entanglement. While maintaining the joint's own features, it effectively models its k-hop neighbor nodes, thereby better capturing the local correlation of human skeletal joints.

[0131] 2. This invention introduces a joint-part feature enhancement attention module, which strengthens the guidance of the importance of a single skeletal joint in the entire joint set based on the importance of body parts in the entire action sequence, thereby enhancing the feature representation of the skeletal sequence and improving the performance and generalization ability of the model.

[0132] 3. This invention introduces a spatiotemporal domain result bias module, which expands the receptive field from a local window to the entire sequence through non-local operations. It performs temporal and spatial bias feature extraction on the skeleton sequence in a global manner, obtains the corresponding recognition results, and integrates and complements them to achieve the final behavior recognition and classification.

[0133] 4. This invention adopts a three-stream structure, using joint, bone, and motion datasets as input data into the network model, and fusing the features learned from the three feature streams to enable the model to learn richer human motion information.

[0134] In summary, this invention can effectively improve the recognition performance of human behavior recognition models by effectively modeling and capturing features of multi-scale joint nodes and body parts, and by fusing the results of joint, bone and dynamic features. Attached Figure Description

[0135] Figure 1 This is a flowchart of the identification method of the present invention.

[0136] Figure 2 This is a multi-scale natural skeleton topology diagram of the present invention.

[0137] Figure 3 This is a structural diagram of the dynamic multi-scale spatial feature extraction module of the present invention.

[0138] Figure 4 This is a structural diagram of the multi-scale temporal feature extraction module of the present invention.

[0139] Figure 5 This is a structural diagram of the joint-site feature enhancement attention module of the present invention.

[0140] Figure 6 This is a structural diagram of the spatiotemporal domain result bias module of the present invention.

[0141] Figure 7 This is a three-stream fusion structure diagram of the joint, bone, and velocity datasets of this invention. Detailed Implementation

[0142] The technical solution of the present invention is described in detail below. The embodiments described with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0143] This embodiment presents an action recognition method for extracting skeleton-scale difference features. The overall framework of the network structure used is described in [reference needed]. Figure 1 Specifically, it includes the following steps:

[0144] Step 1: Capture videos of human behavior and label the videos according to the categories of human behavior to obtain human action samples. Each video contains T frames.

[0145] Step 2: Based on the human motion samples collected in Step 1, use the Openpose algorithm to extract the spatial three-dimensional information of the human skeletal joints to create a human joint dataset.

[0146] The Openpose algorithm was used to process the human behavior video labeled in step 1, and the three-dimensional coordinate information of the human skeleton in each frame was extracted to obtain the human joint dataset.

[0147] Each image frame in the human behavior video contains 25 joint nodes, each joint node contains 3-dimensional coordinate points, and each behavior category contains T frames. The skeletons of the corresponding image frames of each behavior video are stacked together in chronological order to form a tensor of size (C0, T0, N0), where C0 = 3 represents the three-dimensional coordinates x, y, z; N0 represents the 25 joint nodes; and T0 represents the number of video frames.

[0148] Using the feature matrix The feature information of the joints is represented by C0, T0, and N0, which represent the number of channels, frames, and joints in the skeleton sequence, respectively. Based on the joint feature matrix V of the human skeleton, the human joint dataset can be defined as set J.

[0149] J = {V ti | t=1,…,T0,i=1,…,N0}

[0150] Among them, V ti This indicates that the joint point is the i-th joint point of the t-th frame of the video.

[0151] Step 3: By transforming the human joint dataset (Joint DataSet) from Step 2, construct the human skeleton dataset (Bone DataSet) and the human motion velocity dataset (Motion DataSet);

[0152] In a diagram of the human skeleton, the joint closest to the center of gravity of the skeleton on each side is defined as the source joint V. i =(x i y i , z i The joint furthest from the center of gravity of the skeleton is the target joint V. j =(x j y j , z j If each bone is represented as a vector B pointing from the source joint to the target joint, then each bone is represented as a vector B. jThe formula for calculating this skeleton is as follows:

[0153] B j =(x j -x i y j -y i , z j -z i )

[0154] The human joint dataset from step 2 is expanded in spatial dimension to obtain the human skeleton dataset. The skeleton data can contain bone length information and bone orientation information. Each bone can correspond to a unique target joint. Since there is one more joint than bones, the central joint is not assigned to any bone. Therefore, an empty bone with a value of 0 needs to be assigned to the central joint.

[0155] Using the feature matrix The feature information representing the human skeleton is denoted by C0, T0, and N0, which represent the number of channels, frames, and bones in the skeleton sequence, respectively. Based on the skeleton feature matrix P, the human skeleton dataset (Bone DataSet) is defined as set B.

[0156] B = {P} ti |t=1,…,T0,i=1,…,N0}

[0157] Among them, B ti This indicates that the bone is the i-th bone in the t-th frame of the video.

[0158] Extend the human joint dataset from step 2 in the time dimension, defining T as the earlier frame in two adjacent frames of the joint sequence. t-1 Later frames T t Then the velocity S at the joint can be expressed as T t Joint information V t =(x t y t , z i ) and T t-1 Joint information V t-1 =(x t-1 y t-1 , z t-1 The difference between the values ​​of the two points, and the specific formula for calculating the velocity at the joints are as follows:

[0159] S i，t =(x i，t -x i，t-1 y i，t -y i，t-1 , zi，t -z i，t-1 )

[0160] Among them, S i，t This represents the velocity information of the i-th node in the t-th frame. At the same time, it is necessary to add 0 elements in the 0th frame to make the human motion velocity dataset (Motion DataSet) and the human joint dataset (Joint DataSet) have the same dimension.

[0161] Using the feature matrix The velocity information of human joints is represented by C0, T0, and N0, which represent the number of channels, frames, and joints in the skeleton sequence, respectively. Based on the joint velocity feature matrix Q, the human motion velocity dataset (MotionDataSet) is defined as a set M.

[0162] M = {Q ti |t=1,…,T0,i=1,…,N0}

[0163] Among them, Q ti This represents the velocity information of the i-th joint in the t-th frame of the video.

[0164] Step 4: Construct a dynamic multi-scale spatial feature extraction module and a multi-scale temporal feature extraction module. The dynamic multi-scale spatial feature extraction module is used to extract the spatial correlation between joints within the same frame, effectively modeling distant nodes; the multi-scale temporal feature extraction module is used to extract the temporal correlation between joints in different frames, realizing the interaction and integration of feature information between different time scales.

[0165] See Figure 2 Step 4.1: Construct a dynamic multi-scale skeleton topology graph:

[0166] A multi-scale skeletal topology relation matrix A is defined based on the topological information of the human skeletal joints. (k) ∈R k×N×N Where k represents the scale and N represents the number of joints; A (k) Used to describe the key point v based on the deentanglement method i The joint point v that is k jumps away j The topological relationship and strength between them are defined by the following unentanglement relationship:

[0167]

[0168] Wherein d(v i v j ) represents the key point v i With the joint v j The shortest distance between them, A (0) The self-connectivity of joints is described by the identity matrix I; A(1) The adjacency matrix A describes the information between a joint node and its adjacent joints that are 1 unit apart; when k > 1, A (k) Describe the high-scale information of the joint nodes and the joint nodes that are k hops away from them; generate a relation matrix from 1 to k hops by deentanglement to describe the connection information between the joint nodes;

[0169] Due to the spatial aggregation mechanism of graph convolution models, the self-connectivity relationship A of the joints... (0) Learning the relationship between the current joint node and its k-hop neighbors is crucial, as is understanding the self-connectivity relationship A of the joint when no k-hop neighbors are available. (0) It can also preserve the individual information of each joint node. Therefore, when k > 1, A (k) A needs to be added (0) To make corrections, a trainable parameter α is introduced to dynamically adjust the connection strength between the sampling node and nodes k hops away, as shown in the following formula:

[0170]

[0171] See Figure 3 Step 4.2, Construct a dynamic multi-scale spatial feature extraction module:

[0172] Let the features of the original skeleton sequence be denoted as V0∈R. C×T×N Where C, T, and N represent the number of channels, frames, and joints of the skeleton sequence features, respectively;

[0173] Step 4.2.1: Compare the original skeleton sequence features V0 with the dynamic multi-scale skeleton topology map from Step 4.1. The fusion process generates a skeleton sequence feature V1 with skeleton topological information.

[0174]

[0175] Where V1∈R C×T×k×N To adapt to the network, V1 needs to undergo dimensional transformation:

[0176] V1:R C×T×k×N →R C×T×N

[0177] Where C′=C×k;

[0178] Step 4.2.2: Use a convolutional layer with a kernel size of 1×1 to learn the skeleton sequence feature V1 from Step 4.2.1 to obtain the skeleton sequence feature V2. Then, use a convolutional layer with BatchNorm and ReLU functions to enhance the feature representation ability of the skeleton sequence feature V2 and obtain the output feature X. outThe formula is as follows:

[0179] V2 = Conv 2D(1×1) (V1)

[0180] X out =Relu(BN(V2))

[0181] The dynamic multi-scale spatial feature extraction module can effectively model its k-hop neighbor nodes while preserving the joint's own features. It addresses the weak connectivity between high-scale downsampled joints and their k-hop neighbors through de-entanglement. Under multi-scale conditions, feature information corresponding to different scales k is aggregated by addition, ensuring the model remains effective even when k values ​​are large. The complete formula for this module is as follows:

[0182]

[0183] in, express The normalized matrix, σ (·) represents an activation function with convolution operation;

[0184] See Figure 4 Step 4.3: Construct a multi-scale temporal feature extraction module:

[0185] The multi-scale temporal feature extraction module can model feature sequences based on different temporal dilation rates. This module mainly consists of four branches, each containing a 1×1 convolution to reduce channel dimensions. The first branch does not contain any additional convolutional layers, indicating that sampling is performed frame-by-frame in the temporal dimension. The second branch contains a temporal convolutional layer with a dilation rate of 1, indicating that sampling is performed on frames with an interval of 1 in the temporal dimension. The third branch contains a temporal convolutional layer with a dilation rate of 2, indicating that sampling is performed on frames with an interval of 2 in the temporal dimension. The fourth branch contains a max pooling layer to downsample the feature sequence. Finally, the results of the four branches are summed according to their weights to obtain the output features.

[0186] See Figure 5 Step 5: Construct a joint-site feature enhancement attention module for feature representation of strong skeleton sequences, so as to achieve mutual complementarity between joint features and human body site features;

[0187] The joint-site feature enhancement attention module strengthens the importance of individual skeletal joints within the entire joint set based on the importance of body parts in the overall action sequence. This enhances the feature representation of the skeletal sequence, improving model performance and generalization ability. This module primarily consists of two branches: a joint attention branch and a site attention branch.

[0188] The input skeleton sequence features are denoted as V0∈RC×T×N Where C, T, and N represent the number of channels, frames, and joints of the skeleton sequence, respectively;

[0189] Step 5.1, Construct joint attention branches

[0190] Step 5.1.1: Perform dimensionality transformation on the skeleton sequence feature V0. The joint attention branch focuses on learning the spatial features of the skeleton sequence. The specific transformation is as follows:

[0191] V0:R C×T×N →RV ×T×N

[0192] Step 5.1.2 involves pooling the skeleton sequence feature V0 into a tensor of size (N×1×1) using adaptive average pooling. Adaptive average pooling is a pooling operation that can accept inputs of any size and transform them into a specified output size. Through the adaptive average pooling layer, the input feature map is compressed into a single value, thereby capturing the global information of the entire feature map. Here, the output size is specified as 1×1. The skeleton sequence feature V0 undergoes data compression in both the time and channel dimensions, preserving more important joint features and reducing the model's parameters and computational cost, providing a more compact input for subsequent convolutional operations.

[0193] Step 5.1.3: Use a convolutional layer with a kernel size of 1×1 to learn features from the skeleton sequence feature V0 in Step 5.1.2. Then, use a convolutional layer with BatchNorm and ReLU functions to enhance the feature representation ability of the skeleton sequence feature V0 and obtain the joint attention feature Att. join t, the formula is as follows:

[0194] Att joint =Relu(BN(Conv) 2D(1×1) (V0)))

[0195] Joint attention characteristics Att joint The importance of the sampled joints in the entire joint sequence is described;

[0196] Step 5.2, construct the part attention branch:

[0197] Step 5.2.1: Based on the location information of the natural skeleton, the 25 joints of the human body are divided into 5 parts: the torso, left hand, left foot, right hand, and right foot, and all joints of the corresponding parts are included. By manually selecting the corresponding joints, five body part sequences P0 to P4 are obtained from the skeleton sequence feature V0. Where, N i This indicates the number of joints contained in the i-th part;

[0198] Step 5.2.2: Perform dimensional transformation on the five body part sequences P0 to P4 from step 5.2.1, transforming them into sequences with dimension (N). i A tensor of ×T×C, where N is a tensor of ×T×C. i This represents the number of joints contained in the i-th body part; then, each body part set sequence P0 to P4 is pooled into a tensor of size (1×1×1) using adaptive average pooling; finally, P0 to P4 are concatenated to obtain the body part feature sequence V1∈R. 5×1×1 It is represented as follows:

[0199] V1 = Concat({P i |i=0,1,2,3,4})

[0200] Step 5.2.3: Use a convolutional layer with a kernel size of 1×1 to learn features from the body part sequence feature V1 in Step 5.2.2. Then, use a convolutional layer with BatchNorm and ReLU functions to enhance the feature representation ability of the body part sequence feature V1 and obtain the body part attention feature Att. part The formula is as follows:

[0201] Att part =Relu(BN(Conv) 2D(1×1) (V1)))

[0202] Body Part Attention Characteristics Att part It describes the importance of body parts throughout the sequence of movements;

[0203] Step 5.3: Construct a joint-site feature enhancement attention module

[0204] Att joint The importance of the sampled joints in the entire joint sequence is described. Att part The importance of body parts in the entire action sequence has been described. Now, the body part attention feature Att from step 5.2.3 is used. part Enhance the joint attention feature Att in step 5.1.3 joint The feature representation is ultimately obtained by enhancing the attention att of joint-site features. en九ance The formula is as follows:

[0205] Att enhance [i] = Att joint [i]+δAtt part [j]

[0206] Where i represents the i-th joint, j represents the location of the i-th joint, and δ represents Att. par The weight of t.

[0207] The Joint-Location Feature Enhancement Attention Module learns the relationships between joints and body parts, and strengthens joint features based on location characteristics, thereby dynamically adjusting the importance of joints. The formula is as follows:

[0208] X out =Att en九ance ×V0

[0209] Where V0 is the input skeleton sequence feature, X out It is the output skeleton sequence feature;

[0210] See Figure 6 Step 6: Construct a spatiotemporal domain result bias module to extract temporal and spatial bias features from the skeleton sequence in a global manner, obtain the corresponding temporal and spatial bias recognition results respectively, and then weight and fuse the recognition results to complement each other.

[0211] The input skeleton sequence features are denoted as V0∈R C×T×N Where C, T, and N represent the number of channels, frames, and joints of the skeleton sequence, respectively;

[0212] Step 6.1, Construct the time-domain biased branch

[0213] Step 6.1.1: This branch is designed based on the transformer structure and needs to obtain the time-biased recognition result. First, the skeleton sequence feature V0 is transformed to obtain the time-biased skeleton sequence feature V1. The transformation is as follows:

[0214] V0∈R C×T×N →V1∈R N×C×T

[0215] A convolutional layer with a kernel size of 1×1 is used to perform a convolution operation on the skeleton sequence feature V1 to obtain the sequence V. T Sequence V T The formula representing the time information of the time frame itself is as follows:

[0216] V T =Conv 2D(1×1) (V1)

[0217] Then, the skeleton sequence feature V1 is compressed in the spatial dimension to obtain the skeleton sequence feature V2∈R that retains only the temporal features. C×T Next, a convolutional layer with a kernel size of 1×1 is used to projectively encode the skeleton sequence feature V2, resulting in sequence Q. T and sequence K T , sequence Q TThis represents the information that the current time frame hopes to obtain from other time frames, sequence K. T The formula represents the information that the current time frame can provide to other time frames, as shown below:

[0218] K T =Conv 2D(1×1) (V2)

[0219] Q T =Conw 2D(1×1) (V2)

[0220] Step 6.1.2, by analyzing sequence K from step 6.1.1 T and the sequence Q T Dot product operations are performed to obtain the similarity matrix between each time frame and other time frames. Then, a softmax operation is performed on the similarity matrix to obtain the weight matrix. Finally, this weight matrix is ​​applied to sequence V. T Thus, the time-biased skeleton sequence features X are obtained. t_out The corresponding formula for this process is as follows:

[0221]

[0222] Where C is the sequence K T The number of channels can increase the stability of gradients during training;

[0223] Then for X t_out The dimensional transformation is performed for subsequent operations, and the dimensional transformation operation is as follows:

[0224] X t_out :R C×T →R N×C×T →R C×T×N

[0225] Step 6.1.3, regarding X in step 6.1.2 t_out Compression is performed on the time dimension T and the spatial dimension N, retaining only the channel dimension C, to obtain the compressed sequence input V3∈R. C A fully connected linear layer with an output channel equal to the number of action categories is used on the sequence input V3 to obtain the temporally biased recognition result Label. T The formula is as follows:

[0226] Label T =softmax(linear(V3))

[0227] Step 6.2, Construct Spatial Domain Biased Branches

[0228] Step 6.2.1: This branch is similar to the time-domain biased branch structure. Based on the transformer structure design, it is necessary to obtain the spatial bias recognition result. First, the skeleton sequence feature V0 is transformed to obtain the spatially biased skeleton sequence feature V4. The transformation is as follows:

[0229] V0∈R C×T×N →V4∈R T×C×N

[0230] A convolutional layer with a kernel size of 1×1 is used to perform a convolution operation on the skeleton sequence feature V4 to obtain the sequence V. S Sequence V S The formula representing the spatial information of the joint node itself is as follows:

[0231] V S =Conv 2D(1×1) (V4)

[0232] Then, the skeleton sequence feature V4 is compressed in the time dimension to obtain the input sequence V5∈R that retains only the spatial features. C×N Next, a convolutional layer with a kernel size of 1×1 is used to projectively encode the skeleton sequence feature V5, resulting in sequence Q. S and sequence K S , sequence Q S The sequence K represents the information that the current joint node wants to obtain from other joint nodes. S The formula represents the information that the current joint node can provide to other joint nodes, as shown below:

[0233] K S =Conv 2D(1×1) (V5)

[0234] Q S =Conv 2D(1×1) (V5)

[0235] Step 6.2.2, by analyzing sequence K from step 6.2.1 S and the sequence Q S Dot product operations are performed to obtain the similarity matrix between each joint node and other joint nodes. Then, a softmax operation is performed on the similarity matrix to obtain the weight matrix. Finally, this weight matrix is ​​applied to the sequence V. S Thus, the spatially biased skeleton sequence features X are obtained. s_out The corresponding formula for this process is as follows:

[0236]

[0237] Where C is the sequence K S The number of channels can increase the stability of gradients during training;

[0238] Then for X s_out The dimensional transformation is performed for subsequent operations, and the dimensional transformation operation is as follows:

[0239] X s_out :R C×N →R T×C×N →R C×T×N

[0240] Step 6.2.3, for X s_out Compression is performed on the time dimension T and the spatial dimension N, retaining only the channel dimension C, to obtain the compressed sequence input V6∈R. C A fully connected linear layer with an output channel equal to the number of action categories is used on the sequence input V6 to obtain the spatial bias recognition result Label. S The formula is as follows:

[0241] Label S =softmax(linear(V6))

[0242] Step 6.3, Construct the spatiotemporal domain result bias module

[0243] A dual-stream fusion architecture is adopted to label the time-biased identification results from step 6.1.3. T Label of spatial bias identification result in step 6.2.3 s Each item is summed according to its weight α to obtain a fusion score, and the action label is predicted using softmax. predict The formula is as follows:

[0244] Label predict =softmax(Label) S +αLabel T )

[0245] See Figure 7 Step 7: Input the human joint dataset (Joint DataSet) from Step 2, the human skeleton dataset (Bone DataSet) from Step 3, and the human motion velocity dataset (Motion DataSet) into the network respectively. These datasets will then undergo feature learning through the modules constructed in Steps 4, 5, and 6 to obtain corresponding recognition results. Finally, the recognition results will be weighted and fused to achieve behavior recognition and classification. Details are as follows:

[0246] First, the human joint dataset from step 2 is processed through the modules constructed in steps 4, 5, and 6 to perform feature learning and obtain the corresponding recognition results (Labels). jointNext, the human skeleton dataset (BoneDataSet) and human motion velocity dataset (Motion DataSet) from step 3 are also processed through the modules constructed in steps 4, 5, and 6 to perform feature learning and obtain the corresponding recognition results (Labels). bone and Label motion Ultimately, a three-stream fusion approach was adopted to integrate the Label. joint Label bone and Label motion The results are weighted and fused item by item to obtain the final classification result Label for behavior recognition. final The formula is as follows:

[0247] Label final =softmax(Label) joint +Label bone +Label motion ).

[0248] This invention also provides an action recognition system for extracting skeleton scale difference features, comprising:

[0249] The joint feature extraction module is used to extract the spatial three-dimensional information of human skeletal joints from human motion samples in step 2 using the Openpose algorithm, and create a human joint dataset.

[0250] The skeletal feature extraction module is used to transform the human joint dataset (Joint DataSet) in step 3 to construct the human skeleton dataset (Bone DataSet) and the human motion velocity dataset (Motion DataSet);

[0251] The dynamic multi-scale spatial feature extraction module is used to extract the spatial correlation between joints within the same frame in step 4.

[0252] The multi-scale temporal feature extraction module is used to extract the temporal correlation between joints in different frames in step 4.

[0253] The joint-site feature enhancement attention module is used to realize the feature representation of the strong skeleton sequence in step 5, and to achieve mutual complementarity between joint features and human body site features;

[0254] The spatiotemporal domain result bias module is used to perform temporal bias feature extraction and spatial bias feature extraction on the skeleton sequence in a global manner in step 6, obtain the corresponding temporal bias and spatial bias recognition results respectively, and weightedly fuse the recognition results to complement each other.

[0255] The behavior recognition and classification module is used to implement the dynamic multi-scale spatial feature extraction module, multi-scale temporal feature extraction module, joint-part feature enhancement attention module, and spatiotemporal domain result bias module in step 7 to learn features and obtain corresponding recognition results. Finally, the recognition results are weighted and fused to achieve behavior recognition and classification.

[0256] The present invention also provides an action recognition device for extracting skeleton scale difference features, comprising:

[0257] Memory: A computer program that stores the above-mentioned action recognition method for extracting skeleton scale difference features, and is a computer-readable device;

[0258] Processor: Used to implement the action recognition method for extracting skeleton scale difference features when executing the computer program.

[0259] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the action recognition method for extracting skeleton scale difference features.

[0260] Experimental data

[0261] 1) Experiment 1: Ablation Comparison Experiment for Dynamic Multi-Scale Spatial Feature Extraction Module

[0262] This experiment primarily evaluates the impact of the introduced dynamic multi-scale spatial feature extraction module on network performance. Experiments were conducted to test the network performance after adding the traditional matrix power-based multi-scale structure (MS-AGCN), the non-dynamic multi-scale structure based on disentanglement (MS-DGCN), and the dynamic multi-scale structure (MS-DDGCN) proposed in this work. Comparative experiments were performed on the NTU-RGB 60 dataset, using the joint dataset as network input, and employing the top-1 evaluation criterion in cross-view.

[0263] The specific performance of each module is shown in Table 1. The baseline model without the multi-scale module performed the worst. This is because the human skeleton map contains rich information about human joint connections, and multi-scale k-distance sampling can preserve more skeleton information compared to adjacent local sampling. Analysis of the experimental results shows that the proposed dynamic multi-scale module significantly improves the network model recognition performance compared to the matrix exponentiation method and the non-dynamic multi-scale method, achieving a model recognition accuracy of 93.76%.

[0264] Table 1. Comparative experiments on MS-DDGCN

[0265]

[0266] 2) Experiment 2: Ablation Experiment of Attention Enhancement Module Targeting Joint-Site Features

[0267] This experiment primarily explores the impact of the joint-location feature enhancement attention module on the network model, and compares its performance with that of a standalone joint attention module. All tests were conducted on the NTU-RGB 60 dataset, using joint data as network input, and employing the top-1 evaluation criterion in cross-view analysis.

[0268] The test experiments on the joint-part feature enhancement attention module are shown in Table 2. Based on the optimal network model in Experiment 1, the joint-part feature fusion attention module was added. According to the results, the joint-part feature fusion attention module proposed in this invention exhibits good performance with a recognition accuracy of 94.24%. Compared with joint attention, the joint-part feature enhancement attention module shows better performance and effectively extracts the fusion features of joints and parts of the skeleton.

[0269] Table 2. Ablation experiments on joint-site feature-enhanced attention modules

[0270]

[0271] 3) Experiment 3: Ablation Experiment for the Spatiotemporal Domain Result Bias Module

[0272] This experiment primarily explores the impact of spatiotemporal domain result bias modules on the network model. All tests were conducted on the NTU-RGB 60 dataset, using joint data as network input and employing the top-1 evaluation criterion in cross-view.

[0273] The test experiments on the spatiotemporal result bias module are shown in Table 3. Based on the optimal network model in Experiment 2, the spatiotemporal result bias module was added. According to the results, the spatiotemporal result bias module proposed in this invention exhibits good performance, with a recognition accuracy of 95.10%. This shows that by extracting temporally biased and spatially biased features from the skeleton sequence in a global manner, obtaining the corresponding recognition results, and fusing and complementing each other, this method can effectively improve the recognition performance of the network model.

[0274] Table 3. Ablation experiments on the biased module in the spatiotemporal domain results

[0275]

Claims

1. An action recognition method for extracting skeleton scale difference features, characterized in that: Includes the following steps: Step 1: Record videos of human behavior and label the videos according to the categories of human behavior to obtain human action samples; Step 2: Using the Openpose algorithm, extract the spatial three-dimensional information of the human skeletal joints from the human motion samples obtained in Step 1 to create a human joint dataset. Step 3: By transforming the human joint dataset (Joint DataSet) from Step 2, construct the human skeleton dataset (Bone DataSet) and the human motion velocity dataset (Motion DataSet); Step 4: Construct a dynamic multi-scale spatial feature extraction module and a multi-scale temporal feature extraction module. The dynamic multi-scale spatial feature extraction module is used to extract the spatial correlation between joints within the same frame, and the multi-scale temporal feature extraction module is used to extract the temporal correlation between joints within different frames. Step 5: Construct a joint-site feature enhancement attention module for feature representation of strong skeleton sequences, achieving mutual complementarity between joint features and human body site features; Step 6: Construct a spatiotemporal domain result bias module to extract temporal and spatial bias features from the skeleton sequence in a global manner, obtain the corresponding temporal and spatial bias recognition results respectively, and then weight and fuse the recognition results to complement each other. Step 7: The human joint dataset (Joint DataSet) from Step 2, the human skeleton dataset (Bone DataSet) from Step 3, and the human motion speed dataset (Motion DataSet) are processed through the modules constructed in Steps 4, 5, and 6 to learn features and obtain corresponding recognition results. Finally, the recognition results are weighted and fused to achieve behavior recognition and classification.

2. The action recognition method for extracting skeleton scale difference features according to claim 1, characterized in that: The specific steps of step 2 include: The Openpose algorithm was used to process the human behavior video labeled in step 1, and the three-dimensional coordinate information of human skeleton in each frame was extracted to obtain the human joint dataset. Each image frame in the human behavior video contains 25 joint nodes, each joint node contains 3-dimensional coordinate points, and each behavior category contains T frames. The skeletons of the corresponding image frames of each behavior video are stacked together in chronological order to form a tensor of size (C0, T0, N0), where C0 = 3 represents the three-dimensional coordinates x, y, z; N0 represents the 25 joint nodes; and T0 represents the number of video frames. Using the feature matrix The feature information of the joints is represented by C0, T0, and N0, which represent the number of channels, frames, and joints of the skeleton sequence, respectively. Based on the joint feature matrix V of the human skeleton, the human joint dataset is defined as set J. J={V ti |t=1,…,T0,i=1,…,N0} Among them, V ti This indicates that the joint point is the i-th joint point of the t-th frame of the video.

3. The action recognition method for extracting skeleton scale difference features according to claim 1 or 2, characterized in that: The specific steps of step 3 include: In a diagram of the human skeleton, the joint closest to the center of gravity of the skeleton on each side is defined as the source joint V. i =(x i ,y i ,z i The joint furthest from the center of gravity of the skeleton is the target joint V. j =(x j ,y j ,z j If each bone is represented as a vector B pointing from the source joint to the target joint, then each bone is represented as a vector B. j The formula for calculating this skeleton is as follows: B j =(x j -x i ,y j -y i ,z j -z i ) Expand the human joint dataset (Joint DataSet) from step 2 in the spatial dimension to obtain the human skeleton dataset (Bone DataSet); the skeleton data includes bone length information and bone orientation information, and each bone corresponds to a unique target joint; at the same time, an empty bone with a value of 0 is assigned to the central joint; Using the feature matrix The feature information representing the human skeleton is denoted by C0, T0, and N0, which represent the number of channels, frames, and bones in the skeleton sequence, respectively. Based on the skeleton feature matrix P, the human skeleton dataset (Bone DataSet) is defined as set B. B={P ti |t=1,…,T0,i=1,…,N0} Among them, B ti This indicates that the bone is the i-th bone in the t-th frame of the video; Extend the human joint dataset from step 2 in the time dimension, defining T as the earlier frame in two adjacent frames of the joint sequence. t-1 Later frames T t The velocity S at the joint is then expressed as T. t Joint information V t =(x t ,y t ,z t ) and T t-1 Joint information V t-1 =(x t-1 ,y t-1 ,z t-1 The difference between the values ​​of the two points, and the specific formula for calculating the velocity at the joints are as follows: S i,t =(x i,t -x i,t-1 ,y i,t -y i,t-1 ,z i,t -z i,t-1 ) Among them, S i,t This represents the velocity information of the i-th node in the t-th frame, while adding 0 elements in the 0th frame to make the human motion velocity dataset (Motion DataSet) and the human joint dataset (Joint DataSet) have the same dimensions. Using the feature matrix The velocity information of human joints is represented by C0, T0, and N0, which represent the number of channels, frames, and joints in the skeleton sequence, respectively. Based on the joint velocity feature matrix Q, the human motion velocity dataset (MotionDataSet) is defined as a set M. M={Q ti |t=1,…,T0,i=1,…,N0} Among them, Q ti This represents the velocity information of the i-th joint in the t-th frame of the video.

4. The action recognition method for extracting skeleton scale difference features according to claim 1, characterized in that: The specific steps of step 4 include: Step 4.1, Construct a dynamic multi-scale skeleton topology graph: A multi-scale skeletal topology relation matrix A is defined based on the topological information of the human skeletal joints. (k) ∈R k×N×N Where k represents the scale and N represents the number of joints; A (k) This indicates that, based on the deentanglement method, the key point v i The joint point v that is k jumps away j The topological relationship and strength between them are defined by the following unentanglement relationship: Wherein d(v i ,v j ) represents the key point v i With the joint v j The shortest distance between them, A (0) The self-connection relationship of the joints is represented by the identity matrix I; A (1) The information between a joint node and its adjacent joints at a distance of 1 is represented by an adjacency matrix A; when k>1, A (k) It represents the high-scale information of the joint node and the joint nodes that are k hops away from it; it generates a relation matrix from 1 to k hops by unentanglement to describe the connection information between the joint nodes; When k>1, A (k) Join A (0) To make corrections, a trainable parameter α is introduced to dynamically adjust the connection strength between the sampling node and nodes k hops away, as shown in the following formula: Step 4.2, Construct a dynamic multi-scale spatial feature extraction module: Let the features of the original skeleton sequence be denoted as V0∈R. C×T×N Where C, T, and N represent the number of channels, frames, and joints of the skeleton sequence features, respectively; Step 4.2.1: Compare the original skeleton sequence features V0 with the dynamic multi-scale skeleton topology map from Step 4.

1. The fusion process generates a skeleton sequence feature V1 with skeleton topological information. Where V1∈R C×T×k×N Dimensionally transform V1 to fit the network: V1: R C×T×k×N →R C′×T×N Where C′=C×k; Step 4.2.2: Use a convolutional layer with a kernel size of 1×1 to learn the skeleton sequence feature V1 from Step 4.2.1 to obtain the skeleton sequence feature V2. Then, use a convolutional layer with BatchNorm and ReLU functions to enhance the feature representation ability of the skeleton sequence feature V2 and obtain the output feature X. out The formula is as follows: V2=Conv 2D(1×1) (V1) X out =Relu(BN(V2)) Under multi-scale conditions, the feature information corresponding to different scales k is aggregated by addition, so that the model can still remain effective when the value of k is large. The complete formula of this module is as follows: in, express The normalized matrix, σ(·) represents the activation function with convolution operation; Step 4.3, Construct a multi-scale temporal feature extraction module: The multi-scale temporal feature extraction module models the feature sequence based on different temporal dilation rates. This module contains four branches, each containing a 1×1 convolution. The first branch does not contain any additional convolutional layers, indicating that sampling is performed frame-by-frame in the temporal dimension. The second branch contains a temporal convolutional layer with a dilation rate of 1, indicating that sampling is performed on frames with an interval of 1 in the temporal dimension. The third branch contains a temporal convolutional layer with a dilation rate of 2, indicating that sampling is performed on frames with an interval of 2 in the temporal dimension. The fourth branch contains a max pooling layer to downsample the feature sequence. Finally, the results of the four branches are accumulated according to their weights to obtain the output feature.

5. The action recognition method for extracting skeleton scale difference features according to claim 1, characterized in that: The specific steps of step 5 are as follows: The joint-site feature enhancement attention module includes: joint attention branch and site attention branch; The input skeleton sequence features are denoted as V0∈R C×T×N Where C, T, and N represent the number of channels, frames, and joints of the skeleton sequence, respectively; Step 5.1, Construct joint attention branches Step 5.1.1: Perform dimensionality transformation on the skeleton sequence feature V0. The joint attention branch focuses on learning the spatial features of the skeleton sequence. The specific transformation is as follows: V0:R C×T×N →R V×T×N Step 5.1.2: The skeleton sequence feature V0 is pooled into a tensor of size (N×1×1) through adaptive average pooling operation, and the output size is specified as 1×1. The skeleton sequence feature V0 is compressed in the time dimension and channel dimension. Step 5.1.3: Use a convolutional layer with a kernel size of 1×1 to learn features from the skeleton sequence feature V0 in Step 5.1.

2. Then, use a convolutional layer with BatchNorm and ReLU functions to enhance the feature representation ability of the skeleton sequence feature V0 and obtain the joint attention feature Att. joint The formula is as follows: To joint =Relu(BN(Conv 2D(1×1) (V0))); Step 5.2, construct the part attention branch: Step 5.2.1: Based on the location information of the natural skeleton, the 25 joints of the human body are divided into 5 parts: the torso, left hand, left foot, right hand, and right foot, and all joints of the corresponding parts are included. By manually selecting the corresponding joints, five body part sequences P0 to P4 are obtained from the skeleton sequence feature C0. Where, N i This indicates the number of joints contained in the i-th part; Step 5.2.2: Perform dimensional transformation on the five body part sequences P0 to P4 from step 5.2.1, transforming them into sequences with dimension (N). i A tensor of ×T×C, where N is a tensor of ×T×C. i This represents the number of joints contained in the i-th body part; then, each body part set sequence P0 to P4 is pooled into a tensor of size (1×1×1) using adaptive average pooling; finally, P0 to P4 are concatenated to obtain the body part sequence feature V1∈R. 5×1×1 It is represented as follows: V1=Concat({P i |i=0,1,2,3,4}) Step 5.2.3: Use a convolutional layer with a kernel size of 1×1 to learn features from the body part sequence feature V1 in Step 5.2.

2. Then, use a convolutional layer with BatchNorm and ReLU functions to enhance the feature representation ability of the body part sequence feature V1 and obtain the body part attention feature Att. part The formula is as follows: To part =Relu(BN(Conv 2D(1×1) (V1))) Step 5.3: Construct a joint-site feature enhancement attention module Using the body part attention feature Att from step 5.2.3 part Enhance the joint attention feature Att in step 5.1.3 joint The feature representation is used to ultimately obtain the attention feature Att enhanced by joint-location features. enhance The formula is as follows: To enhance [in]=To joint [i]+δAt part [j] Where i represents the i-th joint, j represents the location of the i-th joint, and δ represents Att. part Weighting; The formula for the joint-site feature-enhanced attention module is as follows: X out =To enhance ×V0 Where V0 is the input skeleton sequence feature, X out It is the output skeleton sequence feature.

6. The action recognition method for extracting skeleton scale difference features according to claim 1, characterized in that: The specific steps of step 6 are as follows: Step 6.1, Construct the time-domain biased branch Step 6.1.1: Perform a dimensionality transformation on the skeleton sequence feature V0 to obtain the time-biased skeleton sequence feature V1, as follows: V0∈R C×T×N →V1∈R N×C×T A convolutional layer with a kernel size of 1×1 is used to perform a convolution operation on the skeleton sequence feature V1 to obtain the sequence V. T Sequence V T The formula representing the time information of the time frame itself is as follows: V T =Conv 2D(1×1) (V1) Then, the skeleton sequence feature V1 is compressed in the spatial dimension to obtain the skeleton sequence feature V2∈R that retains only the temporal features. C×T Next, a convolutional layer with a kernel size of 1×1 is used to projectively encode the skeleton sequence feature V2, resulting in sequence Q. T and sequence K T , sequence Q T This represents the information that the current time frame hopes to obtain from other time frames, sequence K. T The formula represents the information that the current time frame can provide to other time frames, as shown below: K T =Conv 2D(1×1) (V2) Q T =Conv 2D(1×1) (V2) Step 6.1.2, by analyzing sequence K from step 6.1.1 T and the sequence Q T A dot product operation is performed to obtain the similarity matrix between each time frame and other time frames. Then, a softmax operation is performed on the similarity matrix to obtain the weight matrix. Finally, this weight matrix is ​​applied to sequence V. T The time-biased skeleton sequence features X are obtained from the above. t_out The corresponding formula for this process is as follows: Where C is the sequence K T The number of channels; Then for X t_out Dimensional transformation is performed as follows: X t_out :R C×T →R N×C×T →R C×T×N Step 6.1.3, regarding X in step 6.1.2 t_out Compression is performed on the time dimension T and the spatial dimension N, retaining only the channel dimension C, to obtain the compressed sequence input V3∈R. C A fully connected linear layer with an output channel equal to the number of action categories is used on the sequence input V3 to obtain the temporally biased recognition result Label. T The formula is as follows: Label T =softmax(linear(V3)); Step 6.2, construct the spatial domain biased branch Step 6.2.1: Perform a dimensionality transformation on the skeleton sequence feature V0 to obtain the spatially biased skeleton sequence feature V4, as follows: V0∈R C×T×N →V4∈R T×C×N A convolutional layer with a kernel size of 1×1 is used to perform a convolution operation on the skeleton sequence feature V4 to obtain the sequence V. S Sequence V S The formula representing the spatial information of the joint node itself is as follows: V S =Conv 2D(1×1) (V4) Then, the skeleton sequence feature V4 is compressed in the time dimension to obtain the input sequence V5∈R that retains only the spatial features. C×N Next, a convolutional layer with a kernel size of 1×1 is used to projectively encode the skeleton sequence feature V5, resulting in sequence Q. S and sequence K S , sequence Q S The sequence K represents the information that the current joint node wants to obtain from other joint nodes. s The formula represents the information that the current joint node can provide to other joint nodes, as shown below: K S =Conv 2D(1×1) (V5) Q S =Conv 2D(1×1) (V5) Step 6.2.2, by analyzing sequence K from step 6.2.1 S and the sequence Q S A dot product operation is performed to obtain the similarity matrix between each joint node and other joint nodes. Then, a softmax operation is performed on the similarity matrix to obtain the weight matrix. Finally, this weight matrix is ​​applied to the sequence V. S Thus, the spatially biased skeleton sequence features X are obtained. s_out The corresponding formula for this process is as follows: Where C is the sequence K S The number of channels; Then for X s_out Dimensional transformation is performed as follows: X s_out :R C×N →R T×C×N →R C×T×N Step 6.2.3, for X s_out Compression is performed on the time dimension T and the spatial dimension N, retaining only the channel dimension C, to obtain the compressed sequence input V6∈R. C A fully connected linear layer with an output channel equal to the number of action categories is used on the sequence input V6 to obtain the spatial bias recognition result Label. S The formula is as follows: Label S =softmax(linear(V6)) Step 6.3, Construct the spatiotemporal domain result bias module A dual-stream fusion architecture is adopted to label the time-biased identification results from step 6.1.

3. T Label of spatial bias identification result in step 6.2.3 S Each item is summed according to its weight α to obtain a fusion score, and the action label is predicted using softmax. predict The formula is as follows: Label predict =softmax(Label s +αLabel T )。 7. The action recognition method for extracting skeleton scale difference features according to claim 1, characterized in that: The specific steps of step 7 are as follows: First, the human joint dataset from step 2 is processed through the modules constructed in steps 4, 5, and 6 to perform feature learning and obtain the corresponding recognition results (Labels). joint Next, the human skeleton dataset (Bone DataSet) and human motion velocity dataset (Motion DataSet) from step 3 are also processed through the modules constructed in steps 4, 5, and 6 to perform feature learning and obtain the corresponding recognition results (Labels). bone and Label motion Ultimately, a three-stream fusion approach was adopted to integrate the Label. joint Label bone and Label motion The results are weighted and fused item by item to obtain the final classification result Label for behavior recognition. final The formula is as follows: Label final =softmax(Label joint +Label bone +Label motion )。 8. An action recognition system for extracting skeleton scale difference features, characterized in that: include: The joint feature extraction module is used to extract the spatial three-dimensional information of human skeletal joints from human motion samples using the Openpose algorithm, and create a human joint dataset. The skeletal feature extraction module is used to transform the human joint dataset to construct the human skeleton dataset and the human motion velocity dataset. The dynamic multi-scale spatial feature extraction module is used to extract the spatial correlation between joints within the same frame; A multi-scale temporal feature extraction module is used to extract the temporal correlation between joints in different frames; The joint-site feature enhancement attention module is used for feature representation of strong skeleton sequences, enabling the mutual complementarity of joint features and human body site features; The spatiotemporal domain result bias module is used to extract temporal and spatial bias features from the skeleton sequence in a global manner, obtain the corresponding temporal and spatial bias recognition results respectively, and then weight and fuse the recognition results to complement each other. The behavior recognition and classification module is used to learn features from the human joint dataset (Joint DataSet), human skeleton dataset (Bone DataSet), and human motion velocity dataset (Motion DataSet) through a dynamic multi-scale spatial feature extraction module, a multi-scale temporal feature extraction module, a joint-part feature enhancement attention module, and a spatiotemporal domain result bias module, and obtain corresponding recognition results. Finally, the recognition results are weighted and fused to achieve behavior recognition and classification.

9. An action recognition device for extracting skeleton scale difference features, characterized in that: include: Memory: A computer program for an action recognition method for extracting skeleton scale difference features as described in any one of claims 1-7, which is a computer-readable device; Processor: Used to implement the action recognition method for extracting skeleton scale difference features as described in any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of an action recognition method for extracting skeleton scale difference features as described in any one of claims 1-7.

Images