Behavior recognition method, device and system and storage medium
Through the convolutional learning model of sparse and dense graphs, the problem of difficulty in perceiving joint relationships and rapid change details in the existing technology is solved, and the accurate recognition of actions is achieved, avoiding action confusion.
Patent Information
- Application Number
- CN202411012446.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-07-26
AI Technical Summary
Existing behavioral recognition models based on graph convolutional networks are difficult to perceive the interrelationships and the details of rapid changes between joints, resulting in frequent confusion of actions with similar motion trajectories.
Adopt a network model based on sparse and dense graph convolution learning, including an incremental feature enhancement module, an enhanced spatiotemporal convolution module and a sparse dense path module. The skeleton data is captured through the multi-head self-attention mechanism, mask adjacency matrix and attention interaction module. Intra-space features and inter-frame time dependencies in the frame reduce feature inconsistency and highlight global and local features.
Effectively avoiding action confusion, improving the ability to perceive rapid changes in details, and enhancing the accuracy of action recognition.
Smart Images

Figure CN119091496A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a behavior recognition method and device, a system, and a storage medium. Background Art
[0002] In recent years, human action recognition (HAR), as an extremely important component in the field of computer vision, has been widely studied in human-computer interaction, intelligent monitoring, medical education, etc. In particular, human action recognition based on skeleton data has received increasing attention due to the robustness of skeleton data to environmental noise. Many existing deep learning-based methods use time series of 2D or 3D joint vectors or pseudo images to construct skeleton sequences, and input these data into recurrent neural networks (RNNs) or convolutional neural networks (CNNs) to capture the spatiotemporal feature information of the skeleton. However, since the skeleton is naturally constructed as a graph in a non-Euclidean space, CNN- and RNN-based methods have difficulty in effectively perceiving the topological structure and motion changes of the skeleton.
[0003] Recent studies have demonstrated the effectiveness of graph convolutional networks (GCNs) in capturing spatiotemporal features in natural skeleton structures. ST-GCN is a pioneering framework that defines a skeleton spatiotemporal graph that reflects the inherent structure and temporal dependencies in human motion, and successfully captures human skeleton motion data by stacking spatial graph convolution operations and temporal modeling operations. Based on ST-GCN, some variants propose to enhance the exploration of skeleton topology by introducing some incremental modules, such as attention modules and semantic guidance modules. In addition, some models also adopt a two-stream architecture to explore the feature information of joints and bones, and design different feature fusion methods from the perspective of separate processing or motion transmission to improve recognition performance.
[0004] The problems with the above algorithm are:
[0005] Existing action recognition models based on graph convolutional networks mainly focus on the joint positions and motion information of the skeleton in space, as well as the overall dynamics along the time dimension. Therefore, the potential relationships between joints are ignored and the details of rapid changes cannot be perceived, resulting in frequent confusion of actions with similar motion trajectories. Summary of the invention
[0006] The technical problem to be solved by the present invention is to provide a behavior recognition method and device, system, and storage medium based on sparse and dense graph convolution learning to confuse action representation, which can perceive rapidly changing details and avoid confusing actions with similar motion trajectories.
[0007] To achieve the above object, the present invention adopts the following technical solution:
[0008] A behavior recognition method, comprising:
[0009] Step S1, obtaining skeleton data;
[0010] Step S2: Based on the skeleton data, human behavior recognition is performed by using a network model based on dense and sparse graph convolution learning to confuse action representation and a classification operation module.
[0011] Preferably, the skeleton data includes: joint information, bone features and angle features; wherein the joint information is represented as the coordinates of each joint point, the bone features are represented as vectors pointing from joints close to the weight to joints far from the center of gravity, and the angle features are represented as angle vectors between different bone vectors.
[0012] Preferably, the network model for learning confused action representation based on sparse-dense graph convolution includes: a progressive feature enhancement module, an enhanced spatiotemporal convolution module and a sparse-dense path module; wherein the sparse-dense path module includes an attention interaction module.
[0013] The present invention also provides a behavior recognition device, comprising:
[0014] Acquisition module, used to obtain skeleton data;
[0015] The recognition module is used to perform human action recognition based on skeleton data through a network model based on dense and sparse graph convolution learning to confuse action representation and a classification operation module.
[0016] Preferably, the skeleton data includes: joint information, bone features and angle features; wherein the joint information is represented as the coordinates of each joint point, the bone features are represented as vectors pointing from joints close to the weight to joints far from the center of gravity, and the angle features are represented as angle vectors between different bone vectors.
[0017] Preferably, the network model for learning confused action representation based on sparse-dense graph convolution includes: a progressive feature enhancement module, an enhanced spatiotemporal convolution module and a sparse-dense path module; wherein the sparse-dense path module includes an attention interaction module.
[0018] The present invention also provides a behavior recognition system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes the behavior recognition method when executed by the processor.
[0019] The present invention also provides a storage medium, on which a computer program is stored, and the computer program executes the behavior recognition method when running.
[0020] The present invention adopts a network model based on dense-sparse graph convolutional network to learn confusion representation and a classification operation module to realize human behavior recognition, effectively learning the spatial features within a frame and the temporal dependency between frames. The network model based on dense-sparse graph convolutional network to learn confusion representation consists of a progressive feature enhancement (PFE) module, an enhanced spatiotemporal convolution (ESTC) module and a sparse dense path (SDP) module; the PFE module uses a multi-head self-attention mechanism to generate different types of feature attention maps respectively, and by aggregating these feature attention maps step by step, different types of feature information are gradually enhanced between layers; in order to effectively model the skeleton data, the ESTC module is designed as a backbone network, and the ESTC module adds A masked adjacency matrix is added to infer the potential dependencies between nodes within a frame, and multi-scale fused temporal convolution is used to alleviate the problem of inconsistent features and semantics at different time scales. To further capture the subtle differences in dynamic temporal changes, an SDP divided at different spatiotemporal resolutions is proposed, and an attention interaction module (AIM) is designed in the SDP, which promotes the communication of important information in the two branches while ensuring frame alignment between the two paths and highlighting the global common features and local difference features captured by different paths. The classification operation module uses the softmax activation function and the fully connected layer to map the prediction probabilities of different paths, and achieves the adaptive fusion output of the prediction by adjusting the weights of the regression parameters. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0022] Figure 1 is a flow chart of a behavior recognition method according to an embodiment of the present invention;
[0023] Figure 2 The architecture diagram of the network model for learning confusing action representation based on dense and sparse graph convolution;
[0024] Figure 3 It is the feature attention generation mechanism;
[0025] Figure 4 is the cross-channel attention module. DETAILED DESCRIPTION
[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0027] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0028] Embodiment 1:
[0029] like Figure 1 As shown, an embodiment of the present invention provides a behavior recognition method based on dense and sparse graph convolution learning to confuse action representation, including:
[0030] Step S1, obtaining skeleton data;
[0031] Step S2: Based on the skeleton data, human behavior recognition is performed by using a network model based on dense and sparse graph convolution learning to confuse action representation and a classification operation module.
[0032] As an implementation method of an embodiment of the present invention, the skeleton data includes: joint information, bone features and angle features; wherein the joint information is represented as the coordinates of each joint point, the bone features are represented as vectors pointing from joints close to the weight to joints far from the center of gravity, and the angle features are represented as angle vectors between different bone vectors. The above information is encapsulated into a tensor form (B*W, C, T, N) that can be input into a graph convolutional network, wherein B represents the training batch, C represents the number of channels, T represents the number of frames, N represents the number of joint points, and W represents the number of people moving.
[0033] The GCN-based skeleton data is represented as a skeleton graph sequence. The skeleton graph of each frame can be represented as {(N,E)}, where N represents joints and E represents bones. The low-level information of the skeleton is defined as joint information and bone features. The joint information is (N 1 ,N 2 ,…,N n ), the coordinates of the i-th joint in the t-th frame are Bone features usually refer to vectors pointing from joints near the center of gravity to joints far from the center of gravity. They include the length and direction of the bone, expressed as
[0034] Since the angle change between joints hides potential features, the angle distance is defined as a high-order feature to measure the relative motion of body parts. Assume that in the tth frame, joint A on the human skeleton is the source joint, and its coordinates are expressed as The coordinates of the target joints B and C are expressed as and The length from A to B and C is given by and It means that it can be calculated as and (x c -x a ,y c -y a ,z c -z a ). The angle feature can be expressed as:
[0035]
[0036] In addition, the dynamic change of the angle feature between frames can be expressed as D a (A)=R a (A t+1 )-R a (A t ).
[0037] As an implementation method of the embodiment of the present invention, Figure 2 As shown, the network model for learning confused action representation based on sparse-dense graph convolution includes: a progressive feature enhancement (PFE) module, an enhanced spatiotemporal convolution (ESTC) module and a sparse-dense path (SDP) module; among which the sparse-dense path module includes an attention interaction module.
[0038] Furthermore, the workflow of the progressive feature enhancement (PFE) module is:
[0039] The progressive feature enhancement module is designed to use a multi-head attention mechanism to process joint information, bone information, and angle information into corresponding feature attention maps, and then different feature attention maps are gradually aggregated from low-order to high-order to enrich the high-order semantic information of the action. The feature attention map can better capture the dependencies between dynamic joints, use it to assign weights to different joints, and emphasize the feature information of key joints in a self-attention manner.
[0040] PFE modules such as Figure 2 First, the joint, bone, and angle feature information processed from the skeleton sequence are mapped into query features Q, key features K, and value features V through convolutional layers respectively:
[0041]
[0042] Among them, Q, K and
[0043] Take the angle feature as an example, Figure 3As shown in Figure 1, the feature weights are extracted by calculating the similarity between the transposes of Q and K. Subsequently, the Tanh function is applied to process the weights to generate a feature attention map in the form of (B*W,C,N,N). Similar to the angle feature, the joint M j and skeleton M b The attention map of is obtained in the same way and added to the angle attention map M at the same time a In order to perform the dot product with V. Then, multi-head self-attention is used to connect the relevant information of different subspaces. The above operation can be expressed as:
[0044]
[0045] in, T is the transposed matrix, It can increase the stability of the gradient during training. H represents the multi-head parameter. Subsequently, the tail processing is added including the feedforward network, batch normalization and two residual connections. This processing is expressed as:
[0046]
[0047] in, and is a linear transformation, ReLU activation function. Finally, the joint and bone features are concatenated with the angle features by the same method to achieve a more comprehensive representation of the skeleton features. The data format is (B*W,C,T,N), and the formula is as follows:
[0048]
[0049] Furthermore, the workflow of the enhanced spatiotemporal convolution (ESTC) module is:
[0050] In order to further explore the implicit associations between joints, an enhanced spatiotemporal convolution is designed as the backbone network, which fuses multiple types of feature flows through the PFE module. As input, it is output after a layer of mask fusion graph convolution (MF-GC) and three layers of multi-scale fusion temporal convolution (MF-TC). Figure 2 (b), MF-GC is composed of K-adjacency matrix Learnable residual matrix and the skeleton mask matrix Among them, Aggregate multi-hop neighbor information and effectively avoid transition smoothing caused by stacking multiple layers of graph convolution. Dynamically enhance and weaken edges to increase the flexibility of MF-TC. From the adjacency matrix
[0051] The adjacency matrix of key points learned in the training data, including nine joint nodes such as head, hand, elbow and foot, can usually express action information in more detail. By capturing the non-physical dependencies between key joints, more discriminative feature representations between categories are learned. The calculation process of MF-GC is as follows:
[0052]
[0053] MF-TC is based on multi-scale temporal convolution, which inputs the feature information processed by MF-GC into multi-scale temporal convolution, and then aggregates local and global temporal features through attention in the channel dimension. The temporal context information from different scales is fused into a refined feature map through the weighted input feature map and the attention map. The final output after ESTC processing can be expressed as:
[0054]
[0055] in, Represents the computation of the ESTC block. L(·) represents the local context channel with dimensions matching the input features, which is used to preserve and emphasize the details within the features, while G(·) is the global channel context.
[0056] Furthermore, the workflow of the Sparse Dense Path (SDP) module is:
[0057] In order to capture the dynamic relationship of motion at different frequencies on the relative motion scale, such as Figure 2 As shown in (c), a stacked double-layer sparse-dense channel is designed to obtain the feature data after the ESTC block. The input is divided into a sparse path and a dense path with different temporal sampling rates. Specifically, the sparse path is a path with low input resolution and high channel capacity to capture the correlation between joints. It uses the frame extraction function (ζ extrac ) in the time dimension Down-sampling is performed, that is, one frame is extracted every τ frames to obtain the feature map of the sparse path. Then, the feature information enters a The coarse-grained GCN block represented by , the output features can be expressed as:
[0058]
[0059] It is worth noting that the coarse-grained GCN block removes the attention fusion operation in the multi-scale fused temporal convolutions of the first two layers to emphasize the spatial information.
[0060] Since the learning goal is to capture subtle temporal motion differences using dense pathways, it is necessary to reduce the focus on spatial dimensions to reduce information redundancy. The temporal frequency is taken as input and the embedding function (ζ embed ) reduces its channel dimension by half, thereby reducing the representation ability of spatial semantic features. Then, a fine GCN block is applied to learn the feature information of dense paths, expressed as It has the same detailed structure as ESTC, and the output of the dense path is represented as follows:
[0061]
[0062] Furthermore, the workflow of the attention interaction module is:
[0063] The features processed by ESTC in the dense and sparse paths are used as input respectively, and then flow to the feature transfer attention and cross-channel attention between the sparse-dense paths in sequence, as shown in Figure 2 The design of the attention interaction module mainly considers two motivations: (i) it is able to filter out important feature information that can be transferred in the two branch paths and enhance each other; (ii) it should learn and benefit from the discriminative feature information existing in the two branches.
[0064] Considering that sparse channels have higher spatial resolution, feature transfer attention is achieved through frame attention The output of the sparse path is used as input to flow into the dense channel to provide richer spatial and semantic information. The complete frame attention process is as follows: The temporal dimension T' of is upsampled to T at a sampling rate of 2 times, ensuring that the relative temporal positions of the sparse path and the dense path are aligned. Then, a linear layer is used to project the channel size to align with the dense path, and the inter-frame attention weights are generated by convolution. Finally, the inter-frame attention allocation is achieved on the dense path by matrix addition, expressed as:
[0065]
[0066] In order to filter important context features in the dense path to be passed to the sparse path, a lightweight self-attention mask To filter the fine-grained features from the dense path and input them to the sparse path. The mask processing includes pooling, point-by-point convolution and Sigmoid function, and after dot product with the dense path features, it flows to the sparse path. The final sparse feature is expressed as:
[0067]
[0068] In addition, in order to automatically extract the importance difference between sparse and dense channels during training, cross-channel attention is designed at the end of the two branches, and the output after feature transfer attention processing is used as input to dynamically adjust the responses of different channels, such as Figure 4 Specifically, for the channel features in different branches The weight of The interaction between its k neighboring nodes. Share the same learnable parameters through 1d convolution of size k, and then get the output of different branches through Sigmoid function. Its data format is (B*W, C', T', N'), as shown below:
[0069]
[0070] As an implementation method of an embodiment of the present invention, the workflow of the classification operation module is as follows:
[0071] Data flow of different branches After cross-channel attention, it enters the fully connected (FC) layer with Softmax activation function and is mapped to the probability distribution of the candidate category. The adaptive fusion output of the prediction is achieved by adjusting the weights of the regression parameters. Specifically, each branch has a global average pooling (GAP) layer, a fully connected (FC) layer, and a Softmax layer. By passing the feature map from these network layers in turn, the prediction S of each branch is obtained respectively. sp and S de Considering the characteristics of different paths, such as “reading” and “writing”, the dense branch can effectively characterize the movement details of the hand. In contrast, for “standing”, the global features of the sparse branch are more important. Therefore, two parameters ω are regressed according to the importance of different paths for different actions. s and ω d To adjust the output weights of sparse and dense branches, adaptively fuse and output the final classification result (N, class), which can be expressed as:
[0072] S out =ω s ·S sp +ω d ·S de (17)
[0073] Embodiment 2:
[0074] The embodiment of the present invention further provides a behavior recognition device, including:
[0075] Acquisition module, used to obtain skeleton data;
[0076] The recognition module is used to perform human action recognition based on skeleton data through a network model based on dense and sparse graph convolution learning to confuse action representation and a classification operation module.
[0077] As an implementation method of an embodiment of the present invention, the skeleton data includes: joint information, bone features and angle features; wherein the joint information is represented as the coordinates of each joint point, the bone features are represented as vectors pointing from joints close to the center of gravity to joints away from the center of gravity, and the angle features are represented as angle vectors between different bone vectors.
[0078] As an implementation method of an embodiment of the present invention, a network model for learning confused action representation based on sparse-dense graph convolution includes: a progressive feature enhancement module, an enhanced spatiotemporal convolution module and a sparse-dense path module; wherein the sparse-dense path module includes an attention interaction module.
[0079] Embodiment 3:
[0080] An embodiment of the present invention further provides a behavior recognition system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a behavior recognition method when executed by the processor.
[0081] Embodiment 4:
[0082] An embodiment of the present invention further provides a storage medium, on which a computer program is stored, and the computer program executes the behavior recognition method when running.
[0083] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.
Claims
1. A behavior recognition method, characterized in that: include: Step S1, obtaining skeleton data; Step S2: Based on the skeleton data, human behavior recognition is performed by using a network model based on dense and sparse graph convolution learning to confuse action representation and a classification operation module.
2. The behavior recognition method according to claim 1, characterized in that: Skeleton data includes: joint information, bone features and angle features; among them, joint information is represented by the coordinates of each joint point, bone features are represented by vectors pointing from joints close to the weight to joints far from the center of gravity, and angle features are represented by angle vectors between different bone vectors.
3. The behavior recognition method according to claim 2, characterized in that: The network model based on sparse-dense graph convolution learning for confusing action representation includes: a progressive feature enhancement module, an enhanced spatiotemporal convolution module and a sparse-dense path module; wherein the sparse-dense path module includes an attention interaction module.
4. A behavior recognition device, characterized in that: include: Acquisition module, used to obtain skeleton data; The recognition module is used to perform human action recognition based on skeleton data through a network model based on dense and sparse graph convolution learning to confuse action representation and a classification operation module.
5. The behavior recognition device according to claim 4, characterized in that: Skeleton data includes: joint information, bone features and angle features; among them, joint information is represented by the coordinates of each joint point, bone features are represented by vectors pointing from joints close to the weight to joints far from the center of gravity, and angle features are represented by angle vectors between different bone vectors.
6. The behavior recognition device according to claim 5, characterized in that: The network model based on sparse-dense graph convolution learning for confusing action representation includes: a progressive feature enhancement module, an enhanced spatiotemporal convolution module and a sparse-dense path module; wherein the sparse-dense path module includes an attention interaction module.
7. A behavior recognition system, characterized in that: include: A memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the behavior recognition method according to any one of claims 1 to 3 is executed.
8. A storage medium, characterized in that: The storage medium stores a computer program, which executes the behavior recognition method according to any one of claims 1 to 3 when running.
Citation Information
Patent Citations
Action recognition method and system based on multi-stream information enhancement graph convolutional network, and storage medium
CN113408455A
Three-dimensional human body posture estimation method, system and device and medium
CN113989854A
Abnormal behavior recognition method, system and equipment with enhanced skeleton joint points and medium
CN114782998A
Human body behavior recognition method and system based on self-attention enhancement graph neural network
CN116863531A
Human body behavior recognition method and system based on multi-channel directed graph convolution
CN116895097A
Cited By
Automatic spontaneous development identification method based on enhanced spatial-temporal characteristic network
CN119399566A
Automated spontaneous imaging recognition method based on enhanced spatiotemporal feature network
CN119399566B
