Behavior recognition method, device, system, and storage medium

Abstract

The present invention discloses a method, device, system, and storage medium for action recognition, comprising: step S1, acquiring skeleton data; and step S2, using the skeleton data, identifying human actions using a network model and a classification operation module based on dense-sparse graph convolution learning to confuse action representations. The technical solution of the present invention can perceive rapidly changing details and avoid confusing actions with similar motion trajectories.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a behavior recognition method and device, a system, and a storage medium. Background Art

[0002] In recent years, human action recognition (HAR), as an extremely important component of computer vision, has been widely studied in human-computer interaction, intelligent monitoring, medical education, and other fields. In particular, human action recognition based on skeleton data has attracted increasing attention due to the robustness of skeleton data to environmental noise. Many existing deep learning-based methods use time series of 2D or 3D joint vectors or pseudo images to construct skeleton sequences and input these data into recurrent neural networks (RNNs) or convolutional neural networks (CNNs) to capture the spatiotemporal feature information of the skeleton. However, since the skeleton is naturally constructed as a graph in a non-Euclidean space, CNN- and RNN-based methods have difficulty effectively perceiving the topological structure and motion changes of the skeleton.

[0003] Recent studies have demonstrated the effectiveness of graph convolutional networks (GCNs) in capturing spatiotemporal features in natural skeletal structures. ST-GCN is a groundbreaking framework that defines a spatiotemporal skeleton graph that reflects the inherent structure and temporal dependencies in human motion, successfully capturing human skeletal motion data by stacking spatial graph convolution operations and temporal modeling operations. Based on ST-GCN, some variants propose to enhance the exploration of skeletal topology by introducing some incremental modules, such as attention modules and semantic guidance modules. In addition, some models also adopt a two-stream architecture to explore the feature information of joints and bones, and design different feature fusion methods from the perspective of separate processing or motion transmission to improve recognition performance.

[0004] The problems with the above algorithm are:

[0005] Existing action recognition models based on graph convolutional networks primarily focus on the spatial joint positions and motion information of the skeleton, as well as the overall dynamics along the time dimension. As a result, they ignore the potential interrelationships between joints and are unable to perceive rapidly changing details, resulting in actions with similar motion trajectories often being confused. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a behavior recognition method and device, system, and storage medium based on dense and sparse graph convolution learning to confuse action representation, which can perceive rapidly changing details and avoid confusing actions with similar motion trajectories.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A behavior recognition method, comprising:

[0009] Step S1, obtaining skeleton data;

[0010] Step S2: Based on the skeleton data, human action recognition is performed by using a network model based on dense and sparse graph convolution learning to confuse action representation and a classification operation module.

[0011] Preferably, the skeleton data includes: joint information, bone features and angle features; wherein the joint information is represented as the coordinates of each joint point, the bone features are represented as vectors pointing from joints close to the weight to joints away from the center of gravity, and the angle features are represented as angle vectors between different bone vectors.

[0012] Preferably, the network model for learning confused action representation based on sparse-dense graph convolution includes: a progressive feature enhancement module, an enhanced spatiotemporal convolution module and a sparse-dense path module; wherein the sparse-dense path module includes an attention interaction module.

[0013] The present invention also provides a behavior recognition device, comprising:

[0014] Acquisition module, used to obtain skeleton data;

[0015] The recognition module is used to recognize human actions based on skeleton data through a network model based on dense and sparse graph convolution learning to confuse action representation and a classification operation module.

[0016] Preferably, the skeleton data includes: joint information, bone features and angle features; wherein the joint information is represented as the coordinates of each joint point, the bone features are represented as vectors pointing from joints close to the weight to joints away from the center of gravity, and the angle features are represented as angle vectors between different bone vectors.

[0017] Preferably, the network model for learning confused action representation based on sparse-dense graph convolution includes: a progressive feature enhancement module, an enhanced spatiotemporal convolution module and a sparse-dense path module; wherein the sparse-dense path module includes an attention interaction module.

[0018] The present invention also provides a behavior recognition system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes the behavior recognition method when executed by the processor.

[0019] The present invention also provides a storage medium, on which a computer program is stored, and the computer program executes the behavior recognition method when running.

[0020] The present invention adopts a network model based on dense and sparse graph convolutional network to learn confusion representation and a classification operation module to realize human behavior recognition, effectively learning the spatial features within the frame and the temporal dependency between frames. The network model based on dense and sparse graph convolutional network to learn confusion representation consists of a progressive feature enhancement (PFE) module, an enhanced spatiotemporal convolution (ESTC) module and a sparse dense path (SDP) module; the PFE module uses a multi-head self-attention mechanism to generate different types of feature attention maps respectively, and by aggregating these feature attention maps step by step, different types of feature information are gradually enhanced between layers; in order to effectively model the skeleton data, the ESTC module is designed as a backbone network, and the ESTC module adds A masked adjacency matrix is ​​added to infer the potential dependencies between nodes within the frame, and multi-scale fused temporal convolution is used to alleviate the problem of inconsistent features and semantics at different time scales. To further capture the subtle differences in dynamic temporal changes, an SDP divided at different spatiotemporal resolutions is proposed, and an attention interaction module (AIM) is designed in the SDP. This module promotes the communication of important information between the two branches, while ensuring frame alignment between the two paths and highlighting the global common features and local difference features captured by different paths. The classification operation module uses the softmax activation function and the fully connected layer to map the prediction probabilities of different paths, and achieves adaptive fusion output of the prediction by adjusting the weights of the regression parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0022] Figure 1 This is a flow chart of a behavior recognition method according to an embodiment of the present invention;

[0023] Figure 2 The architecture diagram of the network model for learning confusing action representation based on dense and sparse graph convolution;

[0024] Figure 3 It is the feature attention generation mechanism;

[0025] Figure 4 is the cross-channel attention module. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0027] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] Example 1:

[0029] like Figure 1 As shown, an embodiment of the present invention provides an action recognition method for confusing action representation based on sparse-dense graph convolution learning, including:

[0030] Step S1, obtaining skeleton data;

[0031] Step S2: Based on the skeleton data, human action recognition is performed by using a network model based on dense and sparse graph convolution learning to confuse action representation and a classification operation module.

[0032] As an implementation method of an embodiment of the present invention, the skeleton data includes: joint information, bone features, and angle features; wherein the joint information is represented as the coordinates of each joint point, the bone features are represented as vectors pointing from joints close to the center of gravity to joints far from the center of gravity, and the angle features are represented as angle vectors between different bone vectors. The above information is encapsulated into a tensor form (B*W,C,T,N) that can be input into a graph convolutional network, where B represents the training batch, C represents the number of channels, T represents the number of frames, N represents the number of joints, and W represents the number of people moving.

[0033] The GCN-based skeleton data is represented as a skeleton graph sequence. The skeleton graph of each frame can be represented as {(N,E)}, where N represents joints and E represents bones. The low-level information of the skeleton is defined as joint information and bone features. The joint information is (N1, N2, ..., N n ), the coordinates of the i-th joint in the t-th frame are Bone features usually refer to vectors pointing from joints near the center of gravity to joints far away from the center of gravity. They include the length and direction of the bone and are expressed as

[0034] Since the angle changes between joints hide potential features, the angle distance is defined as a high-order feature to measure the relative motion of body parts. Assume that in the t-th frame, joint A on the human skeleton is the source joint, and its coordinates are expressed as The coordinates of the target joints B and C are expressed as and The length from A to B and C is given by and It means that it can be calculated as and (x c -x a ,y c -y a ,z c -z a ). The angle feature can be expressed as:

[0035]

[0036] In addition, the dynamic change of angle features between frames can be expressed as D a (A)=R a (A t+1 )-R a (A t ).

[0037] As an implementation method of the embodiment of the present invention, Figure 2 As shown in the figure, the network model for learning confused action representation based on sparse-dense graph convolution includes: progressive feature enhancement (PFE) module, enhanced spatiotemporal convolution (ESTC) module and sparse-dense path (SDP) module; among them, the sparse-dense path module includes an attention interaction module.

[0038] Furthermore, the workflow of the Progressive Feature Enhancement (PFE) module is as follows:

[0039] The progressive feature enhancement module uses a multi-head attention mechanism to process joint, skeletal, and angle information into corresponding feature attention maps. These maps are then aggregated from low-level to high-level to enrich the high-level semantics of the action. These feature attention maps better capture the dependencies between dynamic joints, which can be used to assign weights to different joints and emphasize the features of key joints through self-attention.

[0040] PFE modules such as Figure 2 First, the joint, bone, and angle feature information processed from the skeleton sequence are mapped into query features Q, key features K, and value features V through convolutional layers respectively:

[0041]

[0042] Among them, Q, K and

[0043] Take the angle feature as an example, Figure 3As shown in Figure 2, feature weights are extracted by calculating the similarity between the transposes of Q and K. Subsequently, the Tanh function is applied to process the weights to generate a feature attention map in the form of (B*W,C,N,N). Similar to the angle feature, the joint M j and skeleton M b The attention map of is obtained in the same way and added to the angle attention map M at the same time a In order to perform the dot product with V. Then, multi-head self-attention is used to connect the relevant information of different subspaces. The above operation can be expressed as:

[0044]

[0045] in, T is the transposed matrix, This increases the stability of the gradient during training. H represents the multi-head parameter. Subsequently, the tail processing is added, including the feedforward network, batch normalization, and two residual connections. This process is expressed as:

[0046]

[0047] in, and is a linear transformation, Represents the ReLU activation function. Finally, the joint and bone features obtained are concatenated with the angle features using the same method to achieve a more comprehensive representation of the skeleton features, and the output is The data format is (B*W,C,T,N), and the formula is as follows:

[0048]

[0049] Furthermore, the workflow of the enhanced spatiotemporal convolution (ESTC) module is:

[0050] In order to further explore the implicit associations between joints, we design enhanced spatiotemporal convolution as the backbone network, which fuses multiple types of feature flows through the PFE module. As input, it is output after a layer of mask fusion graph convolution (MF-GC) and three layers of multi-scale fusion temporal convolution (MF-TC). Figure 2 (b), MF-GC is composed of K-adjacency matrix Learnable residual matrix and the skeleton mask matrix Among them, Aggregating multi-hop neighbor information effectively avoids transition smoothing caused by stacking multiple layers of graph convolution. Dynamically enhance and weaken edges to increase the flexibility of MF-TC. From the adjacency matrix

[0051] The adjacency matrix of key points learned in the training data, including nine joint nodes such as the head, hands, elbows, and feet, can generally express action information in more detail. By capturing the non-physical dependencies between key joints, more discriminative feature representations between categories are learned. The calculation process of MF-GC is as follows:

[0052]

[0053] MF-TC is based on multi-scale temporal convolution. It inputs the feature information processed by MF-GC into the multi-scale temporal convolution, and then aggregates local and global temporal features through attention in the channel dimension. The temporal context information from different scales is fused into a refined feature map through the weighted input feature map and the attention map. The final output after ESTC processing can be expressed as:

[0054]

[0055] in, Represents the computation of the ESTC block. L(·) represents the local context channel with dimensions matching the input features, which is used to preserve and emphasize details within the features, while G(·) is the global channel context.

[0056] Furthermore, the workflow of the Sparse Dense Path (SDP) module is as follows:

[0057] In order to capture the dynamic relationship of motion at different frequencies on the relative motion scale, e.g. Figure 2 As shown in (c), a stacked double-layer sparse-dense channel is designed to pass the feature data obtained after the ESTC block. The input is divided into sparse path and dense path with different temporal sampling rates. Specifically, the sparse path is a path with low input resolution and high channel capacity to capture the correlation between joints. It uses the frame extraction function (ζ extrac ) in the time dimension Downsampling is performed, that is, one frame is extracted every τ frames to obtain the feature map of the sparse path. Subsequently, the feature information enters a The coarse-grained GCN block represented by , the output features can be expressed as:

[0058]

[0059] It is worth noting that the coarse-grained GCN block removes the attention fusion operation in the multi-scale fused temporal convolution of the first two layers to emphasize spatial information.

[0060] Since the learning goal is to capture subtle temporal motion differences using dense pathways, it is necessary to reduce the focus on spatial dimensions to reduce information redundancy. The temporal frequency is taken as input and the embedding function (ζ embed ) reduces its channel dimension by half, thereby reducing the representation ability of spatial semantic features. Then, a fine GCN block is applied to learn the feature information of dense paths, which is expressed as It has the same detailed structure as ESTC, and the output of the dense path is represented as follows:

[0061]

[0062] Furthermore, the workflow of the attention interaction module is as follows:

[0063] The features processed by ESTC in the dense and sparse paths are used as input respectively, and then flow to the feature transfer attention and cross-channel attention between the sparse-dense paths in sequence, as shown in Figure 2 (c) The design of the attention interaction module mainly considers two motivations: (i) it is able to filter out important feature information that can be transferred in the two branch paths and enhance each other; (ii) it should learn and benefit from the discriminative feature information existing in the two branches.

[0064] Considering that sparse channels have higher spatial resolution, feature transfer attention is achieved through frame attention The output of the sparse path is used as input to flow into the dense channel to provide richer spatial and semantic information. The complete frame attention process is as follows: The temporal dimension T' of is upsampled to T at a sampling rate of 2 times, ensuring that the temporal relative positions of the sparse path and the dense path are aligned. Then, a linear layer is used to project the channel size to align with the dense path, and convolution is used to generate inter-frame attention weights. Finally, inter-frame attention allocation is achieved on the dense path through matrix addition, expressed as:

[0065]

[0066] In order to filter important context features in the dense path to pass to the sparse path, a lightweight self-attention mask To filter the fine-grained features from the dense path and input them to the sparse path. The mask processing includes pooling, point-by-point convolution and Sigmoid function, and then flows to the sparse path after dot product with the dense path features. The final sparse feature is represented as:

[0067]

[0068] In addition, in order to automatically extract the importance difference between sparse and dense channels during training, cross-channel attention is designed at the end of the two branches, and the output after feature transfer attention processing is used as input to dynamically adjust the response of different channels, such as Figure 4 Specifically, for the channel features in different branches The weight of The interaction between its k neighboring nodes. Share the same learnable parameters through 1d convolution of size k, and then get the output of different branches through Sigmoid function. Its data format is (B*W,C',T',N'), as shown below:

[0069]

[0070] As an implementation method of an embodiment of the present invention, the workflow of the classification operation module is as follows:

[0071] Data flows in different branches After cross-channel attention, it enters the fully connected (FC) layer with Softmax activation function and is mapped to the probability distribution of the candidate category. The adaptive fusion output of the prediction is achieved by adjusting the weights of the regression parameters. Specifically, each branch has a global average pooling (GAP) layer, a fully connected (FC) layer, and a Softmax layer. By passing the feature map from these network layers in sequence, the prediction S of each branch is obtained. sp and S de Considering the characteristics of different paths, such as “reading” and “writing”, the dense branch can effectively represent the details of the hand movement. In contrast, for “standing”, the global features of the sparse branch are more important. Therefore, two parameters ω are regressed based on the importance of different paths for different actions. s and ω d To adjust the output weights of sparse and dense branches, adaptively fuse and output the final classification result (N, class), which can be expressed as:

[0072] S out =ω s ·S sp +ω d ·S de (17)

[0073] Example 2:

[0074] An embodiment of the present invention further provides a behavior recognition device, comprising:

[0075] Acquisition module, used to obtain skeleton data;

[0076] The recognition module is used to recognize human actions based on skeleton data through a network model based on dense and sparse graph convolution learning to confuse action representation and a classification operation module.

[0077] As an implementation method of an embodiment of the present invention, skeleton data includes: joint information, bone features and angle features; wherein the joint information is represented as the coordinates of each joint point, the bone features are represented as vectors pointing from joints close to the center of gravity to joints away from the center of gravity, and the angle features are represented as angle vectors between different bone vectors.

[0078] As an implementation method of an embodiment of the present invention, a network model for learning confused action representation based on sparse-dense graph convolution includes: a progressive feature enhancement module, an enhanced spatiotemporal convolution module and a sparse-dense path module; wherein the sparse-dense path module includes an attention interaction module.

[0079] Example 3:

[0080] An embodiment of the present invention further provides a behavior recognition system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a behavior recognition method when executed by the processor.

[0081] Example 4:

[0082] An embodiment of the present invention further provides a storage medium having a computer program stored thereon, and the computer program executes the behavior recognition method when running.

[0083] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A behavior recognition method, characterized in that: include: Step S1, obtaining skeleton data; Skeleton data includes: joint information, bone features, and angle features; joint information is represented by the coordinates of each joint point, bone features are represented by vectors pointing from joints close to the center of gravity to joints far from the center of gravity, and angle features are represented by the angle vectors between different bone vectors; Step S2: Based on the skeleton data, human action recognition is performed using a network model based on dense-sparse graph convolution learning to represent confused action representations and a classification operation module; the network model based on dense-sparse graph convolution learning to represent confused action representations includes: a progressive feature enhancement module, an enhanced spatiotemporal convolution module, and a sparse-dense path module; wherein the sparse-dense path module includes an attention interaction module; The workflow of the progressive feature enhancement module is: For angle features, the joint, bone, and angle feature information processed from the skeleton sequence are mapped into query features Q, key features K, and value features V through convolutional layers respectively: Among them, Q, K and The feature weights are extracted by calculating the similarity between the transpose of Q and K; the Tanh function is applied to process the weights to generate a feature attention map, which is in the form of (B*W,C,N,N), where B represents the training batch, C represents the number of channels, T represents the number of frames, N represents the number of joint points, and W represents the number of people moving. Similar to the angle feature, the joint M j and skeleton M b The attention map of is obtained in the same way and added to the angle attention map M at the same time a In order to perform the dot product with V, multi-head self-attention is used to connect the relevant information of different subspaces. The above operation can be expressed as: in, T is the transposed matrix, It can increase the stability of the gradient during training. H represents the multi-head parameter. Adding tail processing includes feedforward network, batch normalization and two residual connections. This process is expressed as: in, and is a linear transformation, Represents the ReLU activation function; The joint and bone features obtained are spliced ​​with the angle features by the same method to achieve a more comprehensive representation of the skeleton features and output The data format is (B*W,C,T,N), and the formula is as follows: The workflow of the sparse-dense path module is: Design a stacked double-layer sparse-dense channel to obtain the feature data after the ESTC block. Input into sparse path and dense path divided by different time sampling rates respectively; using frame extraction function ζ extrac In the time dimension Downsampling is performed, that is, one frame is extracted every τ frames to obtain the feature map of the sparse path; the feature information enters a matrix composed of z c (·) represents the coarse-grained GCN block, outputting features It can be expressed as: Dense paths are maintained The temporal frequency is taken as input and the embedding function ζ is used embed Its channel dimension is reduced by half; a fine GCN block is applied to learn the feature information of the dense path, denoted as z f (·), which has the same detailed structure as ESTC, the output of the dense path It is expressed as follows:

2. A behavior recognition device for implementing the behavior recognition method according to claim 1, characterized in that: include: Acquisition module, used to obtain skeleton data; The recognition module is used to recognize human actions based on skeleton data through a network model based on dense and sparse graph convolution learning to confuse action representation and a classification operation module.

3. A behavior recognition system, characterized in that: include: A memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the behavior recognition method according to claim 1 is executed.

4. A storage medium, characterized in that The storage medium stores a computer program, which executes the behavior recognition method according to claim 1 when running.

Images