Behavior recognition method based on space-time mixed attention module

Abstract

The invention relates to the technical field of computer vision, in particular to a behavior recognition method based on a space-time mixed attention module, which comprises the following steps of: S1, dividing joint points into regions: dividing each joint point into regions, and respectively performing feature enhancement on the sub-regions to obtain corresponding feature weights; and S3, space-time stream fusion: combining the attention scores of the space-time streams to obtain a space-time attention score statt, and endowing the obtained space-time attention score statt to an input feature map to realize enhancement of time domain and space domain features. According to the method, the perception ability of the model to the local space structure and the time dynamic change is enhanced, the interaction and information exchange between the time domain and the space domain characteristics are enhanced, and the understanding ability and prediction precision of the model to the complex space-time mode are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a behavior recognition method based on a spatiotemporal hybrid attention module. Background Art

[0002] With the advancement of deep learning technology, behavior recognition technology has developed unprecedentedly and has been applied to many fields such as public safety, traffic safety, and medical health. For example, in the field of public safety, behavior recognition technology is used to monitor suspicious and dangerous behaviors; in the field of traffic safety, behavior recognition technology is used to identify traffic violations; in the field of medical health, it is used to monitor patient activities to prevent falls. In the field of deep learning, behavior recognition technology identifies and classifies human behaviors by analyzing video or image sequences, and relies on large labeled data sets (such as UCF101, Kinetics, etc.) to train models. Traditional feature extraction methods have gradually been replaced by convolutional neural networks (CNNs), combined with time series modeling technologies such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), and 3D CNNs, which have achieved effective extraction of spatial and temporal features. In addition, the application of attention mechanisms and transfer learning has further improved the accuracy and efficiency of recognition, making behavior recognition widely used in monitoring, smart homes, and medical health.

[0003] However, CNN technology mostly processes RGB data and lacks spatial structure information. In order to capture the spatial structure of nodes in the image, it is obvious to use graph convolutional network (GCN) to process skeleton point data. At present, skeleton point data has been applied in many fields. In sports training and health monitoring, accurate joint position information is used to analyze and guide movements, and it is also used for health monitoring, physical condition assessment and rehabilitation progress assessment; in human-computer interaction and virtual reality, users can interact with the virtual environment through body movements to achieve a more immersive experience; in the field of intelligent monitoring and security, skeleton point data is used for abnormal behavior detection, such as fall detection and intrusion detection, to improve the intelligence level of the monitoring system. However, since the topological map formed by the skeleton points does not meet the translation invariance, CNN is no longer applicable. Graph convolutional network (GCN) can effectively process non-Euclidean data such as topological maps and capture spatiotemporal information in the spatial and temporal domains.

[0004] Graph convolutional networks are used to achieve behavior recognition. Based on decoupled graph convolution, spatial and temporal domain features are extracted through spatial graph convolution and temporal convolution. Spatial graph convolution obtains neighborhood information through the adjacency matrix, and then uses aggregation mechanisms such as global averaging and maximum pooling to collect and integrate feature information from the neighborhood to obtain the spatial features of each node, thereby capturing local dependencies in the graph structure. Temporal convolution usually uses one-dimensional convolution kernels to process time series data. These convolution kernels slide along the time axis to capture local patterns in the time series. Although feature extraction based on spatial graph convolution and temporal convolution can capture local structural information between nodes and the changing trend of time series data to a certain extent, it usually cannot fully consider factors such as long-distance dependencies between nodes, complex temporal dynamic changes, and multimodal information fusion. It is not enough to obtain highly discriminative and significant features, which limits the performance of the model when processing complex tasks.

[0005] The spatial attention module obtains the importance between different nodes and dynamically adjusts the weight of each node feature, thereby effectively capturing the important relationship between key nodes in the graph structure. The temporal attention module focuses on the time domain, automatically assigns different weights to each time step by analyzing the importance of different frame data, and thus better captures the long-term dependencies and instantaneous changes in the time series. The combination of the two can significantly improve the performance and accuracy of the model when processing spatiotemporal data.

[0006] At present, most attention modules have the following problems: The attention spatiotemporal enhancement modules are independent of each other. Although the key nodes and key frames are enhanced separately, the model performance is limited to a certain extent due to the separation of information and the asynchrony of the enhancement of temporal and spatial domain features. This separation of information makes it difficult for the model to fully capture the complex patterns and deep associations in spatiotemporal data, thus limiting the overall performance and generalization ability of the model to a certain extent. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention provides a behavior recognition method based on a spatiotemporal hybrid attention module, which respectively solves the problem that spatial graph convolution and temporal domain convolution are insufficient to obtain discriminative and significant features, and the problem that the attention spatiotemporal enhancement modules are independent of each other, enhances the model's perception of local spatial structure and temporal dynamic changes, strengthens the interaction and information exchange between time domain and spatial domain features, and also improves the model's understanding of complex spatiotemporal patterns and prediction accuracy.

[0008] The present invention is achieved through the following technical solutions:

[0009] A method for behavior recognition based on a spatiotemporal hybrid attention module is provided, comprising the following steps:

[0010] S1. Divide the joint points into regions: Divide each joint point into regions, perform feature enhancement on the sub-regions respectively, and obtain the corresponding feature weights;

[0011] S2. Design a spatiotemporal hybrid attention module: combine the temporal information evolution with the spatial information evolution to obtain a spatiotemporal attention map;

[0012] The input feature map is denoted as F0, which compresses the time and space information to obtain spatial and temporal feature maps respectively, and then concatenates them by channel to obtain the feature Figure 1 , denoted as F1, expressed as:

[0013] F1=SqueezeT(F0)||SqueezeS(F0) (0-3);

[0014] Among them, SqueezeT(·) and SqueezeS(·) represent the compression of temporal information and spatial information respectively, and || represents the concatenation of feature maps by channel, and the dimension after concatenation is C×(V+T);

[0015] The characteristics Figure 1 After passing through the Fusion module, the features are obtained Figure 2 , denoted as F2, feature Figure 2 It is expressed as:

[0016] F2=PReLU(α2(α1(F1))) (0-4);

[0017] Among them, α1(·) and α2(·) represent the first and second convolution layers respectively, PReLU(·) represents the PReLU activation function; the number of channels of F2 is The characteristics Figure 2 Split by dimension, we get the spatial feature map and the temporal feature map, denoted as F S 、F T , F S The dimension is F T The dimension is Then the spatial feature map F S And the time feature map F T Further extract features along the spatial stream and temporal stream respectively;

[0018] S3, spatiotemporal stream fusion: The attention scores of the spatiotemporal streams are combined to obtain the spatiotemporal attention score st att , which is expressed as follows:

[0019] st att =s att ×t att (1-3);

[0020] Among them, st attThe dimension is C×T×V;

[0021] S4, the obtained spatiotemporal attention score st att By assigning the input feature map, the temporal and spatial domain features are enhanced.

[0022] Furthermore, in step S1, the 25 key nodes of the whole body are divided into four groups, namely, face and trunk (nodes 0, 1, 8, 15-18), upper limbs (2-7), left lower limbs (9-11, 22-24) and right lower limbs (12-14, 19-21).

[0023] Furthermore, in step S2, the Fusion module consists of two layers of convolution and activation functions: the first layer uses 1×1 convolution to reduce the dimension, reduce the number of channels (hyperparameter r1), and reduce the number of parameters; the second layer uses 1×3 convolution to extract features.

[0024] Furthermore, in step S3, the spatial stream is the spatial feature map firstly subjected to pooling operation, including average pooling and maximum pooling, respectively denoted as Avgpool(·) and Maxpool(·). After the spatial stream is subjected to the two pooling operations, the feature map is obtained by channel splicing, denoted as F Sp , expressed as:

[0025] F Sp =Avgpool(F S )||Maxpool(F S ) (1-4);

[0026] The concatenated dimensions are C×1×V, and after two convolutions, the number of channels is (channel reduction hyperparameter r2), we get s attention maps, where s is the number of partitions of the 27 joints, and we give attention to the partition. After the Sigmoid activation function, we get the attention score. The spatial attention map is denoted as F Sa , the spatial attention score is denoted as s att , the expression is:

[0027] F Sa =α4(α3(F Sp )) (1-5);

[0028] s att =Sigmoid(F Sa ) (1-6);

[0029] Among them, the two convolutions are denoted as α3(·) and α4(·), F Sa With s att The dimensions are C×1×V.

[0030] Furthermore, in step S3, the temporal stream is the temporal feature map that undergoes multi-scale convolution and then channels are concatenated to obtain the attention map, and then passes through Sigmoid to obtain the attention score, F Ta represents the temporal attention map, t att represents the temporal attention score, expressed as:

[0031]

[0032] t att =Sigmoid(F Ta ) (1-8);

[0033] in, It is a multi-scale convolution, and the convolution kernels are 1×1, 3×1, and 5×1; || means concatenating feature maps by channel, and the dimension after concatenation is C×T×1.

[0034] Beneficial effects of the present invention:

[0035] In order to solve the problem that spatial graph convolution and temporal convolution are not sufficient to obtain discriminative and significant features, the present invention introduces an attention module into the backbone network to obtain key features in different nodes and time frame data, realize the efficient extraction of spatial features and temporal features, and enhance the model's perception of local spatial structures and temporal dynamic changes.

[0036] In view of the problem that the attention space-time enhancement modules are independent of each other, the present invention proposes a spatial temporal hybrid attention module (Spatial Temporal Hybrid Attention Module, STHA), which always maintains space-time consistency, breaks the limitation of model representation caused by the separation of space-time evolution information, enhances the ability to extract effective information, and obtains significant space-time evolution features under the premise of space-time consistency. By designing a new attention mechanism, while paying attention to the key nodes and key time frames in the graph structure, the significant features and their evolution process in the space-time data are effectively captured. This method not only strengthens the interaction and information exchange between the time domain and space domain features, but also improves the model's understanding ability and prediction accuracy of complex space-time patterns. This module can not only capture the feature changes in the spatial and temporal dimensions at the same time, ensuring the integrity and coherence of the information, but also improves the performance of the model in processing complex scenes.

[0037] In addition, the attention module exhibits good robustness and can be applied "plug and play" in a variety of models based on graph convolutional networks. It is suitable for a variety of tasks such as behavior recognition and sign language gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of 25 joint points of the whole body in the present invention.

[0039] Figure 2 This is the process of generating the spatiotemporal attention map in the present invention.

[0040] Figure 3 This is a table showing the performance improvement of the STHA module on graph convolutional networks on the WLASL dataset. DETAILED DESCRIPTION

[0041] In order to clearly illustrate the technical features of this solution, this solution is described below through a specific implementation method.

[0042] A behavior recognition method based on a spatiotemporal hybrid attention module comprises the following steps:

[0043] S1. Divide the joint points into regions: Divide each joint point into regions, perform feature enhancement on each sub-region, and obtain the corresponding feature weights; Figure 1 As shown, the 25 key nodes of the whole body are divided into 4 groups, namely face and trunk (nodes 0, 1, 8, 15-18), upper limbs (2-7), left lower limbs (9-11, 22-24) and right lower limbs (12-14, 19-21).

[0044] The principles for dividing joint points in this application are as follows:

[0045] (a) Since adjacent nodes or edges tend to have similar attributes, nodes with adjacent or close locations are classified into one category, for example, the left upper limb and the right upper limb are classified into one category.

[0046] (b) Nodes with similar functions are grouped into one category. In action recognition, face nodes and torso nodes mostly play an auxiliary role, so these nodes are grouped into one category.

[0047] (c) According to experience, the number of nodes in each category is generally 5 to 7. Too many nodes may not accurately obtain the feature weight of each node, and too few nodes will cause a waste of memory resources.

[0048] S2. Design a spatiotemporal hybrid attention module (STHA): fuse the evolution of temporal information with the evolution of spatial information to obtain a spatiotemporal attention map.

[0049] The input feature map is denoted as F0, which compresses the time and space information to obtain spatial and temporal feature maps respectively, and then concatenates them by channel to obtain the feature Figure 1 , denoted as F1, expressed as:

[0050] F1=SqueezeT(F0)||SqueezeS(F0) (0-5);

[0051] Among them, SqueezeT(·) and SqueezeS(·) represent the compression of temporal information and spatial information respectively, || means concatenating the feature maps by channels, and the dimension after concatenation is C×(V+T).

[0052] The characteristics Figure 1 After passing through the Fusion module, the features are obtained Figure 2 , denoted as F2, feature Figure 2 It is expressed as:

[0053] F2=PReLU(α2(α1(F1))) (0-6);

[0054] Among them, α1(·) and α2(·) represent the first and second layers of convolution respectively, and PReLU(·) represents the PReLU activation function. The Fusion module consists of two layers of convolution and activation functions: the first layer uses 1×1 convolution for dimensionality reduction, reduces the number of channels (hyperparameter r1), and reduces the number of parameters; the second layer uses 1×3 convolution to extract features.

[0055] The number of channels of F2 is The characteristics Figure 2 Split by dimension, we get the spatial feature map and the temporal feature map, denoted as F S 、F T , F S The dimension is F T The dimension is Then the spatial feature map F S And the time feature map F T Features are further extracted along the spatial stream and the temporal stream respectively.

[0056] Specifically: The spatial stream is the spatial feature map that first undergoes pooling operations, including average pooling and maximum pooling, denoted as Avgpool(·) and Maxpool(·), respectively. After the spatial stream undergoes two pooling operations, it is concatenated by channel to obtain the feature map, denoted as F Sp , expressed as:

[0057] F Sp =Avgpool(F S )||Maxpool(F S ) (1-4);

[0058] The concatenated dimensions are C×1×V, and after two convolutions, the number of channels is (channel reduction hyperparameter r2), we get s attention maps, where s is the number of partitions of the 27 joints, and we give attention to the partition. After the Sigmoid activation function, we get the attention score. The spatial attention map is denoted as F Sa, the spatial attention score is denoted as s att , the expression is:

[0059] F Sa =α4(α3(F Sp ))(1-5);

[0060] s att =Sigmoid(F Sa )(1-6);

[0061] Among them, the two convolutions are denoted as α3(·) and α4(·), F Sa With s att The dimensions are C×1×V.

[0062] The time stream is the time feature map that undergoes multi-scale convolution and then channels are spliced ​​to obtain the attention map, and then the attention score is obtained by Sigmoid. Ta represents the temporal attention map, t att represents the temporal attention score, expressed as:

[0063]

[0064] t att =Sigmoid(F Ta )(1-8);

[0065] in, It is a multi-scale convolution, and the convolution kernels are 1×1, 3×1, and 5×1; || means concatenating feature maps by channel, and the dimension after concatenation is C×T×1.

[0066] The errors in feature extraction mainly include: (1) the limited size of the neighborhood causes the variance of the estimated value to increase. (2) Parameter errors cause the estimated mean to shift. Maximum pooling is used to extract information such as edge features and texture features, select features with higher classification recognition and provide nonlinearity. Average pooling retains the overall data characteristics and reflects the integrity of information during transmission. Generally speaking, average pooling can reduce the first error, and maximum pooling can reduce the second error. Therefore, the advantages of average pooling and maximum pooling are fully utilized in the STHA module.

[0067] S3, spatiotemporal stream fusion: The attention scores of the spatiotemporal streams are combined to obtain the spatiotemporal attention score st att , which is expressed as follows:

[0068] st att =s att ×t att (1-3);

[0069] Among them, st att The dimensions are C×T×V.

[0070] S4, the obtained spatiotemporal attention score st att By assigning the input feature map, the temporal and spatial domain features are enhanced.

[0071] In this application, the STHA module improves the performance of graph convolutional networks:

[0072] The spatiotemporal hybrid attention module is used in three graph convolution-based backbone networks: Decouple-GCN, CTR-GCN and SL-GCN. In the table, "1s" represents a single-stream network, and "4s" represents a four-stream network fusion, including joint stream, skeleton stream, joint motion stream, and skeleton motion stream. "+STHA" represents the experimental effect after adding the STHA module. The recognition accuracy uses two indicators, Top-1 and Top-5. The former indicates the accuracy of the category with the first score in the classification and the actual result, and the latter indicates the accuracy of the category with the top five scores in the classification and the actual result. The "accuracy" mentioned in the experimental results usually refers to the Top-1 accuracy. According to the attached Figure 3 It can be seen that on the WLASL2000 dataset, STHA improves the performance of single-stream networks by 0.87% to 2%, and improves the performance of four-stream networks by 0.31% to 0.94%, with significant effects.

[0073] Graph convolutional networks have significant advantages in the field of behavior recognition. First, GCN can effectively process structured data, such as human joint data, facial key point data, etc., by modeling the human skeleton as a graph structure, with nodes representing joints and edges representing connections between joints, so as to better capture the spatiotemporal relationship of human actions. Secondly, GCN can simultaneously extract the local features of nodes and the global structure of the graph structure, realize the combination of local features and global features, and improve the understanding and modeling capabilities of complex human actions. In addition, GCN has strong robustness in dealing with noise and missing data, and can maintain good recognition performance even when sensor data is incomplete. These advantages enable graph convolutional networks to show higher accuracy and generalization capabilities in behavior recognition tasks. This application designs spatiotemporal hybrid attention STHA to obtain discriminative and significant features under the premise of spatiotemporal consistency, and applies it to GCN to divide the key points of the human body into regions and obtain targeted feature enhancements in each region.

[0074] The spatiotemporal hybrid attention module of this application can be applied to multiple fields due to its powerful spatiotemporal information integration capabilities and efficient information extraction capabilities, especially in video analysis, autonomous driving, and medical image analysis. It shows great potential. For example, in video surveillance, this module can help the system more accurately identify and track moving targets and improve the accuracy of abnormal behavior detection; in the field of autonomous driving, it can enhance the vehicle's perception of the surrounding environment and improve driving safety; in medical image analysis, by integrating time and space information, it helps to diagnose disease progression more accurately. In addition, the thinking expansion of this module can also be extended to multiple cutting-edge fields such as sports analysis, virtual reality, and human-computer interaction, providing strong support for technological innovation in these fields.

[0075] Of course, the above description is not limited to the above examples. Technical features not described in the present invention can be achieved by or by using existing technologies, which will not be described here. The above embodiments and drawings are only used to illustrate the technical scheme of the present invention and are not limitations of the present invention. The present invention is described in detail with reference to the preferred implementation methods. Ordinary technicians in this field should understand that changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention do not depart from the purpose of the present invention, and should also fall within the scope of protection of the claims of the present invention.

Claims

1. A behavior recognition method based on a spatiotemporal hybrid attention module, characterized in that: The following steps are involved: S1. Divide the joint points into regions: Divide each joint point into regions, perform feature enhancement on the sub-regions respectively, and obtain the corresponding feature weights; S2. Design a spatiotemporal hybrid attention module: combine the temporal information evolution with the spatial information evolution to obtain a spatiotemporal attention map; The input feature map is denoted as F0, and its time and space information is compressed to obtain spatial and time domain feature maps respectively. Feature map 1 is obtained by channel splicing, denoted as F1, which is expressed as: F1=SqueezeT(F0)||SqueezeS(F0)(0-1); Among them, SqueezeT(·) and SqueezeS(·) represent the compression of temporal information and spatial information respectively, and || represents the concatenation of feature maps by channel, and the dimension after concatenation is C×(V+T); After the feature map 1 passes through the Fusion module, the feature map 2 is obtained, denoted as F2. The feature map 2 is expressed as: F2=PReLU(α2(α1(F1)))(0-2); Among them, α1(·) and α2(·) represent the first and second convolution layers respectively, PReLU(·) represents the PReLU activation function; the number of channels of F2 is Split feature map 2 by dimension to obtain spatial feature map and temporal feature map, denoted as F S 、F T , F S The dimension is F T The dimension is Then the spatial feature map F S And the time feature map F T Further extract features along the spatial stream and temporal stream respectively; S3, spatiotemporal stream fusion: The attention scores of the spatiotemporal streams are combined to obtain the spatiotemporal attention score st att , which is expressed as follows: st att =s att ×t att (1-3); Among them, st att The dimension is C×T×V; S4, the obtained spatiotemporal attention score st att By assigning the input feature map, the temporal and spatial domain features are enhanced.

2. The behavior recognition method based on the spatiotemporal hybrid attention module according to claim 1 is characterized in that: In step S1, the 25 key nodes of the whole body are divided into 4 groups, namely face and trunk (nodes 0, 1, 8, 15-18), upper limbs (2-7), left lower limbs (9-11, 22-24) and right lower limbs (12-14, 19-21).

3. The behavior recognition method based on the spatiotemporal hybrid attention module according to claim 1 is characterized in that: In step S2, the Fusion module consists of two layers of convolution and activation functions: the first layer uses 1×1 convolution to reduce the dimension, reduce the number of channels (hyperparameter r1), and reduce the number of parameters; the second layer uses 1×3 convolution to extract features.

4. The behavior recognition method based on the spatiotemporal hybrid attention module according to claim 1 is characterized in that: In step S3, the spatial stream is the spatial feature map that first undergoes pooling operations, including average pooling and maximum pooling, denoted as Avgpool(·) and Maxpool(·), respectively. After the spatial stream undergoes two pooling operations, it is spliced ​​by channel to obtain a feature map, denoted as F Sp , expressed as: F Sp =Avgpool(F S )||Maxpool(F S )(1-4); The concatenated dimensions are C×1×V, and after two convolutions, the number of channels is (channel reduction hyperparameter r2), we get s attention maps, where s is the number of partitions of the 27 joints, and we give attention to the partition. After the Sigmoid activation function, we get the attention score. The spatial attention map is denoted as F Sa , the spatial attention score is denoted as s att , the expression is: F Sa =α4(α3(F Sp ))(1-5); s att =Sigmoid(F Sa )(1-6); Among them, the two convolutions are denoted as α3(·) and α4(·), F Sa With s att The dimensions are C×1×V.

5. The behavior recognition method based on the spatiotemporal hybrid attention module according to claim 1 is characterized in that: In step S3, the temporal stream is the temporal feature map that undergoes multi-scale convolution and then channels are concatenated to obtain the attention map, and then passes through Sigmoid to obtain the attention score, F Ta represents the temporal attention map, t att represents the temporal attention score, expressed as: t att =Sigmoid(F Ta )(1-8); in, It is a multi-scale convolution, and the convolution kernels are 1×1, 3×1, and 5×1; || means concatenating feature maps by channel, and the dimension after concatenation is C×T×1.

Images