A hypergraph learning action recognition system based on attention mechanism

Abstract

The present invention discloses a hypergraph learning action recognition system based on an attention mechanism, which realizes the function of recognizing human actions, and includes the following steps: first, a temporal attention module is constructed through a learnable threshold, and video action frames from different perspectives are input into the temporal attention module, thereby extracting the key frames of the action, reducing the computational cost for subsequent network learning; then, the key frames of the action are input into a hypergraph convolution module based on a dynamic spatiotemporal attention mechanism, the salient areas of the action are obtained, and the spatiotemporal high-order semantic features are learned through a multi-scale spatiotemporal residual module; and then data partitioning, model training, and model saving operations are performed. In the process of calling the model detection, the recognition part of the network outputs the action recognition result by loading the model. The present invention solves the problem of a large number of network calculation parameters in the spatiotemporal characteristics of behavioral actions in video sequences, obtains the completeness of human joint information, and improves the accuracy of action recognition.

Description

Technical Field

[0001] The present invention relates to the technical fields of deep learning, machine vision, and action recognition, and specifically to a hypergraph learning action recognition system based on an attention mechanism. Background Art

[0002] Action recognition is a hot topic in the field of machine vision research in the intelligent era. It enables natural interactive cognition between humans and machines and has been applied in many intelligent fields. Among them, the accurate recognition of behaviors such as pedestrians and traffic police is an important practical problem to be solved for the implementation of unmanned driving applications. At present, some methods use attention mechanisms to help neural networks focus on important areas in the video and extract key features, thereby avoiding noise interference and information redundancy and effectively improving model performance. Some researchers use spatiotemporal attention mechanisms, which not only include static information in the spatial domain, but also motion information in the temporal domain, effectively aggregating global and local relationships in the temporal and spatial dimensions. Based on these ideas, this system designs a spatiotemporal hypergraph attention module, which can achieve efficient action recognition tasks by obtaining high-order semantic features between joints in different frames. Summary of the Invention

[0003] To address the shortcomings of the above-mentioned existing technologies, the present invention proposes a hypergraph learning action recognition system based on an attention mechanism, which introduces a learnable threshold to select key frames in the video, thereby removing information of irrelevant redundant frames; the key frames of the action are input into a hypergraph convolution module based on a dynamic spatiotemporal attention mechanism to obtain the salient areas of the action and distinguish similar action pairs within the class; finally, behavioral action recognition is effectively achieved through a multi-scale spatiotemporal residual module.

[0004] The technical solution adopted by the present invention is a hypergraph learning action recognition system based on the attention mechanism, which includes a temporal attention module based on a learnable threshold, a hypergraph convolution module based on a dynamic spatiotemporal attention mechanism, a multi-scale spatiotemporal residual module and an action recognition module;

[0005] The temporal attention module based on the learnable threshold includes various operations such as calculating motion information, setting a learnable threshold, calculating key frame motion features, and enhancing the key frame feature map with a 1×1 convolution kernel. The dynamic temporal feature map is calculated to form temporal attention to obtain the final key frame feature set;

[0006] The hypergraph convolution module based on the dynamic spatiotemporal attention mechanism includes spatiotemporal hypergraph construction, self-attention mechanism and spatiotemporal hypergraph convolution. It constructs a spatiotemporal hypergraph from the key frame feature set using the human body joints in the same part of different frames. Then, it updates the correlation matrix through the self-attention mechanism to obtain the correlation between the joints. The spatiotemporal hypergraph convolution is introduced to extract features from the salient areas of the action.

[0007] The multi-scale spatiotemporal residual module includes a multi-scale hypergraph convolutional layer and five convolutional modules based on multi-scale feature representation for extracting and learning spatiotemporal features, ultimately achieving action recognition;

[0008] The action recognition module verifies the trained model, inputs action video data, and outputs action recognition results through the saved model.

[0009] Furthermore, a temporal attention module based on a learnable threshold is used to extract keyframe information from multi-view videos. First, the temporal difference between the feature maps of the current frame and adjacent frames is calculated to extract motion information. This is then compared with a learnable threshold to obtain a set of keyframe feature maps. The keyframe feature maps are then enhanced using a 1×1 convolution kernel. Finally, these features are concatenated with the features of the current frame to aggregate and obtain the final set of keyframe features.

[0010] Furthermore, the hypergraph convolution module based on the dynamic spatiotemporal attention mechanism divides the human joints of different frames into three parts: torso, hands, and legs from the key frame feature set to construct a spatiotemporal hypergraph, and extracts the high-order features of the multi-view spatiotemporal hypergraph of the human joints. The high-order features are input into the self-attention mechanism module for global modeling of the human joints, and finally the salient areas of the action are obtained through spatiotemporal hypergraph convolution.

[0011] Furthermore, the multi-scale spatiotemporal residual module first uses a multi-scale hypergraph convolutional layer to capture the changes in temporal information between different frames using the salient area features of the action. The features are then input into a convolutional module based on multi-scale feature representation to model long-term and short-term semantic information. Residual connections are then placed between two adjacent modules to obtain the dependency between local and non-local joints, ultimately achieving action recognition.

[0012] Furthermore, the model was trained using the training set of NTU RGB+D and a self-collected traffic police gesture dataset. To objectively reflect the action recognition results, the cross-entropy loss function was used to measure the training effect. After multiple iterations, when the loss value stabilized, the model was saved as the trained model. Furthermore, the action recognition module was validated using the test set of NTU RGB+D and a self-collected traffic police gesture dataset.

[0013] This method, a hypergraph learning action recognition system based on an attention mechanism, can perform pedestrian and action recognition on PC-based interactive devices. This system not only improves action recognition in complex scenarios, but also increases the speed of identifying pedestrians and traffic police around vehicles in unmanned driving environments, and provides accurate action recognition algorithms in cross-domains. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is an overall flow chart of the hypergraph learning action recognition system based on the attention mechanism.

[0015] Figure 2 It is an overall network structure diagram of a hypergraph learning action recognition system based on the attention mechanism. DETAILED DESCRIPTION

[0016] The method is described in detail below with reference to the accompanying drawings and embodiments:

[0017] The system of the present invention constructs a temporal attention module through a learnable threshold, inputs action frames of videos from different perspectives into the temporal attention module, thereby extracting key frames of the action and reducing the computational cost for subsequent network learning; then inputs the key frames of the action into a hypergraph convolution module based on a dynamic spatiotemporal attention mechanism to obtain the salient areas of the action, and learns spatiotemporal high-order semantic features through a multi-scale spatiotemporal residual module; then performs operations such as data partitioning, model training, and model saving. In the process of calling the model detection, the recognition part of the network outputs the action recognition result by loading the model. The present invention solves the problem of a large number of network calculation parameters for the spatiotemporal characteristics of behavioral actions in video sequences, obtains the completeness of human joint information, and improves the accuracy of action recognition.

[0018] Attachment Figure 1 It can be seen that according to a preferred embodiment of a hypergraph learning action recognition system based on an attention mechanism, it includes:

[0019] (1) Using a temporal attention module based on a learnable threshold, this module uses the motion information of the video sequence and introduces a learnable threshold to select key frames in the video, thereby removing the information of irrelevant redundant frames;

[0020] (2) Construct a hypergraph convolution module based on the dynamic spatiotemporal attention mechanism. This module captures the high-order semantic relationship between joints in different frames of the action and inputs it into the dynamic spatiotemporal attention module to learn the association between the features of body parts and joints to obtain the salient regions of the action;

[0021] (3) A multi-scale spatiotemporal residual module combined with a residual connection structure is used to associate the long and short distances of video data, thereby obtaining the dependency between local and non-local joints.

[0022] (3) The NTU RGB+D and self-collected traffic police gesture datasets are divided into training and test sets, and the model is trained and saved;

[0023] (4) Input the action video data to be recognized and load the model for action recognition;

[0024] (5) Output the recognition results.

[0025] Attachment Figure 2It can be seen that according to a preferred embodiment of a hypergraph learning action recognition system based on an attention mechanism, it includes:

[0026] (1) The number of input feature channels of the temporal attention module based on learnable threshold and the hypergraph convolution module based on dynamic spatiotemporal attention mechanism is 64, and the number of output feature channels is 64.

[0027] (2) The number of input feature channels of the multi-scale spatiotemporal residual module is 64, and the number of output feature channels is 128. The final classification is calculated by the Softmax function.

[0028] In the above solution, preferably, the temporal attention module based on a learnable threshold forms temporal attention by obtaining a dynamic temporal feature map. It is divided into the following four steps:

[0029] (1) The joint point information of each frame in the multi-view data As input, where p = 1, 2, ..., P represents the viewing angle, t = 1, 2, ..., T represents the time, I represents the number of human joints, and C represents the coordinates (x, y); Indicates that all numbers in the set are real numbers.

[0030] (2) Calculate the motion information m of the action p (t), which can be expressed as:

[0031] m p (t) = x p (t)-x p (t-1)=x[:,t]-x[:,t-1]

[0032] Among them, the joint point coordinate set C in Indicates coordinate information, T in Indicates a frame.

[0033] (3) Setting a learnable threshold kernel The characteristic mean where m p (t) represents the motion information of the action, represents the feature mean, t=1, 2,…, T represents time.

[0034] (4) Change the movement m p (t) is compared with the difference between TF to determine whether the t-th frame is a key frame. If the comparison result is positive, the t-th frame is marked as a key frame; otherwise, it is not. The feature set KF(t) is finally selected as the key frame = {m p (t)|m p (t)-TF>0}.

[0035] In the above solution, it is preferred that the hypergraph convolution module based on the dynamic spatiotemporal attention mechanism has a The query vector Key vector Sum value vector Node features are obtained by linear transformation The correlation between each pair of nodes is calculated using the SoftMax function use renew, Then add these vectors weightedly to get the final result It can be expressed as:

[0036]

[0037] in,

[0038] Use the multi-head attention mechanism to extract H features to obtain the final combination result

[0039] Then, the related features are extracted through spatiotemporal hypergraph convolution, which can be expressed as follows:

[0040]

[0041] Among them, f out (·) represents the output, K is the predefined maximum distance between nodes, W k and M k Indicates that convolution operations and weight adjustment can learn parameters, represents the updated Laplacian matrix, where and are the diagonal matrices representing the hyperedge degree and vertex degree in the pth space-time hypergraph respectively.

[0042] In the above solution, it is preferred that the multi-scale spatiotemporal residual module is used to extract and learn spatiotemporal features by sequentially stacking a multi-scale hypergraph convolution layer and five convolution module layers based on multi-scale feature representation. It is mainly divided into the following two steps:

[0043] (1) The multi-scale hypergraph convolutional layer adds a simple learnable hypergraph residual mask Dynamically strengthen, weaken, add, or remove edges.

[0044]

[0045] in, Initialized with random values ​​around zero, allowing each multi-scale context to choose the most appropriate mask.

[0046] (2) The above transformations are connected. The convolution module based on multi-scale feature representation is composed of five branches, each of which contains a 1×1 convolution to reduce the channel dimension; the second, third and fourth branches contain three depth-separable convolutions with different scaling factors, respectively, which extract different temporal feature information while reducing the number of parameters; the fifth branch is used to concentrate the features within a single frame.

[0047] Preferably, in the above solution, the hypergraph learning action recognition system based on the attention mechanism uses a cross entropy loss function: where y i is a one-hot vector representing the groundtruth of the action class. After multiple iterations, when the loss value stabilizes, it is saved as the training model.

[0048] In the above solution, preferably, the attention-based hypergraph learning action recognition system is trained with a batch size of 64, an SGD optimization algorithm (momentum of 0.9), a weight decay of 0.0005, 70 epochs, and an initial learning rate of 0.1. Training is performed on the NTU RGB+D and self-collected traffic police gesture datasets.

Claims

1. A hypergraph learning action recognition system based on attention mechanism, characterized by: It includes a temporal attention module based on a learnable threshold, a hypergraph convolution module based on a dynamic spatiotemporal attention mechanism, a multi-scale spatiotemporal residual module, and an action recognition module; The temporal attention module based on the learnable threshold includes various operations such as calculating motion information, setting a learnable threshold, calculating key frame motion features, and enhancing the key frame feature map with a 1×1 convolution kernel. The dynamic temporal feature map is calculated to form temporal attention to obtain the final key frame feature set; The hypergraph convolution module based on the dynamic spatiotemporal attention mechanism includes spatiotemporal hypergraph construction, self-attention mechanism and spatiotemporal hypergraph convolution. It constructs a spatiotemporal hypergraph from the key frame feature set using the human body joints in the same part of different frames. Then, it updates the correlation matrix through the self-attention mechanism to obtain the correlation between the joints. The spatiotemporal hypergraph convolution is introduced to extract features from the salient areas of the action. The multi-scale spatiotemporal residual module includes a multi-scale hypergraph convolutional layer and five convolutional modules based on multi-scale feature representation for extracting and learning spatiotemporal features, ultimately achieving action recognition; The action recognition module verifies the trained model, inputs action video data, and outputs action recognition results through the saved model.

2. The hypergraph learning action recognition system based on attention mechanism according to claim 1, characterized in that: A temporal attention module based on a learnable threshold is proposed to extract key frame information from multi-view videos. The time difference between the feature maps of the current frame and adjacent frames is first calculated to extract motion information, and then a set of key frame feature maps is obtained by comparing them with the learnable threshold. The key frame feature maps are then enhanced by a 1×1 convolution kernel. Finally, the features of the current frame are concatenated and aggregated to obtain the final key frame feature set.

3. The hypergraph learning action recognition system based on attention mechanism according to claim 1, characterized in that: The hypergraph convolution module based on the dynamic spatiotemporal attention mechanism constructs a spatiotemporal hypergraph by dividing the human joints of different frames into three parts: torso, hands, and legs from the key frame feature set, and extracts the high-order features of the multi-view spatiotemporal hypergraph of the human joints. The high-order features are input into the self-attention mechanism module for global modeling of the human joints, and finally the salient areas of the action are obtained through spatiotemporal hypergraph convolution.

4. The hypergraph learning action recognition system based on attention mechanism according to claim 1, characterized in that: The multi-scale spatiotemporal residual module first uses a multi-scale hypergraph convolution layer to capture the temporal information changes between different frames using the salient area features of the action. The features are then input into a convolution module based on multi-scale feature representation to model long-term and short-term semantic information. Residual connections are then placed between two adjacent modules to obtain the dependency between local and non-local joints, ultimately achieving action recognition.

5. The hypergraph learning action recognition system based on attention mechanism according to claim 1, characterized in that: The training set of NTU RGB+D and the self-collected traffic police gesture dataset was used for training. In order to objectively reflect the action recognition results, the cross-entropy loss function was used to measure the training effect. After multiple iterations, when the loss value tends to be stable, it is saved as the training model.

6. The hypergraph learning action recognition system based on attention mechanism according to claim 1, characterized in that: The action recognition module is verified using the test set of NTU RGB+D and the self-collected traffic police gesture dataset.

Images