Action recognition method based on double-flow space-time attention mechanism

Abstract

The invention discloses an action recognition method T-STAM based on a double-flow space-time attention mechanism. The method comprises the following steps: S1, processing a video to obtain an opticalflow graph of an RGB frame; s2, fusing the channel attention network SE-Net into a double-flow basic network BN-Inception to obtain an SE-BN-Inception; s3, inputting the selected RGB frame and optical flow field information into SE-BN-Inception, and modeling the dependency relationship of different channels in the features to obtain a feature vector X of the video; s4, inputting the feature X into a CNN-based time attention network to calculate a time attention score corresponding to each frame; s5, inputting the feature X into a multi-space attention network, and extracting a plurality of motion space salient regions of the frame; s6, fusing the spatial and temporal features to further enhance the feature expression of the video, and fusing the two streams according to different weightsto obtain an action recognition result.

Description

technical field [0001] The invention relates to the fields of computer vision, video classification and the like, and in particular provides an action recognition method T-STAM based on a dual-stream spatio-temporal attention mechanism. Background technique [0002] In recent years, with the rise of deep learning, methods based on convolutional neural networks have been widely used in the field of video action recognition research. Among them, the dual-stream method inputs RGB into the CNN to obtain appearance information, and inputs multi-frame optical flow fields into the CNN to obtain motion information, which can effectively combine the spatiotemporal information in the video and is relatively better in performance. However, the two-stream method ignores the connection of different channel information when extracting video features. In addition, it processes the sampled frames in the video equally, does not distinguish the information of different positions of the frame...

AI Technical Summary

Problems solved by technology

[0004] (1) When extracting motion salient spatial region information, only one spatial attention network is used to focus on multiple salient regions of the frame, resulting in inaccurate extraction of some regions;

[0005] (2) The temporal attention network designed using LSTM has a complex structure and must process video frames in chronological order, and the recognition efficiency is low

Images