A fine-grained video recognition method based on adaptive multi-granularity spatiotemporal modeling

Abstract

The present invention discloses a fine-grained video recognition method based on adaptive multi-granularity spatiotemporal modeling. The method is based on a deep neural network, a fine-grained CPTM module and a coarse-grained CSTP module. The method comprises the following steps: the fine-grained CPTM module is incorporated into the deep neural network to extract fine-grained temporal clues; the fine-grained CPTM module is constructed based on progressive temporal convolution and competitive fusion mechanism; the performance of video action recognition is improved, and the method has the ability to capture spatiotemporal clues of complex scale changes in videos.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning and deep learning, and specifically relates to a video action recognition method based on multi-granularity spatiotemporal cue adaptive modeling. Background Art

[0002] Video action recognition is one of the most representative and important tasks in the field of video understanding. It is applied in many fields in the real world, including behavior analysis, human-computer interaction, game entertainment, and video retrieval. With the explosive progress of deep learning systems in the past decade and the great success in the field of image detection, the field of video understanding has made great progress [1,2,3,4,5,6,7]. In recent years, deep video recognition architectures have emerged in an endless stream. Among them, the three mainstream trends are two-stream networks that open up new paths to capture temporal information [4], 3D deep convolutional networks that extract spatiotemporal feature information [7,2,5], and lightweight 2D deep convolutional networks based on temporal modeling that focus on computational efficiency [8,9,3,10,11]. In addition, people have recently conducted hot research on the application of Transformers to video recognition [12,13]. Although it has achieved better performance, it also consumes a lot of computing and data resources.

[0003] However, efficient video recognition algorithms still face several major challenges. First, existing deep video recognition models capture temporal information, which brings considerable computational costs for training and inference. Second, in terms of temporal modeling, the actions in daily life are complex and varied, and the videos capturing the actions themselves have strong intra-class and inter-class variations. In daily video instances, different action categories are often distinguished by different visual rhythms, such as walking, race walking, running, etc. They have strong apparent similarities, but are in different visual rhythms. In other words, complex actions have complex spatiotemporal scale changes. In particular, they are mainly reflected in the different visual rhythms of the actions, and the spatial scales of the moving objects also vary to varying degrees. This phenomenon also forces action instances to focus more on intra-class and inter-class differences for accurate modeling. In summary, the existence of these special properties of video action instances poses a great challenge to how to effectively model spatiotemporal in video representation learning.

[0004] In order to solve the problem of complex spatiotemporal scale changes in videos, most existing studies are based on input-level [1, 14, 15] or feature-level pyramid mechanisms to extract video clues of action instances

[16] .

[0005] Specifically, the input-level pyramid mechanism obtains richer input data by using different sampling strategies for videos. Among them, the multi-rate video random jump method (RTS)

[15] obtains multiple segments by sampling multiple frames of the sample video at random intervals multiple times, instead of only one segment in the traditional TSN [6]. Dynamic temporal pyramid sampling (DTPN)

[14] extracts features from the sample video using five different temporal sampling rates and uses them as input data. Both RTS and DTPN can be regarded as data enhancement in the temporal dimension, but both of them feed multi-rate inputs into a shared backbone network (such as a two-stream network [4], C3D [2]) to achieve the purpose of extracting richer temporal pyramid features. Therefore, the disadvantages of this type of method are relatively intuitive. Even if inputs of different resolutions are shared by the model, the computational cost will increase rapidly with the increase of the sampling rate. Correspondingly, multi-stream networks, such as multi-rate slow and fast channel networks [1], are designed to achieve a better balance between accuracy and efficiency. Slow and fast channel networks [1] propose a two-branch architecture that uses heavy and lightweight models to process slow and fast time-sampled inputs respectively, but still suffer from expensive computational costs

[16] . In addition, the model is difficult to scale when faced with high sampling rates.

[0006] Recently, existing methods have proposed the Temporal Pyramid Network (TPN)

[16] to fuse features of different spatiotemporal resolutions at the feature level, expanding the input-level pyramid to the feature level. However, it is worth noting that TPN fuses spatiotemporal features from different stages of a deep convolutional network, so these features are coarse-grained and have limited ability to capture subtle changes in visual rhythm. In order to obtain multi-granular spatiotemporal video learning representation capabilities, existing multi-granularity methods

[17] construct a multi-stream model (MSD) by sampling a single frame, multiple consecutive frames, a short clip, and the entire video as inputs of multiple granularities. Obviously, the multi-stream model also bears an expensive computational burden, and it also has the problem of extracting and fusing different granularities in a fixed way, making it difficult to fully interpret the complex scale changes of spatiotemporal cues in the video. Summary of the invention

[0007] In order to overcome the shortcomings of existing methods and technologies, the present invention aims to solve how to efficiently and light-weight spatiotemporal modeling when facing complex video actions, and achieve excellent performance in video action recognition tasks. To this end, the technical solution of the present invention proposes an adaptive multi-granularity spatiotemporal clue modeling network (AMS-Net) based on a single input. By designing a single-scale input single-stream network, multi-granularity spatiotemporal clues are adaptively modeled, aiming to efficiently and effectively capture complex spatiotemporal dynamic processes. Aiming at video recognition scenarios, this method proposes a fine-grained video recognition method based on adaptive multi-granularity spatiotemporal modeling based on the existing deep convolutional network model to solve the problem of capturing complex scale changes in video actions.

[0008] The present invention adopts the following technical solution:

[0009] A fine-grained video recognition method based on adaptive multi-granularity spatiotemporal modeling, the method is based on a deep neural network, a fine-grained CPTM module and a coarse-grained CSTP module; and comprises the following steps:

[0010] The fine-grained CPTM module is incorporated into a deep neural network to extract fine-grained temporal cues;

[0011] The fine-grained CPTM module is constructed based on progressive temporal convolution and competitive fusion mechanism; including:

[0012] A set of multi-scale features is calculated to calculate the temporal features of scale k. It is expressed as:

[0013] y k =G(W k )⊙y k-1 ,k=1,…,K,

[0014] Where y0 = x and G(W k ) indicates that the weight is W k Temporal convolution;

[0015] The fine-grained CPTM module is represented by multi-scale features of a competitive fusion mechanism as follows:

[0016]

[0017] The vector h of dimension C is subjected to global average pooling (GAP) of spatiotemporal features to capture the multi-time scale {y1,…,y K}'s global context, that is is through two weights, It is calculated by the fully connected layer; ⊙ represents the dot product operation along the channel dimension; τ is a balance parameter that determines the competition or collaboration mode between multi-scale features;

[0018] The coarse-grained CSTP module is incorporated into a deep neural network to extract and fuse coarse-grained spatiotemporal features; including:

[0019] The coarse-grained CSTP module adopts a two-stage adaptive fusion mechanism expressed as:

[0020]

[0021] Among them, φ up ,φ dw and φ zare three 2D convolutional layers with a kernel size of 1×1, and the symbol [·;·] represents the connection operation between features; the weight and By input Calculated by function g1; output Z up and Z dw As input to function g2, calculate the weight ω up and ω dw .

[0022] Further, the fine-grained CPTM module and the coarse-grained CSTP module are incorporated into the 2D deep neural network process:

[0023] The CPTM module is selectively inserted between the convolution stages of the 2D deep neural network to extract fine-grained temporal cues of the features between the current consecutive convolution stages:

[0024] set up is the i-th residual block in the l-th stage (res l,i ), where W represents width, H represents height, T represents time dimension and C represents number of channels;

[0025] CPTM module (f CPTM ) Input feature χ in this layer l,Nl The extracted fine-grained temporal cues can be expressed as:

[0026] y(l)=f CPTM (x l,Nl )

[0027] Among them, N l for res l The last residual block of the next stage is res l+1,1 Input;

[0028] The CSTP module uses the following formula to adaptively fuse and output the spatiotemporal resolution features of different convolution stages:

[0029] Z=f CSTP (y(L-1),y(L))

[0030] Where: L represents the total number of stages in the deep convolutional network, and Z is used for the final prediction after fusion of features.

[0031] Further, the fine-grained CPTM module and the coarse-grained CSTP module are incorporated into the 3D deep neural network process:

[0032] In the 3D deep neural network, the CPTM module and the 2D spatial convolution are combined in a residual manner to construct a 3D CPTM layer structure, and through 3D CPTMThe original 3D convolutional layer in the residual network is replaced by the CNN layer to extract fine-grained temporal clues;

[0033] The CSTP module uses the following formula to adaptively fuse and output the spatiotemporal resolution features of different convolution stages:

[0034]

[0035] Where: Z also represents the adaptive fusion of coarse-grained feature information across different spatiotemporal resolutions.

[0036] Beneficial Effects

[0037] The present invention proposes a single-stream network structure (i.e., AMS network) that adaptively extracts multi-granularity spatiotemporal features based on single-scale input, aiming to improve the performance of video action recognition and have the ability to capture spatiotemporal clues of complex scale changes in videos. To this end, two core structures are proposed: a fine-grained time modeling module and a coarse-grained spatiotemporal pyramid module (i.e., a CPTM module and a CSTP module), which extract fine-grained time clues in parallel on a single-resolution feature map and adaptively fuse coarse-grained spatiotemporal features across spatiotemporal resolutions. Given that the CPTM module and the CSTP module are independent of a specific model, the AMS network of the present invention can be flexibly instantiated according to the existing deep 2D / 3D convolutional network architecture. On time-dominant, representation-dominant, and fine-grained video datasets, a large number of experiments clearly demonstrate the effectiveness of the method proposed by the present invention, and show that compared with a deep convolutional network that requires more frames, the present invention is more conducive to fine-grained time clue modeling, while being able to more efficiently capture complex spatiotemporal clues, and more effectively and robustly model time. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a 2D / 3D AMS network framework diagram involved in the present invention;

[0039] Figure 2 CPTM fine-grained temporal modeling module framework diagram;

[0040] Figure 3 CSTP coarse-grained space-time pyramid framework diagram. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and examples. The following embodiments are only descriptive and not restrictive, and the protection scope of the present invention cannot be limited thereto.

[0042] like Figure 1As shown, in the 2D deep convolution architecture based on single-resolution input, the fine-grained time modeling module CPTM proposed in the present invention is plug-and-play inserted between the various stages of the deep convolution network (such as the various stages of the residual network) to extract the fine-grained time clues of the current spatiotemporal resolution; in the 3D deep convolution network, the present invention takes the 3D residual network as an example, and the CPTM module replaces the first 3D convolution layer with a kernel size of 1×1×1 in some 3D residual layers, aiming to achieve a balance between computational complexity and performance. As for the spatiotemporal pyramid module CSTP, in the instantiation of 2D and 3D deep convolution networks, the output features of the stage are obtained at different stages, and coarse-grained adaptive fusion of features with different spatiotemporal resolutions is performed.

[0043] The CPTM module and CSTP module proposed in the present invention respectively adaptively extract fine-grained temporal clues and fuse coarse-grained spatiotemporal features. In particular, the proposed CPTM module is composed of several non-shared temporal convolutions with the same kernel size, and extracts multi-scale features for the same temporal resolution, so as to achieve the purpose of capturing subtle temporal changes with multi-scale features. The captured subtle temporal changes, i.e., the fine-grained temporal clues mentioned above, are finally competitively fused in the form of multi-scale features. For this reason, the CPTM module of the present invention has the ability to adaptively capture subtle changes in the visual rhythm of the video. The collaborative spatiotemporal pyramid (i.e., the CSTP module) is inspired by TPN

[16] , and adaptively fuses spatiotemporal features across different spatiotemporal resolutions in a collaborative manner to better capture coarse-grained spatiotemporal clues.

[0044] The present invention will introduce in detail how to construct the 2D AMS network and 3D AMS network of the present invention.

[0045] 2. Specific plan

[0046] 2.1 Overall Network Architecture

[0047] In order to reasonably construct and widely apply the AMS network, the present invention integrates the proposed CPTM module and CSTP module into the existing 2D / 3D deep convolutional network model, thereby instantiating two architectures, namely 2D-AMS network and 3D-AMS network.

[0048] In order to construct the 2D-AMS network, based on the architecture proposed by previous scholars [8,9,3], the CPTM module is selectively inserted between different convolution stages to extract the fine-grained temporal clues of the features between the current consecutive convolution stages. Specifically, is the i-th residual block in the l-th stage (res l,i ), where W represents width, H represents height, T represents time dimension and C represents number of channels. Then the CPTM module (f CPTM ) Input feature χ in this layerl,Nl The extracted fine-grained temporal cues can be expressed as:

[0049] y(l)=f CPTM (x l,Nl ) (1)

[0050] Among them, N l for res l The last residual block of the next stage is res l+1,1 For the CSTP module, the spatiotemporal features of different convolution stages, that is, the features of different spatiotemporal resolutions, are adaptively fused. In the present invention, in order to balance efficiency and accuracy, the present invention only fuses the outputs of the last two convolution stages. In the following formula (2), L represents the total number of stages in the deep convolution network, and Z is used for the final prediction after the fusion of features:

[0051] Z=f CSTP (y(L-1),y(L)) (2)

[0052] For the construction of 3D AMS network, the present invention combines CPTM module and 2D spatial convolution in a residual manner to construct a 3D CPTM Layer structure, such as Figure 1 As shown in (b), through 3D CPTM The original 3D convolution layer in the residual network is replaced by the CSTP layer to extract fine-grained temporal clues. Consistent with the 2D AMS network, the present invention still uses the CSTP module to fuse the output features of the last two convolution stages, i.e. Where Z also represents the adaptive fusion of coarse-grained feature information across different spatiotemporal resolutions. The AMS network proposed in the present invention models adaptive multi-granularity spatiotemporal cues in the above two ways.

[0053] 2.2 Fine-grained CPTM module

[0054] The CPTM module proposed in the present invention is used to adaptively extract multi-scale, fine-grained temporal cues. Existing methods usually use methods such as dilated convolution or average aggregation to model multi-scale time. Although multi-scale temporal features can be characterized by dilated convolutions with different dilation rates, weight-shared dilated convolutions are difficult to fully capture the scale changes of temporal features; similarly, if non-shared convolutions with different kernel sizes are used, the computational burden increases significantly with the increase in the number of convolutions. Therefore, the CPTM module of the present invention uses several progressive temporal convolutions with the same kernel size but non-shared weights to extract temporal features. The details are as follows: Figure 2 As shown, specifically, if the input feature χ is given, the time feature of the CPTM module of the present invention calculating the scale k is expressed as:

[0055] yk =G(W k )⊙y k-1 ,k=1,…,K, (3)

[0056] Where y0 = x and G(W k ) indicates that the weight is W k Therefore, CPTM can obtain a set of multi-scale features {y1,…,y K Compared with the parallel structure, CPTM extracts features based on the previous scale, takes into account the correlation between features of different scales, and has a lower model complexity. Then, for a set of multi-scale features {y1,…,y K}, the CPTM module of the present invention proposes a competitive fusion mechanism, which can be expressed as:

[0057]

[0058] The vector h of dimension C is subjected to global average pooling (GAP) of spatiotemporal features to capture the multi-time scale {y1,…,y K}'s global context, that is is through two weights, It is calculated by the fully connected layer of ; ⊙ represents the dot product operation along the channel dimension; in particular, τ is a balance parameter that determines the competition or collaboration between multi-scale features. It is easy to see that when τ approaches infinity, the fusion is closer to selecting a certain time scale. When τ = 0, all time scales are equally fused.

[0059] In summary, according to formulas (3) and (4), the proposed CPTM module has the ability to adaptively capture fine-grained temporal cues y on a single-resolution feature map χ.

[0060] 2.3 Coarse-grained CSTP module

[0061] In order to model based on more macroscopic and coarse-grained spatiotemporal cues, inspired by the work of TPN

[16] , this paper proposes a collaborative spatiotemporal pyramid module, which aims to adaptively fuse spatiotemporal features of different resolutions based on the information flows of different spatiotemporal resolutions obtained by calculation. Specifically, according to the given two spatiotemporal resolution features χ i and χ j First, by fully considering the feature size to calculate the information flow feature, a set of alignment features is generated The specific calculation process is as follows

[16] : In order to align the feature sizes of different spatiotemporal resolutions, the i-th stage feature Characteristics of stage j in Both θ and θ consist of 2D convolutional layers and temporal pooling layers to obtain the aligned stage features. Secondly, according to the top-down or bottom-up flow direction of the network forward structure, a 2D convolution φ with a kernel size of 1×1 is used i and φ j , calculate the corresponding stage characteristics after information flow or Finally, in order to fully aggregate the set of aligned features The CSTP module of the present invention adopts a two-stage adaptive fusion mechanism, such as Figure 3 As shown, the final prediction feature Z is calculated by the following formula:

[0062]

[0063] Among them, φ up ,φ dw and φ z are three 2D convolutional layers with a kernel size of 1×1, and the symbol [·;·] represents the connection operation between features. In formula (5), the weight and By input Calculated by function g1; then, Z is calculated up and Z dw As input, the weight ω is calculated through function g2 up and ω dw In particular, functions g1 and g2 are similar to formula (4). It is worth noting that the balance parameter τ in functions g1 and g2 is 0.5, which enables the CSTP module to perform feature adaptive fusion in a collaborative manner.

[0064] For the fine-grained video recognition task, the present invention uses fine-grained datasets: Diving48

[18] and FineGym

[19] to conduct experiments, and the experimental results achieve state-of-the-art performance. Next, the experimental details and implementation methods of the AMS network of the present invention will be described in detail.

[0065] 1. AMS network model construction

[0066] In order to construct a 2D-AMS network, the present invention uses a residual network (R50)

[20] as the backbone network model, and selectively inserts the fine-grained temporal modeling module CPTM proposed in the present invention between different convolution stages (such as after res2 and res4); the coarse-grained cross-temporal and spatial resolution feature fusion selects the output of the last two stages of the residual network as the spatiotemporal features to balance the computational cost and efficiency. To construct a 3D-AMS network, the present invention uses a slow channel network with a network depth of 50 as the backbone network model [1]. In order to balance the computational cost and efficiency, in the last two convolution stages, the 3D CPTM The original 3D convolutional layer is replaced by a residual block every other layer. Therefore, based on the backbone network, the present invention uses 4 3D CPTM layer.

[0067] 2. Data Preparation

[0068] For fine-grained video datasets, the present invention conducts extensive experiments on the diving competition dataset (Diving48) and the fine-grained gymnastics dataset (FineGym), in which the old version (V1) and the updated version (V2) of the Diving48 dataset are experimented. For data enhancement, each sampled frame is randomly cropped to 224×224 with scale jitter. For the sampling of video frames, continuing the settings of [6] and [1], in the 2D-AMS network, each video sample is first uniformly sampled from 16 segments, and one frame is randomly extracted from each segment to obtain 16 frames spanning the entire video timeline; in view of the 3D-AMS network, 16 frames are sampled from 64 consecutive frames with a step size of 4 frames. In the inference stage, the sampling adopts a multi-view strategy, that is, multiple random cropping of video frames and multiple sampling of video segments, for comparison with the performance of the most advanced methods.

[0069] 3. Training of Deep Network Model

[0070] The AMS network proposed in this invention is trained using a stochastic gradient descent (SGD) optimizer with a momentum of 0.9, and the learning rate is updated using a step-down strategy with a decay rate of 0.1. For 16 frames of input, the initial learning rates of the 2D / 3D network model are set to 8e-3 and 5e-3, respectively, the training weight decays are 5e-4 and 1e-4, respectively, and the training sample batch size is 48. The code project of this invention is implemented based on the toolbox MMAction and runs on a server configured with 8 NVIDIA GeForce RTX 2080Ti GPUs.

[0071] 4. Prediction of Deep Network Models

[0072] The experimental results of the embodiments of the present invention are shown in Tables 1 and 2, respectively. The results show the performance of various latest methods on the Diving48 dataset and the FineGym dataset. The video recognition methods include the above-mentioned methods as well as methods such as CorrNet

[21] , GST

[22] , SELFY

[23] , TQN

[24] , and the AMS network method of the present invention. The results show that the method proposed in the present invention can achieve better performance than other methods on fine-grained datasets and achieve the most advanced performance. Therefore, it shows that on the more difficult datasets for capturing time clues, the method of the present invention still has the ability to capture complex time scale changes.

[0073] In summary, the fine-grained video recognition method AMS network based on adaptive multi-granularity spatiotemporal modeling proposed in the present invention has a strong ability to capture complex time change clues through fine-grained adaptive time modeling and adaptive fusion of coarse-grained spatiotemporal features; at the same time, this method can be flexibly inserted into the existing deep convolutional network instantiation, efficiently and lightly realize video action recognition tasks, and improve its performance.

[0074]

[0075]

[0076] Table 1 Comparison of prediction accuracy based on the Diving48 dataset

[0077]

[0078] Table 2 Comparison of prediction accuracy based on FineGym dataset

[0079] 7. Other Information

[0080] References

[0081] [1]Christoph Feichtenhofer,Haoqi Fan,Jitendra Malik,andKaimingHe.Slowfast networks for video recognition.InICCV,2019.1,2,5,7,8.

[0082] [2] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3D convolutional neural networks for human action recognition. IEEE TPAMI, 35(1): 221–231, 2013. 1, 2.

[0083] [3]Ji Lin,Chuang Gan,and Song Han.TSM:Temporal shiftmodule forefficient video understanding.In ICCV,2019.1,2,3,6,7,8.

[0084] [4]Karen Simonyan and Andrew Zisserman.Two-streamconvolutionalnetworks for action recognition in videos.InNeurIPS,2014.1,2.

[0085] [5]Du Tran,Heng Wang,Lorenzo Torresani,Jamie Ray,YannLeCun,andManohar Paluri.A closer look at spatiotemporalconvolutions for actionrecognition.In CVPR,2018.1,2.

[0086] [6]Limin Wang,YuanjunXiong,Zhe Wang,Yu Qiao,DahuaLin,Xiaoou Tang,andLuc Van Gool.Temporal segmentnetworks:Towards good practices for deep actionrecognition.In ECCV,2016.1.

[0087] [7]Christoph Feichtenhofer.X3D:Expanding architectures forefficientvideo recognition.In CVPR,2020.2,7.

[0088] [8]Boyuan Jiang,Mengmeng Wang,WeihaoGan,Wei Wu,andJunjie Yan.STM:Spatiotemporal and motion encoding foraction recognition.In ICCV,2019.2,3.

[0089] [9]Yan Li,Bin Ji,Xintian Shi,Jianguo Zhang,Bin Kang,andLiminWang.TEA:Temporal excitation and aggregation foraction recognition.In CVPR,2020.2,3.

[0090]

[10] Zhaoyang Liu,Donghao Luo,Yabiao Wang,Limin Wang,and TongLu.TEINet:Towards an efficient architecture forvideo recognition.In AAAI,2020.2.

[0091]

[11] Limin Wang,Zhan Tong,Bin Ji,and Gangshan Wu.TDN:temporaldifference networks for efficient action recognition.In CVPR,2021.2,7.

[0092]

[12] Anurag Arnab,Mostafa Dehghani,Georg Heigold,ChenSun,Mario Lucic,and Cordelia Schmid.ViViT:A videovision transformer.ICCV,2021.2,7.

[0093]

[13] Ze Liu,Jia Ning,Yue Cao,Yixuan Wei,Zheng Zhang,Stephen Lin,andHan Hu.Video swin transformer.arXivpreprint arXiv:2106.13230,2021.2.

[0094]

[14] Da Zhang,Xiyang Dai,and Yuan-FangWang.Dynamic temporalpyramidnetwork:A closer look at multi-scale modelingfor activity detection.In ACCV,2018.1,2.

[0095]

[15] Yi Zhu and Shawn D.Newsam.Random temporal skipping for multirate video analysis.In ACCV, 2018.1,2,8.

[0096]

[16] Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition. In CVPR, 2020.1, 2, 4, 6, 7.

[0097]

[17] Qing Li, Zhaofan Qiu, Ting Yao, Tao Mei, Yong Rui, and Jiebo Luo. Action recognition by learning deep multi-granular spatio-temporal video representation. In ACMICMR, 2016.1, 2, 8

[0098]

[18] Yingwei Li, Yi Li, and NunoVasconcelos.RESOUND:Towards actionrecognition without representation bias.InECCV,2018.2,5.

[0099]

[19] Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. FineGym: Ahierarchical video dataset for fine-grained action understanding. In CVPR, 2020.2,5.

[0100]

[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.5.

[0101]

[21] Heng Wang,Du Tran,Lorenzo Torresani,and Matt Feiszli.Videomodeling with correlation networks.In CVPR,2020.8.

[0102]

[22] Chenxu Luo and Alan Yuille.Grouped spatial-temporalaggregation for efficient action recognition.In ICCV, 2019.8.

[0103]

[23] Heeseung Kwon,Manjin Kim,SuhaKwak,and Minsu Cho. Learning self-similarity in space and time as generalized motion for video actionrecognition.In ICCV, 2021.7,8.

[0104]

[24] Chuhan Zhang,Ankush Gupta,and Andrew Zisserman.Temporal querynetworks for fine-grained video understanding.In CVPR,2021.8.

[0105] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solution of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, a person of ordinary skill in the art can also make many forms of specific changes under the guidance of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. A fine-grained video recognition method based on adaptive multi-granularity spatiotemporal modeling, characterized by: The method is based on a deep neural network, a fine-grained CPTM module and a coarse-grained CSTP module; and comprises the following steps: The fine-grained CPTM module is incorporated into a deep neural network to extract fine-grained temporal cues; The fine-grained CPTM module is constructed based on progressive temporal convolution and competitive fusion mechanism; including: A set of multi-scale features is calculated to calculate the temporal features of scale k. It is expressed as: y k =G(W k )⊙y k-1 ,k=1,…,K, Where y0 = x and G(W k ) indicates that the weight is W k Temporal convolution; The fine-grained CPTM module uses a competitive fusion mechanism to fuse multi-scale features, which can be expressed as: The vector h of dimension C is subjected to global average pooling (GAP) of spatiotemporal features to capture the multi-time scale {y1,…,y K }'s global context, that is is through two weights, It is calculated by the fully connected layer; ⊙ represents the dot product operation along the channel dimension; τ is a balance parameter that determines the competition or collaboration mode between multi-scale features; The coarse-grained CSTP module is incorporated into a deep neural network to extract and fuse coarse-grained spatiotemporal features; including: The coarse-grained CSTP module adopts a two-stage adaptive fusion mechanism, which is expressed as: Among them, φ up ,φ dw and φ z are three 2D convolutional layers with a kernel size of 1×1, and the symbol [·;·] represents the connection operation between features; the weight and By input Calculated by function g1; output Z up and Z dw As input to function g2, calculate the weight ω up and ω dw .

2. The fine-grained video recognition method based on adaptive multi-granularity spatiotemporal modeling according to claim 1 is characterized in that: The fine-grained CPTM module and the coarse-grained CSTP module are incorporated into the 2D deep neural network process: The CPTM module is selectively inserted between the convolution stages of the 2D deep neural network to extract fine-grained temporal cues of the features between the current consecutive convolution stages: set up is the output of the i-th residual block (resl,i) in the l-th stage, where W represents width, H represents height, T represents time dimension and C represents the number of channels; CPTM module (f CPTM ) Input features at this layer The extracted fine-grained temporal cues can be expressed as: Among them, N l for res l The last residual block of the next stage is res l+1,1 Input; The CSTP module uses the following formula to adaptively fuse and output the spatiotemporal resolution features of different convolution stages: Z=f CSTP (y(L-1),y(L)) Where: L represents the total number of stages in the deep convolutional network, and Z is used for the final prediction after fusion of features.

3. The fine-grained video recognition method based on adaptive multi-granularity spatiotemporal modeling according to claim 1 is characterized in that: The fine-grained CPTM module and the coarse-grained CSTP module are incorporated into the 3D deep neural network process: In the 3D deep neural network, the CPTM module and the 2D spatial convolution are combined in a residual manner to construct a 3D CPTM layer structure, and through 3D CPTM The original 3D convolutional layer in the residual network is replaced by the CNN layer to extract fine-grained temporal clues; The CSTP module uses the following formula to adaptively fuse and output the spatiotemporal resolution features of different convolution stages: Where: Z also represents the adaptive fusion of coarse-grained feature information across different spatiotemporal resolutions.

Images