A Temporal Bottleneck Attention Architecture for Video Action Recognition

Abstract

The present invention classifies actions performed in a video clip by receiving a video clip for analysis, the video clip including a time series of video frames, applying a bottleneck attention mechanism to the frames in the clip to generate a reduced sequence of keyframes, applying a two-dimensional, 2D, convolutional neural network to the sequence of keyframes to obtain a 3D embedding tensor for each keyframe, applying a multi-head attention mechanism to the 3D embedding tensor to generate a final action context, and applying a classification mechanism to the final action context to obtain a probability for each action class indicating the likelihood that the action specified in the action class occurred in the video clip.

Description

[Technical Field]

[0001] Various embodiments generally relate to methods and systems for classifying actions in videos that augment convolutional neural network (CNN) models with a bottleneck attention mechanism. [Background technology]

[0002] The amount of video content is growing exponentially. Therefore, technologies for analyzing video content must be able to scale efficiently. Maintaining a high level of performance while limiting hardware requirements makes it possible to process these large amounts of video data. In this context, deep neural network architectures for video-based human action recognition (VHAR) enable many real-world applications, such as video data understanding and classification, video surveillance, entertainment, and autonomous driving.

[0003] Extensive research has been proposed on algorithms based on convolutional neural networks (CNNs) for VHAR. One major advantage of two-dimensional (2D) CNN models is their ability to perform fast image calculations and extract meaningful features from high-resolution images through the use of convolutions. However, 2D CNNs perform frame-by-frame operations on video clips and do not explore spatiotemporal relationships between frames. To compensate for the lack of temporal modeling, aggregation algorithms or recurrent neural network models (RNNs), such as long short-term memory (LSTM), a type of RNN, have been combined with 2D CNNs. 2D CNN+LSTM architectures have shown promising results in capturing spatial patterns and long-term dependencies. Because videos contain multiple frames, three-dimensional (3D) CNN architectures are used to retain the feature extraction capabilities of convolutions while also modeling motion dynamics. 3D CNNs rapidly create hierarchical spatiotemporal representations of video data without the need for LSTMs. In this regard, state-of-the-art architectures for VHAR rely on a 2D CNN backbone using residual connections (e.g., ResNet), dilated convolutions, time segment networks also based on residual networks but using shifted convolutions, and 3D CNNs. Nevertheless, both 3D CNNs and LSTMs become increasingly computationally expensive as the number of frames in a video clip increases.

[0004] An efficient way to model temporal dependencies is through the use of attention mechanisms. Initially introduced to analyze long sequences in language tasks, attention models have become an area of ​​extensive research in VHAR. Attention mechanisms are a processing technique for neural networks that allows the network to focus on specific aspects of the data input. In the case of video data, attention mechanisms generate attention scores or weights that indicate the relative importance of regions within frames or images within a clip.

[0005] The model with attention is combined with LSTM, which allows the LSTM to capture the temporal order of frames within a video clip, which may be missed by a pure attention mechanism.

[0006] In recent years, attention models used in natural language processing (NLP), such as Google's Transformer, have been shown to be effective in vision tasks. The use of Transformers for images has been made possible, in part, by the development of bottleneck attention techniques. These solutions reduce the dimensionality of images and vectorize them, e.g., creating 1D vectors from 2D images. Bottleneck techniques are standard building blocks used in 2D CNN models and aim to increase the computational performance of deep neural networks.

[0007] While current bottleneck attention mechanisms are efficient for reducing image dimensions, there is no bottleneck-based solution for quickly reducing the temporal dimension of video while maintaining the exact same image size. Such a solution could be used to reduce the input data volume by generating a keyframe sequence from the raw video input.

[0008] Early research on VHAR suggests that a small, carefully selected set of frames from an input video stream, referred to herein as keyframes, can effectively identify human actions. Keyframe selection builds a sparse sequence representation by discarding frames with multiple, uninformative sequences within a video clip. This representation can then be used to perform action classification. However, finding such keyframes is challenging because selecting specific frames requires detailed frame knowledge of the entire video (e.g., human pose description). Labeling images typically requires human domain expertise, and describing each individual frame can be time-consuming. Furthermore, consider the case where the selected frames are overly sparse. In such cases, motion dynamics may be corrupted. This can degrade action recognition performance due to the high frequencies added to the movements and, consequently, the model parameters.

[0009] Current bottleneck attention mechanisms attempt to compensate for the quadratic scaling problem of the classical Transformer's exhaustive attention. While these solutions are efficient for visual classification, they do not address the time issue faced by video classification models, referred to herein as the clip size dilemma. Longer video clips improve accuracy but require more training time. We note that the use of short video clips contradicts the latest research findings in VHAR. Current long-term temporal convolutions performed on much longer video clips (32- or 128-frame clips) are believed to be necessary for performance gains. While this is true, we propose that compact keyframe sequences (video clips) with more contextual relationships between frames have two main advantages: 1) video recognition models can benefit from compact input representations by learning and modeling data distributions more quickly and accurately. 2) In testing, networks trained on shorter but more informative video clips can utilize human action predictions from longer sequences to improve recognition accuracy. Summary of the Invention [Problem to be solved by the invention]

[0010] It is with respect to these considerations and others, therefore, that the present invention has been made. [Means for solving the problem]

[0011] We use a novel end-to-end deep learning architecture, called Video Human Action Recognition (VHAR), for classifying (recognizing) human actions occurring in video clips. This architecture introduces a temporal bottleneck attention mechanism, referred to herein as TBoT, that constructs sparse keyframe sequences from longer videos. These sparse sequences result in a more useful representation for convolution-based models, significantly reducing computation time while maintaining acceptable results.

[0012] To better model movement dynamics, attention weights are calculated and centered around the mean frame. As a result, actions with short deviations from the mean have small scores, and vice versa. Finally, to enhance the attention representation of complex actions, we develop a residual mechanism that learns to pay attention to specific frames. Here, instead of using fully-connected (FC) layers or dot-product operations, we build a soft residual self-attention mechanism using convolutional and pooling statistics to compute effective context for action prediction.

[0013] TBoT relies on a soft-pooling self-attention mechanism with no learnable parameters, making it flexible enough to be used in different network locations. This allows the use of models pre-trained on large datasets, such as ImageNet, a large visual database designed for use in visual object recognition research. In fact, TBoT is added to the input side of the network with the aim of building a compact, contextualized sequence of keyframes for each clip used, rather than the full set of frames. These keyframes are then used to train the model. In this case, TBoT acts as an effective data augmentation strategy, as it mixes the data input and generates variability in the data.

[0014] TBoT incorporates an attention mechanism that enables convolutional neural networks (CNNs) to find temporal relationships between frames. The convolutional layers complement the attention mechanism by extracting useful image features for video recognition. TBoT does not require human intervention to construct keyframe sequence inputs. The time required to construct a short, contextualized keyframe sequence from any video clip size is relatively short. The attention mechanism performs tensor addition and scalar multiplication, which are efficiently computed by the GPU's tensor cores. The attention mechanism includes a residual self-attention procedure and a pooling self-attention procedure, which process the results generated by the 2D CNN model. Together, the two attention mechanisms function as a multiheaded mechanism to construct a final, compelling behavioral context for classification and prediction.

[0015] In a particular embodiment, the invention classifies actions performed in a video clip by receiving a video clip for analysis, the video clip including a time series of video frames, applying a bottleneck attention mechanism to the frames in the clip to generate a reduced sequence of keyframes, applying a two-dimensional (2D) convolutional neural network to the sequence of keyframes to obtain a 3D embedding tensor for each keyframe, applying a multi-head attention mechanism to the 3D embedding tensor to generate a final behavioral context, and applying a classification mechanism to the final behavioral context to obtain a probability for each behavioral class indicating the likelihood that the behavior specified in the behavioral class occurred in the video clip.

[0016] Non-limiting and non-exhaustive examples of the present invention are described with reference to the following drawings, in which like reference numerals refer to like parts throughout the various views unless otherwise specified.

[0017] For a better understanding of the present invention, reference should be made to the following detailed description, which should be read in connection with the accompanying drawings. [Brief explanation of the drawings]

[0018] [Figure 1] This is a generalized block diagram of a CNN-based system called TBoTNet that performs video-based human action recognition (VHAR). [Figure 2] Figure 1 illustrates an example of a machine learning system called a temporal bottleneck network with visual attention architecture (TBoTNet). [Figure 3A] A diagram showing how a 34-frame video clip can be reduced to 16 keyframes. [Figure 3B] A diagram showing how a 34-frame video clip can be reduced to 11 keyframes. [Figure 4] Figure 1 shows an example of a soft-pooling self-attention method incorporated into TBoTNet. [Figure 5] This figure shows an example of the residual attention mechanism incorporated into TBoTNet. DETAILED DESCRIPTION OF THE INVENTION

[0019] The drawings depict embodiments of the present invention for purposes of illustration only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles of the present invention as described herein.

[0020] The present invention will now be described more fully hereinafter with reference to the accompanying drawings, which form a part of this specification and which show, by way of illustration, specific exemplary embodiments in which the invention may be practiced. However, the present invention may be embodied in many different forms and should not be construed as limiting the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Among other things, the present invention may be embodied as a method, process, system, business method, or device. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Therefore, the following detailed description is not to be taken in a limiting sense.

[0021] As used herein, the following terms have the meanings given below.

[0022] Video Clip, Clip, or Video - Refers to a segment of video containing multiple frames. As used herein, video contains the primary action.

[0023] Human Action or Action - refers to a movement within a video clip by a person. In other examples, an action may refer to an action by another animal or inanimate object.

[0024] VHAR - refers to video human action recognition, a fundamental task in computer vision that aims to recognize or classify human actions based on the actions performed in a video.

[0025] Machine Learning Model—Refers to an algorithm or set of algorithms that takes structured and / or unstructured data inputs and generates a prediction or outcome. A prediction is typically a value or set of values. A machine learning model may itself contain one or more component models that interact and produce a result. As used herein, a machine learning model refers to a neural network, including a convolutional neural network or another type of machine learning mechanism, that receives video clips as input data and generates estimates or predictions for a known validation data set. Typically, a model is trained through successive runs of the model. Typically, a model is run continuously during the training phase, and once successfully trained, it is operationally used to evaluate new data and make predictions. It should be emphasized that this training phase may be run thousands of times to obtain an acceptable model that can predict a success metric. Furthermore, a model may discover thousands, or even tens of thousands, of features, many of which may be completely different from the features provided as input data. Therefore, the model is not known a priori and cannot be computed through mental effort alone.

[0026] Prediction—as used herein—refers to a statistical estimate or estimated probability that an action in a video clip belongs to a particular class or category of action. Prediction may also refer to an estimate or probability assigned to each class or category in a classification system that includes many individual classes. For example, the DeepMind dataset, Kinetics 400, provides up to 650,000 video clips categorized into 400 distinct human actions. This is an example of a commonly used training dataset.

[0027] Architecture - as used herein refers to the overall set of stages, procedures, or processes that are executed sequentially from input data to output data. This is shown in Figure 2 below, and includes pre-processing steps such as bottleneck attention processing that are performed before the data is submitted to a CNN or other machine learning model.

[0028] Generalized Operations The operation of several embodiments of the present invention will now be described with reference to FIGS.

[0029] 1 is a generalized block diagram of a machine learning system 100 that performs video-based human activity recognition (VHAR). A TBoTNet server 120 executes a TBoTNet architecture 125 (or simply TBoTNet 125) on a computer.

[0030] The TBoTNet server 120 accesses data sources 130, which provide video clips for analysis. The video clips may be used during model training or may be live input data used operationally for analysis and classification. For example, YOUTUBE®.COM, a website operated by GOOGLE, Inc., may be one of the data sources 130. Other data sources 130 may include television channels, movies, and video archives. Typically, the TBoTNet server 120 accesses video clips from data sources 130 over a network 140; however, in certain embodiments, clips may be provided on physical media such as USB drives, hard drives, and other electronic communication media such as direct links. The TBoTNet server 120 includes a processor, data storage for storing video clips and intermediate results, and non-volatile memory for storing program code and data.

[0031] TBoTNet Server 120 may be implemented by a single server computer, by multiple server computers working in cooperation, or by a network service or "cloud" service offered by a cloud service provider such as AMAZON AWS. Devices that may operate as TBoTNet Server 120 include, but are not limited to, personal computers, desktop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, servers, network appliances, etc.

[0032] The video clips used by TBoTNet125 include, among others, (1) live video data, training datasets such as the Kinetics 400 dataset, which is used to train machine learning models for classification purposes, and training datasets such as ImageNet, which provides a large number of images and can be used to pre-train machine learning models.

[0033] Users interact with the TBoTNet server 120 to identify and provide training videos or clips to train the TBoTNet model 125. Typically, users interact with a user application 115 running on the user computer 110. The user application 115 may be a native application or a web application running within a web browser such as Mozilla Firefox or Google Chrome.

[0034] User computer 110 may be a laptop computer, a desktop personal computer, a mobile device such as a smartphone, or any other computer running a program that can interact over network 140 and access TBoTNet server 120. In general, user computer 110 may be a smartphone, personal computer, tablet computer, or other computer system with a processor, non-transitory memory for storing program instructions and data, a display, and interactive devices such as a keyboard and mouse.

[0035] TBoTNet 130 typically stores data and runs TBoTNet 125, which is described below with reference to FIGS. 2 and 3A-3B.

[0036] Network 140 allows user computers 110 and TBoTNet server 120 to exchange data and messages. Network 140 may include a local area network (LAN), a wide area network (WAN), a direct connection, a combination thereof, etc., as well as the Internet.

[0037] Behavioral modeling and MR model A supervised machine learning model provides a score or probability estimate for each class in a classification set. The score (probability) indicates the likelihood that a video clip contains the behavior represented by the class member. The class with the highest score can be selected when a single prediction is needed. This class is considered to represent the behavior most likely to have occurred in the video clip, performed by the subject. A validation dataset of video clips, in which the primary class for each clip is known, is used to train the model by sequentially running it through different clips from the dataset and adjusting the model with each successive model run to minimize error. As used herein, the term "subject" is used generically to refer to behaviors performed by people, animals, or other objects in a video clip. While the present invention is primarily intended to apply to behaviors performed by human subjects, it is not so limited and may also be applied to other moving objects, such as animals, and inanimate objects, such as cars, balls, etc.

[0038] VHAR Bottleneck Architecture Figure 2 shows an example of a machine learning architecture called TBoTNet125, which extends convolutional neural networks (CNNs) with bottleneck and multihead attention mechanisms. TBoTNet125 is particularly well-suited for video-based human action recognition (VHAR). In general, TBoTNet125 is a temporal bottleneck network architecture with visual attention. TBoTNet125 first constructs a sparse or compact representation of each input video clip, i.e., a representation with a reduced temporal dimension, and then uses this reduced input to efficiently learn and classify the human actions represented in the video clip. While processing for a single video clip is described below, it can be appreciated that typically a large number of video clips are processed.

[0039] A raw video X is provided as input to TBoTNet125. The raw input video is preprocessed to generate video inputs X1, X2, …, X, which are a sequence of video frames in a video clip. t These video frames are then processed by TBoTNet125 and output

number

[0040] First, in step 202, a raw video file is preprocessed to generate a series of video clips. The video clips are typically processed in parallel by TBoTNet 125 to achieve high throughput, but may also be processed sequentially. Preprocessing typically involves (1) reducing the resolution of the video through averaging, subsampling, or another process to reach a desired video frame size, and (2) cropping or selecting a rectangular region within the reduced-resolution frame for further processing. The resulting smaller video clips are then provided as input to Bottleneck Attention Mechanism 204 (or simply TBoT 204).

[0041] Temporal bottleneck attention, referred to as TBoT204, is applied to a sequence of incoming video frames with the goal of constructing a reduced, representative sequence of a video clip. The TBoT204 process is described in more detail below with reference to Figures 3A and 3B. TBoT204 uses a bottleneck attention procedure to reduce the number of frames in a clip to construct a smaller sequence of keyframes, thereby improving performance. In general, TBoT204 uses the bottleneck attention mechanism to construct a more compact temporal representation of the raw input data, reducing the volume of data while preserving important image details. Traditionally, attention has been used to weight frames and clips, thereby improving results, but not to reduce the volume of data provided to neural networks for subsequent processing.

[0042] Next, a 2D convolutional neural network (2D CNN) 206 is applied to the novel sequence of clips. The 2D CNN 206 computes the

number

[0043] Next, an additive pooling self-attention procedure 208 and a residual self-attention procedure 210 are applied to the embedding output to obtain the soft context (ctx s ) and residual context (ctxr ) and

[0044] A concatenation procedure 212 then combines the two contexts into a single effective behavioral context (ctx) for each clip.

[0045] Finally, the behavioral context ctx is fed into the classifier 214 to generate a logit

number

[0046] While the training of a CNN is considered outside the scope of the present invention, the overall system is trained using a loss function such as cross-entropy loss or mean square error (MSE).

[0047] In general, the architecture in Figure 2 can be grouped into four overall components: (1) video clips and embedded representations, (2) early attention, (3) additive self-attention, and (4) action recognition. Each of the four components is discussed below.

[0048] 2 may represent a procedure that is a step of an overall method that is performed entirely in software or hardware, or a combination of hardware. Furthermore, in some cases, two or more boxes with the same reference number, e.g., TBoT Attention 204, are shown to suggest that processing may be performed in parallel, although such processing may also be performed sequentially.

[0049] Video Clips and Embedded Presentations We first consider the representation of a video clip as a sequence of images, i.e., frames. Formally, a video clip represented as a 4D tensor is

number

[0050] In normal operation, several clips are passed to TBoTNet 125 in batches of clips. For example, batches of 8, 16, 32, and 64 clips may be passed. Therefore, to represent a full video or a sequence of videos, an additional index may be added indicating the sequence number in the batch. This results in a tensor of dimensions B x C x τ x M x M, where B is the number of clips in the batch to process. The batch index is omitted here for notational simplicity. The spatial resolution or size per clip is typically M = 112 or M = 224, although any resolution may be used.

[0051] Time Bottleneck Attention Mechanism To compute the sequence of key frames, a temporal bottleneck attention procedure or mechanism, TBoT204, is employed. TBoT204 automatically constructs compact and contextualized video clips without human intervention. As a result, the network is able to learn human actions from smaller clips, allowing the training phase to occur quickly and accurately.

[0052] Formally, TBoT204 processes a video clip with s frames, size σ, which is formulated as follows:

number

number

number

[0053] In a particular embodiment, a pooling self-attention mechanism 208, described below with reference to Figure 4, calculates temporal attention weights that are then applied when combining a sequence of temporally consecutive frames to generate keyframes. In another embodiment, a residual self-attention mechanism 210, described below with reference to Figure 5, is used. In yet another embodiment, other attention mechanisms are used.

[0054] Figure 3A shows one embodiment of a method performed by TBOT 204 to reduce the number of frames in a clip from 34 to 16, and Figure 3B shows one embodiment of a method performed by TBOT 204 to reduce the number of frames in a clip from 34 to 11. In general, various methods for reducing the size of a video input stream are within the scope of the present invention. In Figure 3A, three consecutive, i.e., temporally consecutive, input frames are combined to produce each keyframe, beginning with the sequence x1, x2, x3 that together produce the first keyframe TBoT1. Meanwhile, in Figure 3B, four consecutive input frames are combined to produce each keyframe.

[0055] Each keyframe is a weighted average of three frames (Figure 3A) or four frames (Figure 3B), where the weights are the temporal attention weights calculated by the bottleneck attention mechanism.

[0056] The TBoT 204 generates a new sequence of frames, i.e., a new video clip, defined in Equation 3 below, which is then provided or fed into the 2D CNN model in step 206. The difference between the input and output is the number of keyframes versus the number of frames in the input video clip.

[0057]

number

[0058] Embedded Representations After applying pre-attention to the input clip X, we have the frame sequence defined in Equation 3.

number

[0059] In a specific implementation, the output of the last convolutional layer of ResNet50 is used for the embedded representation of each frame clip. It may be noted that the CNN may be pre-trained on a dataset such as ImageNet, which is a widely available dataset.

[0060] Fine-tuning is then performed on the target dataset by updating all layers of the model and unfreezing the batch norm layer.

[0061] The embedding representation of the output of step 206 is described in Equation 4 below.

number

number

[0062] Attention Mechanism Although 2D CNNs excel at extracting image features and perform computations faster than 3D CNNs, their lack of equal treatment of video frames is a weakness when it comes to video analysis. In other words, 2D CNNs are limited in quantifying contextual information from video sequences. To mitigate this issue, a combination of temporal contextual frame dependency and soft and residual self-attentive mechanisms is used, as shown in Figures 4 and 5.

[0063] Residual Self-Attention Figure 4 shows an example of an attention mechanism based on pooling statistics, called soft-pooling self-attention or pooling self-attention208, incorporated into TBoTNet125. This technique is used because it provides improved performance for fine-grained classification problems. In general, compared to fully connected (FC) and dot-product attention mechanisms, pooling is more efficient because it only performs tensor additions and scalar multiplications. Furthermore, when used as the attention mechanism in TBoTNet125, the keyframes are generated by the CPU, allowing the model to analyze reduced video clip representations, i.e., keyframe sequences, without constraints, saving GPU memory and shortening training time.

[0064] More formally, given the embedding output, Pooling and Self-Attention 208 first computes a weight vector α by applying 3D average pooling over the channels and spatial locations (C'M'M'). α t =AvgPool3D(e t ) Equation 5 where:

number

[0065] Then, to better model the motion displacement between clips, the attention weights are centered around the mean frame μ α is defined in Equation 6 below.

number

[0066] As a result, movements with small deviations from the mean have small values, and vice versa.

[0067] The centered weights are then processed by a softmax function to normalize their values ​​between 0 and 1. This is formulated in Equation 7 below as follows:

number

[0068] where:

number

number

[0069] As a final step, an additive mechanism is formulated in Equation 8 below, which multiplies each weight by its corresponding frame to expand the differences between frames in the same clip or sub-clip.

number

[0070] Residual Self-Attention The ability of an attention mechanism to maintain and learn weight vectors that represent complex actions relies on how the input is transformed throughout the mechanism. Although linear mapping (e.g., fully connected (FC) layers) is widely accepted for increasing the representability of features, convolutional neural networks (CNNs) are widely used to extract image representations. Therefore, to enhance the attention representation of complex actions, a residual attention mechanism that calculates attention weights for every frame is incorporated into TBoTNet125.

[0071] Figure 5 shows an example of the residual attention mechanism 210 built into TBoTNet125. The embedding output E of the model is a two-dimensional 1x1 kernel with a stride of 1.

number

number

[0072] Next, β is fed into a softmax function as given in Equation 10 below.

number

number

[0073] Finally, the residual attention vector is formulated in Equation 11 as follows:

number

[0074] behavioral recognition Attention can better capture contextual semantic representations when computed as a multi-head attention mechanism. The term head refers to the final processing step in a neural network architecture that produces the final result. As used herein, multi-head attention runs through at least two different attention mechanisms in parallel. The independent attention outputs are then concatenated or otherwise combined to obtain a single output. Different attention heads may be used to analyze portions of the input sequence in different ways (e.g., long-term versus short-term dependencies). In the example of FIG. 2, multi-head attention is used to run pooling self-attention 208 and residual attention 210 in parallel, then concatenate their respective outputs in step 212 to produce a single vector of attention weights, called the final behavioral context. The final behavioral context is

number

[0075] In step 214 of Figure 2, ctx is fed into a batch normalization (BN) layer, followed by conv(2*C',C' / 2), ReLU

[38] , global average pooling, and logit vector

number

[0076] The backpropagation process of training the CNN weights to reduce the error on the target dataset is not considered part of this invention, but a loss function such as minimum square error (MSE) cross-entropy loss is employed.

[0077] Training and Testing As mentioned above, in a specific embodiment, TBoTNet125 includes a convolutional neural network (CNN) model. This model may be pre-trained on the ImageNet dataset. The CNN model is fine-tuned with the BN layer enabled and without dropout. Input video clip frames are resized to 128x240 for scale enhancement. 112x112 pixel crops are randomly sampled from each frame-clip at the same location in a Cartesian coordinate system. These crops also undergo random horizontal flipping and random RGB and grayscale enhancement with a probability of 0.5 and a color distortion of 0.2. From consecutive frames in each training video, several consecutive frames τ are sampled, and the remaining frames are discarded. Tests were performed on clips with 8, 11, and 16 keyframes. The number of input clips tested was τ = 34, with stride = {3, 4, 5}. For clips of size 8 and 11, a batch size of 128 clips per GPU was used, and for clips of 16 frames, a batch size of 120 clips was used due to GPU memory capacity.

[0078] The predictions of TBoTNet125 are compared to a training dataset, such as the Kinetics 400 dataset, and the error is determined according to a loss function.

[0079] Overall, the performance of the TBoTNet125 architecture was tested using multiple ablation experiments and showed significant improvement in action recognition rate.

[0080] The above specification, examples, and data provide a complete description of the manufacture and use of the composition of the invention. Since many embodiments of the invention can be made without departing from the spirit and scope of the invention, the invention resides in the claims hereinafter appended.

Claims

1. 1. A computer-implemented method for classifying actions performed in a video clip, comprising: receiving a video clip for analysis, the video clip including a time sequence of video frames; applying a bottleneck attention mechanism to the frames in the clip to generate a reduced sequence of keyframes; applying a two-dimensional (2D) convolutional neural network to the sequence of keyframes to obtain a 3D embedding tensor for each keyframe; applying a multi-head attention mechanism to the 3D embedding tensor to generate a final behavioral context; applying a classification mechanism to the final behavioral context to obtain a probability for each behavioral class, the probability indicating the likelihood that a behavior specified in the behavioral class occurred in the video clip; 20. A computer-implemented method comprising:

2. The method of claim 1 , wherein each keyframe represents a different subset of temporally consecutive frames in the video clip.

3. 3. The method of claim 2, wherein the bottleneck attention mechanism generates either 16 keyframes or 11 keyframes from a video clip of 34 video frames.

4. The multi-head attention mechanism comprises: applying a pooling self-attention procedure; applying a residual self-attention procedure; concatenating the results of the pooling self-attention procedure and the results of the residual self-attention procedure to obtain the final behavioral context; The method of claim 1 , comprising:

5. the pooling self-attention procedure comprising: Calculating attention weights for each keyframe in the clip based on 3D average pooling; centering the attention weights around an average frame; normalizing the centered attention weights to create a normalized attention vector; multiplying the normalized attention weights by the attention weights of each keyframe to expand the differences between the keyframes within the clip; and The method of claim 4, comprising:

6. The bottleneck attention mechanism: calculating a temporal attention weight for each of said video frames; computing keyframes, each keyframe being a weighted average of a subset of temporally consecutive frames, the weights being the calculated temporal attention weights; The method of claim 4, comprising:

7. The method of claim 6 , wherein the temporal attention weights are generated by the pooling self-attention procedure.

8. the residual self-attention procedure: convolving the embedding tensor output with a two-dimensional 1×1 kernel; Computing attention weights for each convolved tensor based on 2D average pooling; applying softmax to the attention weights to generate a normalized attention vector; multiplying the weights by the embedding tensor and scaling the result to obtain a residual behavioral context; The method of claim 4, comprising:

9. The method of claim 6 , wherein the temporal attention weights are generated by the residual self-attention procedure.

10. The method of claim 1 , further comprising selecting a highest probability from the behavioral context probabilities to predict the behavior that is most likely to have occurred in the video.

11. 1. A server computer, comprising: a processor; a communications interface in communication with the processor; a data storage for storing the video clips; a memory in communication with the processor for storing instructions that, when executed by the processor, cause the server to: receiving a video clip for analysis, the video clip including a time sequence of video frames; applying a bottleneck attention mechanism to the frames in the clip to generate a reduced sequence of keyframes; applying a two-dimensional (2D) convolutional neural network to the sequence of keyframes to obtain a 3D embedding tensor for each keyframe; applying a multi-head attention mechanism to the 3D embedding tensor to generate a final behavioral context; applying a classification mechanism to the final behavioral context to obtain a probability for each behavioral class, the probability indicating the likelihood that a behavior specified in the behavioral class occurred within the video clip; memory and A server computer comprising:

12. 12. The server computer of claim 11, wherein each keyframe represents a different subset of temporally consecutive frames in the video clip.

13. 13. The server computer of claim 12, wherein the bottleneck attention mechanism generates either 16 keyframes or 11 keyframes from a video clip of 34 video frames.

14. The multi-head attention mechanism comprises: applying a pooling self-attention procedure; applying a residual self-attention procedure; concatenating the results of the pooling self-attention procedure and the results of the residual self-attention procedure to obtain the final behavioral context; 12. The server computer of claim 11, comprising:

15. the pooling self-attention procedure comprising: Calculating attention weights for each keyframe in the clip based on 3D average pooling; centering the attention weights around an average frame; normalizing the centered attention weights to create a normalized attention vector; multiplying the normalized attention weights by the attention weights of each keyframe to expand the differences between the keyframes within the clip; and 15. The server computer of claim 14, comprising:

16. The bottleneck attention mechanism: calculating a temporal attention weight for each of said video frames; computing keyframes, each keyframe being a weighted average of a subset of temporally consecutive frames, the weights being the calculated temporal attention weights; 15. The server computer of claim 14, comprising:

17. 17. The server computer of claim 16, wherein the temporal attention weights are generated by the pooling self-attention procedure.

18. the residual self-attention procedure: convolving the embedding tensor output with a two-dimensional 1×1 kernel; Computing attention weights for each convolved tensor based on 2D average pooling; applying softmax to the attention weights to generate a normalized attention vector; multiplying the weights by the embedding tensor and scaling the result to obtain a residual behavioral context; 15. The server computer of claim 14, comprising:

19. 17. The server computer of claim 16, wherein the temporal attention weights are generated by the residual self-attention procedure.

20. The server computer of claim 11 , further comprising selecting a highest probability from the behavioral context probabilities to predict a behavior that is most likely to have occurred in the video.

Images