Adaptive Use of Video Models for Holistic Video Understanding

Abstract

Systems and techniques are provided for performing holistic video understanding. For example, the process may include obtaining a first video and using a machine learning model determination engine to determine a first machine learning model from a set of machine learning models to use for processing at least a portion of the first video. The first machine learning model may be determined based on one or more characteristics of at least the portion of the first video. The process may include using the first machine learning model to process at least the portion of the first video.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates generally to models for understanding video content. In some examples, aspects of the disclosure relate to systems and techniques for selection of machine learning models based on characteristics of video content. [Background technology]

[0002] Many machine learning models are available for analyzing and categorizing information contained in video data (or a sequence of image frames). To obtain a holistic understanding of the information contained in the video data, the video data should be understood across various categories, such as actions, attributes, events, objects, scenes, etc., among others. Machine learning models trained to detect actions (e.g., three-dimensional (3D) convolutional neural networks (CNNs), optical flow neural networks, etc.) can be overly computationally intensive when applied to video data containing a small number of actions. On the other hand, computationally efficient machine learning models that are suitable for detecting objects in static video scenes may not be able to adequately detect actions in videos containing large movements. In some cases, it is desirable to have a machine learning model that is capable of performing a holistic understanding of videos across multiple categories in as efficient a manner as possible. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Ali Diba et al., “Large scale holistic video understanding”, 2020 [Non-Patent Document 2] Kaiming He et al., “Deep residual learning for image recognition”, 2016 [Non-Patent Document 3] Christoph Feichtenhofer et al., “Slowfast networks for video recognition”, 2019 [Non-Patent Document 4] Mark Sandler et al., “MobileNetV2: Inverted Residuals and Linear Bottlenecks”, 2019 Summary of the Invention [Means for solving the problem]

[0004] In some examples, systems and techniques for adaptively selecting between video processing models for holistic video understanding are described. According to at least one illustrative example, a method for processing video data is provided. The method includes obtaining a first video, determining a first machine learning model from a set of machine learning models using a machine learning model determination engine, where the first machine learning model is determined based on one or more characteristics of at least a portion of the first video, and processing at least the portion of the first video using the first machine learning model.

[0005] In another example, an apparatus is provided for processing video data, the apparatus including a memory configured to store at least one video or portion of a video, and one or more processors coupled to the memory (e.g., implemented in a circuit), the one or more processors configured and capable of obtaining a first video, determining a first machine learning model from a set of machine learning models using a machine learning model determination engine, where the first machine learning model is determined based on one or more characteristics of at least the portion of the first video, and processing at least the portion of the first video using the first machine learning model.

[0006] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to obtain a first video; determine, using a machine learning model determination engine, a first machine learning model from a set of machine learning models, where the first machine learning model is determined based on one or more characteristics of at least a portion of the first video; and process at least the portion of the first video using the first machine learning model.

[0007] In another example, an apparatus for processing video data is provided that includes means for obtaining a first video, means for determining a first machine learning model from a set of machine learning models using a machine learning model determination engine, where the first machine learning model is determined based on one or more characteristics of at least a portion of the first video, and means for processing at least the portion of the first video using the first machine learning model.

[0008] In some aspects, the methods, apparatus, and computer-readable media described above further include determining a classification of at least the portion of the first video based on processing at least the portion of the first video using the first machine learning model.

[0009] In some aspects, the methods, apparatus, and computer-readable media described above further include extracting a first one or more features from at least the portion of the first video using a first machine learning model, processing the first one or more features using a first auxiliary model, and determining a first classification of at least the portion of the first video based on processing the first one or more features using the first auxiliary model. In some cases, the first auxiliary model includes a neural network. In some cases, the first auxiliary model includes a one-dimensional convolutional neural network. In such cases, the first one or more features include a first feature vector based on a first frame of at least the portion of the first video and a second feature vector based on a second frame of at least the portion of the first video. In such cases, the first auxiliary model generates temporal information of at least the portion of the first video from at least the first feature vector and the second feature vector. In some cases, the first auxiliary model includes a multi-layer perceptron. In some cases, the first auxiliary model includes a graph convolutional network.

[0010] In some aspects, the methods, apparatus, and computer-readable media described above further include using the machine learning model determination engine to determine a second machine learning model from the set of machine learning models, where the second machine learning model is determined based on one or more characteristics of at least another portion of the first video, and using the second machine learning model to process at least another portion of the first video. In some aspects, the second machine learning model includes a neural network.

[0011] In some aspects, the methods, apparatus, and computer-readable media described above further include extracting a second one or more features from at least another portion of the first video using a second machine learning model, processing the second one or more features using a second auxiliary model, and determining a second classification of the at least another portion of the first video based on processing the second one or more features using the second auxiliary model.

[0012] In some aspects, the first machine learning model includes a neural network. In some aspects, the first machine learning model includes a two-dimensional convolutional neural network. In some cases, the two-dimensional convolutional neural network extracts the first one or more features from at least the portion of the first video in two spatial dimensions.

[0013] In some aspects, the first machine learning model includes a three-dimensional (3D) CNN. In some cases, the 3D CNN extracts the first one or more features from at least the portion of the first video in two spatial dimensions and a temporal dimension.

[0014] In some aspects, the methods, apparatus, and computer-readable media described above further include obtaining a second video; determining a second machine learning model from a set of machine learning models using a machine learning model determination engine, where the second machine learning model is determined based on one or more characteristics of at least a portion of the second video; and processing at least the portion of the second video using the second machine learning model.

[0015] In some aspects, the machine learning model determination engine shares a common neural network with the first machine learning model. In some aspects, the methods, apparatus, and computer-readable media described above further include using the first machine learning model to extract a first one or more features from at least the portion of the first video in parallel with determining the first machine learning model from the set of machine learning models for use in processing at least the portion of the first video.

[0016] In some aspects, the one or more characteristics of at least the portion of the first video include spatial characteristics and temporal characteristics.

[0017] In some aspects, the one or more characteristics of at least the portion of the first video include audio characteristics.

[0018] In some aspects, one or more of the devices described above may include or be part of a mobile device (e.g., a mobile phone, or so-called "smartphone," or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR), augmented reality (AR), or mixed reality (MR) device), a vehicle (e.g., a vehicular computing device), a personal computer, a laptop computer, a server computer, or other device. In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device further includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors that may be used to determine the location and / or attitude of the device, the state of the device, and / or for other purposes.

[0019] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter, which subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0020] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0021] Exemplary embodiments of the present application are described in detail below with reference to the following figures. [Brief description of the drawings]

[0022] [Figure 1] 1 is a block diagram illustrating an example holistic video understanding system, in accordance with some examples. [Figure 2A] FIG. 2 illustrates an exemplary operation of a one-dimensional convolution filter, in accordance with some examples. [Figure 2B] FIG. 2 illustrates an exemplary operation of a one-dimensional convolution filter, in accordance with some examples. [Figure 2C] FIG. 2 illustrates an exemplary operation of a one-dimensional convolution filter, in accordance with some examples. [Figure 2D] FIG. 2 illustrates an exemplary operation of a one-dimensional convolution filter, in accordance with some examples. [Diagram 3] 1 is a block diagram illustrating an example of a holistic video understanding system, in accordance with some examples. [Figure 4] 1 is a block diagram illustrating another example of a holistic video understanding system, in accordance with some examples. [Diagram 5] 1 is a block diagram illustrating another example of a holistic video understanding system, in accordance with some examples. [Figure 6] 1 is a block diagram illustrating another example of a holistic video understanding system, in accordance with some examples. [Figure 7]1 is a block diagram illustrating another example of a holistic video understanding system, in accordance with some examples. [Figure 8] 1 is a plot illustrating the performance of an exemplary holistic video understanding system, according to some examples. [Figure 9] FIG. 2 is a flow diagram illustrating an example of a process for processing video data, according to some examples. [Figure 10] FIG. 1 is a block diagram illustrating an example of a deep learning network, in accordance with some examples. [Figure 11] FIG. 1 is a block diagram illustrating an example of a convolutional neural network, in accordance with some examples. [Figure 12] FIG. 1 illustrates an example of a computing system for implementing some aspects described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Some aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the purposes of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting.

[0024] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0025] Machine learning models may be used to process and / or analyze video data. Video understanding using machine deep learning systems can often be challenging. For example, video data includes many interconnected categories of information, such as objects, actions, events, attributes, concepts, and scenes. These categories may be highly interconnected, which may make it difficult for machine learning models to perform some functions. For example, machine learning models may not be able to perform action classification without recognizing objects, because actions often involve objects. As an example, machine learning models may be used to distinguish between playing a piano versus playing a guitar. The playing motions vary depending on the type of object (e.g., playing a guitar versus hitting piano keys), indicating that actions and objects are interconnected. For any particular video, one or more of the categories associated with the video may be more important than others for understanding the video content. In one example, in a video showing a static scene, recognizing objects may be more important than detecting actions. In another example, in a video showing a sporting event with a moving camera following the action, it may be equally important to recognize the action as well as to recognize objects as they move.

[0026] A machine learning model may have prior knowledge of important categories present in a video dataset that the machine learning model will process. Based on the prior knowledge, the machine learning model may be specifically designed to efficiently process that dataset. For example, a three-dimensional (3D) convolutional neural network (CNN) that analyzes videos across two spatial dimensions and a time dimension may be optimized using such prior knowledge to reduce computational costs while detecting actions in various types of videos with dynamic content (e.g., videos of sports matches, concerts, etc.). However, processing videos showing static scenes through a 3D CNN may incur significant computational costs (e.g., many floating-point operations (FLOPs)) that may not be necessary in light of the content. This additional computational cost may result from the 3D convolutional network performing some functions (e.g., convolutions) across multiple frames simultaneously, which may not be needed when objects in the scene are not moving between frames.

[0027] Two-dimensional (2D) CNNs can recognize objects in a video based on a single video frame, rather than performing functions across multiple frames simultaneously, and as a result can be more efficient at identifying objects in one or more video frames. However, because 2D CNNs do not operate on multiple frames captured at different times, 2D CNNs do not return any temporal information that indicates how objects in a scene move over time. For example, 2D CNNs, by themselves, may not be suitable for detecting actions (e.g., that occur across multiple video frames).

[0028] To overcome the limitation that 2D CNN only produces spatial information for individual frames, various approaches can be used. In one example, one-dimensional (1D) convolution can be used as an auxiliary model to 2D CNN. ID convolution can be used to generate temporal information from features generated by 2D CNN across consecutive frames. In another example, a non-local neural network with forward skip can be used to detect temporal relationships between features generated by 2D CNN across discontinuous frames. In another example, a graph convolution network can be used as an auxiliary model to 2D CNN. A graph convolution network can be used to generate relationships between concepts in a video based on spatial features generated by 2D CNN.

[0029] When the most important categories are known for a particular video dataset, it may be efficient to design a machine learning model (e.g., a neural network) specifically tailored to that particular category. However, relying on a single model to analyze a video dataset that includes a variety of videos with different characteristics and containing different categories of information within them may result in excessive computational effort (e.g., when the model performs calculations that attempt to detect categories that are not present in the sample videos), insufficient accuracy (e.g., when the model is not suitable for detecting one or more of the categories of information present in the sample videos), or both. When a new dataset is encountered that does not fit the model (e.g., because the model was not trained to understand the type of content in the new dataset), a new targeted model may be developed to solve the new problem presented by the new dataset. However, developing a new model may be costly in that new models must always be developed. Developing a new model may also not actually address the problem of holistic video understanding, such as when the newly developed model is designed to solve a narrow problem or to understand videos that contain one or more distinct categories of information.

[0030] Table 1 below shows the relative performance of five different video understanding models expressed in units of mean average precision (mAP). The contents in Table 1 provide an example showing the results of selecting one model over another for a video. In Table 1, the mAP values ​​of five different machine learning models (Models 1-5) are shown for the categories of action, attribute, concept, event, object, and scene. In some examples, machine learning models can apply certain inductive biases, which can be prior assumptions that limit the model's capabilities by restricting the model's search space. In Table 1, the model or models with the highest mAP in each category (e.g., Models 2 and 4 have similar values ​​for the category "events") are underlined. As shown, no single model has the highest accuracy among all of the categories. As a result, a fixed application of any one of the models in Table 1 to a video dataset that contains video data encompassing multiple relevant categories will not achieve optimal accuracy for that video dataset.

[0031] [Table 1]

[0032] In one illustrative example, the first model is HATNet, as described in Ali Diba et al., "Large scale holistic video understanding," 2020, which is incorporated herein by reference in its entirety for all purposes. In another illustrative example, the second, third, and fourth models share a common 2D CNN as a first stage, which is built in a Resnet-50 neural network architecture, such as that described in Kaiming He et al., "Deep residual learning for image recognition," 2016, which is incorporated herein by reference in its entirety for all purposes. The Resnet-50 2D CNN generates spatial features. For each of the second, third, and fourth models, the spatial features are further processed by an auxiliary model. In the second model, a non-local auxiliary model is applied to the features generated by the 2D CNN. In the third model, 1D convolutions may be applied to the features generated by the 2D CNN to generate temporal information from the spatial features (generated by the 2D CNN) evaluated across multiple frames. ID convolution may be applied to the features generated by the 2D CNN across multiple frames to generate temporal information for the features in the time dimension (over time). In a fourth model, graph convolution may be applied to the features generated by the 2D CNN. The application of graph convolution may result in the generation of relational information from the spatial features generated by the 2D CNN to the available classes that the neural network is trained to classify. In one illustrative example, the fifth model may be a 3D CNN using the Slowfast model described in Christoph Feichtenhofer et al., "Slowfast networks for video recognition," 2019.

[0033] Described herein are systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively "systems and techniques") for performing holistic video understanding. As used herein, holistic video understanding refers to understanding the content of a video across multiple interrelated categories of data contained in the video. In some examples, the systems and techniques are described herein as video understanding, analysis, and / or classification. However, the systems and techniques described herein as video understanding, analysis, and / or classification may be applied to any sequence of frames or images.

[0034] As described in more detail below, the holistic video understanding system may include one or more model decision engines (which may be machine learning models or systems), a feature extraction machine learning model (or machine learning system), and an auxiliary machine learning model (or machine learning system). The one or more model decision engines may be trained to classify input video data to determine which feature extraction model to use to process a particular video or portion of a video (e.g., based on a combination of computational efficiency and accuracy of the feature extraction model combined with the auxiliary model available for selection). In one illustrative example, the model decision engine may be trained to select the most accurate model while keeping the average computation per video or portion of a video below 40 GigaFLOPs (GFLOPs). The portion of a video may be a video clip (e.g., a video clip υ described below). ijDuring inference (after one or more model determination engines are trained), the classification output by the model determination engine may be used to determine which one or more feature extraction models will be used in the neural network to process the input video or video portion. In some cases, the model determination engine may also be trained to learn features associated with the input video or frames of the input video. In some examples, the model determination engine may be trained using video frames and labels as training data, such as using one or more supervised or semi-supervised training techniques. During inference (e.g., after the model determination engine is trained to extract features), the model determination engine may determine that features it generated during the classification process should be provided to an auxiliary model and subsequently used to classify the input video without requiring the use of any of the other available feature extraction models.

[0035] In some examples, the model decision engine is a neural network, or a combination of multiple neural network architectures. In some examples, the model decision engine is a 2D CNN, or any other suitable lightweight neural network architecture. As mentioned above, 2D CNN is lightweight because it can recognize objects in a video based on a single video frame, rather than simultaneously performing functions (e.g., convolutions) across multiple frames. Using a lightweight neural network architecture for the model decision engine can reduce the computational complexity introduced by utilizing the model decision engine in the holistic video understanding system. Keeping the computational cost of including the model decision engine to a minimum (e.g., by using a lightweight neural network architecture) can lead to an overall increase in computational efficiency due to the computational savings of being able to choose a lightweight feature generation model for some input videos. In one illustrative example, the neural network of the model decision engine includes a MobileNetV2 neural network architecture, such as that described in Mark Sandler et al., "MobileNetV2: Inverted Residuals and Linear Bottlenecks," 2019, which is incorporated by reference in its entirety for all purposes. In another illustrative example, the neural network of the model decision engine includes a Resnet-50 neural network architecture, such as that described in Kaiming He et al., “Deep residual learning for image recognition,” 2016, which is incorporated by reference in its entirety and for all purposes.

[0036] In some examples, as described below, the model decision engine may be trained together with the feature extraction model and any auxiliary models in an end-to-end process. In some examples, as described below, the model decision engine may be trained in the final stage of a three-stage process. For example, the three-stage process may include a first step of training the feature extraction model until a certain accuracy level is achieved, a second step of training the auxiliary model using features from the trained feature extraction model with frozen weights (e.g., when the weights of the trained feature extraction model do not change), and a third step of training the model decision engine using the trained feature extraction model and the auxiliary model with frozen weights. In some cases, the decision engine is trained using supervised learning with pseudo labels that tell the decision engine which model is most accurate. During the training iterations, the decision engine attempts to maximize the accuracy of choosing the most accurate model, and parameters (e.g., weights, biases, etc.) are adjusted in each iteration based on the determined loss.

[0037] In one illustrative example, as described in more detail below, the second step of the three-step process is the cross-entropy loss (L CE ) or the binary cross-entropy loss (L BCE (denoted as L, L) and the third step may include training a model decision engine using the following loss: L=L π +αL uniform +βL flops training a model decision engine using L π is the cross entropy (CE) loss (or softmax loss), and L uniform is the uniform loss, and L flops is the FLOP loss. BCE loss (L BCE) trains the decision engine to maximize the accuracy of obtaining the correct classification based on binary cross entropy. flops ) trains the decision engine to select efficient models more frequently, since it is beneficial to minimize the expected (e.g., average over the training data) FLOPs that will be used. The FLOPs that will be used are calculated based on the number of decisions a from the decision engine. V The beta (β) parameter controls or adjusts the tradeoff between accuracy and complexity or computational efficiency (e.g., as measured by FLOPs). For example, if β is set to 0, the result is that the decision engine will learn to use the most accurate model. Setting the β parameter to a high value (e.g., close to 1, such as 0.75, 0.80, 0.85, etc.) will result in the decision engine selecting and preferring the model with the lowest computational cost. The uniform loss (L uniform ) can be used to ensure that the model decision engine does not always select the same model that results in a certain trade-off. The alpha (α) parameter is used to adjust the uniform loss. An example of an optimal balance is by selecting the most accurate model that averages under 40 GFLOPS.

[0038] The systems and techniques described herein can provide more accurate and efficient holistic video understanding by adaptively selecting between one or more machine learning models for processing video data. For example, as described above, many video understanding systems apply a single fixed machine learning model to analyze all videos, regardless of whether the categories of data contained in a particular input video are suitable for analysis by the fixed machine learning model. By adaptively selecting between multiple different machine learning models to perform video understanding based on the characteristics of the particular video or portion of a video being analyzed, the systems and techniques can analyze each video or portion of a video with a model that is suitable for the characteristics of the particular video or portion of a video (e.g., the categories of data contained within the video or portion of a video). Furthermore, by avoiding unnecessary calculation of computationally expensive models that are preferred for categories not present in the video or portion of a video, the computing and power resources of the device performing the video understanding techniques can be conserved for other operations.

[0039] Although examples are described herein for classifying videos into specific classifications (e.g., playing piano, kids playing soccer, etc.), the holistic video understanding systems and techniques described herein may be used to generate other classifications not specifically listed. In some examples, the holistic video understanding systems and techniques described herein may be used to determine activity of people or objects in other categories and classifications, such as a person sitting at a desk in front of a computer, a person holding a mobile device, a robotic device in a factory environment, and / or any other actions, events, attributes, concepts, and / or scenes.

[0040] Various aspects of the techniques described herein are described below with reference to the figures. Figure 1 is a block diagram illustrating an example of a holistic video understanding system 100. The holistic video understanding system 100 includes various components used to process video data (e.g., one or more videos) and detect or recognize categories of information (e.g., objects, actions, events, attributes, concepts, and scenes). As shown, the components of the holistic video understanding system 100 include a video retriever 102, a storage device 104, a model decision engine 106, a feature extraction model 108, an auxiliary model 110, and an output classifier 112.

[0041] The holistic video understanding system may include or be part of a mobile or fixed telephone handset (e.g., a smartphone, a cellular telephone, etc.), a server computer (e.g., in communication with a vehicle computing system), a vehicle (e.g., a driver monitoring system (DMS) in the vehicle), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video streaming device, or any other suitable electronic device. In some examples, the holistic video understanding system 100 may include one or more wireless transceivers (or separate wireless receivers and transmitters) for wireless communication, such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, Bluetooth or other short-range communication, any combination thereof, and / or other communication. In some implementations, the components of the holistic video understanding system 100 (e.g., the video retriever 102, the storage device 104, the model determination engine 106, the feature extraction model 108, the auxiliary model 110, and the output classifier 112) may be part of the same computing device. In some implementations, the components of the holistic video understanding system 100 may be part of two or more separate computing devices. In some cases, the holistic video understanding system 100 may be implemented as part of a computing system 1200 shown in FIG. 12.

[0042] Although holistic video understanding system 100 is shown to include several components, one skilled in the art will appreciate that holistic video understanding system 100 may include more or fewer components than those shown in FIG. 1. In some cases, the additional components of holistic video understanding system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some cases, holistic video understanding system 100 may include one or more sensors (e.g., one or more cameras, inertial measurement units (IMUs), radar, light detection and ranging (LIDAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or one or more other software and / or hardware components not shown in FIG. 1. In some implementations, additional components of holistic video understanding system 100 may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., digital signal processors (DSPs), microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), any combination thereof, and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing holistic video understanding system 100.

[0043] The video retriever 102 may retrieve frames (e.g., video frames, images, etc.) from the storage device 104. The storage device 104 may include any suitable type of storage (e.g., any type of storage described with respect to storage device 1230 in FIG. 12 below). The video retrieved by the video retriever 102 may include a sequence of frames. For example, the sequence of frames may include a group of consecutively captured images or other sequences of frames. In one illustrative example, the frames may include red-green-blue (RGB) images, luma, chroma-blue, chroma-red (YCbCr or Y′CbCr) images, monochrome images, and / or any other suitable type of image. In some cases, the video retrieved by the video retriever 102 may also receive an audio component from the storage device 104.

[0044] In some aspects, the video retriever 102 may partition the video retrieved from the storage device 104 into evenly spaced portions or clips (e.g., each clip may contain an equal number of frames). ijAn example of is described below. In some examples, the video retriever may also partition the audio data into individual audio segments such that the audio data matches the time associated with a corresponding sequence of frames for a portion of the video. In some aspects, the video may be stored as video portions or clips in the storage device 104, and the video retriever may retrieve clips from the storage device 104 rather than retrieving the entire video. In some aspects, the video retriever 102 may retrieve the complete video from the storage device 104. In such aspects, the complete video may be divided into portions or clips elsewhere in the holistic video understanding system 100 (e.g., before being input to the model determination engine 106). In some aspects, a portion may include a subset of frames of a video, such as two frames, three frames, five frames, or any suitable number of frames. In some examples, the portions may overlap (e.g., the portions may share a common frame with each other from a sequence of frames). In one illustrative example where each video portion includes three frames, a first portion of a first video may include a first, second, and third frame of the video, while a second portion of the first video may include a second, third, and fourth frame of the video.

[0045] The frames retrieved by the video retriever 102 may be provided as input to the model decision engine 106. In some examples, the retrieved frames of video may have dimensions larger than the dimensions of an input layer (not shown) of the model decision engine 106. In such cases, each input frame having a larger dimension may be normalized to a resolution that matches the input dimensions or a resolution that the model decision engine 106 is configured to process. For example, the model decision engine 106 (or other components of the holistic video understanding system 100) may normalize each frame having a larger dimension to the input dimensions or resolution of the model decision engine 106 by downsampling or downscaling each frame. In one illustrative example, a video portion may be resized to match the input resolution of 224×224 (that the model decision engine 106 is configured to process) by resizing the smallest dimension of the retrieved video frame to 224 and extracting a center crop (e.g., a center portion of the video frame). In some examples, the downsampled, downsized, or otherwise resized frames may be stored in the storage device 104 for use by subsequent stages of the holistic video understanding system 100.

[0046] In some examples, the model determination engine 106 may be trained to determine which one or more models from the set of feature extraction models 108 to use to process an input video or a portion of an input video. In some cases, the model determination engine 106 may be trained to determine which one or more models to use based on a combination of computational efficiency and accuracy of the feature extraction models 108 in combination with the auxiliary models 110 that are also available for selection by the model determination engine 106. For example, the model determination engine 106 may be trained to choose between a first machine learning model and a second machine learning model (from the feature extraction models 108 and / or the auxiliary models 110) to process an input video. In some examples, the model determination engine 106 may be trained to determine which set of “N” number of feature extraction models 108 to use to process an input video, where “N” is an integer equal to or greater than one. The number of feature extraction models 108 may be made to include as many machine learning models as needed for a given application. Some exemplary feature extraction models 108 may include 2D CNN, 3D CNN, audio processing models, optical flow networks, and any other neural network architectures that may be used to process video data. In some examples, the model determination engine 106 may include a classification neural network trained to be able to classify N output classes equal to the number of feature extraction models 108. For example, the model determination engine 106 may process video data and determine probabilities for N classes based on processing the video data. The model determination engine 106 may select a given model (from the feature extraction models 108) that is associated with a class that has the highest probability value among the N classes. In some examples, the neural network of the model determination engine 106 may include an inner layer that generates one or more feature vectors from the input video frames and a fully connected layer that synthesizes features of the one or more feature vectors into N classes. An example of training the model determination engine 106 is provided below.

[0047] During inference (after the model determination engine 106 is trained), the model determination engine 106 may receive one or more frames of video representing an input video or video portion from the video retriever 102 and / or from the storage device 104. In some examples, the one or more frames of video representing an input video or video portion received by the model determination engine 106 from the video retriever 102 and / or the storage device 104 are portions of the input video as described above. In some examples, the model determination engine 106 may generate a classification of the input video or video portion (e.g., a class with the highest probability among the N classes). In some cases, the model determination engine 106 may process each frame of the input video or video portion in sequence to generate a classification for each input video, video portion, or video frame. In some implementations, the model determination engine 106 can generate a classification from multiple consecutive frames of an input video or video portion (e.g., by averaging features from multiple consecutive frames and processing the averaged features using a multi-layer perceptron (MLP), fully connected layer, or other neural network component to generate a classification).

[0048] In some examples, the classification generated by the model determination engine may be used to determine which one of the feature extraction models 108 to use to further process the input video (e.g., the entire input video). In some examples, the classification may be used to determine which model from the feature extraction models 108 to use to further process a portion of the input video (e.g., one or more frames of the input video), where each portion of the video may receive a classification separate from other portions of the video. As an example, the first 90 frames of a video may include a static scene, and action may begin within the video during the last 150 frames of the video. In such an example, the model determination engine 106 may determine to use a 2D CNN to process the first 90 frames (based on the one or more classifications determined by the model determination engine 106) and to use a 3D CNN to process the last 150 frames. In some examples, the classifications output by the model determination engine 106 may be stored in the storage device 104.

[0049] In some examples, in addition to generating a classification, the model determination engine 106 can also generate features based on the input video or video portion. For example, the model determination engine 106 may include a 2D CNN that generates features from the input video. The features may later be input to a layer (e.g., a fully connected layer or other layer) of the model determination engine 106 to generate a classification (e.g., indicating which feature extraction model 108 should be used to process the input video or portion thereof, as described above). In some examples, the features generated within the model determination engine 106 may also be stored in the storage device 104. The features generated by the model determination engine 106 may be used as input by one or more of the auxiliary models 110 for additional processing or may be used directly in the output classifier 112 to classify the video. In some examples, the features generated by the model determination engine 106 may be stored in the storage device 104.

[0050] The feature extraction models 108 may be trained to learn features associated with an input video or video portion. In some examples, the feature extraction models 108 may include at least two feature extraction models from which the model determination engine 106 may select. In some examples, the feature extraction models 108 may each have a different neural network backbone that uses a different neural network architecture to extract features from the video data. For example, one feature extraction model from the feature extraction models 108 may include a 2D CNN that may be trained to extract spatial features from individual video frames (or images that are not part of a video). Another feature extraction model from the feature extraction models 108 may include a 3D CNN that may be trained to extract both spatial and temporal features from a sequence of frames. In some examples, the 3D CNN may be trained to generate spatial and temporal features by performing convolutions over a portion of the video that includes three frames, five frames, or any other number of frames that provides a desired tradeoff between temporal resolution and computational cost for the features generated by the 3D CNN. In some cases, another feature extraction model from the feature extraction models 108 may be an audio model that may be trained to extract features from audio information associated with the input video. Another feature extraction model from the feature extraction models 108 may include an optical flow model that may be trained to detect the movement of objects or features (e.g., edges, shapes) across multiple video frames. In some examples, two or more feature extraction models from the feature extraction models 108 may use similar modalities (e.g., 2D CNN, 3D CNN, or other CNNs) but operate with different architectures. Different architectures may have different computational costs. For example, two of the feature extraction models 108 may both include 2D CNN architectures, where a first of the 2D CNNs has lower accuracy in generating spatial features relative to the second 2D CNN, but also has lower computational costs.The feature extraction model 108 may be used to extract features that may later be used to classify the input video. An example of training the feature extraction model 108 is provided below.

[0051] During inference (after the feature extraction model 108 is trained), the selected feature extraction model 108 (e.g., selected by the model determination engine 106) may receive one or more frames from the video retriever 102, the storage device 104, and / or the model determination engine 106. The selected feature extraction model 108 may be activated based on the classification generated by the model determination engine 106, as described above. In some examples, the model determination engine 106 may directly activate the selected feature extraction model 108 (e.g., by sending a command or signal instructing the selected feature extraction model 108 to start). In some examples, the feature extraction model 108 may retrieve the classification output from the model determination engine 106 (e.g., directly from the model determination engine 106, from the storage device 104, etc.). The features extracted by the selected feature extraction model 108 may be provided as input to the auxiliary model 110.

[0052] Depending on the type of model selected by the model determination engine 106, the selected feature extraction model 108 may process each frame of the one or more frames sequentially (e.g., one frame at a time) or multiple frames simultaneously. For example, when a 2D CNN is selected from the feature extraction model 108, the 2D CNN may extract spatial features from each frame (e.g., based on processing the frames individually). In another example, when a 3D CNN is selected from the feature extraction model 108, the 3D CNN may extract spatial and temporal features from multiple consecutive frames (e.g., based on processing multiple frames simultaneously). In some cases, selecting a 2D CNN may be preferable. For example, a 2D CNN may have a relatively low computational cost (e.g., measured by the number of floating point operations (FLOPs) required to generate features using the model) compared to a 3D CNN or other models. In some examples, the ratio of computational costs between 2D CNN and 3D CNN may be a function of the number K of video frames simultaneously processed by the 3D CNN, where K may be equal to the depth of the 3D CNN kernel in the time dimension. In some cases, selecting a 3D CNN may be preferable. For example, a 3D CNN may provide a strong temporal bias for understanding actions within a particular video or portion of a video. In one example, a 2D CNN may be selected by the model determination engine 106 when the video or portion of a video being analyzed shows a static scene (e.g., from a wall-mounted camera) with little or no motion within the scene. In such an example, the model determination engine 106 may determine that the characteristics of the video (e.g., a static scene) are such that a 2D CNN should be used for the video. In another example, a 3D CNN may be selected by the model determination engine 106 when the video or portion of a video being analyzed shows a dynamic scene (e.g., a scene captured by a static or moving camera that includes moving objects, such as a sporting event).In such an example, the model determination engine 106 can determine that the characteristics of the video (e.g., motion within a scene) are such that a 3D CNN should be used for the video. By extracting features from video data across multiple frames simultaneously, 3D CNN's ability to detect actions can produce a more accurate understanding of the video content compared to 2D CNNs that can only extract features from one frame at a time.

[0053] In some cases, the output of the feature extraction model from the feature extraction model 108 may include a feature vector (or other feature representation) representing features extracted from each frame or from a sequence of frames. In some cases, a single feature vector may be extracted from one input frame, with the feature vector representing the features of that input frame (e.g., in the case of a 2D CNN that generates spatial features per frame). The features extracted from the input frames (e.g., represented as a feature vector) provide a representation of the frame that includes the information contained within the input frames. In some cases, a single feature vector may be extracted from multiple input frames, with the feature vector containing all information associated with the features of the multiple input frames (e.g., in the case of a 3D CNN or optical flow network that generates spatial and temporal features across multiple frames). In one illustrative example, the feature vector generated by the feature extraction model for each frame or for multiple frames may be a 1×2048 vector (indicating that the feature vector is a one-dimensional feature vector with a length of 2048 values). In some examples, the storage device 104 may be used to store features extracted from the input video or video portions by the feature extraction model 108 (e.g., by storing an extracted feature vector for each frame or group of frames). In some cases, feature representations other than feature vectors may be used, such as tensors or other representations.

[0054] The auxiliary model 110 may be trained to supplement the features extracted by the feature extraction model 108 with additional information that may be used to classify the video. During inference (e.g., after the auxiliary model 110 is trained), the auxiliary model 110 may receive the features output by the feature extraction model 108 and provide further processing. In some examples, a fully connected layer (not shown) may be included between the feature extraction model 108 and the auxiliary model 110. In one example, as described above, a feature extraction model utilizing a 2D CNN may have a strong spatial bias in the features it generates, but may not include any temporal information across multiple frames, because the 2D CNN only operates on a single frame of a video at a time. In some examples, an auxiliary model from the auxiliary model 110 may be applied to the features generated by the 2D CNN for multiple frames to generate temporal information across multiple frames. In one example, the auxiliary model may include a 1D convolutional neural network that may be applied to the features generated by the 2D CNN (selected from the feature extraction model 108) for multiple frames to generate temporal information for the multiple frames. An illustrative example of a 1D convolutional neural network (as an example of an auxiliary model) is described below with respect to FIGS. 2A-2D.

[0055] 2A-2C illustrate an exemplary convolution operation that may be used as an auxiliary model 110 applied to features generated by feature extraction model 108 in the holistic video understanding system 100 of FIG. 1. FIGS. 2A-2C illustrate composite feature vectors, including composite feature vector 210 (in FIG. 2A), composite feature vector 212 (in FIG. 2B), and composite feature vector 214 (in FIG. 2C), which include a combination of five 1×2048 feature vectors from five consecutive frames output by a 2D CNN feature extraction model, as described above with respect to FIG. 1. The resulting composite feature vectors 210, 212, and 214 each have a dimension of 5×2048 (indicating that each composite feature vector 210, 212, and 214 includes features from five consecutive frames output from the 2D CNN). In the illustrated example of Figures 2A-2C, each cube (whether black or white in color) in the composite feature vector 210, 212, and 214 represents one value of the composite feature vector. The black shading in the cubes of the composite feature vectors 210, 212, 214, and 216 of Figures 2A-2D indicates different convolution filters that may be applied to the composite feature vector (where the convolution filters move across the data over time). In the example of Figure 2A, the selected convolution filter 211 is a 5x3 kernel that spans all five values ​​in the time dimension and across the three channels (in the direction indicated by arrow 221). In the example of Figure 2A, the kernel moves along the channel dimension (as indicated by arrow 221) to perform the convolution.

[0056] In the example of FIG. 2B, the selected convolution filter 213 is a 3×1 kernel that spans three values ​​in the time dimension and one value in the channel dimension. The 3×1 kernel moves along both the time dimension and the channel dimension (as indicated by arrow 223 in the channel dimension and arrow 225 in the time dimension). In some cases, the 3×1 kernel first moves in the time dimension for each channel one “column” at a time (corresponding to a stride value of 1) to generate three new convolution output values ​​for the first channel. In some cases, the 3×1 kernel then moves to the next “row” (corresponding to a stride value of 1) in the channel dimension of the composite feature vector. The 3×1 kernel can then repeat the three convolutions moving in the time dimension to generate three additional new convolution output values ​​for the second channel. In some examples, this process can be repeated for each “row” in the channel dimension to generate a 3×2048 convolution output. In some cases, additional neural network layers may be used to generate 1×2048 outputs from the 3×2048 convolution outputs. In one illustrative example, an additional convolution (e.g., using a 3×1 kernel) moving along the channel dimension one “row” at a time (corresponding to a stride value of 1) may be applied to the 3×2048 convolution output to produce a 1×2048 composite feature vector. The convolution filter shown in FIG. 2B may be considered a 2D convolution because the kernel moves along both the time and channel dimensions.

[0057] In the example of FIG. 2C, the selected convolution filter 215 is a 3×2048 filter kernel that spans three values ​​in the time dimension and all 2048 values ​​in the channel dimension. The first 3×2048 kernel moves along the time dimension (as indicated by arrow 227), one cube at a time, two cubes at a time, etc. In one illustrative example, the first 3×2048 kernel creates a first convolution output value based on the convolution of 3×2048 features at a first position. In such an example, the first 3×2048 kernel may move in the time dimension one “column” at a time, generating two additional values ​​for a total of 3×1 convolution output values. In such an example, this process may be repeated with 2047 additional 3×2048 kernels to generate a 3×2048 composite convolution output vector. In some examples, additional neural network layers (e.g., convolutional layers) may be used to generate a 1×2048 output from the 3×2048 convolution output. In some implementations, the convolutional filter 215 described in FIG. 2C may be used to capture recurrent patterns from the composite feature vector 214 that exist in the time dimension of the data. The convolution shown in FIG. 2C may be considered a 1D convolution because each kernel moves only along the time dimension.

[0058] FIG. 2D illustrates an example of the output of the 1D convolution shown in FIG. 2A. The convolution filter 211 (indicated by the black cube in the composite feature vector 210) may be used to generate a new feature vector 216 of dimension 1×2048. As shown, the first element of the new feature vector 216 (indicated by the black cube 217) is generated based on a 5×3 convolution filter kernel applied to the first three “rows” (in the time dimension) of the composite feature vector 210. Similarly, the second element of the new feature vector 216 may be generated by stepping the 5×3 convolution filter by one row (e.g., along the direction of arrow 221) and applying the convolution to each element of the composite feature vector 210. As the filter kernel moves in the channel dimension (as indicated by arrow 221), each subsequent element of the new feature vector 216 may be generated based on applying a 5×3 convolution filter. The resulting new feature vector 216 may have dimensions 1×2048, as described above, where each element of the new feature vector 216 represents the result of a convolution filter applied to the composite feature vector 210. Using this technique to composite features from multiple frames, temporal information for the features may be generated even though the original features were extracted from individual frames without any temporal information. Such a technique offers the advantage of including temporal information for a more holistic understanding of the input video, without incurring the full computational cost of performing 3D convolution.

[0059] While the above examples with respect to Figures 2A-2D are described with respect to a composite feature vector being output from a 2D CNN, it should be understood that the convolutional filters described in Figures 2A-2D may be applied to any feature vector, regardless of the type of neural network from which the feature vector originates. In addition, while the examples of Figures 2A-2C show a composite feature vector having a dimension of 5x2048, one skilled in the art will appreciate that the filters described may be applied to feature vectors having different dimensions. For example, a composite feature vector representing only three frames of feature data may have a dimension of 3x2048. In another example, a composite feature vector representing five frames of feature data may have fewer features, such as 1024 features, in which case the composite feature vector may have a dimension of 5x1024.

[0060] Returning to FIG. 1, an additional auxiliary model 110 may be provided to further process the features from the feature extraction model 108, as described above. In some examples, the auxiliary model (from the auxiliary model 110) applied to the features generated by the feature extraction model 108 may not apply additional inductive bias to the output of the feature extraction model 108. For example, as described above with respect to FIG. 2D, a 1D convolutional auxiliary model (selected from the auxiliary model 110) may be used to generate temporal information for a video based on the video features extracted by the 2D CNN feature extraction model (from the feature extraction model 108) when the video does not contain motion (e.g., a static scene) or contains a small amount of motion that is not sufficient to benefit from the increased computational complexity of a 3D CNN. As an example, a 1D convolutional auxiliary model may be selected when the input video contains a static scene, such as a scene from an IP camera aimed at a fixed location, where there may be occasional movement of objects or people within the scene to analyze. Such an approach can reduce computational complexity because temporal information can be processed from spatial features of video frames extracted by 2D CNN, rather than simultaneously extracting spatial and temporal information directly from video frames (e.g., using 3D CNN), which may require processing significantly more data.

[0061] Another example of an auxiliary model is a non-local model (also called an attention model or a non-local attention model). A non-local model can extract temporal information from features extracted by a 2D CNN feature extraction model by comparing features from each video frame with features from one or more other video frames (e.g., every other video frame) in the set of video frames. For example, for a set of five video frames, a first frame feature can be compared separately with features from each of the second, third, fourth, and fifth frames to determine whether events or actions in far-away frames (e.g., the first and fifth frames, the first and fourth frames, the second and fifth frames, the second and fourth frames, etc.) are related. For a large number of frames, the number of comparisons by a non-local auxiliary model can be large. The benefit of a non-local model is that it can detect relationships that are not detected by models that only analyze information from closely grouped frames (e.g., consecutive frames). For example, in contrast to 1D convolutional auxiliary models, the inductive bias of non-local models does not necessarily assume that related actions will be grouped together in time. By attempting to determine these types of non-local relationships in a video, the holistic video understanding system 100 can be more holistic based on its ability to analyze characteristics across a broader range (e.g., across more frames that are separated in time) in the video. One illustrative example in which the non-local auxiliary model 110 may be useful is a video of dominoes falling, where the effect of a domino falling in the first frame may not be seen in the last domino until many frames later. The relationship between the first domino falling and the last domino falling may not be detected by a 3D CNN or 2D CNN supplemented with 1D convolutions, but may be detected by a 2D CNN supplemented with a non-local model.

[0062] Another exemplary auxiliary model that may be applied to the features extracted by the 2D CNN feature extraction model 108 is a graph convolutional network. In some examples, the graph convolutional network may be used as an auxiliary model (of the auxiliary models 110) to relate features (from the feature extraction model 108) to some concepts, utilizing knowledge of the last classifier to be applied by the classifier 112. In some examples, the graph convolutional network applies a concept or relational bias to the features it receives. In one example, the graph convolutional network may be able to infer that the classification "kids playing soccer" applies to a video or video portion based on the presence of features (e.g., ball, child, grass, etc.) generated by the feature extraction model. In some implementations, this may be done by relating the features to concepts and / or output classifiers or classifications. In some cases, the auxiliary model may not apply any additional inductive bias to the features generated by the feature extraction model. For example, the auxiliary model may serve the purpose of connecting the features generated by the feature extraction model to one of the classes that the output classifier 112 is trained to classify. In one illustrative example where the feature extraction model from feature extraction model 108 is a 3D CNN, auxiliary model 110 may be a multi-layer perceptron (MLP) or a fully connected layer. For example, model determination engine 106 may select a 3D CNN from feature extraction model 108 along with MLP auxiliary model 110 based on the video or video portion containing a large amount of motion (e.g., a video of a sporting event where the camera is also moving). In some examples, the output of auxiliary model 110 may be stored in storage device 104.

[0063] The output classifier 112 may be trained to classify the input video or video portion (or features associated with the input video or video portion) with one or more classifications. A training dataset including exemplary videos and classification labels may be used to train the output classifier 112, such as using techniques described with respect to Figures 10 and 11. During inference (e.g., after the output classifier 112 is trained), the output classifier 112 may receive output from the auxiliary model 110 and may use the output to classify the input video or video portion. Exemplary classifications may include "playing piano," "kids playing soccer," and any other classification that the output classifier 112 is trained to classify.

[0064] In some cases, the system including the model determination engine 106, the feature extraction model 108, the auxiliary model 110, and the output classifier 112 may be trained in a single end-to-end training sequence. In one implementation of the end-to-end training sequence, a Gumbel Softmax layer (also referred to as a Gumbel layer) may facilitate the end-to-end training. The Gumbel layer may enable the model determination engine 106 to make discrete decisions while still being differentiable so that a backpropagation training process may be performed. In some examples, the backpropagation training process may be performed to adjust the weights (and possibly other parameters such as biases) of each node of the neural network (e.g., the model determination engine 106, the feature extraction model 108, the auxiliary model 110, and the output classifier 112). As described in more detail with respect to FIG. 10, the backpropagation may include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter updates may be performed for each training iteration (e.g., for each batch of data from the training set, or for the entire training data set). The backpropagation process may be repeated for each set of training data for a certain number of iterations until the weights of the parameters of the model decision engine 106, the feature extraction model 108, the auxiliary model 110, and the output classifier 112 are precisely tuned. In some examples, the loss function used in the end-to-end training process may include a cross-entropy (CE) loss, a binary cross-entropy (BCE) loss, a FLOP loss, a uniform loss, any combination thereof, or any other suitable loss function. In some examples, the loss function used in the end-to-end training process may include a weighted combination of multiple loss functions (e.g., a combination of BCE loss and FLOP loss with different weightings).

[0065] In some cases, it may be difficult to train the model determination engine 106 using an end-to-end training process with the entire holistic video understanding system 100. For example, the model determination engine 106 may predict which model to select based on past statistics of the feature extraction model 108, the auxiliary model 110, and the classifier 112, which change during each iteration of training as a result of the end-to-end nature of the training. In some examples, rather than training the holistic video understanding system 100 end-to-end, a three-stage training process may be utilized. For example, during a first stage of the three-stage training process, the feature extraction model 108 may be trained. Similar to the end-to-end training process described above, a backpropagation training process may be performed to adjust the weights (and possibly other parameters, such as biases) of each node of the feature extraction model 108. A loss function may be used to analyze the error in the output of the feature extraction model 108 relative to a known output or ground truth (e.g., based on ground truth labels associated with the training data). In one illustrative example, a cross-entropy loss function may be used. In some cases, other loss functions may be used. After training the feature extraction model 108, the weights (and possibly other parameters, such as biases) of the feature extraction model 108 may be fixed (or "frozen").

[0066] In some examples, the auxiliary model 110 may be trained in a second stage of a three-stage training process using features extracted by the trained feature extraction model 108. In some cases, the auxiliary model may be trained using a backpropagation training process similar to the process described for the feature extraction model 108. After training, the weights (and possibly other parameters such as biases) of the auxiliary model 110 may be fixed or frozen. In some examples, the model determination engine 106 may be trained in the last stage of a three-stage training process after the feature extraction model 108 and the auxiliary model 110 are trained. In some cases, during the three-stage training process, the model determination engine 106 may be trained to learn which one or more models among the set of fixed models (e.g., the feature extraction model 108 and the auxiliary model 110) should be selected to process the input video or video portion. In some examples, the same data set may be used for all three training stages of the three-stage training process. Additional details of training a holistic video understanding system, including example loss functions that may be used to train the model determination engine 106, are provided with respect to the description of FIG. 3 below.

[0067] FIG. 3 illustrates another example of a holistic video understanding system 300. As illustrated, components of the holistic video understanding system 300 include a model determination engine 306, video processing models 313A, 313B-313M, and a classifier 312. One or more components of the holistic video understanding system 300 of FIG. 3 may be similar to the components of the holistic video understanding system 100 of FIG. 1 and may perform similar operations as those components. For example, the model determination engine 306 and the classifier 312 may be similar to the model determination engine 106 and the output classifier 112 of FIG. 1 and may perform similar operations. The video processing models 313A, 313B-313M are examples of the feature extraction model 108 of FIG. 1.

[0068] In some examples, the model determination engine 306 may generate a video clip υ from the input video 302 or a video portion (such as a video clip υ ij ) to process the input video 302. In some cases, the model determination engine 306 may be trained to determine which model or models to use based on a combination of computational efficiency and accuracy of the video processing models 313A, 313B-313M in processing the input video 302. In some examples, the model determination engine 306 may include a classification neural network trained to classify N classes equal to the number of video processing models 313A, 313B-313M. For example, the model determination engine 306 may process video data and determine probabilities for the N classes based on processing the video data. The model determination engine 306 may select a video processing model (from the video processing models 313A, 313B-313M) associated with a class having a highest probability value among the N classes. In some examples, the neural network of the model determination engine 306 may include an inner layer that generates one or more feature vectors from the input video 302 or video portion. In some cases, the neural network of the model determination engine 306 may further include a fully connected layer that combines features of one or more feature vectors into N classes.

[0069] During inference (after the model determination engine 306 is trained), the model determination engine 306 can determine which one or more of the video processing models 313A, 313B-313M to use to process the input video 302 or video portions. Referring again to FIG. 1 , in some implementations, the model determination engine 306 can receive the input video 302 or video portions from the video retriever 102 and / or from the storage device 104. In some examples, the input video is divided into J equal-sized portions vij , n=1, n=1, n=2, n=3, n=4, n=5, n=6, n=7, n=8, n=9, n=10, n=11, n=2, n=3, n=4, n=5, n=6, n=11, n=12, n=13, n=2, n=14, n=15, n=2, n=3, n=15, n=16, n=2, n=17, n=18, n=2, n=3, n=19, n=20, n=21, n=32, n=19, n=22, n=33, n=19, n=23, n=34, n=35, n=36, n=37, n=4, n=5, n=6, n=19, n=24, n=17, n=18, n=25, n=39, n=46, n=19, n=25, n=38, n=19, n=26, n=39, n=19, n=27, n=39, n=39, n=47, n=19, n=28, n=39, n=19, n=29, n=30, n=31, n=32, n=29, n=31, n=32, n=33, n=47, n=19, n=28, n=34, n=19, n=29, n=35, n=36, n=37, n=47, n=19, n=29, n=39, n=39, n=47, n=19, n=29, n=39, n=39, n=29, n=39, n=39, n=47, n=39, n=29, n=39, n=39, n=47, n=3 In one illustrative example, for a holistic video understanding system 300 including two video processing models, the classification vector {circumflex over (x)} is generated when the model decision engine 306 determines whether the input video 302 or a video portion v ij It can be shown that a first video processing model is selected from the video processing models 313A, 313B to 313M to process the video.

[0070] In some examples, the model determination engine 306 may select two or more video processing models from the video processing models 313A, 313B-313M to process the input video 302. For example, the model determination engine 306 may select two or more video processing models to process the input video 302 or video portion when there is no clear best-choice model from the video processing models 313A, 313B-313M. In some examples, the outputs of the two or more selected video processing models from the video processing models 313A, 313B-313M may be combined before the final classification of the input video 302 or video portion by the classifier 312. In one illustrative example, the outputs of the two selected models may be combined by averaging. In some cases, the classifier 312 may receive the output of the selected video processing model from the video processing models 313A, 313B-313M (or the combined output from the two or more selected video processing models). Using the output, the classifier 312 may classify the input video 302 or video portion. Exemplary categories include "playing piano," "kids playing soccer," "making sandwiches," and "listening to classical music." Referring to Figure 1, the storage device 104 may be used to store features extracted from frames of the video 302 by a selected video processing model (or models) from the video processing models 313A, 313B-313M.

[0071] In one illustrative example of a training process for the holistic video understanding system 300 of FIG. 3, a dataset D containing V videos may be used for the training process. For dataset D, v ij (which may correspond to input video 302 in FIG. 3) is a video from video V (where this video is i ∈V), a video clip υ ij is used as an example of a portion of the input video 302 that may be processed by the holistic video understanding system 300 (e.g., each video clip vij (includes a part). φ m The term v represents the neural networks (e.g., video processing models 313A, 313B-313M in FIG. 3) that are available for selection by the model decision engine 306. ij For example, the neural network selected from the video processing models 313A, 313B to 313M predicts

[0072]

number

[0073] In some cases, each individual neural network φ m can be trained separately using the cross-entropy (CE) loss as follows:

[0074]

number

[0075] however,

[0076]

number

[0077] is the expectation, and the loss function is ij and the associated label y i , which is iteratively evaluated over the dataset D. In some cases, a binary cross-entropy (BCE) loss function is applied to each individual neural network φ m In some aspects, the BCE loss function may be used to train a neural network φ to apply multiple classification labels to a single video. m can be used to train

[0078] In one illustrative example, all models φm can be trained for 10 epochs with an Adam optimizer with an initial learning rate of 1e-4. An epoch is one pass through the entire dataset through a given model. In such an example, model φ m may be adjusted to 1e-5 and 1e-6 in epochs 2 and 7, respectively. ij The input resolution of is 224×224 after resizing the smallest dimension of the input video 302 to 224 and extracting the center crop as described above. In an illustrative example, during training, each video processing model 313A-313M may be terminated with a classification layer including a dropout layer with (p=0.5) and a linear layer to the classes of a particular dataset.

[0079] As described above, in a three-stage training process, the feature extraction models 108 may be trained individually in a first stage. Once the individual feature extraction models 108 are trained, the auxiliary model 110 may be trained in a second stage using the trained feature extraction models 108 with fixed or frozen weights. Returning to FIG. 3, in some examples, the video processing models 313A, 313B-313M may incorporate both the feature extraction models 108 and the auxiliary model 110, as described in FIG. 1. In such examples, the first training stage may be applied to the feature extraction models, and the second training stage may be applied to the auxiliary model incorporated within the video processing model 313 using the cross-entropy loss described above. In the third stage, the model determination engine 306 may be trained to predict which neural network (e.g., which of the video processing models 313A, 313B-313M) to apply to a particular video clip. The model determination engine 306 may be trained to predict which neural network (e.g., which of the video processing models 313A, 313B-313M) to apply to a particular video clip using the function π(υ ij In some implementations, the model decision engine 306 selects which model to use to process the video or video portion, where the decision engine 306 selection can be expressed mathematically as: ij=π(υ ij The selection of the decision engine 306 a ij (e.g., the prediction from the selected model m)

[0080]

number

[0081] can be represented by:

[0082]

number

[0083] however,

[0084]

number

[0085] is the indicator function, and φ m (υ ij ) is the output classification of model m.

[0086] The ground truth label y for the model decision engine 306 i is a video clip υ from dataset D ij Let us use the neural network φ m The accuracy of the model determination engine 306 may be evaluated by the following cross-entropy (CE) loss function (also called softmax loss):

[0087]

number

[0088] In some examples, argmax m has the largest negative cross-entropy (for the neural network φ mReturns a one-hot representation with value 1 for the neural network (selected from ).

[0089] The above CE loss L π In addition to , additional loss functions may be applied to optimize the decisions made by the model decision engine 306. For example, a uniform loss term may also be used. The uniform loss term instructs the model decision engine 306 to, on average, calculate the loss ratios for each neural network φ among the total number M of available neural networks (e.g., the video processing models 313A, 313B-313M in FIG. 3) equally many times as follows: m Select the desired option.

[0090]

number

[0091] Finally, a loss function that targets the efficiency of the entire system may be used, where the amount of computational complexity represented by the total number of FLOPs may be optimized. An example of such a loss function (called FLOP loss) is:

[0092]

number

[0093] An overall loss function that combines the above three loss functions may be used to train the model decision engine 306. The overall loss function may be mathematically expressed as follows: L=L π +αL uniform +βL flops

[0094] where α and β are weights that can be applied to adjust the total loss function L. For example, increasing β can increase the extent to which the total loss function takes FLOP loss into account, while setting β to 0 will cause the total loss function to ignore FLOP loss.

[0095] FIG. 4 illustrates an example of a holistic video understanding system 400. As illustrated, components of the holistic video understanding system 400 include a model determination engine 406, a feature extraction model 408, an auxiliary model 410, and a classifier 412. In the example of FIG. 4, the feature extraction model includes a feature extraction portion 429 (which may be part of the model determination engine 406 in some cases), a first feature extraction model 431, and a second feature extraction model 433. The auxiliary model 410 includes a first auxiliary model 435, a second auxiliary model 437, and a third auxiliary model 439. The input video 402 processed by the model determination engine 406 may be a complete video or a video portion (e.g., retrieved from the video retriever 102 and / or the storage device 104 of FIG. 1).

[0096] One or more components of the holistic video understanding system 400 of Figure 4 may be similar to components of the holistic video understanding system 100 of Figure 1 and / or any of the other examples of holistic video understanding systems described herein and may perform similar operations as those components. For example, the auxiliary model 410 and the classifier 412 may be similar to the auxiliary model 110 and the output classifier 112 of Figure 1 and may perform similar operations as those components. The feature extraction model 408 of Figure 4 may be similar to components of the holistic video understanding system of Figure 1 and may perform similar operations as those components, except as described below.

[0097] Similar to what has been described above, the model determination engine 406 may be trained to determine which feature extraction model from the feature extraction models 408 to use to process the input video 402 or video portion. The model determination engine 406 may also be trained to extract features from the input video 402 or video portion. During inference (e.g., after the model determination engine 406 is trained), the model determination engine may extract features from the input video 402 or video portion as a step in the classification process to determine which feature extraction model from the feature extraction models 408 to use to process the input video 402 or video portion. In some examples, the model determination engine 406 may determine that the features generated by a given model from the feature extraction models 408 provide the best desired outcome (e.g., the best combination of accuracy and computational complexity) when compared to features from other models of the feature extraction models 408. As mentioned above, in some implementations, the feature extraction portion 429 may be part of the model determination engine 406 and may be considered one of the feature extraction models 408. As shown in FIG. 4, each feature extraction model 408, including the feature extraction portion 429 of the model determination engine 406, can be associated with an auxiliary model 410, as described above with respect to FIG.

[0098] In one illustrative example of the holistic video understanding system 400, the model decision engine 406 includes a MobilenetV2 neural network, the model 431 includes a Resnet-50 neural network, and the model 433 includes a Slowfast neural network. In such an illustrative example, the auxiliary model 435 and the auxiliary model 437 may each include a 1D convolutional network (e.g., the 1D convolutional network described above with respect to FIG. 2A and FIG. 2D). Continuing with the same example, the auxiliary model 439 may include an MLP. The MobileNetV2 network and the Resnet-50 network are 2D CNNs that extract spatial features from individual video frames. The spatial features from the MobileNetV2 network and the Resnet-50 network may be supplemented with temporal information from the auxiliary 1D convolutional network. The Slowfast network includes a 3D CNN that extracts features that include both spatial and temporal information in the 3D convolution of multiple video frames. In such an example, the MobileNetV2 network (model decision engine 406) may be the least computationally intensive, the Resnet-50 network (model 431) may be more computationally intensive than MobileNetV2 but less computationally intensive than Slowfast, and Slowfast (model 433) may be the most computationally intensive. Examples of situations in which the model decision engine 406 may select each of the available feature extraction models (e.g., feature extraction portion 429, and first feature extraction model 431 and second feature extraction model 433) (e.g., based on the type of content within the input video 402 or portion thereof) are described above with respect to FIG.

[0099] FIG. 5 illustrates a holistic video understanding system 500. As illustrated, components of the holistic video understanding system 500 include a model determination engine 506, a feature extraction model 508, an auxiliary model 510, a synthesizer 511, and a classifier 512. One or more components of the holistic video understanding system 500 of FIG. 5 may be similar to components of the holistic video understanding system 100 of FIG. 1 and / or any of the other examples of the holistic video understanding system described herein and may perform similar operations as those components. For example, the feature extraction model 508, the auxiliary model 510, and the classifier 512 may be similar to the feature extraction model 108, the auxiliary model 110, and the output classifier 112 of FIG. 1 and may perform similar operations. An input video 502 may be processed by the model determination engine 506. Referring to FIG. 1 as an example, the input video 502 may be retrieved from the video retriever 102 or from the storage device 104. As shown in Figure 4, the exemplary holistic video understanding system 400 includes a one-to-one (1:1) relationship between the feature extraction models 408 and the auxiliary models 410. The holistic video understanding system 500 in Figure 5 illustrates a configuration in which each feature extraction model in the feature extraction models 508 may have a one-to-many (1:many) or many-to-many (many:many) relationship with the auxiliary models 510.

[0100] In some examples, the model determination engine 506 may be trained to determine which feature extraction model 508 will be used to process the input video 502 or video portion. The model determination engine 506 may also be trained to determine which auxiliary model from the auxiliary models 510 to apply to features generated by the feature extraction model (selected from the feature extraction model 508) to achieve a target objective. One example of a target objective is to achieve an optimal balance between efficiency and accuracy. In some examples, each pairing of a selected feature extraction model (from the feature extraction model 508) with a selected auxiliary model (from the auxiliary models 510) may be similar to one of the video processing models 313A, 313B-313M of FIG. 3.

[0101] The holistic video understanding system 500 may be trained using the three-step training process described above. For example, the model determination engine 506 may be trained in a third step after the feature extraction model 508 and the auxiliary model 510 are trained (with fixed or frozen weights after training) as described above. During inference (e.g., after the model determination engine 506 is trained), the model determination engine 506 determines which feature extraction model from the feature extraction model 508 and which auxiliary model from the auxiliary model 510 will be used to process the input video 502 or a portion of the input video 502 (e.g., clip υ of the input video 502). ij ) can be determined based on the

[0102] In some implementations, the model determination engine 506 can output a classification that enables the selection of a single feature extraction model 508 paired with one or more auxiliary models from the auxiliary models 510. In one example, the model determination engine can select O (where O is one or more, two or more, or other number) auxiliary models to process the output of the feature extraction model from the feature extraction model 508. In such an example, each of the O auxiliary models from the auxiliary models 510 can receive the features extracted by the selected feature extraction model from the feature extraction model 508. Each auxiliary model among the O auxiliary models can process the features and produce an output, resulting in O total outputs (e.g., feature vectors) from the O auxiliary models. In some examples, the combiner 511 can receive the O outputs and combine the O outputs into a single output (e.g., a single feature vector). In an illustrative example, the combiner 511 can average the O outputs and combine the O outputs into a single output that can be provided to the classifier 512 to classify the input video 502 or video portion. In one illustrative example, the combiner 511 can concatenate the O outputs to combine the O outputs into a single output, which can then be provided to the classifier 512 to classify the input video 502 or video portion.

[0103] 5 highlights an exemplary case in which model determination engine 506 selects a single feature extraction model (e.g., feature extraction model 541) from feature extraction models 508. Features from selected feature extraction model 541 are received by two selected auxiliary models (e.g., auxiliary model 543 and auxiliary model 545) from auxiliary models 510. Further, in such an illustrative example, output 547 of selected auxiliary model 543 and output 549 of selected auxiliary model 545 are received by combiner 511. As discussed above, combiner 511 can combine the two outputs 547 and 549 (e.g., by averaging the values ​​of the feature vectors associated with outputs 547 and 549, by concatenating the values ​​of the feature vectors associated with output 547 and output 549, or by otherwise combining outputs 547 and 549).

[0104] In some examples, the model decision engine 506 may output a one-hot classification (e.g., selecting a single feature extraction model 508 paired with a single auxiliary model 510) for processing the input video 502. In some implementations, when only one auxiliary model from the auxiliary models 510 produces an output, the combiner 511 may pass the output directly (without modifying the output) to the classifier 512 for classification.

[0105] FIG. 6 illustrates another example of a holistic video understanding system 600. As shown, components of the holistic video understanding system 600 include a model determination engine 606, a feature extraction model, an auxiliary model 610, a synthesizer 611, a classifier 612, and a second auxiliary model determination engine 651. As shown, the feature extraction model includes a first feature extraction model 608A and a second feature extraction model 608B. The auxiliary model 610 includes a first auxiliary model 653, a second auxiliary model 655, a third auxiliary model 657, and a fourth auxiliary model 659. One or more components of the holistic video understanding system 600 of FIG. 6 may be similar to components of the holistic video understanding system 100 of FIG. 1 and / or any of the other examples of the holistic video understanding system described herein and may perform similar operations as such components. For example, the model determination engine 606, the feature extraction models 608A and 608B, the auxiliary model 610, and the classifier 612 may be similar to and perform similar operations as the feature extraction model 108, the auxiliary model 110, and the output classifier 112 of Figure 1. In some cases, the compositor 611 of the holistic video understanding system 600 of Figure 6 may be similar to and perform similar operations as the compositor 511 of the holistic video understanding system 500 of Figure 5.

[0106] The input video 602 may be processed by the model determination engine 606. Referring to FIG. 1 as an example, the input video 602 or video portion may be retrieved from the video retriever 102 or the storage device 104. FIG. 6 introduces the concept of a secondary model determination engine 651, which may be disposed between the feature extraction models 608A and 608B and the auxiliary model 610. As shown, the secondary model determination engine 651 may receive features from the first feature extraction model 608A. The secondary model determination engine 651 may be trained to determine which auxiliary model from the auxiliary models 610 (e.g., auxiliary models 653, 655, and 657) coupled to the first feature extraction model 608A to use to process features generated by the first feature extraction model 608A. In some cases, the model determination engine 651 can be trained to determine which of the auxiliary models 653, 655, and 657 to use to further process features of the first feature extraction model 608A based on a desired combination of computational efficiency and accuracy.

[0107] During inference (after the secondary model determination engine 651 is trained), the secondary model determination engine 651 may receive features from the first feature extraction model 608A based on a determination by the model determination engine 606. In one example, based on characteristics of the features received from the first feature extraction model 608A, the secondary model determination engine 651 may determine which one or more of the auxiliary models 653, 655, and / or 657 should be used to further process the features from the first feature extraction model 608A. In some examples, the secondary model determination engine 651 may select two or more of the auxiliary models 653, 655, and / or 657. In one illustrative example, the secondary model determination engine 651 may select the auxiliary models 653 and 655. In such an example, the selected auxiliary models 653 and 655 may each receive features extracted from the input video 602 by the first feature extraction model 608A, process the received features, and provide a respective output to the synthesizer 611. The combiner 611 may combine (e.g., by averaging, concatenating, or otherwise combining the outputs) the outputs received from the auxiliary models 653 and 655. The combiner 611 may pass the combined output to a classifier 612 for classification of the video or video portion.

[0108] In another example where the model determination engine 606 selects the second feature extraction model 608B to extract features from the input video 602 or video portion, the model determination engine 651 may not perform any action on the input video 602 or video portion. In such an example, the features generated by the second feature extraction model 608B may be output to the auxiliary model 659 for further processing. The auxiliary model 659 may provide its output to the combiner 611. The combiner 611 may then forward the output to the classifier 612 for classification of the input video 602 or video portion.

[0109] Those skilled in the art should understand that the principle of utilizing the secondary model determination engine 651 is not limited to a particular topology of two feature extraction models 608A and 608B, where three auxiliary models 653, 655, 657 are configured to receive input from a first feature extraction model 608A, and one auxiliary model 659 is coupled to receive input from a second feature extraction model 608B. Any other suitable configuration may be used with the model determination engine 651. For example, in some cases, the secondary model determination engine 651 may be used in any configuration in which there are at least two auxiliary models that may be used to process features generated by one of the feature extraction models 608A and 608B. In some examples, multiple secondary model determination engines 651 may be used, where two or more of the feature extraction models 608A and 608B are coupled to two or more auxiliary models 610.

[0110] FIG. 7 is a diagram illustrating an alternative configuration for a holistic video understanding system 700. As shown, the components of the holistic video understanding system 700 include a feature extraction model, an auxiliary model 710, a synthesizer 711, and a classifier 712. The feature extraction model includes a first feature extraction model 708A and a second feature extraction model 708B. The auxiliary model 710 includes a first auxiliary model 753, a second auxiliary model 755, a third auxiliary model 757, and a fourth auxiliary model 759. The components of the holistic video understanding system 700 of FIG. 7 may be similar to the components of the holistic video understanding system 600 of FIG. 6 and / or any of the other examples of holistic video understanding systems described herein and may perform similar operations as those components. For example, each component of the holistic video understanding system 700 may be similar to the corresponding component of FIG. 6 and may perform similar operations. The holistic video understanding system 700 of FIG. 7 does not include a model determination engine (e.g., model determination engine 606) or a secondary model determination engine (e.g., secondary model determination engine 651), but instead utilizes both available feature extraction models 708A and 708B and corresponding auxiliary models 753, 755, 757, and 759 to process the input video 702 or video portions.

[0111] The input video 702 may be processed by the first feature extraction model 708A and the second feature extraction model 708B. Referring to FIG. 1 as an example, the input video 702 may be retrieved from the video retriever 102 or the storage device 104. The auxiliary models 753, 755, and 757 may process the features (e.g., one or more feature vectors) output by the first feature extraction model 708A. The auxiliary model 759 may process the one or more features (e.g., feature vectors) output by the second feature extraction model 708B. In the example of FIG. 7, the combiner 711 combines (e.g., by averaging, concatenating, etc.) all of the outputs of the auxiliary models 753, 755, 757, and 759 and provides the combined output to the classifier 712. The classifier 712 may classify the input video 702 or a video portion based on the output (e.g., a combined feature vector) from the combiner 711.

[0112] In the configuration of Figure 7, processing the input video 702 through all available paths (e.g., through all of the feature extraction models and all of the auxiliary models 710) occurs regardless of the characteristics of the input video. The holistic video understanding system 700 shown in Figure 7 is holistic in the sense that multiple feature extraction models (e.g., feature extraction model 708A and feature extraction model 708B) and multiple auxiliary models 710 (which may each apply a different inductive bias as described with respect to Figure 1 above) may be applied, as compared to application of a single fixed model to the input video 702.

[0113] Table 2 below shows experimental results comparing two configurations utilizing the four models 2-5 shown above in Table 1 and described thereafter. Table 2 compares the potential performance (measured in mAP) of using a model decision engine to select between models against a strategy that averages results from multiple models. For example, the first row of Table 2 shows experimental results for a configuration similar to the holistic video understanding system 700 of FIG. 7 that processed an input video through all four models 2-5 simultaneously. The synthesizer 711 of FIG. 7 averaged the outputs of the four models 2-5. In the experiment, the synthesized output was classified by the classifier 712 of FIG. 7.

[0114] [Table 2]

[0115] The second row of Table 2 shows experimental results for selecting the best model or subset of models to process each input video or video portion based on experimental data on which models provide the best accuracy. The values ​​in the second row of Table 2 were obtained by selecting one or more of the models averaged in row 1 for each input video or video portion. As shown in the second row of Table 2, the average accuracy and accuracy across individual categories (e.g., actions, concepts, events, objects, and scenes) were improved compared to the averaging shown in the first row. Table 2 shows that by utilizing a model determination engine described in this disclosure (e.g., model determination engine 106 in FIG. 1), accuracy can be improved compared to an approach that selects all models and averages the results.

[0116] FIG. 8 shows experimental plots of the relative performance of the holistic video understanding system 600 using the model decision engine 606 of FIG. 6 compared to an approach that utilizes only a fixed model and an approach that randomly selects between processing paths (e.g., each path can represent a pairing of a feature extraction model and an auxiliary model). The mAP for the experimental results is plotted on the vertical axis, and the computational complexity measured in GFLOPs per clip is plotted on the horizontal axis. In the experiments related to the plots in FIG. 8, the model decision engine 606 was implemented using MobileNetV2. In the experiments, each configuration described herein was tested using the same video dataset. A 2D CNN implemented using Resnet-50 was used as the first feature extraction model 608A. The second model decision engine 651 was configured to apply all three auxiliary models 653, 655, and 657 to the output of the first feature extraction model 608A. For example, the model decision engine 606 may be configured to apply the mAP of a video (or a video clip υ ijWhenever the first feature extraction model 608A was selected to process a segment of the video (e.g., a portion of the video), the features generated by the first feature extraction model 608A were processed through all three auxiliary models 653, 655, and 657. The combiner 611 combined the outputs from the three auxiliary models (e.g., by averaging, concatenating, or otherwise combining the features from the auxiliary models). The combiner 611 provided the combined output to the classifier 612 for classification. In the experiments, a 1D convolution over the time dimension (as described with respect to Figures 2A and 2B) was used for the auxiliary model 653, a non-local time model was used for the auxiliary model 655, and graph convolution was used for the auxiliary model 657. A 3D CNN implemented using the Slowfast model was used for the second feature extraction model 608B, and an MLP was used as the auxiliary model 659. In the experiments, whenever the model decision engine 606 selected the second feature extraction model 608B to process a video (or a portion of a video), the features of the second feature extraction model 608B were input to the auxiliary model 659, the synthesizer 611 did not perform any action (because only a single result was passed to its input), and the output of the auxiliary model 659 was classified by the classifier 612.

[0117] 8 plot show the results of using the model decision engine 606 to choose between the first feature extraction model 608A and the second feature extraction model 608B. For example, data point 836A represents the results of the model decision engine 606 always selecting the first feature extraction model 608A, and data point 836G represents the results of the model decision engine 606 always selecting the second feature extraction model 608B. Groups of data points 836A, 836B, 836C, 836D, 836E, 836F, and 836G respectively represent the results of randomly selecting between the first feature extraction model 608A and the second feature extraction model 608B according to the ratios [0.0, 0.1, 0.3, 0.5, 0.7, 0.9, 1.0], where 0.0 represents always selecting the first feature extraction model 608A and 1.0 represents always selecting the second feature extraction model 608B. As shown, the mAP for data point 830 exceeds both data points 836A and 836G, indicating that using the model decision engine 606 improves accuracy over using either the first feature extraction model 608A or the second feature extraction model 608B exclusively. Based on the average improved accuracy, the plot in FIG. 8 shows that the model determination engine 606 chooses a model based on visual or temporal features of the video.

[0118] As mentioned above, the holistic video understanding systems (e.g., holistic video understanding systems 100, 300, 400, 500, and / or 600) and related techniques described herein can enable a system to adaptively select among video processing machine learning models to efficiently and accurately classify video content across multiple categories (e.g., actions, attributes, events, objects, and scenes). For example, using the holistic video understanding system 100, a first video portion containing a static scene may be analyzed by a 2D CNN to extract spatial features for detecting objects in the scene from the first video portion, and a second video portion containing a sporting event may be analyzed by a 3D CNN to extract spatial and temporal features for detecting objects and actions associated with those objects from the second video portion. By utilizing the model determination engine 106 to select a feature extraction model (from the feature extraction models 108) that is suitable for the characteristics of the relevant video portion (e.g., the type of information, motion, etc. of the relevant video portion) and is less computationally intensive than other available feature extraction models 108, the computing and power resources of the system (e.g., computing system) may be saved for other operations. In addition to saving the computing and power resources of the system, including various auxiliary models 110 with different inductive biases may make the holistic video understanding system 100 more holistic by providing models that specialize in recognizing a wider variety of categories. In comparison, a system that includes a fixed model for all input videos may perform well when the video being analyzed contains the category of data that the fixed model is optimized to detect, but may lose accuracy and / or consume unnecessary computational effort when the video being analyzed contains different categories of information.

[0119] 9 is a flow diagram illustrating an example of a process 900 for processing a video using a holistic video understanding system, according to some examples. At block 902, the process 900 includes obtaining a first video. In some examples, the process 900 may obtain the first video from a storage device. In an illustrative example, the storage device includes the storage device 104 shown in FIG. 1 and / or the storage device 1230 shown in FIG. 12.

[0120] At block 904, the process 900 includes using a machine learning model determination engine to determine a first machine learning model from the set of machine learning models (e.g., for use in processing at least a portion of the first video). The first machine learning model may be determined based on one or more characteristics of at least the portion of the first video. In one illustrative example, the model determination engine includes the model determination engine 106 illustrated in FIG. 1. In some examples, the model determination engine includes a neural network (e.g., a MobileNetV2 neural network and / or a Resnet-50 neural network). In one illustrative example, the set of machine learning models includes the feature extraction model 108 illustrated in FIG. 1. In another illustrative example, the set of machine learning models includes models 313A, 313B-313M illustrated in FIG. 3.

[0121] In some examples, the first machine learning model includes a convolutional neural network (CNN), as described above. For example, in some examples, the CNN includes a two-dimensional CNN (e.g., MobileNetV2 or Resnet-50), a three-dimensional CNN (e.g., Slowfast), or other CNN. In some cases, the first machine learning model may include a first type of CNN that is different from the CNNs included in at least some of the other machine learning models in the set of machine learning models. In some examples, the one or more characteristics of at least the portion of the first video include spatial characteristics, temporal characteristics, and / or audio characteristics.

[0122] At block 906, the process 900 includes processing at least the portion of the first video using a first machine learning model. As described herein, processing at least the portion of the first video may include, among other processing, extracting features from at least the portion of the first video, processing the features using one or more auxiliary models, and classifying at least the portion of the first video.

[0123] In some examples, the process 900 may include determining a classification of at least the portion of the first video based on processing at least the portion of the first video using a first machine learning model. In some examples, the classes determined during the classification may include playing soccer, playing the violin, and listening to music, among others. In one illustrative example, the output classifier 112 shown in FIG. 1 determines the classification. In some examples, the process 900 includes determining a classification that identifies a plurality of classes included in at least the portion of the first video, as described above.

[0124] In some examples, the process 900 includes extracting a first one or more features from at least the portion of the first video using a first machine learning model. In some cases, the first machine learning model includes a neural network such as a two-dimensional (2D) CNN. In such cases, the 2D CNN can extract the first one or more features from at least the portion of the first video in two spatial dimensions. In some implementations, the first machine learning model includes a 3D CNN. In such implementations, the 3D CNN can extract the first one or more features from at least the portion of the first video in two spatial dimensions and a temporal dimension.

[0125] The process 900 may include using a first auxiliary model to process the first one or more features extracted from at least the portion of the first video by the first machine learning model. In an illustrative example, the first auxiliary model includes an auxiliary model included in the auxiliary model 110 illustrated in FIG. 1. In some cases, the first auxiliary model includes a neural network such as a one-dimensional (1D) CNN. For example, when the first auxiliary model includes a 1D CNN, the first one or more features may include a first feature vector based on a first frame and a second feature vector based on a second frame of at least the portion of the first video. In such a case, the 1D CNN may generate temporal information of at least the portion of the first video from at least the first feature vector and the second feature vector. In an illustrative example, the 1D CNN performs the convolution illustrated in FIG. 2D and described above. In some cases, the first auxiliary model includes a multi-layer perceptron. In some implementations, the first auxiliary model includes a graph convolutional network. In some cases, the first auxiliary model includes a non-local model. In some examples, the process 900 includes determining a first classification of at least the portion of the first video based on processing the first one or more features using the first auxiliary model.

[0126] In some examples, the process 900 includes using a machine learning model determination engine to determine a second machine learning model from the set of machine learning models (e.g., for use in processing at least another portion of the first video). The second machine learning model may be determined based on one or more characteristics of the at least another portion of the first video. In some cases, the process 900 includes using a second machine learning model to process at least another portion of the first video. In one aspect, the second machine learning model may include a neural network (e.g., a CNN) different from the one or more neural networks (e.g., a CNN) of the first machine learning model. In some cases, the process 900 includes using the second machine learning model to extract a second one or more features from at least another portion of the first video. The process 900 may include using a second auxiliary model to process the second one or more features extracted from at least another portion of the first video by the second machine learning model. In some examples, the process 900 may determine a second classification of at least another portion of the first video based on one or more features from the second auxiliary model.

[0127] In some examples, the process 900 may obtain a second video. The process 900 may determine a second machine learning model (e.g., for use to process at least a portion of the second video). The second machine learning model may be determined based on one or more characteristics of at least that portion of the second video. In some examples, the process 900 may obtain the second video from a storage device. In an illustrative example, the storage device includes the storage device 104 shown in FIG. 1 and / or the storage device 1230 shown in FIG. 12. The process 900 may process at least that portion of the second video using the second machine learning model.

[0128] In some examples, the process 900 includes using a first machine learning model to extract a first one or more features from at least the portion of the first video in parallel with determining a first machine learning model from the set of machine learning models for use in processing at least the portion of the first video. In some cases, the machine learning model determination engine shares a common neural network with the first machine learning model. In one illustrative example, the model determination engine that shares a common neural network with the first machine learning model may include the model determination engine 406 and the feature extraction component 429, as shown in FIG. 4.

[0129] In some examples, the processes described herein (e.g., process 900 and / or other processes described herein) may be performed by a computing device or apparatus. In one example, one or more of the processes may be performed by the holistic video understanding system 100 of FIG. 1. In another example, one or more of the processes may be performed by the computing system 1200 shown in FIG. 12. For example, a computing device with the computing system 1200 shown in FIG. 12 may include components of the holistic video understanding system 100 and may perform operations of the process 900 of FIG. 9 and / or other processes described herein.

[0130] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a vehicle or a vehicular computing device (e.g., a vehicle's driver monitoring system (DMS)), a server computer, a robotic device, a television, and / or any other computing device having resource capabilities to perform the processes described herein, including process 900 and / or other processes described herein. In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0131] Components of a computing device may be implemented with circuitry. For example, components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0132] Process 900 is illustrated as a logical flow diagram, whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.

[0133] Additionally, process 900 and / or other processes described herein may be executed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0134] As mentioned above, various aspects of the present disclosure may use machine learning models or systems. FIG. 10 is an illustrative example of a deep learning neural network 1000 that may be used to implement the holistic video understanding system described above. The input layer 1020 includes input data. In one illustrative example, the input layer 1020 may include data representing pixels of an input video frame. The neural network 1000 includes multiple hidden layers 1022a, 1022b-1022n. The hidden layers 1022a, 1022b-1022n include "n" hidden layers, where "n" is an integer equal to or greater than 1. The number of hidden layers may be made to include as many layers as required for a given application. The neural network 1000 further includes an output layer 1021 that provides output resulting from the processing performed by the hidden layers 1022a, 1022b-1022n. In one illustrative example, the output layer 1021 may provide a classification for an object in the input video frame. The classification may include classes that classify types of activities (eg, playing soccer, playing piano, listening to piano, playing guitar, etc.).

[0135] Neural network 1000 is a multi-layered neural network of interconnected nodes. Each node can represent information. The information associated with a node is shared between different layers, with each layer retaining the information as it is processed. In some cases, neural network 1000 can include a feedforward network, in which there are no feedback connections where the output of the network is fed back to itself. In some cases, neural network 1000 can include a recurrent neural network, which can have loops that allow information to be carried across nodes while reading at the input.

[0136] Through node-to-node interconnections between the various layers, information may be exchanged between the nodes. The nodes of the input layer 1020 may activate a set of nodes in the first hidden layer 1022a. For example, as shown, each of the input nodes of the input layer 1020 is connected to each of the nodes of the first hidden layer 1022a. The nodes of the first hidden layer 1022a may transform the information of each input node by applying an activation function to the input node information. The information derived from the transformation may then be passed to the nodes of the next hidden layer 1022b, which may activate such nodes, which may perform their own designated functions. Exemplary functions include convolution, upsampling, data transformation, and / or any other suitable functions. The output of the hidden layer 1022b may then activate the nodes of the next hidden layer, and so on. The output of the last hidden layer 1022n may activate one or more nodes of the output layer 1021, where an output is provided. In some cases, a node in neural network 1000 (e.g., node 1026) is shown as having multiple output lines, however, the node has a single output and all lines shown as outputting from the node represent the same output value.

[0137] In some cases, each node or interconnections between nodes can have weights, which are a set of parameters derived from training the neural network 1000. Once the neural network 1000 is trained, it may be referred to as a trained neural network and may be used to classify one or more activities. For example, the interconnections between nodes may represent information learned about the interconnected nodes. The interconnections may have adjustable numerical weights that may be adjusted (e.g., based on a training data set), allowing the neural network 1000 to be adaptive to the input and to learn as more and more data is processed.

[0138] The neural network 1000 is pre-trained to process features from data in the input layer 1020 using different hidden layers 1022a, 1022b-1022n to provide output through the output layer 1021. In an example where the neural network 1000 is used to identify activities being performed by a driver in frames, the neural network 1000 may be trained using training data including both frames and labels, as described above. For example, training frames may be input to the network, with each training frame having a label indicating the feature in the frame (for a feature extraction machine learning system) or the class of activity in each frame. In an example using object classification for illustrative purposes, the training frames may include images of the number 2, in which case the label for the image may be [0 0 1 0 0 0 0 0 0 0].

[0139] In some cases, the neural network 1000 may use a training process called backpropagation to adjust the node weights. As described above, the backpropagation process may include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update are performed during one training iteration. The process may be repeated for a number of iterations for each set of training images until the neural network 1000 is trained well enough that the layer weights are accurately adjusted.

[0140] For the example of identifying objects in a frame, the forward pass may include passing a training image through the neural network 1000. Before the neural network 1000 is trained, the weights are first randomized. As an illustrative example, the frame may include an array of numbers representing pixels of the image. Each number in the array may include a value from 0 to 255 representing the pixel intensity at that location in the array. In one example, the array may include a 28×28×3 array of numbers with 28 rows and 28 columns of pixels and three color components (e.g., a red component, a green component, and a blue component, or a luma component and two chroma components, etc.).

[0141] As mentioned above, in the first training iteration for the neural network 1000, due to the weights being randomly selected in the initialization, the output may contain values ​​that do not give a preference to any particular class. For example, if the output is a vector with the probability that the object contains different classes, the probability values ​​of each of the different classes may be equal or at least very similar (e.g., for 10 possible classes, each class may have a probability value of 0.1). With the initial weights, the neural network 1000 is unable to make an accurate determination of what the classification of the object is likely to be, since it is unable to determine the low-level features. A loss function may be used to analyze the error in the output. Any suitable loss function definition may be used, such as cross-entropy loss. Another example of a loss function is:

[0142]

number

[0143] The loss includes the mean squared error (MSE), defined as total may be set equal to the value of

[0144] The loss (or error) is large for the first training images because the actual values ​​are very different from the predicted outputs. The goal of training is to minimize the amount of loss so that the predicted outputs are the same as the training labels. The neural network 1000 can perform a backward pass by determining which inputs (weights) contributed most to the network's loss, and the weights can be adjusted so that the loss is reduced and eventually minimized. To determine the weights that contributed most to the network's loss, the derivative of the loss with respect to the weights (denoted as dL / dW, where W is the weight at a particular layer) can be calculated. After the derivatives are calculated, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated such that the weights change in the opposite direction of the gradient. The weight update can be

[0145]

number

[0146] where w denotes the weight and w i denotes the initial weights and η denotes the learning rate, which can be set to any suitable value, with higher learning rates inducing larger weight updates and smaller values ​​indicating smaller weight updates.

[0147] The neural network 1000 may include any suitable deep network. An example includes a convolutional neural network (CNN), which includes an input layer and an output layer with multiple hidden layers between the input layer and the output layer. The hidden layer of a CNN includes a series of convolutional layers, nonlinear layers, pooling layers (for downsampling), and fully connected layers. The neural network 1000 may include any other deep network other than a CNN, such as an autoencoder, a deep belief net (DBN), a recurrent neural network (RNN), among others.

[0148] FIG. 11 is an illustrative example of a convolutional neural network (CNN) 1100. The input layer 1120 of the CNN 1100 includes data representing an image or frame. For example, the data can include an array of numbers representing pixels of the image, with each number in the array including a value between 0 and 255 representing the pixel intensity at that location in the array. Using the previous example from above, the array can include a 28×28×3 array of numbers, with 28 rows and 28 columns of pixels and three color components (e.g., a red component, a green component, and a blue component, or a luma component and two chroma components, etc.). The image is passed through a convolutional hidden layer 1122a, an optional nonlinear activation layer, a pooling hidden layer 1122b, and a fully connected hidden layer 1122c to obtain an output at the output layer 1124. Although only one of each hidden layer is shown in Figure 11, one skilled in the art will appreciate that multiple convolutional hidden layers, nonlinear layers, pooling hidden layers, and / or fully connected layers may be included in the CNN 1100. As previously explained, the output may indicate a single class of object, or may include a probability of the class that best represents the object in the image.

[0149] The first layer of the CNN 1100 is the convolutional hidden layer 1122a. The convolutional hidden layer 1122a analyzes the image data of the input layer 1120. Each node of the convolutional hidden layer 1122a is connected to a region of nodes (pixels) of the input image called the receptive field. The convolutional hidden layer 1122a can be thought of as one or more filters (each filter corresponds to a different activation or feature map), and each convolutional iteration of a filter is a node or neuron of the convolutional hidden layer 1122a. For example, the region of the input image that a filter is responsible for in each convolutional iteration is the receptive field of the filter. In one illustrative example, if the input image includes a 28×28 array and each filter (and corresponding receptive field) is a 5×5 array, there are 24×24 nodes in the convolutional hidden layer 1122a. Each connection between a node and the receptive field for that node learns a weight, and possibly an overall bias, so that each node learns to analyze its particular local receptive field in the input image. Each node in the hidden layer 1122a has the same weights and biases (called shared weights and shared biases). For example, a filter has an array of weights (numbers) and the same depth as the input. The filter has a depth of 3 in the example video frame (according to the three color components of the input image). An illustrative example size of the filter array is 5x5x3, which corresponds to the size of the receptive field of the node.

[0150] The convolutional nature of the convolutional hidden layer 1122a is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 1122a may start at the upper left corner of the input image array and convolve around the input image. As mentioned above, each convolutional iteration of the filter may be considered as a node or neuron of the convolutional hidden layer 1122a. At each convolutional iteration, the value of the filter is multiplied with a corresponding number of original pixel values ​​of the image (e.g., a 5×5 filter array is multiplied with a 5×5 array of input pixel values ​​at the upper left corner of the input image array). The multiplications from each convolutional iteration may be added together to obtain a total for that iteration or node. The process is then continued at the next location in the input image according to the receptive field of the next node in the convolutional hidden layer 1122a. For example, the filter may be moved to the next receptive field by a certain step amount (called a stride). The stride may be set to 1 or other suitable amount. For example, if the stride is set to 1, the filter is moved one pixel to the right in each convolution iteration. Processing the filter at each unique location of the input volume produces a number representing the filter result for that location, resulting in an aggregate value being determined for each node in the convolutional hidden layer 1122a.

[0151] The mapping from the input layer to the convolutional hidden layer 1122a is called an activation map (or feature map). The activation map contains the value of each node that represents the filter result at each location of the input volume. The activation map may contain an array containing various aggregate values ​​resulting from each iteration of the filter on the input volume. For example, if a 5x5 filter is applied to each pixel of a 28x28 input image (a stride of 1), the activation map contains a 24x24 array. The convolutional hidden layer 1122a may contain several activation maps to identify multiple features in an image. The example shown in FIG. 11 contains three activation maps. Using the three activation maps, the convolutional hidden layer 1122a can detect three different types of features, each detectable throughout the image.

[0152] In some examples, a nonlinear hidden layer may be applied after the convolutional hidden layer 1122a. The nonlinear layer may be used to introduce nonlinearity into a system that was previously computing linear operations. One illustrative example of a nonlinear layer is a rectified linear unit (ReLU) layer. The ReLU layer may apply a function f(x)=max(0,x) to all of the values ​​in the input volume, which changes all negative activations to 0. Thus, the ReLU may enhance the nonlinear nature of the CNN 1100 without affecting the receptive field of the convolutional hidden layer 1122a.

[0153] After the convolutional hidden layer 1122a (and after the nonlinear hidden layer, when used), a pooling hidden layer 1122b may be applied. The pooling hidden layer 1122b is used to simplify the information in the output from the convolutional hidden layer 1122a. For example, the pooling hidden layer 1122b may take each activation map output from the convolutional hidden layer 1122a and use a pooling function to generate a compressed activation map (or feature map). Max pooling is one example of a function performed by the pooling hidden layer. Other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions, may be used by the pooling hidden layer 1122b. A pooling function (e.g., a max pooling filter, an L2 norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 1122a. In the example shown in FIG. 11, three pooling filters are used for the three activation maps in the convolutional hidden layer 1122a.

[0154] In some examples, max pooling may be used by applying a max pooling filter (e.g., having a size of 2×2) with a stride (e.g., equal to the dimension of the filter, such as a stride of 2) to the activation map output from the convolutional hidden layer 1122a. The output from the max pooling filter contains the maximum number of any subregion around which the filter convolves. Using a 2×2 filter as an example, each unit in the pooling layer may summarize a region of 2×2 nodes (each node is a value in the activation map) in the previous layer. For example, four values ​​(nodes) in the activation map are analyzed by a 2×2 max pooling filter at each iteration of the filter, and the maximum value from the four values ​​is output as the “max” value. If such a max pooling filter is applied to an activation filter from the convolutional hidden layer 1122a with dimensions of 24×24 nodes, the output from the pooling hidden layer 1122b is an array of 12×12 nodes.

[0155] In some examples, an L2 norm pooling filter may also be used, which involves computing the square root of the sum of the squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (rather than computing the maximum value as is done in max pooling) and using the computed value as the output.

[0156] Intuitively, a pooling function (e.g., a max pooling function, an L2 norm pooling function, or other pooling function) determines whether a given feature is found anywhere within a region of an image and discards the exact location information. This can be done without affecting the outcome of feature detection because once a feature has been found, the exact location of the feature is not as important as its approximate location relative to other features. Max pooling (as well as other pooling methods) offers the advantage that there are many fewer features to pool, thus reducing the number of parameters required in later layers of the CNN 1100.

[0157] The final layer of connections in the network is a fully connected layer that connects every node from the pooling hidden layer 1122b to every one of the output nodes in the output layer 1124. Using the above example, the input layer includes 28×28 nodes that encode pixel intensities of the input image, the convolutional hidden layer 1122a includes 3×24×24 hidden feature nodes based on application of 5×5 local receptive fields (for the filters) to three activation maps, and the pooling hidden layer 1122b includes a layer of 3×12×12 hidden feature nodes based on application of max pooling filters to 2×2 regions across each of the three feature maps. Extending this example, the output layer 1124 may include 10 output nodes. In such an example, every node of the 3×12×12 pooling hidden layer 1122b is connected to every node of the output layer 1124.

[0158] The fully connected layer 1122c can take the output of the previous pooling hidden layer 1122b (which should represent the activation map of high-level features) and determine the features that are most correlated to a particular class. For example, the fully connected layer 1122c can determine the high-level features that are most strongly correlated to a particular class and can include weights (nodes) for the high-level features. The products can be calculated between the weights of the fully connected layer 1122c and the pooling hidden layer 1122b to obtain probabilities for different classes. For example, if the CNN 1100 is used to predict that an object in a video frame is a person, there will be high values ​​in the activation map that represents high-level features of people (e.g., there are two legs, the face is at the top of the object, two eyes are at the top left and top right of the face, the nose is in the center of the face, the mouth is at the bottom of the face, and / or other features that are common to people).

[0159] In some examples, the output from the output layer 1124 may include an M-dimensional vector (in the previous example, M=10). M indicates the number of classes the CNN 1100 must choose from when classifying an object in an image. Other example outputs may also be provided. Each number in the M-dimensional vector may represent a probability that the object is a certain class. In one illustrative example, if a 10-dimensional output vector represents 10 different classes of objects are [0 0 0.05 0.8 0 0.15 0 0 0 0], the vector indicates that there is a 5% probability that the image is a third class of object (e.g., a dog), an 80% probability that the image is a fourth class of object (e.g., a human), and a 15% probability that the image is a sixth class of object (e.g., a kangaroo). The probability of a class may be considered a confidence level that the object is part of that class.

[0160] 12 is a diagram illustrating an example of a system for implementing some aspects of the present technology. In particular, FIG. 12 illustrates an example of a computing system 1200, which may be any computing device, such as an internal computing system, a remote computing system, a camera, or any components thereof, in which the components of the system communicate with each other using a connection 1205. The connection 1205 may be a physical connection using a bus, or a direct connection to a processor 1210, such as in a chipset architecture. The connection 1205 may also be a virtual connection, a network connection, or a logical connection.

[0161] In some embodiments, computing system 1200 is a distributed system in which the functionality described in this disclosure may be distributed across a data center, multiple data centers, a peer network, etc. In some embodiments, one or more of the described system components represent many components, each performing some or all of the functionality for which the component is described. In some embodiments, the components may be physical or virtual devices.

[0162] The exemplary system 1200 includes at least one processing unit (CPU or processor) 1210 and connections 1205 coupling various system components to the processor 1210, including system memory 1215, such as read only memory (ROM) 1220 and random access memory (RAM) 1225. The computing system 1200 may include a cache 1212 of high-speed memory directly connected to the processor 1210, in close proximity to the processor 1210, or integrated as part of the processor 1210.

[0163] The processor 1210 may include any general-purpose processor, as well as hardware or software services, such as services 1232, 1234, and 1236 stored in a storage device 1230, configured to control the processor 1210, as well as special-purpose processors where software instructions are embedded into the actual processor design. The processor 1210 may essentially be a completely self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0164] To enable user interaction, computing system 1200 includes input devices 1245, which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gestural or graphical input, a keyboard, a mouse, motion input, speech, etc. Computing system 1200 can also include output devices 1235, which can be one or more of several output mechanisms. In some instances, a multimodal system may allow a user to provide multiple types of input / output to communicate with computing system 1200. Computing system 1200 can include a communication interface 1240, which can generally govern and manage user input and system output.The communications interface may be an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet port / plug, a fiber optic port / plug, a proprietary wired port / plug, BLUETOOTH® wireless signal transmission, BLUETOOTH® Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, Radio Frequency Identification (RFID) wireless signal transmission, Near Field Communication (NFC) wireless signal transmission, Dedicated Short Range Communications (DSRC) wireless signal transmission, 802.11 The communication interface 1240 may perform or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including those utilizing Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or any combination thereof. The communication interface 1240 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine the location of the computing system 1200 based on reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States Global Positioning System (GPS), the Russian Global Navigation Satellite System (GLONASS), the Chinese BeiDou Navigation Satellite System (BDS), and the European Galileo GNSS.There is no constraint to operating on any particular hardware configuration, and therefore the basic features herein may be easily substituted for improved hardware or firmware configurations as they are developed.

[0165] The storage device 1230 may be a non-volatile and / or non-transitory and / or computer readable memory device, such as a hard disk, or a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid state memory, a compact disk read only memory (CD-ROM) optical disk, a re-writable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disk (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick card, a smart card chip, It may be another type of computer readable medium capable of storing data that is accessible by a computer, such as an EMV chip, a Subscriber Identity Module (SIM) card, a mini / micro / nano / pico SIM card, another Integrated Circuit (IC) chip / card, a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read Only Memory (EPROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a Resistive Random Access Memory (RRAM / ReRAM), a Phase Change Memory (PCM), a Spin Transfer Torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0166] Storage devices 1230 may include software services, servers, services, etc. that cause the system to perform functions when code defining such software is executed by processor 1210. In some embodiments, hardware services that perform specific functions may include software components stored in a computer-readable medium in conjunction with the necessary hardware components, such as processor 1210, connections 1205, output devices 1235, etc., to perform the functions.

[0167] The term "computer-readable medium" as used herein includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, storing, or transporting instructions and / or data. Computer-readable media may include non-transitory media on which data can be stored and that do not include carrier waves and / or transitory electronic signals propagating wirelessly or via wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory, or memory devices. A computer-readable medium may store code and / or machine-executable instructions, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted using any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0168] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0169] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that the present embodiments may be practiced without these specific details. For clarity of explanation, in some cases, the present technology may be presented as including individual functional blocks, including functional blocks comprising devices, device components, and steps or routines in a method embodied in software, or a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, or other components may be shown as components in block diagram form to avoid obscuring the embodiments in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0170] Particular embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0171] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a processing device to perform or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a particular function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, network-attached storage devices, etc.

[0172] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, and the like. The functions described herein may also be embodied in peripheral devices or add-in cards. Such functions may also be implemented on a circuit board in different chips or different processes executed in a single device, as further examples.

[0173] The instructions, media for carrying such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0174] In the above description, aspects of the present application are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, while exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in various other ways, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. Various features and aspects of the applications described above may be used individually or together. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be considered illustrative rather than restrictive. For purposes of illustration, the methods have been described in a particular order. It should be appreciated that in alternative embodiments, the methods may be performed in an order different from that described.

[0175] Those skilled in the art will appreciate that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.

[0176] When a component is described as being "configured to" perform some operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0177] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component and / or any component that is in communication, either directly or indirectly, with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communications interface).

[0178] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can further include items not listed in the set of A and B.

[0179] The various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0180] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a FLASH memory, a magnetic or optical data storage medium, or the like. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0181] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0182] Exemplary aspects of the present disclosure include the following.

[0183] Aspect 1: A method for processing video data, the method including: acquiring a first video; determining a first machine learning model from a set of machine learning models using a machine learning model determination engine (e.g., for use in processing at least a portion of the first video), where the first machine learning model is determined based on one or more characteristics of at least a portion of the first video; and processing at least the portion of the first video using the first machine learning model.

[0184] Aspect 2: The method of aspect 1, further comprising determining a classification of at least the portion of the first video based on processing at least the portion of the first video using a first machine learning model.

[0185] Aspect 3: The method of any one of aspects 1 or 2, further comprising: extracting a first one or more features from at least a portion of the first video using a first machine learning model; processing the first one or more features using a first auxiliary model; and determining a first classification of at least a portion of the first video based on the processing of the first one or more features using the first auxiliary model.

[0186] Aspect 4: The method of aspect 3, wherein the first auxiliary model includes a one-dimensional convolutional neural network, the first one or more features include a first feature vector based on a first frame of at least that portion of the first video and a second feature vector based on a second frame of at least that portion of the first video, and the first auxiliary model generates temporal information of at least that portion of the first video from at least the first feature vector and the second feature vector.

[0187] Example 5: The method of any one of examples 3 or 4, wherein the first auxiliary model comprises a multi-layer perceptron.

[0188] Aspect 6: The method of any one of aspects 3 to 5, wherein the first auxiliary model includes a graph convolutional network.

[0189] Aspect 7: The method of any one of aspects 3 to 6, wherein the first auxiliary model includes a non-local model.

[0190] Aspect 8: The method of any one of aspects 1 to 7, further comprising: determining a second machine learning model from the set of machine learning models using a machine learning model determination engine, wherein the second machine learning model is determined based on one or more characteristics of at least another portion of the first video; and processing at least another portion of the first video using the second machine learning model.

[0191] Aspect 9: The method of aspect 8, further comprising: extracting a second one or more features from at least another portion of the first video using a second machine learning model; processing the second one or more features using a second auxiliary model; and determining a second classification of at least another portion of the first video based on the processing of the second one or more features using the second auxiliary model.

[0192] Aspect 10: The method of any one of aspects 1 to 9, wherein the first machine learning model includes a two-dimensional CNN.

[0193] Example 11: The method of example 10, wherein the 2D CNN extracts a first one or more features from at least a portion of the first video in two spatial dimensions.

[0194] Aspect 12: The method of any one of aspects 1 to 11, wherein the first machine learning model includes a three-dimensional (3D) CNN.

[0195] Example 13: The method of example 12, wherein the 3D CNN extracts a first one or more features from at least a portion of the first video in two spatial dimensions and a temporal dimension.

[0196] Aspect 14: The method of any one of aspects 1 to 13, further comprising: acquiring a second video; determining a second machine learning model from a set of machine learning models using a machine learning model determination engine, wherein the second machine learning model is determined based on one or more characteristics of at least a portion of the second video; and processing at least the portion of the second video using the second machine learning model.

[0197] Aspect 15: The method of any one of aspects 1 to 14, further comprising extracting a first one or more features from at least the portion of the first video in parallel with determining a first machine learning model from the set of machine learning models for use in processing at least the portion of the first video. In some cases, the machine learning model determination engine shares a common neural network with the first machine learning model.

[0198] Aspect 16: The method of any one of aspects 1 to 15, wherein the one or more characteristics of at least the portion of the first video include spatial characteristics and temporal characteristics.

[0199] Aspect 17: The method of any one of aspects 1 to 16, wherein the one or more characteristics of at least the portion of the first video includes an audio characteristic.

[0200] Aspect 18: An apparatus for processing video data, comprising: a memory; and one or more processors coupled to the memory, wherein the one or more processors are configured to: acquire a first video; determine, using a machine learning model determination engine, a first machine learning model from a set of machine learning models (e.g., for use to process at least a portion of the first video), wherein the first machine learning model is determined based on one or more characteristics of at least a portion of the first video; and process at least the portion of the first video using the first machine learning model.

[0201] Aspect 19: The apparatus of aspect 18, wherein the one or more processors are configured to determine a classification of at least the portion of the first video based on processing at least the portion of the first video using a first machine learning model.

[0202] Aspect 20: The apparatus of aspect 18 or 19, wherein the one or more processors are configured to extract a first one or more features from at least a portion of the first video using a first machine learning model, process the first one or more features using a first auxiliary model, and determine a first classification of at least a portion of the first video based on processing the first one or more features using the first auxiliary model.

[0203] Aspect 21: The apparatus of aspect 20, wherein the first auxiliary model includes a one-dimensional convolutional neural network, the first one or more features include a first feature vector based on a first frame of at least that portion of the first video and a second feature vector based on a second frame of at least that portion of the first video, and the first auxiliary model generates temporal information of at least that portion of the first video from at least the first feature vector and the second feature vector.

[0204] Example 22: The apparatus of any one of examples 20 or 21, wherein the first auxiliary model includes a multi-layer perceptron.

[0205] Example 23: The apparatus of any one of examples 20 to 22, wherein the first auxiliary model includes a graph convolutional network.

[0206] Example 24: The apparatus of any one of examples 20 to 23, wherein the first auxiliary model includes a non-local model.

[0207] Aspect 25: The apparatus of any one of aspects 18 to 24, wherein the one or more processors are configured to: determine, using a machine learning model determination engine, a second machine learning model from the set of machine learning models, where the second machine learning model is determined based on one or more characteristics of at least another portion of the first video; and process the at least another portion of the first video using the second machine learning model.

[0208] Aspect 26: The apparatus of aspect 25, wherein the one or more processors are configured to extract a second one or more features from at least another portion of the first video using a second machine learning model, process the second one or more features using a second auxiliary model, and determine a second classification of at least another portion of the first video based on processing the second one or more features using the second auxiliary model.

[0209] Example 27: The apparatus of any one of examples 18 to 26, wherein the first machine learning model includes a two-dimensional CNN.

[0210] Example 28: The apparatus of example 27, wherein the 2D CNN extracts the first one or more features from at least a portion of the first video in two spatial dimensions.

[0211] Example 29: The apparatus of any one of examples 18 to 28, wherein the first machine learning model includes a three-dimensional (3D) CNN.

[0212] Example 30: The apparatus of example 29, wherein the 3D CNN extracts a first one or more features from at least a portion of the first video in two spatial dimensions and a temporal dimension.

[0213] Aspect 31: The apparatus of any one of aspects 18 to 30, wherein one or more processors are configured to: obtain a second video; determine a second machine learning model from a set of machine learning models using a machine learning model determination engine, wherein the second machine learning model is determined based on one or more characteristics of at least a portion of the second video; and process at least the portion of the second video using the second machine learning model.

[0214] Aspect 32: The apparatus of any one of aspects 18 to 31, wherein the machine learning model determination engine shares a common neural network with the first machine learning model, and the one or more processors are configured to use the first machine learning model to extract a first one or more features from at least the portion of the first video in parallel with determining a first machine learning model from the set of machine learning models to use to process at least the portion of the first video.

[0215] Aspect 33: The apparatus of any one of aspects 18 to 32, wherein the one or more characteristics of at least the portion of the first video include spatial characteristics and temporal characteristics.

[0216] Aspect 34: The apparatus of any one of aspects 18 to 33, wherein the one or more characteristics of at least the portion of the first video include an audio characteristic.

[0217] Aspect 35: A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform any of the operations of aspects 1 to 34.

[0218] Example 36: An apparatus comprising means for performing any of the operations of examples 1 to 34. [Explanation of symbols]

[0219] 100, 300, 400, 500, 600, 700 Holistic Video Understanding System 102 Video Retriever 104, 1230 Storage Devices 106, 406, 506, 606 Model Decision Engine 108 Feature Extraction Model, 2D CNN Feature Extraction Model 110 Auxiliary Models, Nonlocal Auxiliary Models, MLP Auxiliary Models 112 Output classifier, classifier 210, 212, 214 Composite feature vector 211, 213, 215 Convolution filters 216 Synthetic feature vectors, new feature vectors 217 Black Cube 221, 223, 225, 227 Arrows 302 Input Video, Video 306 Model Decision Engine, Decision Engine 312, 412, 512, 612, 712 classifier 313 Video Processing Model 313A, 313B~313M Video Processing Models, Models 402, 502, 602, 702 Input Video 408, 508, 541 Feature extraction models 410, 510, 543, 545, 610, 710 Auxiliary Models 429 Feature Extraction Part, Feature Extraction Component 431 First feature extraction model, model 433 Second feature extraction model, model 435, 653, 753 First auxiliary model, auxiliary model 437, 655, 755 Second auxiliary model, auxiliary model 439, 657, 757 3rd auxiliary model, auxiliary model 511, 611, 711 Synthesizer 547, 549 Output 608A, 708A First feature extraction model, feature extraction model 608B, 708B Second feature extraction model, feature extraction model 651 second auxiliary model decision engine, secondary model decision engine, model decision engine, second model decision engine 659, 759 4th auxiliary model, auxiliary model 830, 836A, 836B, 836C, 836D, 836E, 836F, 836G Data Points 1000 Deep Learning Neural Networks, Neural Networks 1020, 1120 Input layer 1021, 1124 output layer 1022a, 1022b~1022n hidden layers 1026 nodes 1100 Convolutional Neural Network (CNN), CNN 1122a Convolutional hidden layer, hidden layer 1122b Pooling hidden layer 1122c Fully connected hidden layer, fully connected layer 1200 Computing system, system 1205 Connection 1210 Processor, Processing Unit (CPU or Processor) 1212 Cache 1215 System Memory 1220 Read-Only Memory (ROM) 1225 Random Access Memory (RAM) 1232, 1234, 1236 Services 1235 Output Device 1240 Communication Interface 1245 Input Devices

Claims

1. 1. An apparatus for processing video data, comprising: Memory, and one or more processors coupled to the memory, the one or more processors: Obtaining the first video; determining a first machine learning model from a set of machine learning models using a machine learning model determination engine, the first machine learning model being determined based on characteristics of a plurality of frames of at least a portion of the first video; and determining a classification of at least the portion of the first video by processing at least the portion of the first video using the first machine learning model; An apparatus configured to:

2. the one or more processors: extracting a first one or more features from at least the portion of the first video using the first machine learning model; processing the first one or more features using a first auxiliary model; and determining a first classification of at least the portion of the first video based on processing the first one or more features using the first auxiliary model; The apparatus of claim 1 , configured to:

3. the first auxiliary model includes a one-dimensional convolutional neural network (CNN); the first one or more features include a first feature vector based on a first frame of at least the portion of the first video and a second feature vector based on a second frame of at least the portion of the first video; The apparatus of claim 2 , wherein the first auxiliary model generates temporal information for at least the portion of the first video from at least the first feature vector and the second feature vector.

4. The apparatus of claim 2 , wherein the first auxiliary model comprises a multi-layer perceptron.

5. The apparatus of claim 2 , wherein the first auxiliary model comprises a graph convolutional network.

6. The apparatus of claim 2 , wherein the first auxiliary model comprises a non-local model.

7. the one or more processors: determining a second machine learning model from the set of machine learning models using the machine learning model determination engine, the second machine learning model being determined based on one or more characteristics of at least another portion of the first video; and determining a second classification of the at least another portion of the first video by processing the at least another portion of the first video using the second machine learning model. The apparatus of claim 1 , configured to:

8. The set of machine learning models: a two-dimensional (2D) CNN that extracts a first one or more features from at least the portion of the first video in two spatial dimensions; and a three-dimensional (3D) CNN that extracts a first one or more features from at least the portion of the first video in two spatial dimensions and a temporal dimension; The apparatus of claim 1 , comprising:

9. the one or more processors: Obtaining a second video; determining a second machine learning model from the set of machine learning models using the machine learning model determination engine, the second machine learning model being determined based on one or more characteristics of at least a portion of the second video; and determining a classification of at least the portion of the second video by processing at least the portion of the second video using the second machine learning model; The apparatus of claim 1 , configured to:

10. The machine learning model determination engine shares a common neural network with the first machine learning model, and the one or more processors: Using the first machine learning model, extracting a first one or more features from at least the portion of the first video in parallel with determining the first machine learning model from the set of machine learning models for use in processing at least the portion of the first video. The apparatus of claim 1 , configured to:

11. The apparatus of claim 1 , wherein the characteristics of the plurality of frames of at least the portion of the first video include spatial and temporal characteristics of the plurality of frames.

12. 1. A method for processing video data, comprising the steps of: acquiring a first video; determining a first machine learning model from a set of machine learning models using a machine learning model determination engine, the first machine learning model being determined based on characteristics of a plurality of frames of at least a portion of the first video; determining a classification of at least the portion of the first video by processing at least the portion of the first video using the first machine learning model; The method includes:

13. determining a second machine learning model from the set of machine learning models using the machine learning model determination engine, the second machine learning model being determined based on one or more characteristics of at least another portion of the first video; processing the at least another portion of the first video using the second machine learning model; and 13. The method of claim 12, further comprising:

14. extracting a second one or more features from the at least another portion of the first video using the second machine learning model; and processing the second one or more features using a second auxiliary model; determining a second classification of the at least another portion of the first video based on processing the second one or more features using the second auxiliary model; 14. The method of claim 13, further comprising:

15. 13. The method of claim 12, further comprising: extracting a first one or more features from at least the portion of the first video in parallel with determining the first machine learning model from the set of machine learning models for use in processing at least the portion of the first video using the first machine learning model, wherein the machine learning model determination engine shares a common neural network with the first machine learning model.

Images